>> A more serious irony is that the automatic control system has been put in place because it can do the job better than the operator, yet the operator is being asked to monitor that it is working effectively.
Automatic control is good for standard case, and the expected errors. For weird cases, for now humans are better to detect them and press the big red button.
Jumping into article, I assumed that the author was going to describe how he enjoys, by drawing it out as far as he can,
the process of surrendering to the machine god..
(Not the machina diabol, as it sadly turned out to be, though some BDSM zealots might disagree.. the things that bureaucracy does to us are mild in comparison? )
But.. we do monitor whether our prayers are effective .. right? Is that a smitable offense?
After that disappointment, I too was "curious": was the author going to talk about "curiosity", that magic virtue that we hackers believe, lifts design into the divine (but more often than not kills the pussy)
Except experience shows they aren't. Experience shows that when humans are just pressing "ok" 99% of the time they'll press it 100% of the time and completely miss the other cases.
Great post. I particularly like the 4 column pr view idea. I’ve built something similar to review speckit specs. The scaffolding to ensure that code actually works makes sense too.
There’s one thing that’s bothering me though — all of these have us consuming what LLMs produce. I learned and internalized how things worked by building systems. At least within the constraints of the abstractions I was using. This makes me think of TikTok and shorts. Everybody is just watching. Maybe that’s not charitable though. Maybe the abstraction has now changed from an “os/gui/event loop” to “llm” and the rest is just accidental complexity.
i sometimes wonder how all carpenters or music producers in the world would react if tomorrow a magic box arrives where they type-in/dictate what they want and voila... out comes a table/composition.. Will the worlds music producers and carpenters just "accept" this new magic genie? I would hope they wont.
Yet here we are as software engineers...ready to surrender our craft to mere "prompts" with reckless abandon.
"Yup, looks good to me!" said the human reviewing thousands of lines of generated code everyday.
This has become common now, and I've seen it ruin large projects until nobody understands the codebase full of verbose nonsensical code and comments. It becomes impossible to further develop the project without AI. Contributors not using AI for any reason are all leaving because no one enjoys working with mindless robots (and humans) as colleagues. It's demeaning and devaluing their labor and humanity. They're not getting "left behind", they are leaving the madness of crowds behind.
> ..the possibility of runaway dynamics: once a critical threshold is crossed, small increases in adoption can trigger rapid population-level shifts toward *persistent dependence, with abrupt losses in cognitive competence*.
We can all see it happening. As this technology permeates society, the challenge to maintain control, coherence, and quality will become greater and ever more serious - for example, AI in healthcare, military, management and governance.
I hope there's still value to be found in a bunch of developers standing around a whiteboard and talking, and I think this scenario is where their ability and understanding really shows these days.
If we don't value that though--if we don't value understanding the system and just thinking through problems as a group--then let's stop pretending and just not do code reviews. Probably nobody cares and just types "LGTM" after having their own LLM do a spot check to cover their ass.
It's really easy to push developers to stop caring. If management says "go fast fast fast, use those LLMs!", then developers can't keep up, get frustrated, stop understanding the system, and at that point they're just there to collect a paycheck before society collapses. Or, if management is making dumb decisions all the time, same thing.
Hopefully as implementing ideas takes less and less time and resources, smart people who actually care will start building things with other smart people who care.
Even if the human's judgement is correct in every LGTM, there's a bias towards merging. Each merge that adds a feature but degrades understandability 1% will be merged. It won't be held back until it improves it by 5% instead. After 100 merges you have degraded understandability 100% even though nobody made an error.
You’re completely right that nobody is stopping a PR that just ever so slightly decreases understandability of something. It may not even be obvious to the reviewer that it’s slightly less understandable overall… Doubly so if they’re just skimming due to the volume, and tripley so if they’re having agents do the most of the review, which lets the human reviewer’s guard down.
But before LLMs, it didn’t matter because on a good team you’d have a balance where maybe some PRs added complexity and some simplified it. Sometimes people would include a small related cleanup or refactor in a PR for code around what they were doing if they noticed complexity. That has become far less common, I think.
I know it’s not your point, but a 1% decrease 100 times is 0.99^100 = 0.366 which is a 63.4% decrease in understandability. Just throwing that out there
To be fair, models are more accurate than how they were earlier this year. In march I was still frustrated and was writing my own event loops because the model simply wasn't built for it. By January I was still hand checking statistical outputs to make sure it wasn't making any errors, and it was. After august it was pretty much over. There were no stats errors, code generated was shit but it could do what is asked. Instruction following is still a problem, and I sometimes catch silly things in the scrolling diffs, but it has been much easier since then.
I am often thinking recently of the littel loop diagrams from the "The Capability Trap" paper. We need to keep a loop reinvesting into our own Human capability in anything we're doing.
This is copium that is gone with the wind when the models perform up to the level of standard that the user requires. I'll give the author 6 months before they feel like its okay to just send it.
Karpathy had the same trajectory. Primeagean had the same trajectory. I think in the end everyone except for Jonathan Blow will have the same trajectory
Automatic control is good for standard case, and the expected errors. For weird cases, for now humans are better to detect them and press the big red button.
But.. we do monitor whether our prayers are effective .. right? Is that a smitable offense?
After that disappointment, I too was "curious": was the author going to talk about "curiosity", that magic virtue that we hackers believe, lifts design into the divine (but more often than not kills the pussy)
There’s one thing that’s bothering me though — all of these have us consuming what LLMs produce. I learned and internalized how things worked by building systems. At least within the constraints of the abstractions I was using. This makes me think of TikTok and shorts. Everybody is just watching. Maybe that’s not charitable though. Maybe the abstraction has now changed from an “os/gui/event loop” to “llm” and the rest is just accidental complexity.
This has become common now, and I've seen it ruin large projects until nobody understands the codebase full of verbose nonsensical code and comments. It becomes impossible to further develop the project without AI. Contributors not using AI for any reason are all leaving because no one enjoys working with mindless robots (and humans) as colleagues. It's demeaning and devaluing their labor and humanity. They're not getting "left behind", they are leaving the madness of crowds behind.
> ..the possibility of runaway dynamics: once a critical threshold is crossed, small increases in adoption can trigger rapid population-level shifts toward *persistent dependence, with abrupt losses in cognitive competence*.
We can all see it happening. As this technology permeates society, the challenge to maintain control, coherence, and quality will become greater and ever more serious - for example, AI in healthcare, military, management and governance.
If we don't value that though--if we don't value understanding the system and just thinking through problems as a group--then let's stop pretending and just not do code reviews. Probably nobody cares and just types "LGTM" after having their own LLM do a spot check to cover their ass.
It's really easy to push developers to stop caring. If management says "go fast fast fast, use those LLMs!", then developers can't keep up, get frustrated, stop understanding the system, and at that point they're just there to collect a paycheck before society collapses. Or, if management is making dumb decisions all the time, same thing.
Hopefully as implementing ideas takes less and less time and resources, smart people who actually care will start building things with other smart people who care.
But before LLMs, it didn’t matter because on a good team you’d have a balance where maybe some PRs added complexity and some simplified it. Sometimes people would include a small related cleanup or refactor in a PR for code around what they were doing if they noticed complexity. That has become far less common, I think.
I know it’s not your point, but a 1% decrease 100 times is 0.99^100 = 0.366 which is a 63.4% decrease in understandability. Just throwing that out there
Karpathy had the same trajectory. Primeagean had the same trajectory. I think in the end everyone except for Jonathan Blow will have the same trajectory
I guess you could ask the model. "Yes well done, you're a visionary software engineer".
The lawyers and judges who trusted an ML system to be up to snuff are currently pounding sand.