80 points doppp 1 hour ago 96 comments
tphyahoo2 51 minutes ago | parent
I like how this captures with a metric (lines of code and cyclomatic complexity, some other basically tractable measures) in an automated way, something we all by now intuitively know.
fosterfriends 48 minutes ago | parent
I believe that the era of “ai writes tons of slop code” will be a stepping stone in the longer story, and is simply a current gap in the reward functions.
Per the author - if we can get strong measurements of what good code is, we can train against it and close the gap fast. Excited to see more thinking in this area
loveparade 46 minutes ago | parent
yehoshuapw 44 minutes ago | parent
However it is more likely to be something which can be detached..
peder 45 minutes ago | parent
Totally agreed that we're not looking at the correct metric right now. Increasingly, code quality will be determined by outcomes.
alecbz 36 minutes ago | parent
It's an LLM concern too. LLMs seem to do better with well-organized codebases, just like the humans they were trained on.
stiiv 36 minutes ago | parent
On the other hand, there are claims that the best languages for robots tend to be terse (allegedly); I tend to regard a rich domain model (static types, not terse) as a major facet of high-quality code.
FuckButtons 32 minutes ago | parent
siscia 47 minutes ago | parent
yorwba 17 minutes ago | parent
mococa 45 minutes ago | parent
marginalia_nu 44 minutes ago | parent
hankbond 39 minutes ago | parent
thinking about how to accurately describe the problem at hand and figuring out the simplest way to approach it takes a lot of effort well before you get to the implementation step of coding.
mococa 10 minutes ago | parent
softwaredoug 44 minutes ago | parent
Which is more important to you: Velocity to a solution? Or velocity to understanding?
nucleative 9 minutes ago | parent
Velocity to solution is default for almost everyone else, especially one-off or low impact / low consequence of failure projects.
vanschelven 43 minutes ago | parent
Alan Kay called programming "Pop Culture" some 20 years ago[0]
scronkfinkle 43 minutes ago | parent
victorbjorklund 41 minutes ago | parent
shimman 39 minutes ago | parent
Blaming workers is always an excuse by poor management.
Retric 27 minutes ago | parent
Hand those people an LLM and they don’t suddenly become competent, but they do start slinging more code.
shimman 15 minutes ago | parent
These companies pay management more than workers for a reason, if you can't even admit that they are to blame then what are you trying to do here? Just attack workers for what reason exactly? Being anti-worker is a great tell to never trust a person.
softwaredoug 37 minutes ago | parent
baridbelmedar 30 minutes ago | parent
FloorEgg 29 minutes ago | parent
I don't necessarily disagree. That said...
Why?
I've been grappling with this myself. There is an easy/obvious answer, but I wonder how stable/permanent it is. If you feel strongly about this, are you willing to unpack your judgement?
softwaredoug 25 minutes ago | parent
But even still, cognitive debt is a real, documented phenomenon where we lose our ability to change projects.[1]
We also know coding agents tend to accumulate slop in projects over time without some human intervention[2]
There's also the concern of whether its a good idea to be dependent on an addictive slot machine run by a trillion dollar company to do your work.
JodieBenitez 25 minutes ago | parent
Tanjreeve 31 minutes ago | parent
2. A lot of it wasn’t. Low quality code/speed serves a purpose for point solutions and scripts etc. That’s not the same thing as writing a core system and if the user doesn’t put any credentials in for an S3 bucket then it falls back to giving information about your own S3 bucket (as I’ve seen just this week).
3. Plenty of companies you can discern the difference between mission critical systems versus “business” systems where if it falls over it’s annoying but not the end of the world.
ramijames 29 minutes ago | parent
I've been doing development, in one way or another, since the 90s. I've worked with dozens of teams from enterprises to startups. Hundreds of developers. The quality of work has been all over the place, but the majority was not great.
I'm arguing that what people today call "AI slop" is already higher quality than what most developers created historically, and the fact that tests and documentation pretty much come for free now means that the floor has been raised.
The quality of AI generated code is not great. Yes, it will get better. It's already better than 65%+ of what regular devs can do AND it is faster to produce, iterate, and release.
avgDev 29 minutes ago | parent
You can't just rewrite everything. So over many years people are touching small parts of the pie.
If it works it isn't low quality.
Dlemlo 18 minutes ago | parent
harshalizee 15 minutes ago | parent
thi2 26 minutes ago | parent
bunderbunder 14 minutes ago | parent
So much "bad" enterprise code evolved into that state over years or even decades of small changes. Meanwhile, last year I got to watch an LLM-authored codebase speedrun itself into a similar state in only a couple months. And I would say that the enterprise code was actually better. It at least did its job fairly reliably. The LLM codebase was riddled with defects, so much so that it ate up all our time and our feature delivery rate ground to a halt.
There are two observations that really eat at me:
1. Studies seem to indicate that agentic coding uses 2-10x as many lines of code to accomplish the same task.
2. One of the only really well-established empirical results in software engineering is the strong association between LOC and defect rate.
devin 10 minutes ago | parent
Varelion 43 minutes ago | parent
linsomniac 42 minutes ago | parent
_pdp_ 40 minutes ago | parent
Code is an abstract concept that is not bound to the physical world and I imagine that future will have some much more of it that it is difficult to comprehend. Everything will be code and more code will be written than ever before.
Code will never going to be solved. The question is how much humans will be involved and I think the evidence is that perhaps just a bit. However, because we are talking about vast libraries of code even if we are involved in under 1% of all code and decision making that is needed around the code, there are still not enough developers out there to take on the task.
I might be wrong :)
wang_li 28 minutes ago | parent
danbruc 39 minutes ago | parent
zsoltkacsandi 36 minutes ago | parent
Dlemlo 28 minutes ago | parent
ahalay-mahalay 23 minutes ago | parent
echelon 33 minutes ago | parent
It's astounding to me that people can see coding get solved and not think every single one of these tasks won't be solved too.
Why in the hell do you not think these things aren't going to be completely automated?
Fable and Astra can one-shot video games with compelling novel game loops. They can do systems programming, distributed systems, robotics. I haven't found a weak point.
Seedance 2.5 can make video better than the manual labor of VFX artists, 3D artists, and animators.
Nano Banana and GPT Image can do a better job than graphics designers.
LLMs just solved a Millennium Prize Problem, and there are probably more that will fall in the coming weeks.
Just wait. All of these things will be solved.
There is no "stopping point".
Edit:
Don't anticipate that 2036 will look anything like 2026.
Will Smith spaghetti doesn't stay that way forever. Trillions of dollars will be spent on solving these problems. They will be solved.
dwroberts 28 minutes ago | parent
Will all these things get automation? Yeah sure. But the idea that they will be perfect automated solutions applicable in all cases is just marketing, it’s not reality.
BowBun 27 minutes ago | parent
Tell that to the mountain of failed AI slop games on Steam! As a game dev, building compelling, fun games is not even something humans are good at doing consistently. The AI can build the tech, but it can't make something 'fun' yet (unless your bar for fun is simply that a tool created a thing).
danbruc 27 minutes ago | parent
sigbottle 27 minutes ago | parent
If you value humans intrinsically, this is necessarily the loop that will converge. I don't think humans have deep intensional a priori knowledge of the structure of reality. If we did, then we wouldn't need tools like AI because we'd be a superset of that. We can only observe and judge.
If we don't value humans, then sure, I think AI is at the point where it can kill all humans (conditional on sentience and resources etc). Two ways to solve a problem - solve the problem, or eliminate the problem statement. Plenty of easier vectors to eliminate the "problem statement", than say, try to solve problems such as making human life better. If you do value the latter though, there will necessarily be human judgers. That's how it works.
testaccount121 26 minutes ago | parent
i understand the cope because it legit seems like such a cynical thing to say but we have to face the reality at hand.
echelon 19 minutes ago | parent
_s_a_m_ 26 minutes ago | parent
mbernstein 25 minutes ago | parent
Have you tried one-shotting real distributed systems problems? What was the result and how did you verify correctness?
Dlemlo 19 minutes ago | parent
But lets be fair, if an expert would use AI today to build something with this, I would feel a lot more confident than not doing this.
I would start with the base architecture and add all the guardrails for a distributed system, i might even go so far to leverage the math skills of a frontier model like fable or astra. I would for sure have the proper budget for using Fable/Astra.
_s_a_m_ 29 minutes ago | parent
perchard 29 minutes ago | parent
gedy 27 minutes ago | parent
Really doubt we are near that being solved with non-technical folks + LLMs. I'm seeing people gleefully rebuilding products with the exact same blind spots in their understanding/logic using LLMs. Claude, etc are not seemingly able to "AGI" around goofy asks. The CSS looks a little nicer than their legacy products though, lol.
chucksmash 26 minutes ago | parent
I doubt many people here are brave enough to claim their code does what is supposed to do in every conceivable case. Maybe you have high confidence in the correctness of parts of the code. Correctness of an application is murky though. Things we build are never fully correct, merely correct enough. Like maybe you're responsible for the UI in a web app and you're using your expertise to ensure it gracefully handles display across browsers and a gamut of screen sizes/form factors. But are you also verifying how it works when localized with an RtL script? Are you checking every change you make against CJK?
danbruc 18 minutes ago | parent
binary0010 25 minutes ago | parent
The refactor ended up adding 22,000 loc.
I went in there and quickly read through it, laughed my ass off. Reverted the work tree. Micromanaged a new refactor. Net lines of code for something really elegant and easy to reason about was -3k loc in the project.
In case you are wondering why vibe coders are doing 30k loc a day, this is why.
Supermancho 22 minutes ago | parent
My experience is a little different. For higher abstraction languages the output is largely acceptable in my work. I always consider that LLMs don't know what I don't tell them and they have limited context to work from. Coding issues I often identify:
* Efficiency. Marginal by default. Coding efficiency problems often appear because LLMs dont usually consider the entire codebase or future plans (although they do guess at some futures). Sometimes they write/name things in ways that are lazy/wasted cycles. Most of the time, they don't.
* Security. Marginal by default. I say they do pretty good. Considering all the failure modes, not so much.
* Maintainability. Marginal by default. Mostly due to the careful consideration of modularity, upgrade paths, etc. while often taking wildly different approaches to solutions without having specific broad instructions. Even then, there can be big gaps in quality.
* Observability. Not acceptable by default. There's usually some consideration and can often one-shot.
* Portability. Not acceptable by default. Good, if you specify what those targets are. Regardless, testing validates this above the coding and models are very good at hitting functional test targets. This is less of an issue in something like Java ofc.
danielvaughn 20 minutes ago | parent
dnikolovv 13 minutes ago | parent
heaney-555 5 minutes ago | parent
mr_roboto 16 minutes ago | parent
OtomotO 14 minutes ago | parent
It's just that many (I guess that includes me? :D) assumed that they are better than the actually were.
whatever1 13 minutes ago | parent
Better at writing code within a huge system, definitely not. Maybe in the future, but as of Astra, Fable 5.1, the answer is still no.
danbruc 7 minutes ago | parent
skrebbel 36 minutes ago | parent
Kinrany 34 minutes ago | parent
The only caveat is that it's the same model doing an iteration and then using that iteration as a starting point for the next step. So the model is allowed to write absolutely insane solutions, as long as it can read them back, even if no one else can.
One thing that could be done is to use a separately developed baseline coding model B to evaluate the outcome of each iteration. For model under test X to pass an iteration, not only should it be able to solve starting from the previous solution, but so should B, starting from X's previous solution.
cjalmeida 32 minutes ago | parent
This matches my experience. Before working on an issue, I ask the LLM to estimate net LOCs at the final PR based on the scope. It works well, and review steps do flag inconsistencies. But as the OP mentioned, if you turn this into a hard metric vs "design smell", you can see LLMs code-golfing for oneliners.
Xenoamorphous 31 minutes ago | parent
guayusa 30 minutes ago | parent
cheney_2004 30 minutes ago | parent
avgDev 25 minutes ago | parent
I finally gave up and read documentation for 15 minutes and solved the problem.
I will never push AI generated code to production without understanding it, and this is why I only generate small code snippets and copy/paste most of the time.
tloopff 29 minutes ago | parent
Reality: earandil.com uses 170% CPU in Firefox.
What has this author written before LLMs? Why should we listen to him and his adjudication of "perfect code"?
Cyclomatic complexity is the oldest paper generating grift for college students. There are hundreds of thousands of useless papers about cyclomatic complexity.
antoni4040 29 minutes ago | parent
90% of problems are easy once you know what you actually want well enough for you to be able to ask it from an LLM.
90% of code before LLMs was badly copied from StackOverflow anyway.
That 10% that's remaining, I've see 0, ZERO, nil progress. Windows is still awful. Spotify still doesn't work correctly offline. Youtube search is trash. Jira takes 20 seconds sometimes to load a task. LLMs haven't created a new database or a new game engine or a new renderer or anything like that.
The maths breakthroughs are really more of a testament to the efforts of the last 150 years for maths to be an organised verifiable principle. If LLMs had to practice math they way Euler did, they wouldn't be able to find shit.
(sorry if I sound incoherent, just some thoughts while I'm commuting)
j45 29 minutes ago | parent
Of course, sloppiness to date can be measured by different shared and interpreted preferences and definitions.
justinmarsan 26 minutes ago | parent
It's so easy to ship a lot of code, more effort should be put into ensuring the code is correct, with self-improving feedback loops that involve developers, and dedicated tooling...
But again, a while ago, everything was about prompt engineering, and now you can express you idea vaguely and get a somewhat working result, so this likely will evolve fast as well...
bobkb 21 minutes ago | parent
conqrr 13 minutes ago | parent
If humans increasingly are kept out of coding, then who holds the mental model?
If AI holds the mental model, by definition human prompts will be over lossy channel. This is true without AI too. Software quality is directly dependent on good devs that translate from business/PM speak to technical decisions.
So is coding solved now? it was already solved decades ago.
FiberBundle 13 minutes ago | parent
CuriouslyC 12 minutes ago | parent
Originally I tried to walk the line between improved agent performance and human readability, but current models are so good I don't think human readability matters much, though at a high level, being able to grok the overall folder structure still matters. I've got my hands full polishing a demo for my game, but I intend to revisit Valknut by crafting an eval set that lets me calculate the difference in agent token consumption and task failure rate between ~isomorphic codebase structures. This will let me loop agents to discover organizing policies that improve them.
Truthfully though, with today's models I don't think this sort of codebase optimization is likely to have much impact below 250k-300k LoC projects, and it probably won't be a decisive win till you're near 1M. Also, the shelf life of a product like this isn't infinite as each generation of models pushes those numbers up while also having new policy preferences that require re-evaluating existing policies.
RickJWagner 9 minutes ago | parent
Code will become throwaway stuff, like the results of AI prompts. Cook it up, test it for adequacy, and run it. When something comes along that adds new requirements, just update the requirements/prompt and make a new one.
Test suites will be important.