Yet another Hacker News

Show HN: Parsimony – an open-source benchmark for how much code AI models add

77 points lasu 3 days ago 18 comments

Existing benchmarks measure solve rate and token output/cost, but don't punish excess code, which makes a codebase harder to work with in the long term. I wanted to measure this since it matters most for long-term projects.

Method: it takes the patches that benchmarks already publish (DeepSWE, SWE-bench Verified, SWE-bench Live) and counts "coding units": syntax elements like statements, calls, names and literals. I didn't use lines of code, so cramming code isn't rewarded. Deleted code is subtracted, so it measures net units.

GPT-6 and Claude 5.5 have no public patches yet, so I ran them myself on 3–10 tasks on my own subscriptions (shown as faded circles, not ranked).

Any feedback or model contributions are welcome, especially public patch exports for newer models.

GitHub: https://github.com/La5u/Parsimony

Not affiliated with any of the benchmarks.

fastball 3 days ago | parent

On the one hand, I absolutely agree that we should try to measure and reduce verbosity that LLMs are slopping into our codebases.

On the other hand, the fact that Astra is "winning" this bench is not great signal for me, since imo Astra produces basically unreadable code. Astra plays code golf. My own revealed preference is that I'd rather have the excessive comments and tests of Claude rather than the inscrutability of Astra.

ndr 3 days ago | parent

I keep going in between the two and I'm currently in an Astra season, deeply disliking the tone and modes of Claude.

I wonder if the two will diverge enough that this becomes the tab-vs-spaces/vim-vs-emacs of the LLM era and they'll manage to keep users moving over at some point.

demibabs 3 days ago | parent

Yeah the issue is that while code that’s too verbose is bad, code that’s too terse is as well.

Astra codes like it has to pay by the keystroke (although in a way, it indeed does)

happyPersonR 3 days ago | parent

we actually need good metrics for readable code that is clear and understandable, and not just someone's lint rules lol. Maybe that's the next part of this and where this goes?

One approach, skill's etc for good clean readable understandable code.

i bet there's a better one involving something with metrics and scoring etc

Vakaiser 3 days ago | parent

Does anyone have ways to reduce the complexity and verbosity of code generated via agents?

Anything that can increase code quality would help.

hlipusch 7 hours ago | parent

I am trying my new compress skill: After claude opus is done being verbose, i have gpt 6.1 sol do one or mulitple cleanup passes: - names - types - functions - files - comments and docs - tests In each step it merges duplicate code and deletes what isn't needed.

Xeoncross 3 days ago | parent

At this point, I really want all AI benchmarks to use the best coding/writing model that normal people can run locally.

For now that seems to be Qwen 3.8 27B that requires ~28GB of RAM (Q4) which is pretty doable for anyone with a beefy GPU or slightly-above-average mac.

I want to know how close open weights (open models) are to all these leading commercial models. Yes, yes, local models aren't as good, but that distance is closing as local models are already better than what we had just over a year ago in many cases.

Same with Text-to-Image generation (Flux 2, Z-Image-Turbo, etc..) or text-to-speech (Parakeet, Whisper, etc..).

vunderba 3 days ago | parent

FWIW, I have mixed feelings about Artificial Analysis but they at least let you filter by size class and by proprietary versus open.

https://artificialanalysis.ai/?intelligence=agentic-index

smilekzs 3 days ago | parent

On a related note: Would love to see cyclomatic complexity as a supporting metric for readability. I increasingly find AI-adjacent and AI-written code suffering from this.

chengjunsuger 3 days ago | parent

Nice project! But I wonder does less coding actually means less work for whoever reviews it?

KasianFranks 3 days ago | parent

Nobody is going to care about how much code an AI system generates in the very near future, some even now, once everyone understands the scale at which a vector space truly operates at. Complexity is not much of an issue for an AI system.

nostrebored 3 days ago | parent

LLM systems start to Frankenstein things together at a very modest amount of code. The lack of abstractions cause huge and glaring problems for anything that needs to run predictably.

Abstraction is an issue for every AI coding agent. They are painfully useful, but need constant badgering in the right direction.

KasianFranks 3 days ago | parent

After 22,948 commits, 13,720 bugs and -2,899,391 lines eliminated, I've seen this too but a properly designed AI pipeline's ability to optimize and course correct with a little badgering supersedes any Frankenstein-like outcomes.

plusmax1 3 days ago | parent

I agree with you. an LLM will do what you guide it to do, give it the proper guardrails and instructions and it will produce code in any style you wish.

Be it abstract or simple, more comments or no comments, adhering to clean code principles or following whatever software architecture you want...

Include style guides, static analysis and formatting tools in the development loop, use skills and hooks properly... I started to have a much better time with agentic coding once I spent time setting things up properly for my projects and writing down how I wanted things to be done.

Tanjreeve 3 days ago | parent

Why even benchmark anything? I have faith that the AI will just get better over time anyway. I think to even attempt to measure any usability is an insult to the infinite exponential curve of improvement.

lasu 3 days ago | parent

The benchmarks are what allows the AI researchers to improve the models. The AI doesn't get better on its own.

Tanjreeve 3 days ago | parent

AI should be free to evaluate itself free from arbitrary human constraints like benchmarks imo.

tinthedev 3 days ago | parent

Yeah, I love this. We can't manage what we don't measure, and adjusting the mesurement to be on the abstract syntax level removes an entire layer of "mess" that comes from comments and formatting.

I'll bookmark it and probably adjust the measurement methods for future review purposes. I've often found my LLM workflows to produce inane and overly complex code, and allowing me to see a single figure will allow me to more easily eval generated code.

I'd love to hear people's lived experience, though. I think heuristics in this space still allow for a lot of optimizations of code generation and pipelining.