Yet another Hacker News

MaxxBench – "Nice DeepSWE score. But is it true?"

77 points antiuniverse 2 hours ago 16 comments

brcmthrowaway 1 hour ago | parent

Two brothers created a benchmark, family is so proud.

Neywiny 1 hour ago | parent

This is perfect. Real programming has a ton of these instances of minor changes.

saejox 1 hour ago | parent

There is always a huge discrepancy between benchmarks and real world capabilities.

For my use case deepseek 4.1 flash is almost as dumb as haiku 5.5. Both are fast garbage producers.

jeffbee 1 hour ago | parent

I was beginning to worry I had overlooked the inner beauty of DeepSeek. When I have tried it, it was totally useless. Never understood HN cheering for it.

AlotOfReading 36 minutes ago | parent

Don't know what you're doing, but it's good enough for the reasonably sized tasks I give LLMs. It's also a much better price. My browser stopped rendering the UI frame recently because I messed up permissions on /usr/share/icons. Took DS 4.1 flash about 3 minutes and a few cents to diagnose it. Same harness with opus 5 took twice as long and several dollars.

Moreover, the months I've used anthropic/OAI exclusively my API token costs are in the hundreds (or sometimes thousands). With DS flash as the first agent I reach for, I rarely exceed $40/mo.

jeffbee 22 minutes ago | parent

I meant for generative coding. I'm sure it can do a few things here and there.

andreypopp 7 minutes ago | parent

How did you use it? I was using it as "implementor" subagents orchestrated by opus 5.5 and it was pretty good and fast.

pixelesque 10 minutes ago | parent

I've had significant success using DS 4 Pro and now 4.1 flash for fairly complex coding tasks on C++, Rust and Python projects at home.

I have access to Claude Code at work (and use it quite a bit), so I feel I can honestly compare them.

You do have to plan things first a bit more with DS (than say Opus can (Opus will have more chance of one-shotting something, although I generally don't think that's a good idea), although it often over-engineers and tries to be too helpful in my experience) and be a bit more explicit at times, but I feel it's quite close to Sonnet in performance for implementing plans, debugging issues and explaining how codebases work.

embedding-shape 1 hour ago | parent

> There is always a huge discrepancy between benchmarks and real world capabilities.

I mean of course, the real world is messy, the models are only good for what they've been trained on, and post-training frequently takes benchmark scores into account too, leading to this sort of "Great score, sucks in the real world" syndrome that seemingly most models are affected by.

YetAnotherNick 55 minutes ago | parent

DeepSWE was released on May and Deepseek v4 flash was released on April. By definition it couldn't be benchmaxxed on this particular benchmark. My guess is the mutation is not like real world and it is tripping small models, not benchmaxed model.

viraptor 6 minutes ago | parent

It's not clear which v4 flash they used. There are 3 different releases for that one.

peri-cl 54 minutes ago | parent

The big trend in that graph is that smaller models collapse the most. Everything with "-flash" in the name, plus OpenAI Luna.

Look at GPT 5.6-Sol vs. GPT 5.6-Luna in particular.

(The GLM pair shouldn't be expected follow this pattern, because they're misnamed: GLM-5.3 and GLM-5.3-flash are distinct lineages. The -flash is an early release of a new pretrain on a new architecture).

If this *is* the main factor, I'd expect Kimi K3.1 and Gemini Carbon would also do well on this test (unlike Gemini-flash); while Haiku 6 would collapse (breaking the trend of the other recent Claude models). It's a falsifiable prediction.

andai 27 minutes ago | parent

So basically it turns out the good models are good and the benchmaxxed models are benchmaxxed.

Edit: another comment pointed out it's probably not benchmaxxing due to the chronology.

https://news.ycombinator.com/item?id=50406948

But why should minor changes to the task cause catastrophic performance drops? How is that not evidence of benchmaxxing?

andai 19 minutes ago | parent

> Most models tried pleasing an imagined grader over the prompt

>When we read the trajectories, we saw that many models had figured out they were in a benchmark and started second-guessing the prompt. Instead of just building what we asked for, they started coding to the tests they thought the grader would run.

I ran into the same problem on my English exams.

citizenpaul 6 minutes ago | parent

I liked the GLM models until 5.3 and this site seems to explain perhaps some of the reason why.