145 points ilreb 1 hour ago 28 comments

dan_thinks 1 hour ago | parent

This is exciting. The continued focus on efficiency alongside quality is encouraging - it makes me hopeful to see what smaller models will become capable of.

Looking forward to the "Flash" and 27B versions.

seizethecheese 59 minutes ago | parent

I hope they are able to serve 4 faster. Qwen 3.8 is a great model but I’ve regretted including it in the battle system for opencode [1] we have been building because it scores so low.

And it’s a shame because the only reason it scores so low is that it’s slooooow. (Like 30 tokens per second vs opus near 80.) If you exclude voting, Qwen 3.8 does well.

(We allow early voting contrary to Arena.ai because I think it makes sense for agents, where it doesn’t for chat. This means our data takes speed into account.)

Also, it would be ideal if they served this with third parties. Part of the slowness with 3.8 is just not a lot of providers. Their open source version is much faster, but we couldn’t include it because it doesn’t handle images (another own goal by Alibaba).

[1]: https://agentduel.com/leaderboard

forgingahead 44 minutes ago | parent

Qwen3.8 doesn't handle images?

seizethecheese 40 minutes ago | parent

Sorry that was confusing. Qwen 3.8 2.4T is faster than what we have on leaderboard but it doesn’t handle image input: https://openrouter.ai/qwen/qwen3.8-2.4t-a95b

anana_ 43 minutes ago | parent

Artificial Analysis score doesn't look too hot. Doesn't even surpass Kimi 3.1

selectodude 41 minutes ago | parent

Anthropic finally figured out how to block them from distilling Opus.

m00dy 22 minutes ago | parent

how so ?

anon373839 39 minutes ago | parent

AA has become pretty meaningless. They just keep adjusting the recipe to get a preferred ranking result, with no supporting evidence for their decisions.

gr_norm 16 minutes ago | parent

Yeah, I don't know why people bother with it anymore. It no longer reflects what you're likely to see in real-world usage, at least IME.

stratts 15 minutes ago | parent

One thing I find confusing about AA is that they show GPT 6.1 dominating the lower end of the Pareto line, with eg, GPT 6.1 Sol at medium supposedly costing far less per task than DeepSeek V4.1 Flash at high.

Yet every time I compare the two in something like Pi, Sol ends up being 5-10x the price for a simple query, something like 7-10c vs. less than 0.5c for DeepSeek. What's the deal?

blackoil 10 minutes ago | parent

Same for me coding tasks taking $20-50 are easily done by deepseek in $5-10

the_duke 3 minutes ago | parent

To be fair, a lot of the popular benchmarks have become pretty meaningless.

Models often get great scores despite being a lot worse than the frontier in practice.

AAisthetiobeofA 21 minutes ago | parent

> Artificial Analysis

Ah yes, the reverse-benchmaxers of OpenAI models.

m00dy 31 minutes ago | parent

I bet most of us waiting for 27b model and want to see how good it will be.

McGlockenshire 28 minutes ago | parent

My 3060 yearns for the next generation MoE!

kylehotchkiss 18 minutes ago | parent

M5U locked and loaded.

I wish there were more ~60b dense options like ripple too!

maplet 20 minutes ago | parent

It scores a 43 on Artificial Analysis with an average cost of $3.27 [1]. For reference, GPT-6.1 Sol (medium) gets a 46 with an average of $0.36.

[1]: https://artificialanalysis.ai/models/qwen4-max

esperent 14 minutes ago | parent

The table on that page shows gpt 6.1 sol (max) at $1.27 per task and it's the only model cheaper than Qwen 4 max.

By comparison opus 5.5 (max) is $10.01.

If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.

nixosbestos 13 minutes ago | parent

Really would love if these publications have a huge, bright red bolded section "How to use this right now". Is it on openrouter? Is Qwen hosting it already? Maybe I'm just ignorant?

EDIT: oh, I guess this isn't open weights, so presumably I'll have to get it through Qwen or whoever they might partner with (Bedrock?).

nelaggy 8 minutes ago | parent

is the open weights lag widening? these open models don't seem to be improving as quickly as ant/oai

taneq 6 minutes ago | parent

Might just be the commercial offerings scaling up parameters so much? Like for like the open models don’t seem to be lagging.