37 points hn-aarvin 1 hour ago 15 comments

quantisan 25 minutes ago | parent

When and why should I trust this index over others?

ummonk 21 minutes ago | parent

I wouldn't trust any particular one, but obscure / indie indexes are going to be less affected by benchmaxxing than the well known ones, so it's a positive sign for Neon that it is performing so well on a lot of them including this one.

js4ever 20 minutes ago | parent

Is it just me or all Google models are terribly bad? They all seems benchmaxed to hell and producing total crap compared to Claude or openai models from my own experience (mostly coding)

copperx 18 minutes ago | parent

Are you saying Neon is just as bad?

mjburgess 17 minutes ago | parent

And tokens/task is very high. They don't seem to have figured out making the models usable in a typical interactive workflow, ie., they're in a Opus-4.7 era.

Very high hallucination rates too.

SwellJoe 16 minutes ago | parent

I think it's just you. Carbon is excellent but extremely slow. If Neon is better than or equal to Carbon while also being a lot faster, that'd be a great model. I haven't tried Neon, yet, but they seem to have understood the assignment with the latest round of models.

azan_ 15 minutes ago | parent

Yes. Every Google model so far has been disappointment. And their biggest moat, the integration into other google products has been laughable. Today I've asked gemini in gmail to download all attachments from one e-mail address and it turned out it's unable to do it. It's unbelievable how incompetent they are, Google has such a massive edge over OAI and Anthropic and it's still losing.

andunieee 6 minutes ago | parent

You mean they are not providing you with their best models for free?

hadlock 13 minutes ago | parent

After hearing some good things from my tokenmaxxing buddies (or whatever you want to call them, AI enthusiasts), I got a $20/mo sub to try out Neon and it's just breezed through some hardware graphics bring-up problems I've been struggling with on Sol/Astra and Fable. I'm gonna drop my $200 OpenAI sub this coming month in favor of Google's $200 plan just for more Neon tokens.

Yahivin 11 minutes ago | parent

With Neon Gemini is good now.

asadm 4 minutes ago | parent

this week I ran out of codex astra quota and switched to antigravity, boy is Gemini Carbon amazing model! It actually helped achieve a breakthrough I was stuck on for weeks with Astra.

YuechenLi 13 minutes ago | parent

Ok, so I can see that this is a benchmark the submitter made and submitted themselves, which is kinda misleading and should at least be disclosed.

But my question is, what does this benchmark actually measure about the LLM's engineering abilities? It just looks like a page of random bar graphs to me.

andunieee 7 minutes ago | parent

How do I use it? Do I have to create a Google Console account and navigate through 15 menus in order to generate an API key? Or are there another 7 different ways to achieve similar outcomes?