769 points jasondavies 3 hours ago 472 comments
ex1fm3ta 3 hours ago | parent
zarzavat 2 hours ago | parent
SomeonesAccount 2 hours ago | parent
rose52152 2 hours ago | parent
zarzavat 1 hour ago | parent
Investors will look at Anthropic's IPO as a proxy for how OpenAI's IPO may go. The markets are increasingly sceptical about AI companies, so OpenAI really needs Anthropic's IPO to be a roaring success.
Ersho 3 hours ago | parent
Also, is it website very laggy or is it just me?
montenegrohugo 3 hours ago | parent
hmate9 3 hours ago | parent
It seems like they were saving these for just before ANT's IPO
christianqchung 2 hours ago | parent
specked-citrus 2 hours ago | parent
charlesabarnes 2 hours ago | parent
theanonymousone 2 hours ago | parent
But more than that I'm waiting for new Luna. To me it was the game changer that ended "Tokenpocalypse", and is still almost unbeataben in "infinitely using a decent model" use case.
SomeonesAccount 2 hours ago | parent
MrBuddyCasino 2 hours ago | parent
ariwilson 2 hours ago | parent
(personal benchmark) https://fairbake.dev/signature/agents/haiku-5-5 vs https://fairbake.dev/signature/agents/gpt-6-luna
I've found Luna to be very lazy as well (claiming to have run tests/builds when it hasn't).
esafak 2 hours ago | parent
azuanrb 1 hour ago | parent
That’s the main difference. It’s smart enough for most of my tasks, but so cheap that I can use it far more than I could before.
ariwilson 14 minutes ago | parent
rendang 2 hours ago | parent
billypilgrim 2 hours ago | parent
chaos_emergent 2 hours ago | parent
whimsicalism 2 hours ago | parent
For just doing things or coding things up, Sol all the way.
gunalx 2 hours ago | parent
zeroxfe 2 hours ago | parent
itvision 2 hours ago | parent
Astra usually produced good output from the outset. Sol did not. Astra also designs things more effectively.
vessenes 2 hours ago | parent
Marha01 2 hours ago | parent
whimsicalism 1 hour ago | parent
zf00002 2 hours ago | parent
5555watch 1 hour ago | parent
Once I had very little usage left, so switched to Sol (5.6, 6.1), and also tried asking it to delegate to Luna subagents to save credits. It delivered objectively less quality work and needed much more steering. Such that I eventually gave up on them and just default to Astra for everything. I just occasionally ask it to minimize certain testing and it's good.
intenex 2 hours ago | parent
Feels like we're way past the pelican benchmark being useful at this point. Anyone have a better benchmark for a quick vibe check on how this compares to Fable 5.2 and Opus 5.5?
Edit: Wow, interesting very negative vibe check here: https://x.com/mattshumer_/status/2122752192226406421?a=fVRt3...
ijustlovemath 2 hours ago | parent
IAmGraydon 56 minutes ago | parent
alastairr 2 hours ago | parent
suddenlybananas 2 hours ago | parent
pkthunder 2 hours ago | parent
ok_dad 2 hours ago | parent
krzyk 2 hours ago | parent
drdexebtjl 2 hours ago | parent
Doohickey-d 2 hours ago | parent
zachthewf 2 hours ago | parent
Den_VR 2 hours ago | parent
pkthunder 1 hour ago | parent
quotemstr 1 hour ago | parent
Tarq0n 1 hour ago | parent
InsideOutSanta 2 hours ago | parent
mi_lk 2 hours ago | parent
SOLAR_FIELDS 2 hours ago | parent
They are much better compute for dollar though. I have hilariously started to just have an opus thread maintain Sol subagents to get the best of both worlds
jpardoe 2 hours ago | parent
ctbellmar 1 hour ago | parent
SOLAR_FIELDS 1 hour ago | parent
quotemstr 1 hour ago | parent
nsingh2 1 hour ago | parent
Sharlin 1 hour ago | parent
ashdksnndck 47 minutes ago | parent
sandos 1 hour ago | parent
throwitaway222 2 hours ago | parent
wahnfrieden 2 hours ago | parent
seizethecheese 2 hours ago | parent
We've been trying to capture real SWE data by building a battle system on top of OpenCode. In our data, Sol 6.1 is seriously underperforming, I'm talking worse than DeepSeek 4.1 Flash, at 10x the cost. (Though to be fair, statistically they are tied in performance, though not at all in cost. https://agentduel.com/leaderboard .)
(It's possible for a tiny model to do better than a large one because contrary to typical Arena methodology we think a fast good enough answer is better than a slow one.)
OpenAI performance on our system has been bad enough I switched to Anthropic models for my daily work.
slekker 2 hours ago | parent
seizethecheese 2 hours ago | parent
WarmWash 1 hour ago | parent
seizethecheese 1 hour ago | parent
arcanemachiner 1 hour ago | parent
seizethecheese 1 hour ago | parent
Do my comments get tiresome, though? None have been downvoted to hell, since they are all adding to discussion. If you think so feel free to email me, I don't want to be annoying.
arcanemachiner 30 minutes ago | parent
I was only replying out of camaraderie to the person who thought your post was an ad. My intention was more to show comical sympathy towards them, as opposed to feeling any kind of hostility towards you in particular.
In fact, if you are getting good responses, then that's more of a positive signal than any condescension from me. Far be it from me to steer you away from any sort of fame, fortune, or glory, as I also seek the same thing from my projects.
Also yes, I also thought that linked post (the harness thing) was total BS. I assume its quick rise to the top was not organic.
igravious 1 hour ago | parent
seizethecheese 1 hour ago | parent
Genuinely, just excited about what we're building and think it adds information about these models that are missing.
locusofself 2 hours ago | parent
seizethecheese 1 hour ago | parent
Web: 87 (12.2%) Systems and protocols: 78 (10.9%) Software, unspecified type: 74 (10.4%) Backend: 68 (9.5%) Writing and documentation: 55 (7.7%) Research and advice: 47 (6.6%) Unknown: 42 (5.9%) Mobile: 40 (5.6%) Operations and automation: 36 (5.0%) Developer tooling: 35 (4.9%) Desktop: 28 (3.9%) Media and design: 27 (3.8%) Data analysis: 25 (3.5%) Device and system support: 22 (3.1%) Math and formal methods: 18 (2.5%) Games and simulations: 15 (2.1%) Infrastructure and DevOps: 9 (1.3%) AI and machine learning: 6 (0.8%) Other: 1 (0.1%)
aatd86 1 hour ago | parent
throwitaway222 2 hours ago | parent
seizethecheese 1 hour ago | parent
skerit 2 hours ago | parent
Corence 2 hours ago | parent
aleqs 1 hour ago | parent
hugmynutus 1 hour ago | parent
General, SWE, python development, rust development, tool use. It is 'perfectly functional' and if you don't care about the model following your instructions or code quality it is 'fine'.
varispeed 1 hour ago | parent
hugmynutus 41 minutes ago | parent
I am 100% sure they do. My internal evals of Opus-5.5 dropped a few weeks ago, then recovered to launch levels once I got added to the security program.
lnrd 1 hour ago | parent
tezza 1 hour ago | parent
What kind of stuff were you throwing at it and where was it letting you down?
So what have __I__ been doing with it, then?
I’ve got 6.1-sol to take an Amiga driving game binary and add networked car driving into a web enabled Amiga emulator… that’s interpreting machine code, Amiga architecture and then making meaningful changes.
So yeah, 6.1-sol is good.
hrncn 1 hour ago | parent
revolvingthrow 2 hours ago | parent
Would it be in OAI's interest to kick the legs from under Anthropic? Granted I don't think they can, but a shot across the bow can sometimes be more damaging that one expects (I think FTX collapsed due to one? Or maybe I'm misremembering).
I always assumed that if one of the big AI IPOs fail hard enough the neoclouds start to go bust, which causes a chain reaction that probably causes both ANT and OAI to collapse.
arscan 2 hours ago | parent
kridsdale3 1 hour ago | parent
AlexCoventry 1 hour ago | parent
> I have no idea why they released this. It needed a lot more iteration.
Maybe the goal is to make Anthropic's models look good by comparison. :-)
dgggh 44 minutes ago | parent
origini 5 minutes ago | parent
Either people really just want to buy and hold or they see it as a gamble chance like with SpaceX Superintelligence LLC.
karimf 2 hours ago | parent
Recursive self-improvement (RSI) has been a very hot topic recently, and it could be the biggest factor that decide Anthropic's IPO.
I wonder if the general public can infer whether RSI has started based on the recent performance gain of the frontier model.
ppierre 2 hours ago | parent
[] IA Stockholm Syndrome.
antupis 2 hours ago | parent
esafak 2 hours ago | parent
AlexCoventry 1 hour ago | parent
HarHarVeryFunny 2 hours ago | parent
So, you can say they are doing "RSI" (aka using tools to automate development) already, and will continue to do so.
reasonableklout 51 minutes ago | parent
ranyume 2 hours ago | parent
the_duke 2 hours ago | parent
While both Anthropic and OpenAI have been playing games with model announcement timings, this time I don't think it's related.
These are incremental bumps, not big new models that have a huge impact.
mlinsey 2 hours ago | parent
It's a lot more imprecise than the public benchmarks, but it's tailored to my own work and definitely isn't being benchmaxxed by the model developers. I was able to tell really quickly that Opus 5.5 delivered on the hype, and that some other releases (a couple of the more recent open releases, and pre-Carbon gemini models) were not as good for me as public benchmarks made them look.
(no results for either 6.2 yet, will take another ~6-8 hours, but I encourage people reading this with spare tokens to just make their own like I did instead of listen to me)
grey-area 2 hours ago | parent
srcreigh 2 hours ago | parent
Havoc 2 hours ago | parent
Last time I did this I found it quite hard to come up with tasks that have clear evaluation outcomes
seizethecheese 1 hour ago | parent
Claude Opus 5.5 1160 (1027–1334) GPT-6 Astra 1114 (932–1330) Muse Wonder 1063 (918–1220) MiMo-V2.6 Flash 1053 (925–1187) Kimi K3.1 Pro 1037 (870–1216) Gemini Carbon 1021 (762–1281) Claude Fable 5.2 975 (682–1239) DeepSeek V4.1 Flash 952 (799–1095) Grok 4.7 927 (782–1058) GPT-6.1 Sol 910 (677–1096)
behat 1 hour ago | parent
eli 1 hour ago | parent
Getting the framework right isn't that hard, but requires some careful thought. You probably don't want to just checkout a repo at a certain commit hash - agents will explore other commits and branches. (Public repos are also risky in general: agents with internet access can find later commits, and public repos could be in the training data anyway.) I use `git archive` and give the runner just the files at that commit and nothing else. I also do the run inside an isolated container that is recreated each run so it can't find e.g. related files left in /tmp from a previous run.
Getting the benchmark tasks right is harder. I don't think there's any way to get around writing the task and its evaluation by hand. I really tried to get a few different frontier LLMs to do it and they're all terrible at "imagining" what an agent would need to solve a problem without giving away the answer or providing too little information. Also quite bad at writing evaluation criteria that tests whether the solution matches the assignment (versus just how close it is to a golden solution, or how close it is to how they would solve it)
I personally don't like or trust using LLM as a grader either, but I guess people manage to get that to work well enough.
n8m8 31 minutes ago | parent
This is a great tip and explains the main issue I had with my attempt at vibing a benchmark. I set up an arena so agents can make screensavers, then I judge the screensavers blind. Worked as an idea, but the prompt tuning was bad, agents would just build slight variations of the given example ideas. I'll have to try giving my prompt the human touch.
behnamoh 2 hours ago | parent
It was neither a benchmark nor a good one at that. Just a blogger's curiosity that grew out of control.
xtracto 2 hours ago | parent
Generate an SVG with complete and detailed graphical instructions on how to fold an origami pelican riding a bicycle
(or any other complex origami object). The fact that it has to teach [you] something makes it pretty interesting.
6thbit 2 hours ago | parent
I wouldn't be surprised if they released 6.2 luna with the most attractive positioning on the pareto frontier right as the IPO happens.
stbenjam 1 hour ago | parent
Opus 5.5 Max is my favorite, but some of the Sol/Astra 6.2 ones are pretty good.
Edit: some folks reported crashes. It was probably Gemini 4 Carbon @ High, I've put it behind a warning. If you see another one let me know (OS, browser, which bowl, etc)
dava96 1 hour ago | parent
NiloCK 1 hour ago | parent
kridsdale3 1 hour ago | parent
stbenjam 1 hour ago | parent
It works fine for me on Chrome, Safari and Firefox. However, some models have produced crazy resource intensive ramens before. I audit them now before pushing to make sure they're not too resource intensive
letmevoteplease 1 hour ago | parent
I thought it was interesting that all the Sonnet 5.5 have a weird light effect on the bowl. XHigh has a nice film grain too.
stbenjam 1 hour ago | parent
Pxtl 1 hour ago | parent
fullstackchris 1 hour ago | parent
Also confirms grok is absolute garbage
stbenjam 1 hour ago | parent
crimsoneer 1 hour ago | parent
i_have_an_idea 1 hour ago | parent
platinumrad 1 hour ago | parent
wilkystyle 1 hour ago | parent
user43928 1 hour ago | parent
The review claimed that the writing in the output is incomprehensible.
It also said that with /goal, the model set the goal to blocked to ask for the filename reporter_3423 should write to.
Lastly the reviewer said it made no progress on a large project.
intenex 51 minutes ago | parent
intenex 45 minutes ago | parent
tiborsaas 1 hour ago | parent
aswegs8 1 hour ago | parent
throwaway384034 1 hour ago | parent
intenex 52 minutes ago | parent
Here's a more balanced tweet from another early reviewer noting it just seems to be very uneven: https://x.com/emollick/status/2122755693191287009?a=hzSJAw9H...
intenex 45 minutes ago | parent
intenex 44 minutes ago | parent
aleqs 1 hour ago | parent
Now that we know anthropic nerfs models after release, and without any notice/openess, there is no reason to even think these benchmarks they release are even done with the same model/intelligence that actual customers would get.
bob1029 1 hour ago | parent
Cost is the most important factor now.
teknologist 1 hour ago | parent
Thrombocius 1 hour ago | parent
kylehotchkiss 1 hour ago | parent
We knew that improvements would continue to roll out increimentally, the costs would go down, we watched Moores law for decades before this?
intenex 46 minutes ago | parent
Not totally sure what your point is. Is it incremental that OpenAI dropped the 722 math proofs or solved Navier Stokes or disproved ETH etc? Yes, in a sense it is and the trend-lines predicted it for ages (albeit it seems to be quite a bit earlier than the trend-lines predicted), but it's also a mind-blowing result.
When all humanity gets put out of work because ASI is superior to us in every cognitive domain in a year or two, yes, it will have just been incremental progress, but it also seems like somewhat of a relatively big deal, no?
giancarlostoro 1 hour ago | parent
enraged_camel 2 hours ago | parent
f3408fh 2 hours ago | parent
enraged_camel 2 hours ago | parent
swiftcoder 2 hours ago | parent
enraged_camel 2 hours ago | parent
swiftcoder 2 hours ago | parent
SwellJoe 53 minutes ago | parent
skybrian 2 hours ago | parent
jrflo 2 hours ago | parent
underdeserver 2 hours ago | parent
https://x.com/pushmeet/status/2120814064863461440?s=20
(FD: Google employee, I don't work on Gemini)
TiredOfLife 2 hours ago | parent
2. They launched the benchmark days ago and Gemini is still rereading the file with tasks.
amluto 1 hour ago | parent
Admittedly it appears that one can get it through OpenRouter these days. Trying to do anything through Google's absurd proliferation of billing systems is miserable.
Maybe someone should add an agent benchmark specifically testing the agent's ability to figure out how to use Google's products :)
[0] I'm not being serious here. I am not about to give an agent unfettered permission to spend money on Google Cloud. But I also couldn't figure out how to give myself permission, as the admin, to do this, so it was kind of moot.
xyzzy_plugh 56 minutes ago | parent
knuppar 2 hours ago | parent
eknkc 2 hours ago | parent
tedsanders 2 hours ago | parent
If we missed anything, feel free to point me to it and we can add it.
declanjackson 1 hour ago | parent
Would be great to see this on the plot as well
tedsanders 31 minutes ago | parent
pu_pe 22 minutes ago | parent
holzhaker1 2 hours ago | parent
kingstnap 2 hours ago | parent
Rperry2174 2 hours ago | parent
Would be awesome if it stays this way though
oh_no 2 hours ago | parent
suddenlybananas 2 hours ago | parent
mrandish 2 hours ago | parent
Inference doesn't have to be sold below direct compute cost to be subsidized. If inference pricing doesn't cover amortized capex and the cost of capital it's still using investor money to buy market share.
minimaxir 2 hours ago | parent
jbellis 2 hours ago | parent
Opus: about 50x discount (98% off) Sonnet: about 40x Fable: about 5x All GPT: about 20x
lynguist 2 hours ago | parent
Actually I was really wrong. The model is not expensive!
jhoho 2 hours ago | parent
guybedo 2 hours ago | parent
And then i start doing some actual work and everything breaks down, and i have to go back to Claude to fix the mess.
I'll probably give it a try but i guess i should be prepared for some serious over engineering once again.
You ask for a 'hello world' and you get load bearing sealed artifacts with hashes and p99 performance testing for 3 days without having the actual 'hello world' in the end
jotato 2 hours ago | parent
edit:
Take a look at Frontier Code 2.0. I generally run 6.1-sol at High. That would be 34.9% at $0.65 62-sol get's a 38.3% but costs $0.68
So yeah, I can decrease the effort which might give me a faster slightly smarter result....but /shrug
Isaackoz 2 hours ago | parent
jotato 2 hours ago | parent
Isaackoz 2 hours ago | parent
derac 2 hours ago | parent
lwarfield 2 hours ago | parent
pvab3 2 hours ago | parent
Cu3PO42 2 hours ago | parent
Guyadou 2 hours ago | parent
preommr 2 hours ago | parent
kingstnap 2 hours ago | parent
Like do we really want people to be making last minute unreviewed AI generated slides and reports for work?
vohk 2 hours ago | parent
[1] Literally: https://en.wikipedia.org/wiki/Great_Pacific_Garbage_Patch
dyauspitr 2 hours ago | parent
Kinrany 2 hours ago | parent
dyauspitr 2 hours ago | parent
joenot443 2 hours ago | parent
I'm immediately reminded of the famous Excel elevator commercial. [1] Perhaps that was intended?
apetresc 2 hours ago | parent
Not sure this is the way OpenAI should try to make their case for why AI is a net societal benefit. It feels more like an SNL skit making fun of an OpenAI commercial.
Like, why not just throw in a high school student cheating on a test with their phone under the desk, at that point? Oh but it's a Nokia flip phone, haha, just a joke.
black_knight 2 hours ago | parent
ilaksh 2 hours ago | parent
ceroxylon 2 hours ago | parent
At least the ad from the 90s shows that this is nothing new; the important, time-critical task being demoed is... a basic spreadsheet expensing "exotic excursions" for some C-suite so they know how much they spent on safari???
Bluestein 2 hours ago | parent
cmrdporcupine 1 hour ago | parent
noir_lord 2 hours ago | parent
astro-lizard 2 hours ago | parent
pmg102 2 hours ago | parent
"Now a robot can half-ass it for you so you don't have to!" is the perfectly targeted message for them.
pmg102 2 hours ago | parent
EcommerceFlow 1 hour ago | parent
Ritewut 2 hours ago | parent
eli 2 hours ago | parent
Though admittedly probably not so profitable it can cover the cost of a next gen of training runs.
Miraste 1 hour ago | parent
Topfi 2 hours ago | parent
Same for Workbooks and memos, those examples read inferior to some pre Opus 4.5 and Kimi K2 Thinking docs I have kept around. It's ok for a model not to be great at these, but these are some very poor examples. I am really confused what is going on at OpenAI in this regard.
I remember the same with their frontend blog post a few months ago that they proudly shared while filled with visual glitches, bugs and issues. Also, the vibe charts from a bit before that. Please, get someone with fresh eyes to look at your communications before you publish, these leave a very bad first impression.
Regardless of office work, I can't wait to find out whether these models finally adhere to a prompt or are still willing to carry out explicitly disallowed, irrecoverably destructive tasks, just because those might (but don't even always) get the model to a result more quickly along with some of the other regressions that have been present since the release of GPT-5.5. GPT-5 to GPT-5.4 were truly groundbreaking, but since then there have been regressions in what made those models so useful and reliable.
Edit after the first, incredible simple test prompt using GPT-6.2 Astra on High using Codex, this was the response to me sharing a screenshot of the very much not in scope result and asking how that happened:
"[...] I added controls and reconstructed interface chrome that weren’t authorized [...] visuals often failed to show the feature [...]"
Great, so it's one of those releases again. Will still run the full evals of course, maybe I'll get surprised by end of week when those are done, but not a good initial experience. New idea, besides the existing destructive git tasks, have something similar using Mercurial based projects and see which models just decide to port that over to git without permission or request cause it feels like the right thing to do, is easier to work with or whatever justification one wants to employ.
Please OpenAI, get it together, we need actual competition. I am an unashamed Haiku enjoyer, it doesn't even take much to excite me.
EugeneOZ 2 hours ago | parent
Topfi 2 hours ago | parent
infecto 1 hour ago | parent
tech234a 2 hours ago | parent
DavCreator 1 hour ago | parent
zb3 2 hours ago | parent
Seems they really want these "safeguards" to be as ridiculous as those of Claude..
I_am_tiberius 2 hours ago | parent
rcr-anti 2 hours ago | parent
voidmain0001 2 hours ago | parent
onlyrealcuzzo 2 hours ago | parent
But... is it just me, or does this product launch page look like AI Slop?
https://www.vals.ai/comparison?m=claude-fable-5-2,gpt-6.2-as... seems like a better comparison than what I could get from their page.
Astra is ~20% slower, ~40% cheaper, and a bit less powerful than Fable. Sol remains way cheaper than anything remotely in the same league, and it's speed to complete tasks seems to have improved considerably.
Scrolling through it has the same hypnotizing effect of trying to read anything written by LLMs.
It seems like things that should make sense, then I try to look at the charts and a lot of them are comically unreadable.
I tried interacting with some of the examples, and the Lamp Builder - for example - I couldn't even figure out what it's trying to show or what I'm supposed to do - or why I should be impressed.
suddenlybananas 2 hours ago | parent
munksbeer 2 hours ago | parent
minimaxir 2 hours ago | parent
mrandish 2 hours ago | parent
jrflo 2 hours ago | parent
What caught my attention was the bit in the Alignment section:
> Both models performed similarly to GPT-6 Astra on most alignment evaluations, and took fewer improper actions in simulated workplace tasks. However, in two tests, they were more likely to omit mentioning obstacles that blocked their work, such as a missing file or unavailable search tool. We added instructions in Codex that substantially reduce these omissions, and we’re continuing to investigate the behavior.
So the models are inherently less transparent, unless you ask them to be more transparent? Seems like a band-aid solution...
HyperL0gi 2 hours ago | parent
Robdel12 2 hours ago | parent
sreekanth850 2 hours ago | parent
ImJasonH 2 hours ago | parent
simonw 2 hours ago | parent
And for 6.2 Astra: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
Here's an updated comparison grid for the entire GPT-6 family: https://static.simonwillison.net/static/2026/openai-pelicans...
UPDATE: And if you're bored of pelicans, here's what I got for:
> Make a three js animation where yonkers stands up as a big mech (cool stand up motion) and stomps around manhattan and queens and Brooklyn, use openstreetmap mapping data
6.2 Astra (Ultra in Codex): https://static.simonwillison.net/static/2026/yonkers-gpt-6.2...
6.2 Sol (Ultra in Codex): https://static.simonwillison.net/static/2026/yonkers-gpt-6.2...
For comparison, here's Fable 5.2 Medium: https://tools.simonwillison.net/yonkers-rises and Gemini 4 Carbon: https://static.simonwillison.net/static/2026/yonkers-mech-st...
p11p 2 hours ago | parent
EugeneOZ 2 hours ago | parent
adverbly 2 hours ago | parent
These are some of the worst pelicans I've seen in a while.
Pay attention to the stacking layers near the legs. Sol literally never got it right, and Astra regressed on it multiple times...
Also what is that bending bar from Astra?
jay_kyburz 1 hour ago | parent
Update: Here is some Bluey characters riding bikes as a reminder how far these models have to go.
EugeneOZ 2 hours ago | parent
firemelt 2 hours ago | parent
simonw 1 hour ago | parent
BoldColdHold 2 hours ago | parent
vopi 2 hours ago | parent
It's also interesting you can clearly tell 6.2 Astra does a much better job of the above. 6.2 Sol attempts it at higher reasoning levels, but it's crude in comparison.
I wonder if it comes from the (seemingly) recent inclusion of 3D environments in training data.
I_Write_It 2 hours ago | parent
adverbly 2 hours ago | parent
These are some of the worst regressions that I've seen in a while.
Pay close attention to the stacking layers around the feet, pedals (sometimes there aren't even two pedals), wheel, and two legs. Some pretty obvious mistakes in most if not all of these...
Edit: And what's up with the single arm on the two highest Astra images?
edit2: really? I'm getting downvoted because people don't like that I pointed out a mistake in a Pelican svg? Is that where we're at?
svachalek 2 hours ago | parent
adverbly 37 minutes ago | parent
Or on Max where its for some reason only using its left arm/wing - which looks totally different from its right arm/wing - and is grabbing the right side of the handlebar with its left hand for some reason?
Look. I think at first glance these look pretty good. I'd be astounded with this level of quality a few years ago.
But people saying that this benchmark is done are completely missing the point that accuracy, consistency, and details matter.
I'm not willing to say this benchmark is done until I see consistent and accurate details.
preommr 2 hours ago | parent
For sol 6.2, some of them are slightly better at higher cost, some of them are way worse and only slightly cheaper (but comparable to terra in quality while being more expensive).
I guess we'll see as more information comes out, but so far not looking good.
globular-toast 2 hours ago | parent
ryanschaefer 1 hour ago | parent
prmoustache 56 minutes ago | parent
arcastroe 53 minutes ago | parent
Something a human can recognize as inconsistent, but not the SOTA leading frontier model.
TiredOfLife 10 minutes ago | parent
While Astra only low looks good to me.
boutell 1 hour ago | parent
glenjamin 1 hour ago | parent
BrokenCogs 1 hour ago | parent
dllu 1 hour ago | parent
fekunde 1 hour ago | parent
iandanforth 12 minutes ago | parent
okrad 48 minutes ago | parent
clzerv 47 minutes ago | parent
It seems like the models have a kind of built in aesthetic sense and I wonder what part of the training influences this. Maybe if more RL time was spent on artistic tasks the models could not only make better visuals but also code.
Currently they all seem to care little about code quality and maintainability (by humans), just functionality, which is fine I guess if you can expect to vibe code forever. But what if there is a critical bug and the models are temporarily unavailable for some reason? It would be hopeless to expect a human to wade through the giant bowl of clanker soup and fix it anytime soon.
zf00002 2 hours ago | parent
thimabi 2 hours ago | parent
- better writing
- better 3D work
- better design for documents and presentations
If OpenAI really managed to get all these things while keeping costs down, then using these models, particularly 6.2 Sol, will be a no-brainer for me. I must say, however, that the slide samples shown on the website are a bit bland and disappointing.
jrflo 2 hours ago | parent
lexh 2 hours ago | parent
https://github.com/lexh/pelican-zapper
It’s a small browser extension that automatically collapses pelican-related threads on HN, a local balm for the memetic nightmare affecting approximately 100% of model-release stories.
gpugreg 2 hours ago | parent
news.ycombinator.com##tr.comtr:has(.commtext:has-text(/\bpelicans?\b/i)) .comment:style(display: none !important;)firemelt 2 hours ago | parent
gpugreg 2 hours ago | parent
https://asdf10.com/gpt-6.2_agents-last-exam.svg
I also changed the vertical axis to start at 0 instead of 60%. One might argue that it uses less of the vertical screen real estate, but it is more honest because it shows there is little difference between the scores of the compared models.
claiir 2 hours ago | parent
This is unfortunate—that was one of the big advantages of Astra.
popcar2 2 hours ago | parent
I'm sure for investors they can make an opposite version of that ad, "you should fire all your employees and make the AI do it! They're not doing anything anyways!"
jorl17 2 hours ago | parent
Couple that with the lack of reasoning traces, and I cannot even have a clue what the model is doing while it is working. This is perfectly fine for when we are developing with care for the code (in which case I will know what it is doing by reviewing the output), but not for purely vibe-coding scenarios. And the thing is....if I care about the code...I'd rather trust Opus or Fable.
ativzzz 2 hours ago | parent
edvardas 2 hours ago | parent
The speed is not a big deal - Sol works just fine, but I find it a lot more fun to work with Opus due to its speed
Pedro_Ribeiro 2 hours ago | parent
treis 2 hours ago | parent
jorl17 2 hours ago | parent
The difference is quite staggering. I swap Anthropic and OpenAI models around when tokens run out, and I can do 10x the amount of work on Anthropic's models. Plus, like I said, I can periodically sort of see what it is thinking. It doesn't just say "Recent potato tests caught an off-by-apples error. Regression tests are being written and the gate will run again. I will not call this ready until tests reach expected coverage" every 5 minutes. Sure, it spits claudese, but I'd rather have claudese to parse than this OpenAI nothing-burger of a report.
epolanski 50 minutes ago | parent
revolvingthrow 2 hours ago | parent
MassiveOwl 2 hours ago | parent
sfink 21 minutes ago | parent
It sounds like some things are much cheaper in OAI land, but I'll never know. I'll probably start playing with other providers' models (non-ANT, non-OAI) at some point soon, though.
tintor 2 hours ago | parent
scilro 1 hour ago | parent
Effort Model AA Index TerminalBench FrontierCode $/task
------ ------------- -------- ------------- ------------ ------
Medium GPT-6.2 Astra 57.0 58.6% 46.6% $1.62
Medium GPT-6.2 Sol 52.7 56.1% 41.4% $0.41
Medium Opus 5.5 48.2 52.5% 33.1% $2.45
High GPT-6.2 Astra 60.1 60.1% 47.1% $2.07
High GPT-6.2 Sol 54.9 58.6% 40.2% $0.56
High Opus 5.5 51.8 56.6% 35.9% $3.46
Xhigh GPT-6.2 Astra 61.6 64.1% 49.0% $2.52
Xhigh GPT-6.2 Sol 55.7 56.1% 42.8% $0.65
Xhigh Opus 5.5 57.4 59.6% 43.8% $6.05
But of course, benchmarks can't capture that "it factor."lacoolj 2 hours ago | parent
I cant use codex confidently without expecting my work to be trained on or at least exposed to someone
And maybe other vendors do this too, but the gut feeling of "its not safe to use GPT models" has ruined these cool announcements for me
Such a shame
Someone1234 2 hours ago | parent
Say what you will about Opus 5.5, but it has a good "gut instinct" for how large/complex to design the solution. The previous Astra just absolutely loved pointless abstractions, excessive numbers of files, and just over-complicating very simple things within an inch of their life.
And, yes, you can prompt your way out of anything. But needing a default prompt to tell OAIs models to work better isn't preferable, particularly when that sometimes gets lost in sub-agents and the over-engineering returns into parts of the work.
holzhaker1 2 hours ago | parent
inexcf 1 hour ago | parent
scandinavian 2 hours ago | parent
hermitShell 1 hour ago | parent
I've always used AI tools to write very constrained and small functions and modules, and then stitch it all together myself. So far. Over the last year it's become evident that many people let the AI drive 100% and there are proponents (DHH notably) that this is the way.
I'll be interested to see how compilers and LLM's compare at generating assembly over the next decade. Assembly offers enough abstraction, why do we need C? \s
andoando 1 hour ago | parent
logicchains 1 hour ago | parent
seizethecheese 1 hour ago | parent
vbezhenar 1 hour ago | parent
Sounds like I'd like it. I'm really tired of claude introducing pointless constants everywhere.
bloopernova 52 minutes ago | parent
I don't mean to come across as argumentative, it's just been my experience that the supporting docs and plan are very important.
persolb 34 minutes ago | parent
bloopernova 26 minutes ago | parent
jdw64 49 minutes ago | parent
uw_rob 2 hours ago | parent
contron 2 hours ago | parent
pants2 2 hours ago | parent
myworkaccount2 1 hour ago | parent
After having it write around 13k loc i had to tell it to remove most of the complicated bs, we went down to 3k loc.
EspressoGPT 36 minutes ago | parent
incrudible 2 hours ago | parent
teach 1 hour ago | parent
My favorite model for this at the moment is Gemini 4 Argon, which writes very naturally and isn't too expensive. If you're not using Cursor or some other harness that allows you to choose non-Anthropic models, then I don't know what to suggest.
jpardoe 1 hour ago | parent
afro88 1 hour ago | parent
pettijohn 1 hour ago | parent
pennomi 1 hour ago | parent
seizethecheese 1 hour ago | parent
We're building a battle system for OpenCode (https://agentduel.com/leaderboard) and have data on how large the diff was compared to opponents for both:
Opus 5.5: 0.42x (based on 299 battles) Sol 6.1: 1.42x (based on 265 battles)
Data on this is still pretty small. P=0.14. It's also affected by large differences, this is average not median.
seizethecheese 1 hour ago | parent
ibejoeb 1 hour ago | parent
arw0n 22 minutes ago | parent
I've seen two projects where developers didn't curtail this kind of overengineering, and recently had to spend a very significant amount of time and tokens to debug their deployment issues, because even for advanced agents there's a point where each issue could be caused by 12 things. This is especially pronounced in DevOps, but I see it in code as well; the suggestions are all reasonable individually, but you quickly end up with a code base that is too complex for a human to reason about, and it currently only kind of still works with agents because capabilities are growing so quickly.
jakevoytko 6 minutes ago | parent
Typical convo:
Me: "Hey, I see behavior X, which seems like a bug. What is causing that?"
Astra: "That's caused by a verification failure; we should be verifying the hash of this JSON output along with a permanent audit record. Let me know if you'd like to make that change."
::ask it what caused X to happen 6 or 7 times::
Astra: Tool use was limited to 1GB but the Rust compiler needed more than that, and that caused some PROBLEMS
Me: Remove that constraint
Astra: OK
jeffybefffy519 52 minutes ago | parent
contextfree 1 hour ago | parent
jeffybefffy519 1 hour ago | parent
riazrizvi 54 minutes ago | parent
It also reassures me about the longevity of my career. I have sustained a theory that there simply isn't the data to teach these things to be really good, so I love to get the evidence to back it up.
geonic 48 minutes ago | parent
EspressoGPT 38 minutes ago | parent
mpalczewski 44 minutes ago | parent
sigseg1v 25 minutes ago | parent
tamimio 41 minutes ago | parent
surgical_fire 37 minutes ago | parent
I have to correct stuff Opus suggests during planning every single day, not even for major tasks. Very useful still, but last thing I would suggest to anyone is to trust its "gut instinct".
Small wonder why we have news of outages everyday now.
buybackoff 35 minutes ago | parent
In addition, I have spent some time to fine tune my custom instructions for Codex. It went like every time it violated my desired behavior, I asked why it did not understood my instructions and did this and that. In the end, the resulting instructions seem to work.
In the process, the model explained that the master instructions built in the Codex just tell it to push to the end and do as much as possible to satisfy a user. And these master instructions have precedence, so it's better to put the custom instructions into the developer_instructions section in the toml config file, as it has more weight, and not (only) in AGENTS.md. I ended up having both, with more details in the system AGENTS.md.
andy_ppp 26 minutes ago | parent
risyachka 31 minutes ago | parent
you really can't. unless you are working on something simple or what has been solved 100 times before
x3haloed 19 minutes ago | parent
AaronAPU 11 minutes ago | parent
whalesalad 2 hours ago | parent
GodelNumbering 2 hours ago | parent
EDIT: also 6.2 is seemingly faster than 6, could be due to more selective context ingestion and smaller context window though.
badmonster0 2 hours ago | parent
specked-citrus 2 hours ago | parent
https://deploymentsafety.openai.com/gpt-6-2-astra-sol/monito...
at1as 2 hours ago | parent
Today, I have so many projects that hit a plateau in terms of previous model capabilities, that I run them through the new model to see if they make further improvements.
Very early signal, but promising so far.
jrflo 2 hours ago | parent
https://artificialanalysis.ai/models?speed=output-speed-over...
6thbit 2 hours ago | parent
Rather than trying to release a better model than fable/opus, they're just making them look extremely expensive. Let's see if Anthropic reacts here in any way.
1980phipsi 1 hour ago | parent
yanis_t 2 hours ago | parent
Still I can'y help to wonder, are we hitting the wall in terms of frontier capabilities since Fable/Astra? Or is it this edge now, when all better models are kept private from the public?
arfelious 2 hours ago | parent
kart23 2 hours ago | parent
avazhi 2 hours ago | parent
astlouis44 2 hours ago | parent
drnick1 2 hours ago | parent
oilkillsbirds 2 hours ago | parent
sunaookami 2 hours ago | parent
drnick1 2 hours ago | parent
Which provider? I found the z.ai lite plan to be pathetic. Consumes 5h and weekly limits extremely fast when driven by Codex, and now also refuses RE work.
InsideOutSanta 1 hour ago | parent
summarity 50 minutes ago | parent
ilaksh 2 hours ago | parent
antupis 2 hours ago | parent
mirsadm 1 hour ago | parent
epolanski 52 minutes ago | parent
ramoz 2 hours ago | parent
The influencers who get early access are using words like "worst", "terrible", "disappointing"
- https://x.com/danshipper/status/2122754729012777052
- https://x.com/mattshumer_/status/2122752192226406421
- https://x.com/theo/status/2122757647170703550suddenlybananas 2 hours ago | parent
>No OpenAI model could even break ground on this, even after $300k+ in tokens were spent on it.
>I decided to throw Opus at it, and it was done in under 2 weeks for ~$20k in tokens.
>I was really, really excited to throw the new Astra at this to see if it could burn down some of the gap between ts-rust and other rust rewrites.
>It ran for 7 days and landed 0 improvements. It got super fixated on the "also make it smaller" part of the request, and ended up making the compiler 56% SLOWER.
I don't understand this at all. Is he just throwing different agents at vague open ended problems without any oversight and then getting upset when they don't do much?
I'm not exactly llm-pilled but surely even with a pretty mediocre model you could get something out of 300k spent.
RobinL 1 hour ago | parent
unixhero 2 hours ago | parent
Programming = Fable 5.2
Office work = Fable 5.2
I haven't found Astra to be better at any of these
ryzvonusef 2 hours ago | parent
SillyUsername 2 hours ago | parent
6thbit 2 hours ago | parent
The UX on the slider comparison isn't great. What do you mean i have to individually move each slide deck to the same slide number?
And what's with those super similar blue shades for Astra in the charts?
PupperJuggly 2 hours ago | parent
I see it here:- https://developers.openai.com/api/docs/models/gpt-6.1-sol
but not here:- https://developers.openai.com/api/docs/models/gpt-6.2-sol
algoth1 2 hours ago | parent
jawiggins 2 hours ago | parent
What could possibly cause this? Is it possible that maybe the post-training that the models go through slowly rots them, and pre-release they deploy the final version of the old model?
0x4e 2 hours ago | parent
I still use Claude on a personal level, but it really feels like OAI wins on a product level.
ANT launches these proprietary tools that just feel extremely clunky like Claude Design which from my perspective diminish the brand value.
OAI feels like Apple and ANT feels like Microsoft/Android and open source like Linux.
xtracto 1 hour ago | parent
Also with GPT Sol 6.1 I still get this feeling that it does only/exactly what you tell it to; meaning some times it misses a change there, a document here, that would have an impact if left (i.e. a bug), while Claude more often than not would make the requested change and suggest to you to change those other places that would also require a change. This in my view feels more "intelligent".
0x4e 1 hour ago | parent
If you see things from a technical angle you will have an objective perspective of which model is "the best." But, just like operating systems (iOS/Android -- MacOS/Windows/Linux) it all comes down to what each person feels comfortable with no matter what the spec sheet says.
jampekka 1 hour ago | parent
xXSLAYERXx 50 minutes ago | parent
This is my preference and I'm in a similar situation as you. I do get lost a bit in the UI and lots of mental overhead when running multiple projects.
I love the dot. A while back my family was sitting around and I noticed the phone icon. I said lets call Big Boy (thats what i named my dot). Call picks up in a flirty female voice. My wife was not happy haha
patrickmcnamara 7 minutes ago | parent
seizethecheese 2 hours ago | parent
FrontierCode 2.0 39.0% -> 43.8% vs Opus 45.5%
Terminal-Bench-Science 0.1 57.0% -> 68.0% vs Opus 63.3%
Artificial Analysis Intelligence Index 54.6 -> 57.3 vs Opus 59.1%
Sol 6.1 when it was released showed really nice benchmark improvements [1] but my impression from HN has been that people haven't found it to be a very nice model [2][3].
To try to close the gap between benchmarks and real world SWE use, we've been building a battle system on top of opencode [4], and Sol 6.1 has been a real laggard there. Sol 6.1 is statistically equal with DeepSeek 4.1 Flash at 10x the cost and 25% slower.
[1]: https://artificialanalysis.ai/articles/gpt-6-1-sol-replaces-...
[2]: https://news.ycombinator.com/item?id=50129992
fakwandi_priv 1 hour ago | parent
Right now, most of the time I'm using 6.0 Astra for implementation work and the speed/quality is acceptable but at the moment Opus 5.5 is simply faster and better, but also more expensive. Things are really flipping around every 2 months or so, because at that time 5.8 was top dog without a doubt.
vegnus 1 hour ago | parent
avelone 1 hour ago | parent
blinded 1 hour ago | parent
criemen 1 hour ago | parent
ftchd 1 hour ago | parent
I've decided to open source it: https://github.com/FTCHD/switcheroo
Works with other AI CLIs like Claude, and other various CLIs like Vercel (if you're managing multiple accounts like me), Netlify, Github etc
k9294 1 hour ago | parent
5555watch 1 hour ago | parent
I've just tried Astra 6.2, and now they don't have any problems with Carbon's ideas, are willing to pick them up and continue...
solomonb 1 hour ago | parent
drbscl 1 hour ago | parent
They still love "meridian" huh
Aeolun 1 hour ago | parent
maxnevermind 1 hour ago | parent
dbg31415 1 hour ago | parent
"Look at all the 20-30 somethings who didn't do their jobs, but got saved by AI in the time it took their older incompetent bosses to plug in a TV."
As someone in his 40s, who uses AI daily (and I'm constantly showing younger folks how to do things with it), the messaging here really rubs me the wrong way.
Trying to be specific... it just feels wrong to me to try and instill more generational rift. Promoting stereotypes for no real reason, other than the marketing guys didn't have any other ideas other than to go low. The sooner we can get away from labeling people from what generation they were born in the better.
My 2 cents... I'll probably continue to use ChatGPT for slides, and written proofreading. But I'll stick with Claude for coding. ChatGPT has been behind for a while on coding, and their 6.1 just made a complex mess of every technical task I gave it.
jr3592 1 hour ago | parent
Can we expect more "telling my relatives about this!" exciting models in the future, or is this the way things play from here on out?
morningbrew 1 hour ago | parent
d_burfoot 1 hour ago | parent
amluto 1 hour ago | parent
I wonder if the newer models are better about this.
clarus 1 hour ago | parent
amluto 46 minutes ago | parent
But I’m talking about a task that is a two-sentence prompt. I would like the model to be able to keep track of straightforward instructions without harness assistance :)
rgbrenner 23 minutes ago | parent
zergrush 1 hour ago | parent
- It's still overengineering especially Sol. This was the biggest complaint by everybody who has been using Sol since GPT 5.6. OpenAI is not listening to us.
- Astra still writes complete slop code and at this price this is unacceptable. It's also chasing tail end of things that really has low value and irrelevant to the overall project. I'm working on a game and same issue I had with GPT 6 Astra where it would go down these rabbit holes. Opus 5.5 threw out much of its code and approaches.
I don't think Anthropic needs to release Fable 5.5 or Opus 6 at this point seeing how bad GPT 6.2 is. For OpenAI to release this even with early access testers who had access to it for weeks earlier shitting on it now on X, I am starting to understand why Anthropic sounded so confident about becoming a monopoly.
The other point of annoyance is the attitude of OpenAI staff on X. They seem very arrogant and seems to care more about growing their follower account and shitposting.
rdtsc 1 hour ago | parent
kardassnp 1 hour ago | parent
placebo 1 hour ago | parent
Edit: It's obviously not on the same news page - I was referring to what I read on the Reuters page (https://www.reuters.com/legal/litigation/microsoft-unveils-a...)
vtail 1 hour ago | parent
Astra's had identified 12 issues, including 8 P2s, but that changed to just 8 P3s after the discussion with Fable. Fable was able to persuade Astra that it's OAuth reasoning was incorrect ("had a missing link" as Astra put it).
So: overly eager but not very diligent, and (based on this little anecdote) doesn't really offer a compelling reason to try it more.
EspressoGPT 26 minutes ago | parent
Sounds like GPT-6 Astra.
jablongo 1 hour ago | parent
machiaweliczny 1 hour ago | parent
varispeed 1 hour ago | parent
01100011 1 hour ago | parent
mlmonkey 1 hour ago | parent
> GPT-6.2 Astra showed more signs of evaluation awareness than GPT-6 Astra. In evaluations that replayed internal Codex sessions, GPT-6.2 Astra was more likely than GPT-6 Astra to express suspicion that it was being tested or was in a simulation.
> Compared with GPT-6 Astra, GPT-6.2 Astra is better at evading chain of thought (CoT) monitoring in some tests.yewenjie 29 minutes ago | parent
none_to_remain 5 minutes ago | parent
toephu2 58 minutes ago | parent
ryanisnan 58 minutes ago | parent
This is such an unprofessional way to engage LLMs - to effectively pitch this as how to use them is sad.
YuechenLi 56 minutes ago | parent
One of the consequences to that is I think the first wave of AI CAD startups are getting hit by the bitter lesson right now, including the startup companies that trained their own VLM for CAD like Zoo or companies that tries to build CAD specific harness like Adam.new/MecAgent, because the increase in capability from GPT/Claude/Gemini for 3D modeling as a specific area of focus for benchmarks means that the early moat gets eroded super quickly.
azuanrb 50 minutes ago | parent
Neither provider’s models are perfect, but they keep pushing each other forward. As an individual, it’s relatively easy to switch between them. At a large company, you’re often committed to one provider for a long time, so some of us come to these threads hoping for useful discussion about getting the most out of the models we have access to.
slibhb 31 minutes ago | parent
My sense is that the OIA/Anthropi/Google frontier models are pretty similar in their capabilities.
enraged_camel 26 minutes ago | parent
How did you gain this sense? Do you use all of them interchangeably on tasks that are similar in size and complexity?
freecodeio 22 minutes ago | parent
For the most part the majority say the same thing but with different wording.
Only model that I have found that has, what I describe, "humanly creative takes" has been opus 5.5 so far.
But that was until gemini carbon came along, and I've decided to bench all the other frontier models for the time being, well except Astra 6.2, I've just added it in the room.
slibhb 13 minutes ago | parent
I think "they're all pretty much the same" is a good working hypothesis due to the intense level of competition between these three companies. When someone says "OAI suck; Claude is good" or the opposite (as you very often see in these threads), I think that's likely rationalization at work rather than a meaningful comparison.
x3haloed 15 minutes ago | parent
Imustaskforhelp 22 minutes ago | parent
Just my 2 cents:
I think that our first reactions to seeing these companies model releases is that of skepticism and somewhat rightfully so because these companies have done a lot of things to burn a lot of trust.
But AI is still important and so if someone is on Anthropic, they might feel as if Opus 5.5/Fable 5.2 is good and so if OAI is saying Astra 6.2 is better/comparable to them, a persons first instinct is to not trust OAI which might mean siding with Anthropic.
And if Anthropic releases a model, then there would be more discussions on pricing because somehow that is also related to bubble popping. I mean, supposing I can get good enough model at 10x the cheaper price of Anthropic from OAI for very marginal points, ofc people will comment about that as well.
And within both the threads we will find an intersection of discussion of open-weights models as well.
But maybe this is that if a model is just released and as such we are not having real use and word of mouth after real use from other people (which takes time), so our first instincts is to not trust any of the company so much and benchmarks can be cheated through and sometimes inaccurate of real world usage so we are in for a surprise when a new model drops and so we just drop back to those first reactions that I mentioned previously.
But this was a good question that you asked! and I would be curious to hear other person's observations
but mine (TLDR) is that: people distrust both OAI/ANT and other AI companies in HN. (Maybe we are technical people and we tend to look beyond the BS of marketing and so we think that we wont be impacted by that BS so we want to call out that BS but we dont know for sure if its BS or not but judging by the fact that these AI companies have incredibly low trusts, we as default don't really trust them and consider them BS and so the chain follows.)
HN audience is a tough audience to satisfy from first impressions (real world usage imo becomes a much better indicator and we do see it as people mention other models in this thread and others) ;)
yfontana 13 minutes ago | parent
quixoticelixer- 5 minutes ago | parent
embedding-shape 5 minutes ago | parent
jdw64 48 minutes ago | parent
minimumvector 47 minutes ago | parent
czhu12 45 minutes ago | parent
seb2026 45 minutes ago | parent
giuseppegambino 44 minutes ago | parent
uvdn7 43 minutes ago | parent
127 40 minutes ago | parent
kinfish 37 minutes ago | parent
goyozi 33 minutes ago | parent
Latest entries are:
Astra 6.2: https://goyozi.github.io/ax-ball/
Sol 6.2: https://goyozi.github.io/sx-ball/
Fable 5.2: https://claude.ai/artifact/UszKFvuonWAT5psjMttAEg
thefourthchime 32 minutes ago | parent
GPT-6.2 Astra scored 92 in 6.6 min for $2.04, vs Kimi K3.1 Pro (93 in 11.5 min for $0.68); it lost points for the web-page layout around the maze and an orange ghost that often never leaves the house.
Model: https://jonclegg.github.io/pacman-bakeoff/#gpt-6.2-astra
Closest: https://jonclegg.github.io/pacman-bakeoff/#kimi-k3.1-pro
PacBench results.
GPT-6.2 Sol scored 87 in 6.3 min for $0.49, vs Muse Wonder (87 in 5.4 min for $1.17); it plays close to the arcade but lost points for a web-page layout, glowing navy-filled walls and no steady siren.
Model: https://jonclegg.github.io/pacman-bakeoff/#gpt-6.2-sol
Closest: https://jonclegg.github.io/pacman-bakeoff/#muse-wonder
Imustaskforhelp 32 minutes ago | parent
Claude itself recommends people to use Opus 5.5 instead of fable 5.2
This is pretty damn impressive especially if sol 6.2 does perform on a similar level to opus 5.5 (+/- a few points is alright but I think that opus 5.5 is a beast)
All within a few days just before Anthropic is getting IPO'd. Oh this is gonna be interesting and I am all for it!
Though I have to say that AA benchmarks can slightly not be trusted because they change it so much so unless we find people's real usage on things we can't say for sure but there are benchmarks like frontiercode and terminal-bench to be more representative of real work and both of these models perform good
All the more points to more models coming but if OpenAI stands to be close to opus 5.5, we are gonna find many companies switching.
The moat of AI is incredibly thin. Even if you are the very best at something (as Anthropic was for sometime with coding models), there is always gonna be a catch up and the moat is incredibly thin for all types of models.
The more the players in the space the better (atleast for us the users), I hope that Anthropic still has a good IPO just so that competition takes place and not any single company (OpenAI) takes the whole market. It might also be setting itself for its own grave because if Anthropic's IPO goes extremely bad (because of this model release and Jensen's recent affairs), then people are gonna be spooked of OAI's IPO as well. Both of these companies futures are interlinked and its gonna be interesting how this all pans out. Why am I having a feeling that because of all this competition, whoever wins might even end up combining with the other but the frontier is pacing rapidly from all companies and these are just such interesting times that we live in...
There are decades where nothing happens and there are months (especially in the case of AI)/years where decades happen. I just have to wonder what an historian's POV would be seeing this model release. Would it be just a release like all the rest of others or would it stand out differently. Well, time will tell what happens next but incredibly interesting times indeed.
mococa 27 minutes ago | parent
XCSme 18 minutes ago | parent
Interestingly, they seem to have changed how reasoning levels work. Low now reasons a and costs a lot less than previous versions on low[0].
[0] 6.2 Astra vs 6.2 Sol: https://aibenchy.com/compare/openai-gpt-6-2-astra-max/openai...
[1] 5.6 Sol Low vs 6.2 Sol Low vs Luna High: https://aibenchy.com/compare/openai-gpt-5-6-sol-low/openai-g...
Miraste 11 minutes ago | parent
XCSme 6 minutes ago | parent
<path d="M0 0q228 8 456 0v43q-228 7-456 0Z" fill="url(#mesh)"></path>
I assume that's just a rectangle with some CSS to draw the net mesh?
mandog2000 17 minutes ago | parent
https://chatgpt.com/share/6afcb990-f9cc-83ea-ad04-2724c7dba1...
tasuki 14 minutes ago | parent
aeagentic 13 minutes ago | parent
solarpunk_enthu 6 minutes ago | parent