YHK - yet another hacker news clone

On the other hand, using AI to trawl through all the grunt work, experiments, and well trodden areas means I can crawl up pretty close to that giants shoulder and spend way less effort myself answering the actually important questions so I can stand on it.

(This isn't supposed to sound self aggrandising - the giants whose shoulders I'm standing on aren't very tall, and they probably have bigger brothers and sisters, but the idiom was too good of a match to pass up using)

These seem pretty easy to solve problems though if I could just afford 100 or 1000 agents. Let them explore and compete and work out their errors for themselves.

At a practical level though, I do a lot more 'bonus' R&D work now by just keeping the agents running and testing ideas in the background.

They aren’t capable of generating truly innovative ideas though. Ask it to come up with a name, it will come up with the same weird slop names over and over. Always gives people the surname Okafor. Because it works by generating probable answers, it can only generate probable names.

I suppose it's possible that frontier labs have internal models that are so far beyond that autonomous research becomes possible, but I'm not sure that's it.

It seems to me like these models are like a very competent, important part of some hypothetical digital brain, but alone they don't have all the facilities they need to be useful, like a genius with severe ADHD and short-term memory loss.

For anything where quality isn't objective in a clear-cut way, very intentional context management is needed to get good outputs. As context size expands, performance degrades. As the number of tasks increases, performance degrees. As the number of independent decisions based on subjective criteria increases, performance degrees.

If you can get a harness that makes sure every decision is made one at a time, focused with only the context it needs, things work great. But making those harnesses is still hard, a lot of work, and something that agents seem to have no clue how to help you with.

It’s a dance, not an orchestration.

If you want autonomous work of such systems you can not expect that: a) each reasoning step will make sense and be correct; b) results will come very quickly - especially in the first stages of the project.

I have very positive experience in non-trivial problems with using AI as an autonomous researcher or engineer. But it worked since an agent had a separated, fully dedicated environment and enough time to build up (emerge) some strong patterns of work. Early days were underperforming and slow progress. Now it is overperforming and frequently surprising me with new stuff.

The worst part of his premise is that they can be overcome by increased intelligence. He’s not describing fundamental limitations. Meaning that it’s inevitable that his examples will be “solved.” At least better than humans. Meaning the R&D is obviously valuable.

If they tackle that, I think models will be perceived as much smarter than they are now - even though now with the right direction they can already do so so much.

Did you see the chart on model performance over time on LifeVal? Ceiling in there. https://openai.com/index/lifeval/

I'm bearish that continued scaling will fully solve this. In the meantime though, as a commentor in the OP points out - by solving the menial stuff (building diagnostic tools, producing visualisations, scaffolding experiments) they're still proving to be extremely useful research assistants.

This sentence resonated strongly with me. Not just in R&D but in every single interaction with the model. From Fable all the way to Sonnet they all very clearly show just zero judgement unless the answer was already seen during training.

Instead of asking Astra to do the thing, the author should research similar games and try to apply the techniques used in those games. For example, Scrabble is a hidden information game. Try using an algorithm like CFR which is used to solve such a game.

Similarly, ultimate tictactoe is a perfect information game so it can be handled with MCTS.

Once you understand the basics, you don’t need an “ingenious” solution. It’s a university project.

I am working on a legacy OSGi/eclipse application. I have a problem where the OSGi bundle is not activated at the point where some code in the bundle is executing. This makes no sense to me. And it makes no sense to the AI either. The AI says: "Since your class is in your own plugin bundle, that bundle must be active for your code to be running at all".

After some more testing we figure out that actually this its possible for eclipse extension points to instantiate objects via reflection which bypasses bundle activation. So now we have a new fact:

-- It's possible for code in a bundle to execute without the bundle being activated.

We also have a new state (another kind of fact)

-- In this system this possibility has occurred. The state of this particular system is that the bundle is not activated.

So then we get around trying to debug how we got here and after trying a few different things the AI still can't believe that this could be possible and it says:

"which is definitely ACTIVE (since your code is running from it),"

Here it is contradicting the knowledge of the state of this system (the bundle is not active) and it is contradicting the fact that was already established where we now know that it's possible for this to be the case.

Then I tell it that the state of the bundle is RESOLVED. As soon as I use this word (first time in the conversation) it immediately gets out of the rabbit hole we were going down and gives me the correct fix.

For a human maintaining this state and logic takes no effort or memory. But it seems that the LLM can't really do this. It is completely biased by the training data. The funniest thing is that AI "reasoning" is the opposite of reasoning. Instead of reducing the problem down to a few bits of information and logic it actually increases information. It makes more words in the hope that those will find a match in the training space.

It is like when you have to deal with the person who says: "When this situation occurs, I always do this". It's just that the LLM has seen so many situations that it is usually correct. In real life these people can be really frustrating to deal with when the situation changes and they are exactly the wrong kinds of people to do research or even just to make decisions in a system that has changing state.

I like this point because it's so hard to define "obvious". I agree they clearly are, but when LLMs are asked to pick the most important they tend to focus on a heuristic like "would show up in a summary", not creativity. Especially in longer texts like books where context is hard to keep they anchor on plot structure, for example time, rather than narrative.

Asked GPT-6 for some historical information today, and it gave me a modern bastardization as a primary source. (It even found actual original sources, but preferred this one...)

AGI right around the corner!

But then I started using the same models for some custom work and in situations that are trivially relatable you could see 0 skill transfer unless explicitly hand held through the process.

For example I built a custom GUI toolkit for my 3D engine, GUI is defined with React components so it should be super familiar to the engine, the layout primitives are very standard and support the stuff needed to make the UI I expected - but the results I would get out of both Opus 5.5 and Astra are garbage. No padding, shit alingment, etc. stuff that it oneshots with web components/frameworks. I prompt the model to "make it look better" without specifics and it's utterly lost - you can see there's no understanding of "design" there's only "trained on a million of HTML/CSS web layouts to spit out what the most common layout patterns in the training set are". When I give it the specifics like "go to x css framework, take the default layout guidelines like sizings/paddings/colors/etc." then it gets closer to what you'd expect but if it truly undrestood design it would realize that text snapped to a vertical divider line is shit design that needs to be fixed when I prompt it to make it less shit.

Why would anyone go to the effort of struggling?

And the latest model that they measured, which is already behind the newest model they're now working on, has an insane jump in capabilities: https://www.reddit.com/r/singularity/comments/1xorf5c/metr_r...

This is easy for us to empirically verify with some back of the envelope math - OpenAI's internal model that solved Navier-Stokes ~2 months ago required 10,000 agents running for 88 hours, or 880,000 total hours, to solve it. A month later, when they dropped the 722 math proofs with some truly stunning results like UGC in there, it took just a single agent an average of 3 hours to get those results. Granted, the results aren't on the same caliber, but some of them genuinely are insane - and even if you knock off multiple digits off the 300,000x speedup there, you're still getting some insane acceleration in capabilities.

And then of course ETH was disproved, and Anthropic claimed it took just 14 million output tokens to do this with their internal research model.

Put simply, the models the frontier labs are using to speed up R&D autonomously that are making them believe RSI is imminent look nothing like the outdated models we have publicly available to us. Saying that the models today don't look like they would do autonomous R&D that well says nothing about how the internal models would perform on similar tasks - in the same way that someone looking at older LLMs and saying they don't code very well autonomously would be mind-blown to see how autonomous coding has turned out today.