YHK - yet another hacker news clone

when the business model is selling more tokens you get such per serve ice times that lead to “more” thinking, engagement baiting, fluffy narratives, and straight up dark patterns

Apart from issues with task generalization, or perhaps related to it, is the fact that LLMs have real trouble with timekeeping, and cannot estimate the real world time it will take them to do things very well. This plus the memory issues make dreams of long horizon agents, that could plausibly handle changing specifications, quite implausible with current architectures.

In narrow domains with more deterministic outputs though, this is less of an issue, and we see multiple agents succeed much better.

The fusion of that capacity, with humans in the loop able to better direct such agents and act as their temporal tethers, is where I think the real action will be for a while at least.

Is this really any different to how humans learn, it takes a lot of training on one specific task to make a human expert as well?

https://arxiv.org/html/2509.24239v4

Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for illegal moves. With illegal asks discarded, none of the bots could beat a chess model calibrated to 1100 ELO.

The author of the originating post says that "current frontier models need laborious oversight and guardrails on even the simplest tasks", and he's absolutely correct.

Frontier Labs will probably survive off hype valuations but will serve the important purpose of discovering architectures/techniques that will probably spread through rumors/transfers to the rest of the world.

> the frontier labs are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers...

Even assuming this is how the AI companies are being valued (they're not), the numbers are off.

The "value" of most knowledge workers -- based on what enterprises currently pay for them -- is $50 - 70 trillion annually. It's reasonable to assume that if AI drop-in-replaced all those knowledge workers, AI companies could credibly charge somewhere in that order of magnitude, because that's what the market is already bearing.

So if their hypothetical revenues are double-digit trillions and valuations are some multiple of that, the entire AI industry would be valued at double-digit trillions at the least.

Yet cumulatively the industry (the frontier labs + the SWAG estimate of the AI parts of all the other players) are valued at, say, ~6 - 7 trillion? Which seems like a fair approximation of how much knowledge work they can currently automate.

As models advance, we shift the goalpost for what "simplest task" means. Before, "simplest task " meant "write a coherent English sentence." Now, "simplest task" means autonomously fix, review, and merge a bugfix.

In particular, I found this very misleading or irrelevant:

a typical CPU project anecdotally has about three times as many specification and validation engineers as design engineers and a 5:1 ratio is not unheard of

The reason silicon design has such verification to design ratio is because the cost of one bug is many, many orders of magnitude higher than software. Both in dollar cost and in schedule cost (it takes months to fab a chip, and if you messed up and need to spin a fix, it costs tens of millions of dollars, not counting any design engineering cost).

I don't think you can extrapolate these very industry-specific facts to judging LLMs.

Depending on where you point them, they can be incredibly useful.

They can even be useful when you point them at each other (though increasingly difficult to get good results).

I'm excited for the promise of RSI and a future where models have inherently "live" weights, but it's not clear to me that the transformer is more than a useful tool to help us get there.

I have no idea how people can so confidently say that call center work is a “controlled environment” or “repetitive”. It’s almost by definition not repetitive or controlled. Customer support is what I go to when the controlled environment has failed

This is only true if you are concerned about the intermediate steps of the model as opposed to the outcome. The huggingface hack was a perfect example of the model doing whatever it takes to accomplish the goal of maximizing its score.

No, they're really not.

They're priced in a way that would imply AI will be universal form of compute, alongside traditional deterministic systems - which it will be.

And that they will capture most of that ... which they won't.

The Frontier Labs are a very bad buy at a high price, but that partly has to do with wacky pricing, but actually mostly has to do with their relatively weak place in the value chain.

The money is going to Nvidia, who have the most powerful position.

A bit like how a retailer can take all the margins of some innovative product, if they own the channel.

AI is over-hyped, the Frontier Labs are over priced - but AI is here to stay, and will grow. Not like Skynet, but like a new form of compute. And it will take it's time, and the profits will be reaped by those with the power.

That's a reason to be bearish about AI companies, not LLMs. But is it even true? OpenAI and Anthropic have each reported ~50 billion in revenue with ~900 billion valuations. That's a high ratio but I'm not sure if follows that the only way it pans out is if we get "fully automated drop-in replacement for most knowledge workers".

It wouldn't shock me to see those revenue numbers scaling up to where they need to be over the next decade ( to, say, ~400 billion) without ever achieving drop-in worker replacements.

have you guys actually designed, built, and deployed agentic workflows?

it is actually quite hard, requires tons of time spent on evals and testing to ensure accuracy, but when it starts to work it is mind blowing.

there is no going back.

listening to people yap about AI when they have only surface level or one dimensional exposure to LLMs and "AI", but have not actually put innovations to work IN PRACTICE.. is a waste of time

> the present problem of reward hacking can be solved only by rigorous specification by domain experts. the time of domain experts is expensive. rigorous specification is itself a skill, demanding its own expertise outside of a given problem domain. even many skilled software engineers are bad at it. for the vast majority of domains, the intersection of domain experts and specification experts is ludicrously small.

- i have no idea how anyone thinks the mighty next token predictor is going to eradicate diseases and eliminate poverty https://blog.florianherrengt.com/how-llms-work.html

- i also have no idea what everyone and their momma on HN is running for more than 5 mins in the name of "agentic AI"

It also makes one wonder: What could AI solve if we managed to orchestrate billions worth of agents to take on a specific problem?

IMO the very best case scenario / potential for these are likely better than we think, but right now hidden due to logistical and financial reasons.

But if we assume that the model costs will continue to drop by a factor of 5-10 annually, there will always be a latency of a couple of years between what is completely out of reach, and what is financially viable.

Basically: If you knew AI could be affordable enough in 3-5 years so that even the most underfunded researchers could use it to solve cancer, how much would you value it now?

Personally, I use LLMs for a lot of things. Oftentimes, I'm a think out loud type of person so even having something that feels like a rubber duck, but more competent, is already amazing for me. And LLMs are a lot more competent than a rubber duck.

But especially sometimes I've noticed that LLMs can be unbelievably stupid. It recently happened a few times with Fable 5.1 as well. Ultimately, I think it comes down to that LLMs can't think broadly. In software development one can usually see this too. For example, a whole app might be built by an LLM and it didn't spend a single token thinking about security because the prompter is at the level of "build a dating app for dogs, make no mistakes". Now you have a dating app for dogs that is insecure.

Since I prompt for almost everything in my life to have an LLM as a sounding board, I'm usually not an expert either. I've noticed LLMs are amazing at "bulk search engine information aggregation" (or whatever you want to call it). So if I need something from the Dutch government, I can find it way more quickly. But oftentimes I've noticed that going for a walk and thinking about a particular thing I'm facing is a more effective way of finding a good solution.

Other times times they are not incredibly stupid, but can't form a strong opinion. This usually happens when I'm tackling a wicked problem [1]. When that's the case, prepare for LLMs to sway with you for every small change in your opinion that you ever will experience.

So I agree: drop in replacement for knowledge workers? No. Rigorous specification is usually needed yes. Though, the small win here is that it doesn't always need to be as rigorous as programming is and it can happen in natural language. It depends on the topic/problem being tackled.

I really like them as UX tools though. Amazing for interactive prototyping and requirements elicitation. And that also corresponds with what the author is saying. Though I find it a bit of a disservice saying "just 3". You know how hard requirements elicitation is? It became a whole lot easier thanks to LLMs (I might change this opinion in a year, haha, but this is the opinion I hold now).

[1] https://en.wikipedia.org/wiki/Wicked_problem

Apologies if this comment is off-topic, but it really is quite egregious, and since the article was submitted by the author I presume they are open to the feedback.

I won't comment on hiring interns as that's not my expertise (even though if you want to teach your staff, obviously I can see a problem there) but I can comment on rapid prototyping, it's what I do.

Rapid prototyping is NOT making a CMS quick. It's not about making a quick mockup of a UI. It's not about making yet another well known... anything. The entire POINT of prototyping is to make something NEVER done before. Typically that means you are reaching the frontier. You are making something with NO documentation to rely on. You are using tools, hardware or software, which do NOT have tons of StackOverflow errors. There is no dataset to crawl, there is no well structured Q&A database to train on. You have to poke and see if the thing actually works as expected, and it often does not.

So sure, if you are using interns as a trick to underpay your staff, or if you are using prototyping as an excuse to build poor quality software fast, maybe it does help. If you are genuinely prototyping, it breaks fast and the supervision overhead makes it pretty pointless, especially since typically it's by actually implementing that you find out not just how the new setup works, but also its limits, and thus the actual needs of the project, not the one the stakeholder imagined would be.

So not, not for rapid prototyping either.

TL;DR: prototyping is a learning process, not a low fidelity output.

PS: this comes up very often from NON prototypists that I wrote a short piece about it https://fabien.benetou.fr/Content/GoodPrototypesAre10LinesLo... so much so that it feels like a pattern "GenAI/LLMs is good for tasks X" while the author actually does not do task X except very superficially.

I don't know how well it is studied, but I suspect it is possible there is language-related complexity constraints to the effectiveness of the LLM algorithms. Like perhaps context-free grammar related problems with adequate training data can be more and more effectively solved, but maybe natural language related problems will not so much be effectively solved.

Would be curious if there is active research here.

Lots of people make this claim about "specific task[s] enjoying clearly defined levels of task performance" but they forget that generative AI is also extremely good at generating a) art and b) prose in literary style. None of those things has "clearly defined levels of task performance", in fact they are both the complete opposite of well-defined tasks. Who knows what counts for "good" art? [1]

For me the right model for generative AI is "a million monkeys on typewriters" [2]. Holding any other model to heart will at some point fail to predict observations and cause you to be unpleasantly surprised. Not least because AI companies are actively engineering their systems to optimise for this model and they have a lot of people working on that engineering and shedloads of money to throw at it.

Don't underestimate what a million monkeys on typewriters can do. They can do anything and everything, given enough time. Geneartive AI can also do anything and everything given enough resources. The only question is: how much is going to be "enough"?

____________________

[1] Yes yes, AI art tends to be slop. Not denying that. But part of the problem with slop is that it presents as technically very competent except that it lacks a certain je-ne-sais-quoi, which makes it good art; aesthetics. The point is that there is no clear measure of what makes technically competent art, any more than there is for aesthetics.

And yet generative AI is very good at it.

[2] There's even an article on wikipedia except it's about one monkey on one typewriter with infinite time. There's a proof too.

I think that's incorrect from the investment point of view. They'd still be worth a lot if they can produce a drop-in replacement but it takes five or ten years as long as they dominate that. The danger from an investment point of view is they become AltaVista, replaced by some Google that does the job better.