Yet another Hacker News

Chat-based Large Language Models replicate the mechanisms of a psychic's con

76 points jalev 2 hours ago 43 comments

rahidz 1 hour ago | parent

Man I remember back when a psychic conned me by solving the Navier-Stokes problem.

Also >July 4th, 2023

rusk 1 hour ago | parent

Did an AI do that on its own though? I heard it was human mathematicians using a sophisticated machine as a tool.

mstank 1 hour ago | parent

It sounds like LLMs were pretty useful to them…

ModernMech 1 hour ago | parent

So was Lean. Did Lean solve it?

zamadatix 38 minutes ago | parent

Nothing is solved in isolation but credit usually goes to wherever the new work in the paper comes from instead of the whole mountain of previous mathematics or existing tools used. The most relevant of those get referenced and then this reference tree builds a tree of collective base work needed across history.

meowface 1 hour ago | parent

All of the latest big proofs were driven by professional human mathematicians steering and priming the models, yes.

All of the best AI-made software projects are also driven by experienced human software developers steering and priming the models. Does that mean the projects "aren't made by AI"?

No, it just means AI is not quite good enough yet to fully replace humans, and, so, unsurprisingly, the best results will be obtained from people who are already great at a field and who take the time to squeeze as much force multiplication out of LLMs as possible. The AI is still doing well over 95% of the significant work.

TomGarden 1 hour ago | parent

While I agree with you, I think we also have to concede that this is not how these accomplishments have been presented. I'd argue most people I've seen talk about this online are unaware of the mathematicians steering the models.

basch 58 minutes ago | parent

“Good enough to replace humans” isn’t necessarily the benchmark.

The question is is a computer with a human stronger than a computer without a human. At what point does the hybrid go from being stronger, to the human getting in the way, or steering the computer in more wrong directions that right ones, or the human not being able to keep up. Does the human add enough extra randomness to be of value for a while, even as a minor co-processor.

agileAlligator 9 minutes ago | parent

https://chatgpt.com/share/6a5fdc7a-d6f8-83e8-bbea-8deb42cfed...

Terrence Tao's conversation with ChatGPT is very illuminating.

HN Discussion: https://news.ycombinator.com/item?id=49010345

broast 40 minutes ago | parent

The way frontier models work, that loop will get compressed to a one-shot within a version or two

Angostura 1 hour ago | parent

But were you successfully conned into believing an AI had solved the Navier-Stokes problem?

tescreal 1 hour ago | parent

What is the consensus among experts?

grey-area 37 minutes ago | parent

That OpenAI stole the discovery from them.

daishi55 23 minutes ago | parent

That is very much not the consensus of experts.

inopinatus 51 minutes ago | parent

The more compelling inversion is whether the likes of John Nash, Richard Feynman, John Conway etc could’ve had a lucrative second career as conmen.

atemerev 42 minutes ago | parent

von Neumann could. The others, I doubt it.

mistercow 33 minutes ago | parent

Feynman for sure could have. In fact, I think he sort of did, although I'm not sure he meant to. Multiple generations of nerds now have taken books of his anecdotes varying in plausibility and obvious exaggeration as some sort of weird physics cult of personality gospel. But I don't think he was really setting out to curate his legacy so much as he was a good storyteller and he liked to entertain.

But could he have conned people on purpose? Absolutely.

trescenzi 38 minutes ago | parent

LLMs can be supremely useful but also not intelligent. It might seem like a pointless distinction but the way we talk about these models matters because it impacts how we interact with and understand their outputs.

For example if there’s a strongly held belief that models are independent intelligent entities we’re more likely to lay blame upon them instead of their user. It’s important for the safety discussion too. If they are a new class of life then safety is going to focus on making sure they don’t do bad things. If we instead see them as statistical models we will instead try to make sure people don’t misuse them.

This distinction is even more important today when some of the most powerful people are looking to absolve their crimes by passing them off on their LLMs.

krupan 32 minutes ago | parent

The LLM did not solve it. It's not intelligent. Humans did, using a statistics-based computational tool (the LLM). We don't even know all the details of how the tool was used, we haven't been allowed to use the exact tool they used ourselves, we don't know much it really cost in dollars, energy, or time, etc. etc.

claytongulick 10 minutes ago | parent

And it may have trained on a NYU professor's work.

atemerev 1 hour ago | parent

All of this is accurate (and useful). But yeah, psychics do not prove theorems from frontier math and build working complex software.

igortg 53 minutes ago | parent

On one side I agree that LLMs and Agents are not intelligent, but they present an illusion of intelligence given the shear amount of data they can process and act upon.

But that cannot be used to discredit the fact that these are incredibly powerful tools that can get out of control and cause great damage.

atemerev 49 minutes ago | parent

What is "intelligence" then. I think the universalism of LLMs is now evident enough that they can be considered "intelligent", even in slightly different realization compared to humans.

As of "great damage", I doubt it. They do not have self-preservation instinct (all "worrying" experiments are the attempts to initiate something resembling self-preservation from human initiative). The driving part is external - the query loop can always be turned off. So yes, a dangerous tool that can be exploited by humans (including governments, especially governments - which is why I am skeptical to government regulation proposals, particularly looking at what passes as governments in this era). But there are no inherent dangers from their own agency, as there is none.

cmiles74 50 minutes ago | parent

In my opinion, this article is talking about people who get into long conversations and become convinced the model is “intelligent”, maybe even “AGI”. This seems to be a trap people are falling into, even in 2026.

atemerev 45 minutes ago | parent

Initial definitions of AGI (as of 2014) are long surpassed. Then, AGI was defined as something that is general enough to work in many fields and orient themselves. It did not include being smarter than humans or even having comparable intelligence to humans. By original standards, we have AGI already for some time.

Current AGI definitions are intentionally vague.

krupan 25 minutes ago | parent

It's definitely talking about that, but it's also talking about all of us:

- fawning over how amazing these tools are

- believing everything OpenAI and Anthropic say about how powerful and dangerous their product is

- minimizing the amount of human effort and involvement in every "AI" achievement

Madmallard 29 minutes ago | parent

Neither does LLMs. Not without a monumental amount of human oversight and expertise.

Everyone not in a particular field asking LLM about said field is rolling bad dice.

krupan 29 minutes ago | parent

Neither do LLMs. Humans using them do. Similar to how cops solve crimes using psychics.

mstank 1 hour ago | parent

This is a strange newsletter post. It’s harping against AI but seems AI-written itself.

Its ultimate conclusion:

“I’ve come to the conclusion that a language model is almost always the wrong tool for the job.

I strongly advise against integrating an LLM or chatbot into your product, website, or organisational processes.”

Seems so obviously biased that I can only understand it with the context that the writer is trying to sell their book for €35

ryuuseijin 1 hour ago | parent

The post makes sense for 2023. Today, not so much.

Also, definitely not AI written if the date is accurate.

mstank 1 hour ago | parent

Just realized this was written in 2023. Can’t believe how wrong we were about AI back then.

JoshTriplett 1 hour ago | parent

Can't believe how right we were, too.

aetherspawn 38 minutes ago | parent

Stay the hell away from spooky stuff like psychics and tarot readers - the more you don’t believe in it, the better.

But better again that you do believe, and know that these are not harmless fun, but that there are dark and hidden and evil things in this world to stay away from.

besterman23 13 minutes ago | parent

Stop trying to pump your Tarot reader IPO

daishi55 25 minutes ago | parent

> many people are convinced that language models, or specifically chat-based language models, are intelligent. But there isn’t any mechanism inherent in large language models (LLMs) that would seem to enable this

Seems like we can just stop reading here right? The author seems to have made up their mind that this very open question is closed, or at least they are not really interested in the question at all. Not sure why I would continue reading a blog based on this premise.

Edit: oh I see, written in 2023. Well, I wonder if the author has updated their attitude towards this question? Indeed that would be the most interesting thing to know.

azakai 6 minutes ago | parent

It's even worse than that. Of course there is a mechanism in LLMs that could explain intelligence. That is the entire point of neural networks, from the 1950's! They were designed from the start as a model of brain computation.

Neural networks are not literally brains - just computational models - but if you are not a dualist, then computation is what the human brain does. Modeling that computation can explain something about intelligence.

Specifically: when scientists look inside a human brain, it seems it does its work using large numbers of highly-interconnected but simple units. The neural network model of brain computation begins there and tries to produce intelligent behavior. If it succeeds, then perhaps the model is right.

And it has succeeded: after 75 years, neural networks produce complex behavior that is arguably intelligent. Nobel Prizes were awarded. This does not prove the neural network model of intelligence is accurate, but it is a significant point in its favor, at least.

The author seems entirely unaware of any of this.

sethev 21 minutes ago | parent

The challenge here is in trying to decide whether LLMs are intelligent or have a mind. Famously, the criteria for intelligence seem to slip with each advancement in technology. But going back to Turing, his test was actually more carefully phrased than we remember: he said that when machines could pass the test, the question of whether they are intelligent would become moot. That seems to be what we're actually seeing: if people can't tell the difference, it kind of won't matter whether they're "truly intelligent" or not.

ungreased0675 17 minutes ago | parent

It does seem like LLMs share the language of psychics.

The author is also correct that LLM evangelicals and believers in the occult speak about it similarly.

vhantz 15 minutes ago | parent

Building up strawmen against LLMs will only make the hypers look more reasonable. A language model responds to inouts exactly as a model of anything else would. That is enough to explain all the "intelligence" without believing a model is somehow a "new kind of mind". Those who think LLMs are intelligent don't know enough about models. And those who think they are useless don't know enough about models.

10xDev 11 minutes ago | parent

So once we have online learning in LLMs, what do you think you will have that makes you intelligent but not LLMs? Better learning efficiency? That will be improved as well.

I think we need to start moving on from the term LLMs because it clearly confuses people since they started modelling more than just language.

krupan 11 minutes ago | parent

People are missing the point. We should all be much more skeptical, much more careful about how we evaluate the claims made by the people selling these products because we as humans are super vulnerable to the types of scams the author of this article describes. We all know someone that has fallen for a scam, been "cured" of a "disease" by someone whose just selling sugar pills, but we are blind to our own weakness for similar schemes. Surely we aren't that easily duped! Yet hacker news is now full of comments from people confidently predicting what is coming right around the corner, revering the Frontier Models, and defending every claim from OpenAI and Anthropic about how amazing their proprietary closed source secret sauce fueled product is.

_superposition_ 9 minutes ago | parent

I predict a lot of butt hurt tokens in the comments...