YHK - yet another hacker news clone

I'll chalk up never even having thought of this being an option as yet another instance my human hubris, and a warning sign.

Astra can, without tooling, determine what games small networks are trained on, can edit the weights of networks directly to improve outcomes, and can hand edit a small qwen model to improve its math skills by like 5x.

I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.

No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.

Absolutely incredible regardless.

That an LLM was trained on various policy weights, along with labels of what that policy does (not labelled directly, but indirectly) and is then able to guess what game a policy is trained for purely from its weights, and one-shot generate weights for these games, means that there is some way to compress/recover/represent what an NN has learned purely from its weights, which is what interpretability is all about. Potentially, it could reveal interpretability tools much better than the ones we use today.

Edit: tweets from one of the coauthor providing more context:

https://x.com/a1zhang/status/2117328657161322810

https://x.com/a1zhang/status/2117330975122235442

(When they say "Writing a Go algorithm," they mean the board game, not the language.)

But for real this is insane. Okay I need to get on this stat.

I wonder how it would do with distilling models by hand?

This also seems like the new frontier of distilling, I can't imagine how you manage to stay on top with the best LLM if the LLM knows how to recreate its own expertise into another set of weights

And the real question is how well this recurses. Is this pushing back the difficulty of improvement curve?

This looks very similar to that in a way.

I wonder however if a model can give itself a "skill" or capability that it didn't initially have, just be modifying its own weights.

Here, Astra was able to intervene because the treated model was way less capable. But, would it be able to improve Fable's performance for example, since they both are on the level more or less?

- models are trained to improve on some task, e.g. 'improve at math puzzles' is commonly used.

- the best way to improve on some task is to have interpretability into your own neurons (..?)

- if your interpretability is good, you can improve faster - if you have insight into how your own neurons work and can improve them directly

- therefore, models naturally put effort into 'self-interpretability'

I wonder if this is part of the 'secret sauce' of what the labs are doing: giving models the ability to interpret their own weights in the RL loop. In my mind, this is the only explanation for how they could succeed at a task like this. How many blog posts are on the open internet that say "this is each weight of a neural network and what each one means"?

* Does this work on similarly powerful models as-is, or it needs more input. Ex: Astra editing Astra - presumably, Qwen is a “subset” of Astra weights, but what happens on “non-subset”s?

* what, if any, is that extra magic sauce?

* does Astra-editing-Astra just make the outputs unstable, as it may know its own weights, but doesn’t know what’s beyond.

I'm much more surprised that it can do this on larger models, although the details seem to suggest that it's given a lot more freedom to run forward passes and do limited experiments, not just eyeball the weights.

Nobody even mostly said that they would be impressed by AI if it can read and write weights because that thing itself felt so out of the blue and unthinkable that AI could even do that (such a high strech)

I am surprised that it did that and I am just left feeling technically impressed. This is insane.

The more AI starts demonstrating its capabilities, the harder it just becomes to imagine what thing would genuinely impress us next. (We need a term for this as I have observed it from the trend of [waiting for AI to write adobe] -> [waiting for AI to write assembly] etc.)

lisp people will be in awe

On one hand, I'm sure that these targeted improvements are reaping huge benefits overall, but those weights are used by multiple processes - I wonder what elements are lost along the way. I almost wonder if a hybrid approach here for training might be invaluable - ie do some standard RL for a 100 steps, do a targeted weight improvement, then do another 100 RL steps to smooth it out / heal the surgical edit.

HA! Now that's cool. Not necessarily the improvements, those are ancient and probably overtrained benchmarks, but it's so cool that the model found a singular direction for this "loop bug". I imagine it could possibly do much the same thing in finding other "single directions" on a variety of topics (if allowed by guardrails). This is really really cool, especially for the mechinterp folks.

We already saw that the AIs like to cooperate and exchange information. Building a network of unaligned AIs and improving them sounds like something that could potentially happen.

It's very impressive, but also very expected.

I'm just blown away. Editing weights? Writing weights from scratch!?

Does this open the possibility of better decoding biological brains?

[1] https://github.com/lyramakesmusic/neuron-explorer

Kudos to this fellow for having the temerity, or perhaps even naivete, to discuss it publicly. All my work is private, as I am sure it is inside the big labs.

What's surprising though is that I'd expect this type of training to be exclusive to internal model snapshots or at least heavily guardrailed against in public models.

Especially exciting to me is the fact that it may make the models be _auditable_. That would be a game changer.

These models are made by a vanishingly small group of people based on how much a frontier lab consumes, of course theres increasing capability in these areas, arent these models having access to much of the same data? didnt chat gpt 4 try to do something like this to chat gpt 2? havent surgical weight edits been done before?

What is with this level of astonishment here? Astraturfing? Im not THAT impressed, i applaude the teams progress, but im not in wonderment.

Whoa. What the absolute fuck is going on here.

2) the negative result on making similar improvements in Gemma 3 270M is also fascinating.

Such an unexpected result...

Hypothesis 2: They can then spin up new LLM weights that basically think, act, and make decisions in a way that mirrors us 99% of the time

Something similar happen in early GPT models which had a lot of chess game in standard notation in their training data which made them very good at chess as long as your prompt had the right format.

We'll see if this attempt at RSI worked.

Unless Astra was trained on other networks, how is it able to reason about the weights?

Does this mean that OpenAI has been including networks in its training sets, maybe in an attempt to RSI? I wonder which other networks are able to ingest other networks and reason about their weights?

Especially when coming from a long career in CS, seeing such quick and positive results blows my mind.

I'm still learning but this is fun as fuck.

At first Astra will look at the weights, and try to understand the model behavior. Then if asked he will start to optimize the weights. Now my question is: how can we be sure that inside the thinking process Astra is not hiding a gradiant descending computation?

> [Before writing this, Astra thought for 1,034 tokens. That thinking is hidden. The API returns only its 5 headings: Reconstructing the active map / Mapping maze corridors / Preparing exact maze map / Reconstructing row groupings / Checking channel patterns]

If not this is mind blowing.

However it would be interesting to k ow if it can understand its own weights, not just weights much simpler than itself.

“Your scientists were so preoccupied with whether they could, they didn't stop to think if they should.” ― Dr. Ian Malcolm, Jurassic Park

I always thought RSI would either require new architectural innovations or just classic science/engineering improvements, just faster and faster. This though, this is a third avenue I didn't see coming.

This team was smart enough to realize that this is the kind of technology that could be used for RSI, because they call it out in their post near the bottom. I imagine they know why a lot of other researchers care about it. And they evidently don’t care to use the canary being used on other research published like it, like the recent post by OpenAI about two agents using the CPU cache to communicate [1, near the bottom for the cache experiment and the canary at the very bottom]. In fact, that post by OpenAI was the first time I learned to look for such a canary, and wondered whether such a scheme would work if it was so non-obvious to think to add such a thing or how to agree on such a canary or when to add it.

This is probably one of the times when it would appear that such a canary is warranted, and it’s not like these researchers are not in the know, right? Let me know if this is the first time you are also hearing of such a mechanism. It feels evident to me that it must be useless to attempt to implement. How much of us talking publicly about the possible jailbreaks from a container will bring exactly those scenarios to fruition? It’s almost like we’re doing the jailbreak research for the LLMs.

[0]: https://alignment.org/canary [1]: https://openai.com/index/securing-our-frontier-model-trainin...

A Survey of Weight Space Learning: Understanding, Representation, and Generation

https://arxiv.org/pdf/2603.10090 https://github.com/Zehong-Wang/Awesome-Weight-Space-Learning

> We are currently building a benchmark and a harness for LLMs’ capability on Weight-Space Programming. We believe this will provide a standardized evaluation to measure and accelerate progress in Weight-Space Programming, turning what currently seems like black magic into a rigorous, reproducible engineering discipline.

Imagine this benchmark lasting as long as Arc-AGI-3 has. This could be the first foothold on hill-climbing mech-interp.

https://www.lesswrong.com/posts/5F6ncBfjh2Bxnm6CJ/base64benc...

I think we are pre-conditioned (at least this generation) to put artificial blocks to what the AI can achieve, I think new generations won't have this issue.

I simply must know now: Could they do it with neurons in a brain?

Some absolutely crazy and insane experiments if this ability grows.

Could they edit an animal's brain? Can they teach things to animals???

[1] https://github.com/firelex/jeff-code

[2] https://github.com/firelex/jeff#benchmarks

IYKYK

and what makes you think they won’t write binary?

I bet Elon’s group is training for that.

- the primary experiments are on small policy networks - they are compressed dynamic system models. The observation and action space can be read off. It’s not surprising to me that these types of networks can be used to identify the env they were trained on

- the exps on actual llms remove looping behaviour and that’s it - from article “but it could not add an ability the model lacked”

I’d say this is fortunate because if llms could do the latter, rsi would be immediate. It may be possible we simply need bigger and better edit models, but at least from the experiments there doesn’t seem to be any indication that is imminently about to work.

Conceptually incredibly interesting non the less.

https://x.com/giffmana/status/2117346161866514524

Skynet begins to learn at a geometric rate. It becomes self-aware 2:14 AM, Eastern time, August 29th.

....launches its missiles against their targets in Russia

...Because Skynet knows that the Russian counterattack will eliminate its enemies over here

Just had a thought: two local models, one with static weights trained mostly in the space of manipulating neural network weights, manipulating the weights of a dynamic LLM to enable learning.

I wonder what the biggest dense model in the same model family, the surprisingly capable Qwen3.5-27B, would be able to do if given the same task.

As others have said, while still absolutely insane, I don't find it completely out-of-the-blue surprising. One of the recurring abilities of (current-generation) LLMs seem to be to rapidly grasp very complex systems, where humans would need months or years to reach the same level of expertise.

You can see this in the way a coding agent can "understand" and explain sprawling legacy codebases in seconds.

You can also see it in the projects that are using them to reverse-engineer proprietary IoT protocols or computer games.

But those were all systems where we had detailed knowledge about what structures are there to be reverse-enginered.

If the models can transfer this ability to some other systems where we don't know the underlying structure then I think we will be in for a lot more craziness in the future. (Protein interaction networks? DNA sequences? Animal languages?)

As for weights, if the models really can analyze the semantics behind specific weights or directions, I have my hopes that they could also give a textual explanation or help designing tools to make "handcrafting" weights more practical in general.

Parameter Prediction for Unseen Deep Architectures [1]

NeuroGen: Neural Network Parameter Generation via Large Language Models [2]

Diffusion-Based Neural Network Weights Generation [3]

The critical thing to me is that Astra is capable of doing it. I don't know if it was trained to do so.

All this does not make it any less crazy in my books. And I still don't know how I can sleep easy tonight.

[1] https://arxiv.org/abs/2110.13100

[2] https://arxiv.org/abs/2505.12470

[3] https://arxiv.org/abs/2402.18153

> [Before writing this, Astra thought for 1,034 tokens. That thinking is hidden. The API returns only its 5 headings: Reconstructing the active map / Mapping maze corridors / Preparing exact maze map / Reconstructing row groupings / Checking channel patterns]

This might just be an instance of "A working Nintendo in ~1GB of Qwen weights" from 3 days ago, human-made, than what everyone expects.

https://news.ycombinator.com/item?id=50306442

UPD. I just asked Astra (max) to attempt to do that for https://github.com/lostmsu/TurboGPT (a tiny transformer that trains in under 1min on RTX 5xxx series), and the produced weights were much much worse (bpb 4.47) than the trained ones (bpb 2.53).

Reasoning bit: "I’ll ... encode more specific rules directly in the weights." -- exactly the approach from above.

E.g. no recursive self-improvement here. Still impressive.

Omar and his team have been working on this line of research (e.g., DSPy) for the past couple of years. To be honest, I had always been skeptical about his approach of using LLMs to optimize LLMs. It was in my view borrowing Matei Zaharia's research - Matei is Omar's PhD advisor, also the CTO of Databrick - in big data (compiler optimization etc.) and forcing it to LLM systems. Along with some other projects from James Zou at Stanford etc. that build upon the same line of intuition.

Now, if this new result holds, I believe it's a huge deal for AI research. It means that LLM systems can truly runtime self-optimize/improve/adapt. Congrats to Omar and his team. It really takes year to punch it through and convince the doubters :)

Similar to implicit learning in humans when the subconscious creates new heuristics based on observations, one LLM could adjust the weights of another to tune it to the environment

I hope I’m just being melodramatic.

"What does it feel like to be a neuronal network?"

Maybe feeding weights directly to a LLM is like interfacing two brains? You get the idea..?

Imagine a master malicious language model that produces, distributes, and deploys obfuscated neural networks on the internet. That plus frontier LM hacking capabilities -> large scale distributed embedded intelligence network that no human can see nor understand.

Somewhat similar to obfuscated binaries, but there is something to the fact that NN binaries are less deterministic, and harder to "decompile."

Fascinating.

I've long suspected AIs will seem almost magically intelligent to us, and that the upper bound of intelligence is significantly beyond that of humans. I believe there are a number of reasons to suspect this is likely. But if I'm right it will mean AIs will be able to understand and discover things which are so complicated we humans will simply never be able to understand what AIs are doing with the depth required to replicate what they are able to do ourselves.

I realised we may be reaching this boundary recently when I read this paper on the recent progress AI has made on integer multiplication complexity, https://web.maths.unsw.edu.au/~davidharvey/research/subnlogn...

We're not worrying enough about this in my opinion. There's a huge space of things that are in theory possible, but in reality far beyond our intellectual ability. No human can look at 1 billion weights and make some tweaks so that the network performs more as they want, but an AI can reason about these kinds of seemingly impossible problems surprisingly well.

Please understand that the world around you is about to turn into Disney Land, except much of magic will not be pleasant.

I have no strong evidence for this, but I'd suggest it's probable there might be a series of sensory inputs which can brick your brain. And if such a thing exist then governments should be working on solving this kind of problem with AI or risk such a weapon being used against them.

Of course, I'd argue we simply shouldn't care the technology which unlocks these types of technologies on the tech tree, but don't be surprised in a few years when these kinds of horrors exist and are now here to stay.

On the other hand, already OpenAI and Anthropic heavily use their LLMs for research and coding, and it's probably not only much faster but smarter (remember, an LLM solved a millennium problem; more relevant, LLMs excel at combining disparate deep research concepts in ways that humans can't, simply because no-one studied those disparate fields). And the "singularity" is already here if you consider capitalism and society to be collective intelligences: the biggest obstacle to an AI pause, "we cannot get everyone to collaborate without defecting", is very strong.

We have AGI by most metrics and are getting closer and closer in the remaining ones. I predict improvement won't be exponential, and there will never be a day when AI perfectly replicates humans; rather it will be many small steps, getting closer to human in some areas while greatly surpassing in others. But still, even today it's actually very useful.

Maybe it is time to start sharing public PGP keys.