107 points lwhsiao 1 hour ago 36 comments
zodwick 57 minutes ago | parent
esafak 50 minutes ago | parent
"Alignment" is not mentioned once in this article. To be fair, I looked up the lead author's home page (https://www.jiajunfan.com/publications/) and found no relevant papers there either. What do you think is going to happen when robots can modify themselves like this without any oversight??
You can't throw rockets up in the air without worrying where they are going to land. It is your department.
lenerdenator 48 minutes ago | parent
This is a field heavily focused on shifting liability away from itself. They don't think about that at all.
snowram 44 minutes ago | parent
Until it doesn't, then comes responsability shifting.
thaumasiotes 37 minutes ago | parent
It works for firing off celebratory bullets.
jrflo 33 minutes ago | parent
That would only work for a closed model though, I agree that an abliterated open model doing this could be very bad.
im3w1l 20 minutes ago | parent
We can say then that a well aligned model will not try to bypass its restrictions by editing its own weights. But it may still accidentally disable them.
jrflo 6 minutes ago | parent
samlinnfer 17 minutes ago | parent
xhrpost 45 minutes ago | parent
Any increase in fidelity must come from somewhere...
>t keeps getting better: it tests its edits on practice problems and submits only what works, each edit improving on the last, and over eight consecutive sessions its accuracy only rises.
I haven't read the full report yet but it seems like this fidelity would ultimately be derived from test problems that it runs itself on? So the weight increase in complexity (English doesn't have a word for the opposite of entropy) would come from the test material as well as any output from what is running the test, or possibly the web if they permit it to search.
Jerrrrrrrrry 39 minutes ago | parent
kazinator 28 minutes ago | parent
The word "entropy" was never needed in the first place; "disorder" would have served just fine. Though entropy is a precise concept in thermodynamics, that is fine; all sorts of ordinary words get extra precise meanings in a specific technical contexts.
pdntspa 40 minutes ago | parent
tsunamifury 35 minutes ago | parent
The issue isn’t training, it’s articulating real world high value outcomes. Something the SOTA labs have no idea what to do about since they are staffed by… not the best people.
Nevermark 9 minutes ago | parent
To me this comes across as word salad.
The edits are done based on reasoning, to accomplish a "target" improvement, that is itself reasoned and designed, taking into account a reasoned understanding of the existing functionality of the weights.
This is the furthest thing possible from any normal optimization algorithm "over fitting" weights toward a simple measure/target.
gavinray 34 minutes ago | parent
https://research.google/blog/the-sleep-framework-a-new-way-f...
> During the offline sleep phase, the model unlocks pre-allocated Mixture-of-Experts (MoE) memory blocks and updates its numerical weights using self-generated practice examples.The-Ergster 18 minutes ago | parent
BoorishBears 11 minutes ago | parent
The original paper that started this hyper was highly misleading because it started with toy networks where the models could directly modify weights, then drew a thread to fixing a reasoning model... but the two approaches were completely unrelated.
The approach here was essentially abliteration, which any model able to write a few hundred lines of Python and generate synthetic examples can so.
vlovich123 29 minutes ago | parent
simonw 25 minutes ago | parent
vlovich123 10 minutes ago | parent
SamInTheShell 27 minutes ago | parent
aesthesia 25 minutes ago | parent
samlinnfer 20 minutes ago | parent
mistercheph 20 minutes ago | parent
ray__ 19 minutes ago | parent
I've noticed this trend in my own sub-field, where companies are frequently releasing "preprints" or "white papers" that are simply marketing pieces, with experiments that lack controls or appropriate benchmarks and just generally misrepresent the work at hand. It's already a significant undertaking to read a paper like this and actually understand it, even if you're in the field—I'm not going to commit an hour of my valuable mental energy to it if it is a press release in disguise.
The work should speak for itself. There is a lot of trust involved in any paper like this, and if the authors feel the need to editorialize the message, how can I trust that the data aren't editorialized as well?
ChaseRensberger 17 minutes ago | parent
damowangcy 8 minutes ago | parent
bagacrap 4 minutes ago | parent
xp84 1 minute ago | parent
Note that the chapter this comes from was not presented as "superhuman AI will kill us if it checks these particular boxes" but as "this is one example way it could go, but IRL we'll probably never be able to predict the details."