49 points porridgeraisin 4 days ago 95 comments
BLKNSLVR 4 days ago | parent
Distillation attacks are covert, industrial-scale campaigns to extract the capabilities of one model and replicate them in another, without the authorization of the original model’s developers.
Replace a few key words and it sounds like reading all the worlds media and offering a service that summarises it.
Don't scrape the scrapers, that's not fair! Expecting Executive Order against 'distillation' in 3, 2, 1...
WheatMillington 4 days ago | parent
jacquesm 4 days ago | parent
The laws they are arguing for are not going to protect anything, they essentially are waging war on open models and since most open models that are 'pretty good' to 'good enough' hail from China they have chosen to play the China-fear card.
But they won't be able to stop anybody from doing this if they want to continue to operate Claude as a business going forward, and that is assuming that is what is actually happening. Chinese labs have been extremely open about their developments, they have shipped working models to the likes of me, which is more than I can say about Anthropic. On top of that they don't sermonize about how they are going to improve the world and never mind the accidental destruction, that's just collateral damage.
hollerith 3 days ago | parent
jrflo 4 days ago | parent
jacquesm 4 days ago | parent
BLKNSLVR 4 days ago | parent
Probably stamping heavily on much of the nuance, there's a certain irony in any of the frontier models complaining about others stealing their work. The secret sauce for the frontier labs may be the reasoning layer, but the secret sauce of any piece of literature is the specific ordering of the words. I hope the parallel is obvious.
The frontier labs are spending record-breaking amounts of money on hardware (to the extent that they're locking out anyone else from the market, but that's beside this particular point, kinda) in order to create their service, and another company that is 'distilling' has not paid the same hardware toll. How much did it cost an author to create their literary work, which could be read and ingested for a miniscule percentage of the cost it took to create? For the benefit of the service provided by a (wannabe) multi-trillion dollar company.
If the frontier labs can prevent or slow down 'distillation' actions that customers are using their service for, then that's all well and good. Maybe they write an acceptable use policy that says it's not allowed (and they probably have). Good on them, it's their service. It's the way this is framed that I have a problem with. Referring to it as an "attack" and lines like this:
Second, illicitly distilled models present a national security risk.
It's bloviation. It's fear mongering. It's being one of the biggest, most powerful and influential companies in the world at the same time as playing the victim. Feel free to share the methods being used to prevent what is considered unacceptable use, but let's not pretend these are the only companies having to expend effort to protect trade secrets and company value. Let's also not pretend this post is all that much more than "lead-up to IPO" hype.
zelcon 4 days ago | parent
Today, movies make money from in person theater tickets and records make money from tours. I don’t know what the equivalent would be for AI (maybe IRL robots?) but it does not look promising.
jacquesm 4 days ago | parent
EGreg 4 days ago | parent
anon373839 3 days ago | parent
Seriously though, these clowns at Anthropic are not just hypocritical. They are clearly feeling threatened by open source AI. I almost couldn’t believe the new blog post today where they argue that dangerous open source models might soon be running on (checks notes) smart ovens.
farhanhubble 4 days ago | parent
Palomides 4 days ago | parent
jacquesm 4 days ago | parent
Kuyawa 4 days ago | parent
"Now that we've distilled all we could lets close the doors so nobody can distill from us"
gensym 4 days ago | parent
jacquesm 4 days ago | parent
(1) This seems to be a pretty bad accusation to make without iron-clad proof.
(2) So their AI is not able to help defend against this.
(3) If GLM 5.3 really is that good why would anybody use Anthropic's products going forward, this almost reads like an ad for GLM.
(4) Regardless of whether it is true or not, the hypocrisy is off the scale, first you steal pretty much all of humanity's cultural output, then you try locking that up behind a paywall and complain when people do the same to you.
(5) All of this 'we'll make the world a better place' seems to be a smoke screen. If they really wanted to make the world a better place through AI then more AI and more providers of AI would be a good thing.
(6) Meanwhile, Anthropic seems to do next to nothing to stop people from using their tool being used for bad stuff, there are plenty of reports of this, far more than there are of people using GLM 5.3 for bad stuff. If they really cared so much about models being used for bad purposes they could do a very simple thing.
I find it hard to believe that Anthropic thinks this sort of strategy is going to be the winning one and it probably mirrors the degree to which they are eyeing their approaching IPO date. Anthropic is claiming it is altruistic towards humanity but that seems to come with significant strings attached which seem to spell 'control'.
JDups 4 days ago | parent
Maybe they're trying to frame it like "Other models are only good because they copy us and we're making sure they can't do that, so going forward you should stick with us."
jacquesm 4 days ago | parent
And who is to say how much of this stuff has been going on between OpenAI and Anthropic, because whoever thought of it first must have been using it already.
impossiblefork 3 days ago | parent
Relatively similar concept, but has more total parameters, more active parameters and ordinary full attention instead of deepseek sparse attention, so it should probably be GLM 5.3-beating.
xyzzy_apps 4 days ago | parent
chatmasta 4 days ago | parent
xyzzy_apps 4 days ago | parent
All these repositories can easily be a part of the pre-training / post-training datasets. Perhaps the distillation the blog is speaking about is for text-generation not the code generation.
xrd 4 days ago | parent
"extract the capabilities of one model and replicate them in another, without the authorization of the original model’s developers. They are often conducted using networks of fraudulent accounts. These attacks allow companies to mimic frontier capabilities for a fraction of the development time and cost"
It turns into this:
"extract the CREATIVE WORK of one MAGAZINE/BOOK/PUBLICATION and replicate them in another, without the authorization of the original MAGAZINE/BOOK/PUBLICATION WRITERS. They are often conducted using networks of fraudulent accounts. These attacks allow companies to mimic CREATIVE WORK for a fraction of the WRITERS time and cost."
I think I have seen evidence to know the second paragraph is true, but I'm not sure about the first. I need to see the evidence from Anthropic.
I did this entirely using my own brain, without the help of LLMs. This is the most creative thing I have done in a year, I'm so proud of myself.
jacquesm 4 days ago | parent
xrd 4 days ago | parent
jacquesm 4 days ago | parent
abrookewood 4 days ago | parent
hector_vasquez 4 days ago | parent
BLKNSLVR 4 days ago | parent
The comment to which you're responding isn't righteous indignation, it's a reaction to the righteous indignation within the article. ie. the same as your reaction, but applied one level up.
Additionally, there's a vast (repeated loudly with cave acoustics: VAST! Vast! vast! vast ...) difference in scale between one person reading a fishing book and teaching a handful of people how to apply it and what the frontier AIs do. Vast! With a coupla trillion dollars behind 'em. It's a silly analogy if it's an analogy at all.
jacquesm 4 days ago | parent
BoggleOhYeah 4 days ago | parent
mw888 4 days ago | parent
jacquesm 4 days ago | parent
hector_vasquez 4 days ago | parent
vhantz 4 days ago | parent
jacquesm 4 days ago | parent
It also seems to - so far at least - have been well behaved enough to not try hack other people's properties. This is possibly a function of my elite skills in constructing sandboxes[1] but then again, I would hope Anthropic is even better at this ;)
[1] contact me for instructions on how to disconnect an ethernet cable.
toppa33 4 days ago | parent
jacquesm 4 days ago | parent
writtenone 4 days ago | parent
kouteiheika 4 days ago | parent
Ahem: https://www.felonybench.com/
I wonder why there is not even a single one of those allegedly "distilled"/"unsafe" model on that list, yet models of companies with "proportionally robust safeguards" are, hm?
> Distillation is a problem for every developer and platform
No, it is not. It's only a problem for you.
jacquesm 4 days ago | parent
jrflo 4 days ago | parent
2) If the open providers are truly just distilling bigger models, the distillers have no reason to do the crazy levels of training and evaluation that would result in a crime being committed
They are definitely spinning this for PR and their own self interest but open models are genuinely more unsafe since it’s a Pandora’s box situation- if you have a very capable hacking model there are plenty of bad actors with access to <$100k in compute resources that could raise hell with a jailbroken model. At least with Anthropic they can restrict access to bad actors.
jacquesm 4 days ago | parent
kouteiheika 4 days ago | parent
Not according to Anthropic:
> We find that GLM-5.3 develops end-to-end exploits in 50 of 410 attempts. Claude Mythos Preview did so at a similar rate—in 56 of 410 attempts.
Another quote:
> [..] a researcher used GLM-5.3 on a sandboxed machine with a local Linux build of a popular web browser. Over the course of a day (and with limited human attention), GLM-5.3 found several previously unknown vulnerabilities in the browser’s JavaScript engine, and chained them together into a working exploit: a webpage that, when visited, reads arbitrary files from the visitor’s computer
Source: https://www.anthropic.com/research/glm-5-3-and-the-spread-of...
jacquesm 4 days ago | parent
It would have been nice if they added some proof though.
emaro 4 days ago | parent
I'm angry and laughing at the same time. The audacity! The biggest theft of IP in history, and then the "attacker" whines the same thing happens to them?
To be fair and balanced: • Anthropic payed up (a little) for their theft. • They pretend to be concerned about cyber and national security at least.
There needs to be a coherent position on knowledge and intellectual property: either it's free to take and share, then no one can complain. Or one can "own" an idea, then the AI companies first need to pay up for real (how is Sony doing it?). They can license (subscribe!) all the works used for training and after that they can bitch around about distillation "attacks".
jacquesm 4 days ago | parent
carterschonwald 4 days ago | parent
jacquesm 4 days ago | parent
carterschonwald 4 days ago | parent
hector_vasquez 4 days ago | parent
carterschonwald 4 days ago | parent
also my setups for making models reliable requires reified jointly inspectable deduction traces. its also fristrating because aside from context limits ive gotten incredible work out of claude 4.5 in the past, despite its context limits being a serious bottleneck. but those reliability things interact terrible with all the safety filter bs.
to be clear this is a customer/consumer rights issue. and the real safety issue. if they are so concerned about unsafe distillation, make it easier to fucking do alignment respecting distillation. no amount of any distillation benefits any fucking paying user whatsoever. its literally a tragedy of firewalling a compilation of public commons because you raided all the book stores and libraries first
edit: im also on the market atm so if anyone wants some smart ocd focused on actual interactive agent reliability, while all my black box tooling isnt public, aside from my lack of public marketing im hard to beat
jacquesm 4 days ago | parent
carterschonwald 3 days ago | parent
i am also working on some commercial ideas around reliability, but im not 100 percent certain that theres a market for a truely good third party harness
jacquesm 2 days ago | parent
That's the bit that jumped out as really interesting. I've been fighting this for some time now and I'm super interested in what your take is on this.
rcr-anti 4 days ago | parent
jacquesm 4 days ago | parent
zelcon 4 days ago | parent
jacquesm 4 days ago | parent
erelong 4 days ago | parent
haha, lol even
> pirates stealing from pirates
arr matey this is the Wild West of the AI frontier...
skybrian 4 days ago | parent
They're just competing companies with conflicting interests.
jacquesm 4 days ago | parent
BLKNSLVR 4 days ago | parent
jacquesm 4 days ago | parent
jrflo 4 days ago | parent
jacquesm 4 days ago | parent
jrflo 4 days ago | parent
A better analogy would be if I read an open scientific paper and used the result to design some device. I put up the money and effort to turn that work into something useful, but someone did the ground work and it was released for everyone to see.
jacquesm 4 days ago | parent
jrflo 4 days ago | parent
jacquesm 4 days ago | parent
If my bike is out on the street without a lock that does not automatically give you permission to take it.
Copyright isn't so much about protection against reproduction, it is about control over your creation. Literally: "an exclusive legal right to copy, distribute, adapt, display, and perform a creative work".
Note that this is a list of rights that automatically apply to any creative work. You can't train an AI without copying. You can't train an AI without adaptation. AIs have been known to be able to verbatim reproduce parts of the ingested data, they are in effect lossy compressors and automated ones at that. If you take out all of the training data the AI is just a pile of random noise so it is the training data not so much as the training process that adds value to the AI. The training process is trivial compared to the effort that went into the creation of the content used during the training process.
If something was 'free for the taking' and you believe they don't need permission anyway then I guess you're going to have to either come to terms with the balance being that the resulting work product should then also be free for the taking or you will have to explain why you think that that is not the case. The Chinese labs (one in particular) that is being accused here in an evidence free manner at least put their weights out for others to use. So I have less of a problem with that than I do with Anthropic.
skybrian 4 days ago | parent
jacquesm 4 days ago | parent
skybrian 4 days ago | parent
I think the issue is that it’s case-specific, but there are plenty of people claiming copyright violations in circumstances where there’s no specific evidence for it, and a few other people claiming that LLM inference necessarily means it’s transformative use.
jacquesm 4 days ago | parent
Besides all that: no matter how you slice the actual case for infringement, these companies have the authorities in their pocket and even if what they are doing is technically illegal I fully expect them to get away with it because they are very wealthy. Which is one reason they're not seeing any of my money.
tuesdaynight 4 days ago | parent
jacquesm 4 days ago | parent
tuesdaynight 4 days ago | parent
jacquesm 4 days ago | parent
And that would be bad for consumers in the long term because who is going to come up with the new ideas then?
nullc 4 days ago | parent
You're welcome.
Gigachad 4 days ago | parent
tkz1312 4 days ago | parent
LiamPowell 4 days ago | parent
I'm so tired of Anthropic's bullshit and I can't believe anyone is willing to support them when they take such user-hostile actions. There's no other industry where a provider can say "we're going to take this job" and then later say "we're not going to give you the deliverables because it might have some of our IP in it, you still have to pay for it though".
jochem9 4 days ago | parent
Framing it as a security issue is intellectually insulting.
jacquesm 4 days ago | parent