158 points dangelosaurus 3 hours ago 99 comments
dangelosaurus 3 hours ago | parent
And we're hiring! https://openai.com/careers/search/?q=cyber
BoppreH 3 hours ago | parent
I understand this work was based on a dated snapshot. Will there be any continuous effort, scanning new PRs as they land, to prevent new weaknesses from being added?
ky1ebot1337 2 hours ago | parent
Lerc 42 minutes ago | parent
It will be difficult to claim that the problem has been solved without an idea of how big the problem is.
Given that vulnerabilities are almost entirely unknown, you are inherently trying to measure the undetected.
Is there a way to hypothesize how many vulnerabilities there are and them make a prediction on how many of those will be found by a model to get a sense of how accurate the estimate is?
farisght 3 hours ago | parent
A system that admits THOUSANDS of such vulnerabilities is certainly flawed in several fundamental aspects
jethro_tell 2 hours ago | parent
esseph 1 hour ago | parent
So you have to chunk it down to workable pieces, and then if you collect enough data of that process you can look at higher level stats.
Same for every other existing piece of software, btw.
konaraddi 2 hours ago | parent
E.g., 948 results is a high number to review in a timely manner unless your team is very large or spending several months combing through the results
EDIT: nvm i shouldve kept reading, answered
> At this scale, manually investigating every report is impractical for our research team. We therefore focused on validating findings that a) were reachable by a low-privileged user, b) required no special capabilities, and c) produced measurable evidence of memory corruption.
ahsillyme 2 hours ago | parent
esseph 2 hours ago | parent
Multiply that behavior times the number of commercial and mobile and b2b apps over 50+ years, and that's where we are.
There will likely be many more leaks, and hacks, and outages. And then in trying to be proactive to fix old systems, that will cause new proactive outages, and then eventually many things will be better.
But, it's going to take a lot of time and investment, and a lot of (majority) of currently unsupported devices will have to be complete replaced / scrapped / ewaste. On the scale of Nation-State initiatives.
This is across all operating systems, firmware, software and devices of all kinds, etc.
LLMs to security are like a bomb went off. We still have to wait for things to fall over before the dust will even clear enough to see what we have to work with.
basilikum 2 hours ago | parent
The article says "The target virtual machines ran a stock desktop operating system". Are the vulns in the kernel specifically or does this include software running as root?
ky1ebot1337 2 hours ago | parent
emmelaich 2 hours ago | parent
0x00cl 1 hour ago | parent
zahlman 3 hours ago | parent
Imagine if open-source organizations other than the Linux Foundation (and Mozilla I guess) could throw $1.2M of API-rate LLM usage at security.
getcrunk 2 hours ago | parent
*buries head in sand*
*cries*
rubinlinux 2 hours ago | parent
kccqzy 2 hours ago | parent
The difference here reminds me of the distinction between the regular Paxos algorithm and the Byzantine Paxos algorithm.
Plasmoid 2 hours ago | parent
Is the kernel simply too complex to reason about? Is it an underlying flaw in the security model or language implementation?
Should we assume that all software is more compromised than the kernel?
antonvs 2 hours ago | parent
We developed AIs that can find them.
spartanatreyu 2 hours ago | parent
> > We developed AIs that can find them.
We could already find them before AI.
We already knew they were there, and we already knew how to prevent them in the first place.
See previous discussion: https://news.ycombinator.com/item?id=49705794
esseph 2 hours ago | parent
Not at this speed and cost.
spartanatreyu 1 hour ago | parent
I'd argue that the resources (i.e. time, money and effort) needed to create the tooling that would have prevented all of these issues would have been orders of magnitude less than the resources that have gone into "AI".
antonvs 53 minutes ago | parent
p-e-w 2 hours ago | parent
0xbadcafebee 2 hours ago | parent
esseph 2 hours ago | parent
First of all... Do you really believe that to be true?
cyberax 2 hours ago | parent
But it also got its share of use-after-free and OOB access vulnerabilities this year: https://feedly.com/cve/vendors/openbsd
icepush 2 hours ago | parent
Is it bulletproof? Definitely! But you need the genius of Albert Einstein as well as the patience of Mother Theresa to actually use it in production.
wahern 2 hours ago | parent
Muromec 1 hour ago | parent
It's the one security practice that actuuuualy works
wahern 2 hours ago | parent
bombcar 2 hours ago | parent
tptacek 2 hours ago | parent
blooalien 1 hour ago | parent
It died in the sense that the "people in charge" hired us for our expertise and then ignored our every suggestion and fought us tooth and nail every inch of the way when we suggested that security or reliability ever mattered in the slightest until after the disaster happened, and then straight-up scapegoated us for "not pushing back harder" against the threats of "go work somewhere else then, code-monkey!" until we gave up and did the "code-monkey" thing for our paycheck and no longer cared about quality or "a job well done".
bombcar 43 minutes ago | parent
hibikir 1 hour ago | parent
mikestylz 2 hours ago | parent
Absolutely, the front door has always been unlocked. We were previously protected by the economics of exploit detection favoring the developers, but now we have entered a new world.
The issues arise almost everywhere we look. It's frustrating because the user experience in edge systems has barely changed in the last fifteen years. We could have rode the advances in semiconductor manufacturing to embed more layers of protection such as CHERI while slightly slowing down the rate of performance improvements. Instead we have only marginally upgraded security and largely added features which the users scarcely notice.
I am hopeful about the long term prospects of formal verification, but having humans reviewing the proof statements and confirming they match our desired properties will take many years. Meanwhile, AI assisted hacking is an urgent issue. So we will have to delegate implementation, proof specification, and proof development to the AIs in the short term. Maybe that goes well, but who knows. Interesting times.
AnimalMuppet 2 hours ago | parent
Now, I don't know how many of those bugs were privilege escalations, or could be leveraged to be. I'm not sure that people tried to exploit in the same way back then (or, if they did, if they said anything publicly about the results). Still, even if the majority of those bugs were not exploitable, OSes were much smaller then.
So that's my conclusion of how we got here: We've always been here.
esseph 2 hours ago | parent
flomo 30 minutes ago | parent
asveikau 2 hours ago | parent
chrismarlow9 2 hours ago | parent
est31 2 hours ago | parent
anonymous908213 1 hour ago | parent
Oh, you strongly recommend rewriting tens of millions of lines of code? Why don't you do it yourself? Just a prompt away, right?
jkelleyrtp 1 hour ago | parent
Muromec 1 hour ago | parent
shoobiedoo 1 hour ago | parent
Gigachad 1 hour ago | parent
We would also have to burn a trillion tokens a day verifying all the C code the hard way forever going forward. It would be cheaper to rewrite the kernel in Rust, subsystem and driver by driver at a time.
shoobiedoo 47 minutes ago | parent
est31 40 minutes ago | parent
AI doesn't give a strong guarantee that things work though, while Rust gives a much stronger (not absolute) guarantee.
But maybe the way of the future is to auto-generate proofs that the code is memory safe and then just leave things where they are, instead of doing rewrites that require maintainer buy ins. The advantage of C plus proofs would be that the AI generated proofs could live separately so it wouldn't touch workflows of the maintainers.
The only required output they have to process is evidence of vulnerabilities in changes proposed to them. They could outsource even that to contributors.
ventedmouse 1 hour ago | parent
est31 1 hour ago | parent
Humans manually typing out the bytes of source code has been an outdated method, since around a year. It simply doesn't scale as much as those same humans directing agentic teams. So yes, it would be about prompts.
The hard part isn't the technology for the language transition, that one is present today. pgrust shows the way. It's ensuring maintainers are cooperative, i.e. a social problem. Eventually you diverge too much if you do a fork, so it's better done in cooperation with upstream.
Just check how long it took LLVM or GCC to switch off svn for example, and a language switch is much more involved than a VCS switch.
anonymous908213 1 hour ago | parent
You mean the pgrust which was recently caught for cheating benchmarks and incapable of even performing basic queries outside the benchmark set? Would love to see pgrust debunked by Jepsen like this other AI-generated, written in Rust, "formally verified" scam.
est31 58 minutes ago | parent
anonymous908213 51 minutes ago | parent
NooneAtAll5 1 hour ago | parent
it's been rule of thumb basically forever that there is 1 bug per ~10KLOC for any project
linux kernel is millions of lines, so thousands of bugs being hidden there is about on point
(tho obviously far smaller serious escalation ones, obviously, but you're exaggerating as well)
jiggawatts 1 hour ago | parent
1 bug per 10 lines -- not kilolines -- is much closer to the actual "bug rate".
You may be thinking "1 serious security vulnerability bug", but in my experience that too is much closer to 1 per 100 lines to 1 per 1,000 lines, depending on the type of application.
Muromec 1 hour ago | parent
There is nothing magical about kernel. Look at the code anywhere and it has bugs, lots of them.
Security bugs are just bugs. More often than not they are silently there and don't prevent you from using the software and you can't react and fix them because of that.
Give it time and they accumulate
pianopatrick 1 hour ago | parent
Speed requires a lot of code in the kernel, where that code can run fast.
There are kernels that do not have a thousand privilege escalation exploits. These kernels are smaller and much easier to check for bugs. But those kernels are significantly slower.
For many use cases, slower is not a problem. But for the large corporations who fund a lot of Linux development and use Linux in large data centers, every 1% slower can literally be millions of dollars worth of additional servers that are required.
OpenBSD shows that you can have a Unix security model without a thousand privilege escalation exploits. But OpenBSD is like 10% slower. So corporations won't use it because it would cost millions of dollars more per year and some of their software requires access that OpenBSD does not grant precisely because that access leads to security exploits (i.e. io_uring, eBPF etc. etc.)
You could also use like seL4, but that's slow too
So that's why. It's a business tradeoff between cost of servers vs cost of getting hacked, and cost of servers wins.
Gigachad 1 hour ago | parent
pianopatrick 14 minutes ago | parent
Like, I bet Anthropic and OpenAI run on a combination of Windows, Mac and Linux. That likely means that most intelligence agencies worth their salt have thoroughly penetrated those companies at one time or the other. So the model weights that those companies spent billions of dollars training are likely in the hands of all sorts of people all over the world.
Similarly for most of our weapons manufacturers. Unless they've really gone to great lengths like air gaps and such - and maintained that - the technical details of most American weapons are likely in the hands of many intelligence agencies.
Same for the banking details of politicians, and the daily schedules of people in power, and the messages everybody in Washington D.C. are sending each other and on and on. All of this stuff exists on computer networks that run on windows / linux with thousands of vulnerabilities. A treasure trove for intelligence agencies.
Similarly for most corporations. All the money spent on R&D and the results of that research can just be hacked away and taken out over the network because we chose speed over security. The whole idea of "Investing in R&D" seems kinda silly in that world. You put the fruits of your expensive research on a computer on a network. That network could be hacked by any hacker who can find any one of the thousand vulnerabilities, including your business competitors and geopolitical rivals.
So I don't think it was really fine.
wormius 25 minutes ago | parent
I'm also pretty hostile to AI, but do recognize that these machines seem to be very good at finding exploits.
I'm sure there've been some some projects out there working to pentest/find cracks in OpenBSD via LLMs/agents/whatever-the-right-term-would-be...
But it is true the attack surface is much lower in OpenBSD on purpose, so hopefully it is overall more secure. But I'd like to see the numbers.
pianopatrick 11 minutes ago | parent
I think the only OS that's truly survived the agents looking at it was sel4, but I could be wrong.
perching_aix 1 hour ago | parent
beala 38 minutes ago | parent
I've been working with a LoRa chip called the LR2021. I plugged a dev board into my computer and told Claude to reverse engineer it. He dissected it with a speed and level of efficiency I couldn't believe. Within a few hours, Claude was writing custom patches for a completely opaque patch RAM area meant only for the manufacture. I wrote about it here: https://beala.substack.com/p/your-node-doesnt-need-a-microco...
verdverm 35 minutes ago | parent
be wary of the affection economy and avoid gendering ai bots
LoganDark 28 minutes ago | parent
verdverm 16 minutes ago | parent
LoganDark 14 minutes ago | parent
verdverm 11 minutes ago | parent
Ai embodied toys are something we ought to get out in front of and ban, companions in general. Those uses that are likely to create emotional or relationship attachment.
dj_gitmo 26 minutes ago | parent
verdverm 20 minutes ago | parent
this is where Zuck is taking Ai, it's more dangerous than what he did with Social
beala 23 minutes ago | parent
verdverm 18 minutes ago | parent
It's dangerous for humans, and using "he" is a slippery slope. Words are more powerful than we tend to acknowledge. Here's a good video with that goes into it.
LoganDark 17 minutes ago | parent
This is why organically written text is superior. Hell yeah.
randallsquared 32 minutes ago | parent
sreekanth850 3 minutes ago | parent
nas 2 hours ago | parent
tamimio 2 hours ago | parent
hoppp 2 hours ago | parent
IcyWindows 1 hour ago | parent
ACLs, Appcontainers with capabilities, etc have been something in the OS for more than a decade.
jasonjayr 1 hour ago | parent
perching_aix 1 hour ago | parent
bzmrgonz 1 hour ago | parent
ksec 1 hour ago | parent
rmbryan 1 hour ago | parent
What are the mitigating factors that prevent this level of vulnerability from being an existential threat to humanity?
nomel 58 minutes ago | parent
[1] https://www.cyberdefensemagazine.com/zero-day-market-the-gov...
rmbryan 54 minutes ago | parent
nomel 32 minutes ago | parent
2026: https://medium.com/@hammaadmughal/the-invisible-bazaar-insid...
etc. Concept of governments being main buyers doesn't change by year.
tantalor 55 minutes ago | parent
Hopefully this type of work to automatically discover & patch vulnerability leads to much more reliable software, and the remove the need for cyber security.
rmbryan 53 minutes ago | parent