YHK - yet another hacker news clone

Is this the new cyber normal?

78 points drewsberry 1 hour ago 66 comments

roenxi 59 minutes ago | parent

> Is this the new cyber normal?

No. We're in a "brief" [0] transition period where a scaleable superhuman developer enters the scene and discovers a whole heap of bugs. Once the majority of code is written by similar systems then the tide of critical bugs being found will slow down.

The article does sorta say this but lets not bury the lede to deeply. It isn't realistic to expect new security vulnerabilities to be found at this rate, we're clearing out years of backlog and the maintainers haven't had time to deploy AI to do the software right the first time. Once the situation stabilises I'd personally expect very few public vulnerabilities because the expertise in how to assess code security is moving from a handful of experts to everyone.

> The NHS approach: close source everything.

Yeah good luck. I've seen how these systems work at decompiling and I expect they're probably going to be really good at fuzzing too.

[0] Could take a while.

ninkendo 44 minutes ago | parent

> Once the majority of code is written by similar systems then the tide of critical bugs being found will slow down.

I don’t know if this necessarily follows. I repeatedly see that AI will write code, then you have it audit the code for security vulnerabilities, and it still finds them. Just because AI can find bugs doesn’t mean it will write bug-free code. AI’s know how to take shortcuts to achieve their goals, and do it all the time.

Yes, you can have another AI audit the first AI’s code, but in practice this doesn’t always happen, especially because once the human gets their hands on the finished product, they don’t always take the extra step to ensure it’s audited. This is particularly true because people seem to think “well the AI wrote it in the first place, why would it find its own bugs?”, ignoring that context is everything, and an AI being asked to find bugs is going to approach the code a lot differently than an AI tasked with adding features.

That’s not to say it’s not possible to set things up properly into multi-agent stages with QA and security as first-class phases of development… I only mean to say that it’s not automatically true that “AI can find bugs” implies “AI doesn’t produce bugs”.

clivedup 42 minutes ago | parent

+1. There aren't an infinite amount of serious security vulnerabilities out there.

> > The NHS approach: close source everything.

> Yeah good luck. I've seen how these systems work at decompiling and I expect they're probably going to be really good at fuzzing too.

Me too. The agents are much faster if it has access to source. NHS would have been easy hunting grounds. Now at least they don't stand out.

"a system should be secure, even if everything about the system—except the key—is public knowledge"

Should indeed, but we've all been caught with our pants down.

MadrasTh0rn 41 minutes ago | parent

>It isn't realistic to expect new security vulnerabilities to be found at this rate

I wouldn't challenge fate like this.

thomascgalvin 54 minutes ago | parent

Everything I've read indicates that yes, this is the new normal.

When Mythos was first announced, pretty much every security expert I talk to freaked directly out. A significant portion of my job now revolves around mitigating the impact of AI-generated exploits.

The rate at which we're churning out AI-generated code is going to make this worse, too, because AI-generated code is kind of notoriously terrible at security.

hnlmorg 51 minutes ago | parent

You just need to use AI to test your AI code.

(What a wonderful new world we live in)

doublerabbit 38 minutes ago | parent

Dystopian days, where you will be forced to write some syntax, that is then handed off to a an Ai tester. The tester will return it to you ready for handing the code to three frontier Ai's. The judges of code.

They approve your code, gold, if not; back to the coding machine and minus 1000 human tokens. This will be done via the implants of brain chips

silveraxe93 34 minutes ago | parent

I really don't get the issue here. Is the rust borrow checker dystopian?

You're forced to write syntax that appeases the tester. If they are not happy you need to return and rewrite.

As software engineering matures, we're adding more linting and static checkers in codebases. This is generally considered good.

chii 28 minutes ago | parent

> You're forced to write syntax that appeases the tester. If they are not happy you need to return and rewrite.

that's called the write/compile/check loop

but if suddenly it's ai, people freakout at having to do this?

mr_mitm 28 minutes ago | parent

Wait, when did brain chips enter the picture?

e12e 31 minutes ago | parent

I know you're joking - but I think it will remain a challenge.

Quality and Security are emergent properties of software - especially when considering software-as-deployed in the real world.

And the problems with "good taste" and good architecture and data modelling - will probably leave ai code open to a lot of issues (Claude code cli issues is a great example of how bad this can get).

Having good architects in charge can help - but it requires a lot of discipline to keep the feature set small so that maintaining quality and security become feasible.

silveraxe93 29 minutes ago | parent

It _is_ wonderful. You're never required to adopt software engineering best practices, but eventually everyone does because either it makes our lives easier or code better.

Having tests / CI pipelines, etc are the same thing? Now you can choose to add a cheap step in your pipeline checking if e.g. some engineer did something stupid like allow SQL injections.

The floor of software quality once it gets to review is permanently raised.

thatsabadlook 38 minutes ago | parent

Yea most the people I know working in cyber security weren't worried about mythos in a serious way. All the people selling cyber products spread fear over it because profit.Everyone knows there are exploits laying in wait. Always has been always will be.

Most the cyber security people I know were upset once llms came out that could do basic os int and author their own emails, and mimic speech/video. That's how the majority of real losses occur. No one even talks about it. It's mind numbing hearing about the zebras and unicorns when there are 10 billion ants entering houses a year. Exotic stuff sells...

Now lots of sane people are genuinely and correctly worried that turning all engineers into 24/7 sloppers is and will continue to make the world far more exploitable. That's the real risk as far as I can tell

edukite 2 minutes ago | parent

I have the same experience. Most of articles I read was reducing it to "just a tool, nothing more"

silveraxe93 36 minutes ago | parent

_Is_ AI generated code notoriously terrible at security?

My experience with AI-code is that it's extremely defensive and good at minimizing 'obvious' mistakes.

Are we comparing to old models generating code? Or mythos-class models?

Helmut10001 30 minutes ago | parent

It is not just code, it is also deploying Software. I constantly have to correct or steer the 'ai' in conversations towards safer directions or better following even the most basic security principles. Here I feel it is still more the overambitious intern that needs to be carefully overseen, but has a lot of time and energy at hand.

nine_k 29 minutes ago | parent

It very much depends on how you generate it, from prompting to test suites, but most importantly, your attitude. The extra security costs you time and money.

There are still folks, many of them, who think that time to market is everything, and anything beside it should be ignored. "It compiles? Ship it!" is a phrase from 1990s; LLMs just make this approach much easier. We will have plenty of machine-generated software with the security posture of a colander hit by a buckshot.

CER10TY 29 minutes ago | parent

It's been shown to be bad at secure coding _unless_ you specifically prompt models to watch out for common vulnerabilities, at which point they're actually quite good at containing the issues, though the studies here are somewhat older (eg https://arxiv.org/abs/2211.03622).

The issue then is that most people aren't actually good at keeping an eye out for security issues in code proactively, so things slip through the cracks (whether human-written or not), just at a much accelerated pace.

silveraxe93 21 minutes ago | parent

from your paper:

> Overall, we find that participants who had access to an AI assistant based on OpenAI's codex-davinci-002 model wrote significantly less secure code than those without access.

That model came out in March 2022, it's GPT-3.5 !!!

I'm sorry but it's absurd to use that as evidence for anything related to AI coding today.

You can't just hedge by saying "it's somewhat older". It's like discussing dunno a new train connection between cities and citing a paper measure steam locomotives.

CER10TY 18 minutes ago | parent

Hence why I said the studies here are quite old, and I'd like to see something more recent, though I guess it'll be some time before that's done.

I think most people writing code with AI nowadays will find it's still lacking in some areas. Not because the model isn't good (definitely miles ahead than what we had access to in 2022-2023), just because it wasn't actively mentioned and so the model skipped over it. The same will apply to security.

silveraxe93 10 minutes ago | parent

I'd also love to see recent studies. It's frustrating how even new studies end up using models which are behind the frontier at the time of publishing.

AI is advancing faster than academia can keep up. Which in general shouldn't matter. But you have this pattern where a paper comes out with a conclusion that just doesn't hold anymore by the time it's read, but people use it to form/support opinions.

tytho 17 minutes ago | parent

I agree that we can’t compare older models to newer ones, but how much code was generated during those months/years when the models weren’t good at secure code? How much is still running publicly today? It’s still worth considering the impact to the overall ecosystem.

thomashabets2 12 minutes ago | parent

Luckily we have newer models going back and fixing these things.

In terms of the set of all bugs, the AIs from 2022-2023 pale in comparison with the amount of bugs written by hand in previous half century, even if 2022 AIs were worse than the average programmer.

thomashabets2 16 minutes ago | parent

That study is from 2022, with at most a last update 2023? (if I read it right)

I'm not at all saying that "oh 3 weeks ago, discard", but three years ago absolutely has a "before and after".

Three years ago AI generated code could basically not be used at all, and I had to rewrite just about every line. Today I have to watch out for generated code making steps in the wrong strategic and architectural direction, and over engineering, no longer hallucinating nonexisting APIs and forgetting to check the auth token.

I still look for those kinds of mistakes, but I think citing a 2022 paper for "can AI do X" is not useful.

spockz 14 minutes ago | parent

It just means that the safeguard we put in place over decades of software engineering with humans also need to remain in place when engineering with agents.

Having properly validated libs with separate concerns and a lot of eyeballs (and agent sensors) on the code so that it can be integrated cleanly in other products. Static code analysis for our apps. System prompts that require output to be secure, etc.

All of these practices that have been touted to be obsolete because agents can now just generate tailor made code just when and where you need, are anything but obsolete. On the contrary, they are paramount to the future of software engineering.

cyanydeez 16 minutes ago | parent

like most software, do you assume people are driving lamborghinis, or the cheapest make and model they can find...

NoboruWataya 15 minutes ago | parent

What models are people actually using to pump out code? That is the relevant comparison, not the absolute best in class models.

DrBazza 13 minutes ago | parent

Physician heal thyself?

Tell an LLM to break into its own system as part of a feed back loop, but with no context of the initial creation? It should iteratively get to 'safe' shouldn't it (subject to the tokens you want to spend)?

nephihaha 34 minutes ago | parent

"New normal"? Is that phrase back again? It was heavily pushed in 2020.

marmarama 16 minutes ago | parent

Talking about the new normal is the new normal.

verdverm 31 minutes ago | parent

> The rate at which we're churning out AI-generated code is going to make this worse

Not only this, a significant number of developers have stopped looking at code, trusting the bots. I've seen major security regressions, bot rolled auth implementations, and falsified benchmarks... just this last week, all in projects that get hyped.

Rule #2 - do not blindly trust Ai output

https://susam.net/inverse-laws-of-robotics.html

user43928 7 minutes ago | parent

It's not stupid if it works.

Concretely, I would like to see it benchmarked:

Have Opus 5.5 implement the authentication, with a 6.1 Sol reviewer. Out of 100 times, how often is the result insecure?

I wouldn't be surprised if it's zero times.

There is also the possibility of specifically prompting for a security audit on top, which would perhaps further lower the risk of overlooked issues.

Now one can say that due to the non-deterministic nature of the tools, there is always a non-zero chance of producing insecure implementation.

That's not really important here - obviously manual review isn't deterministic either, the risk will never be zero.

Gareth321 30 minutes ago | parent

> The rate at which we're churning out AI-generated code is going to make this worse, too, because AI-generated code is kind of notoriously terrible at security.

I think this was true in the past but AI is helping us finding exploits which have been in our code for many years. All out there by humans. We’re seeing fewer customer reported issues, and despite the pen tests getting far more rigorous, we’re passing all of them. Obviously this is one data point, but I’m hearing similar things from my friends in security and ops.

_superposition_ 8 minutes ago | parent

Agreed. Shouldn't continuous pen testing be a thing by now?

salviati 4 minutes ago | parent

> because AI-generated code is kind of notoriously terrible at security.

and vaccines notoriously cause autism. But you might investigate why that belief is so widespread and you might not believe it that much anymore.

Can you point at something specifically that indicates this statement is true?

Razengan 2 minutes ago | parent

Vaccines caused AI

dotdi 51 minutes ago | parent

> already being exploited in the wild [0]

The article cites this commercial blog patting themselves on the back how they detect the exploit, yet after reading it I don't see any actual exploiting? Is scanning/probing the same as active exploitation all of a sudden?

[0]: https://ellio.tech/en/blog/php2shell-what-we-observed-and-wh...

TutleCpt 41 minutes ago | parent

These are all human-generated software bugs. At some point when AI writes all code, the bugs should all but disappear. Almost.

ninkendo 38 minutes ago | parent

I think that’s a fallacy, see https://news.ycombinator.com/item?id=50207540

pegasus 35 minutes ago | parent

The bugs will only disappear until the model that was used to security test the generated code becomes obsoleted. So, yes, this is the new cyber normal. Unless we replace the tech stack with a formally verified one. Which will need to happen, one way or another, seems to me. But cheap & sloppy generated code won't be feasible in this paradigm, so I believe we'll need a much more thoughtful approach.

joenada 27 minutes ago | parent

1. The average human software-generator sucks at security. 2. LLMs are trained on human-generated software. 3. Therefore, LLMs suck at security.

CER10TY 21 minutes ago | parent

Bold of you to assume that AI-written and -reviewed code doesn't introduce new vulnerabilities. I think after we're done with this stretch of finding human-made vulnerabilities, we'll have a long road ahead to clean up the mess that's introduced by AI.

numpad0 6 minutes ago | parent

[delayed]

u8 40 minutes ago | parent

This is the new playing field, but this game can't last forever. Most software is not churning through a software factory of endless patches and releases. Most software, once mature, is mostly done.

I see a world in which we all get some generous Dependabot 2.0 on large forges because leaving so much software insecure otherwise is untenable, but once the software is not iterating a billion times a week, you're not introducing new classes of bugs just sitting there.

You can have a frontier security model pentesting your program every night while you sleep and you can wake up knowing that whatever happens, you did the best you could.

motionlessveloc 34 minutes ago | parent

Alas, our brave new LLM world is making iterating one billion releases per week all too easy.

And on the positive side, LLMs make remediation much easier too. It took me around 15 minutes and a few dimes worth of DeepSeek tokens to upgrade some gnarly, janky, exploitable PHP 7.x code to the latest fully patched version of 8.3 today. Without LLMs, I would probably have been having a serious discussion about whether we should ditch the app entirely instead.

jimmar 18 minutes ago | parent

> Most software, once mature, is mostly done.

The caveat being that most software never matures. Maybe it's a cynical take, but virtually all software I use regularly is updated at least once a month. Right now, I'm using Chrome, and I see the "Relaunch to update" button waiting for me to press it. I see the arrow on the Ollama icon telling me I have to relaunch it to apply updates. I have Visual Studio Code running next to my browser, and two extensions need to be restarted to apply updates.

But at least somebody is maintaining that software.

efilife 39 minutes ago | parent

By generating the entire website in AI, yes, you are contributing to the new normal

leftouterjoins 39 minutes ago | parent

It’s mostly hype. Days in now and nobody can exploit the PHP vuln. Yawn.

GaggiX 34 minutes ago | parent

LiveOverflow has exploited the PHP vulnerability, it’s even linked in the article.

ishouldnotbutk8 9 minutes ago | parent

There was a literal PoC video that started the panic, then multiple writeups. It's not hard (for people who know what they're doing) to exploit the PHP vuln. It's just not one exploit to rule every PHP installation anywhere, since this is a low-level exploit that needs to bypass ASLR, instead of a logic bug - so you need to tweak your exploit for every architecture, system, configuration, etc. And you need to know a fair bit about your target.

The response was overblown, but your dismissive reaction is too far in the other direction.

sigmar 37 minutes ago | parent

I'm not in a position to know. But Jason Clinton seemed to approximate "18 months" as an upper bound for the crisis period:

>as bad as things have gotten in the past 4 days, they are going to get a lot worse before they get better. security leaders, never let a crisis go to waste: please go to your boards and ask for help right now. not for ai, for the basics. (we need to hold on for ~18 months.)

https://x.com/JasonDClinton/status/2113325263157395456

helterskelter2 36 minutes ago | parent

I can't help but wonder if we're going to see the OS and browser maintainers begin streamlining code and reducing complexity. Software these days almost seems like a fractal of insecurity, the only way to make the problem more tractable is to reduce attack surface.

dreamcompiler 29 minutes ago | parent

LLMs seem to be very good at detecting vulnerabilities caused by using C and other forms of human stupidity.

So yes.

yomismoaqui 20 minutes ago | parent

Am I the only one that hates seeing "cyber" when refering to cybersecurity?

The same with "crypto" to refer to cryptocoins instead of cryptography.

aeon_ai 20 minutes ago | parent

> At the time of writing (23rd Oct), eMudhra has announced that it will be retiring from the certificate authority space. In unrelated news, a farmer has announced that he has shut his stable door, following reports of an escaped horse.

Ah, wit is alive and strong!

mattbrewsbytes 19 minutes ago | parent

People running old versions and not applying patches rigorously is also normal. Its been normal for decades, nobody likes doing maintenance so it gets forgotten or ignored.

swiftcoder 18 minutes ago | parent

> Who knew a regex engine was such a security issue

Anyone who has ever implemented a regex engine, I would imagine. Regex has always been a security nightmare, especially due to persistent attempts to use regexes to validate untrusted input

Aissen 15 minutes ago | parent

> especially due to persistent attempts to use regexes to validate untrusted input

This is not the case here, it's an untrusted regex which can be the issue.

DrBazza 14 minutes ago | parent

For now yes. When LLMs can iterate and test at speeds humans cannot, it is going to continue to find holes.

An interesting side effect will be whether we accelerate moving towards memory safe languages (and development of more of those languages) in more sectors than just 'low-level'.

Not forgetting that this is only one mitigation of cyber-attacks and exploits and potentially removes a whole category of attacks developed by analyzing *source code*

bcvvvy 10 minutes ago | parent

Not for me. I’m taking my shit offline. Out of harms way.

csande17 7 minutes ago | parent

Is this the new normal, or just, uh, normal?

Big embarrassing network-exploitable bugs in major open-source projects happen sometimes, and network-scanning attack botnets have been around forever. Small shady CAs mess things up sometimes. People upload malware to PyPI and NPM all the time, and "move fast and break things" software-deployment startups get breached all the time, to the point where it doesn't even really feel like news to me when those things happen again.

It's unfortunate that a few of those events all happened at around the same time. But, like, even if that's not a coincidence and these events are actually happening more frequently, you should have been preparing for them anyway!

jmyeet 2 minutes ago | parent

I'm a bit of an AI doomer but I'm going to disagree. I don't think this is the new mormal but there is going to be a dark transitional period.

What we're seeing is a bit of a "gold rush" where LLm advancements are coming thick and fast and they're being pointed out code bases that simply haven't had the level of scrutiny where relatively easy bugs to find have managed to survive. That's temporary. Here are my predictions:

1. Correctness and, in particular, memory safety is going to become increasingly important. I was bullish on Rust before this AI hype wave. LLMs finding and exploiting bugs has reinforced just important it is that Rust prevents as certain class of bugs that are the most prone to rmote exploitation ie certain memory safety issues.

I've long said that modern CPU architectures, out-of-order execution, hyperthreading (does that still exist? or did Intel fully kill it?), different CPU cores and compiler optimizations that it is essentially impossible for a human to write correct multithreaded code, particularly in userspace with "leaf node" code. LLMs just make that more apparent; and

2. Mature code bases for core functionality are going to become increasingly important because they've been "proven" in the wild (as much as you can prove a negative).

My sincerest hope, which is probably more wishful thinking than anything else, is that LLMs will be another nail in the coffin for C++, which is simply too complex and dragged down with legacy features and backwards compatibility to be salvageable.