Yet another Hacker News

I'm sorry, you're not going to die from an AI-engineered supervirus

48 points jryb 1 hour ago 64 comments

infinite_spin 1 hour ago | parent

mepiethree 54 minutes ago | parent

So somehow AI is able to magically create 100 lethal viruses with no logistical issues but is not able to make a cure due to logistical issues?

anigbrowl 41 minutes ago | parent

My immediate reaction to his

The doomsday virus has a first-mover advantage over the vaccine. The time that people start to drop dead from the virus is the time you start to make and distribute a vaccine. Even if AI can design and synthesize a vaccine within hours, how fast can it be rolled out to the general populace? It took months to distribute the Covid vaccines, even when we already knew how to make them. You have to physically load the doses on trucks[...]

was just 'make a viral vaccine with a shorter incubation period and infect everyone'. Handwavey problems require handwavey solutions.

customguy 36 minutes ago | parent

> but is not able to make a cure due to logistical issues?

It says no such thing:

> they design highly effective mRNA vaccines and antivirals against all the viruses and start shipping them across the world. It’s too late.

mepiethree 52 minutes ago | parent

The author scoffing at the climate scientist is so cringe

Metacelsus 1 hour ago | parent

Just because the author can't think of a way to cause mass death with an engineered virus, doesn't mean an AI won't be able to think of one.

(There are several biological threats the author hasn't considered. I'll just leave it at that.)

phoghed 58 minutes ago | parent

I know what you’re thinking. A modified HIV virus that also makes everyone hot and horny.

jeremyjh 54 minutes ago | parent

The author can't think of what a superintelligence will do, because they are not a superintelligence. But that is not really what people are concerned about, so I think the objections here are quite meaningful.

If there is a fully general superintelligence - in the current state of alignment - we will all die for some reason. It doesn't really matter what the reason is.

cobbzilla 44 minutes ago | parent

please define fully general superintelligence & current state of alignment

is this a tautology?

andai 7 minutes ago | parent

Not the guy but, alignment seems to be several decades away, while self-improvement seems to be several years away.

(Also you don't even need superintelligence to do a lot of damage. In fact you don't need any intelligence!)

andai 9 minutes ago | parent

Well, at that point it doesn't even need to be a virus. It can just be nanobots that atomize your liver. The substrate is a mere technicality.

alevskaya 31 minutes ago | parent

I was a biophysicist and experimental biologist for decades, I designed viral vectors and cell therapies, etc. What protects us here isn't our ignorance vs an "ASI".

It's the fact that the complexity of molecular physics scales exponentially in particle number. No amount of thinking is going to punch through that. You've got to do experiments. The slow loop through reality is not optional, and in the case of "superviruses" would require a lot of iterative experimental development in humans.

(And please knock off the bio infohazard act - I routinely get asked about scenarios that AIxBio safety people cook up and they always involve some howler misunderstanding of basic facts in biology or medicine. Just talk about an idea if you have it.)

naasking 9 minutes ago | parent

We're already doing those experiments. Tailor made mRNA vaccines targeted to one's own specific cancer mutations can be bought right now. The techniques are getting more sophisticated and more targeted every year. Our ability to predict what happens at these levels is improving by leaps and bounds too thanks to AI like AlphaFold. Anyone can be reasonably confident that no AI or teen can do this now or in the next few years, but are you really so confident what might be possible in 10 years?

pama 12 minutes ago | parent

I am not sure what you mean here. There exist plenty high-concern biological threats that dont need any AI help. Human oncovirus design is low on my concern list (immunity is diverse), and in any case it does not need AI—rather labspace. Disgruntled high schoolers or undergrad chemists can do way more damage from readily available materials without AI and without research delay. As can nature (or amateur biologists without AI) by mixing bats with their animal of choice and waiting a while. If/when any scary global events like covid happen again, I sure hope we have true superintelligence to help us navigate it quickly.

ViktorRay 1 hour ago | parent

I wonder if there is a name to this phenomena. Where being knowledgeable in one thing (such as computer science) means you incorrectly believe you are an expert in another domain (such as virology)

throwawayy6767 50 minutes ago | parent

Noah Smith isn't an expert in computer science either, he's just some online guy who posts a lot about his opinions. He used to have semi-decent takes but at some point in the past few years he went completely off the rails.

ViktorRay 49 minutes ago | parent

Maybe my original comment wasn’t clear. If so I apologize. I was actually criticizing Noah Smith. I agree with what the author of the posted article said.

EDIT: actually it isn’t even that. I personally don’t know enough about this topic to really have an opinion or criticize anyone. Still it’s indeed a phenomena I’ve noticed.

ghurtado 46 minutes ago | parent

It's much simpler than that: You don't have to be an expert on anything to be a complete dunce in everything else. Hardly rare enough to call it a "phenomenon"

tayo42 59 minutes ago | parent

My sci fi story I came up with was,not that a virus is good to kill us,but a rogue AI is going to use a virus or something to hijack our minds and have us do the physical work it wants done like an ant colony.

nprateem 52 minutes ago | parent

Maybe it'll call this superpower "money"

dvduval 57 minutes ago | parent

That sounds like a good video game to spin up in Astra. Missile command was pretty cool. We didn’t destroy mankind with that one either.

Mathnerd314 34 minutes ago | parent

Or you could just play the original Pandemic https://www.newgrounds.com/portal/view/360724

I think that's one detail that this analysis glosses over is that you design the virus to mutate and develop lethality. So initially, it's just like, oh, another common cold, but then once it's infected a large number of people, then it mutates and starts being lethal.

intrasight 57 minutes ago | parent

AI is just one of many things that concerns me. But it has lately risen higher on the worry chart.

gerdesj 55 minutes ago | parent

Ye domesday weapon will not be a disenchanted teenager whipping up COVID-19 variants (whatever that means) and spraying them on people.

The real deal needs a lab messing around with viruses and whatever happened a few years ago ... and it did not destroy humanity.

01284a7e 55 minutes ago | parent

"The problems with which the universe can confront any society are, for practical purposes, infinite in number and endless in variety." - Joseph Tainter

AI will really aid the universe in generating more problems. Maybe it's not a supervirus, but hell, maybe it is.

ghurtado 49 minutes ago | parent

That quote is very interesting, because I think I interpreted it the opposite of you:

"we could be attacked by an infinity of problems, which means that worrying about all of them is statistically unpractical"

Dig1t 54 minutes ago | parent

Thank goodness someone wrote this, I was also thinking about writing an article like this.

This is such a massively overblown worry that AI companies who are seeking regulatory capture love to roll out whenever they are trying to gin up some FUD.

Creating viruses at home is extremely difficult, even if you have a perfect instruction manual and someone sitting behind you telling you what to do. It's full of trial and error and validating each step requires rigor.

It's also a process which requires fairly expensive and specific supplies, which can be easily gatekept. Instead of trying to gate-keep knowledge of biology, we can do things like KYC around specific types of lab supplies (like we already do with fertilizer which can make bombs, or specific types of supplies needed to make nukes). Heck the Biden admin already added a KYC requirement for plasmids and plasmid companies already have implemented screening for sequences which could be dangerous.

skew-aberration 46 minutes ago | parent

Viruses probably too hard but I think small scale bioterrorism is much easier and doesn't require fancy enzymes. I think this is actually the best argument for regulation of current models.

It's even happened pre-AI https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack

alevskaya 26 minutes ago | parent

You can read up on the synthesis of sarin on wikipedia.

colingauvin 22 minutes ago | parent

Most people do not understand how easy it would be to make anthrax. Technology is not the reason it basically is never done or deployed. AI cannot meaningfully move this needle.

Let's look at it in an inverse way. Making a single drug that impacts a single protein, intelligently, from the ground up, has arguably never actually been done. Despite unbelievable financial incentive. And with AI.

Dig1t 14 minutes ago | parent

Well we're talking about new types of capabilities here, Sarin is a chemical which can be manufactured with a basic understanding of chemistry and searching with google.com. AI will not allow bad actors to do anything they couldn't have done already with google's 10 blue links.

Again the same logic applies though, it takes significant effort and money to create sarin, it requires a lot of rigor. That is the limiting factor, not the knowledge of how to make it.

If we are worried about people making sarin in their garage we can do many things to control the supply of chemicals needed to make it. We don't need to try to restrict knowledge of chemistry.

"regulation of current models" basically implies that open source models will be heavily restricted if not banned outright and guarantees that intelligence will be owned and gatekept by a duopoly.

gustavus 13 minutes ago | parent

That really proves the point though worst case scenario, releasing a nerve agent into a crowded subway, multiple subways resulted in a grand total of 14 deaths and 50 serious injuries. bum bum bum It's not that that isn't tragic or sad but more people will die or be maimed in a car crash in the US in the next 2 hours than died in the worst case scenario bioterrorism attack.

So if I have to choose between the slim possibility that a crazy person might develop a bio-weapon and the possibility that Anthropic, et. Al convince the government to regulate and control open source models (because I'm sure the government will only do that for good reasons and never things like covering up scandals, squashing dissent or enhancing their own power) and any competitors. I'm probably going to put my chips on a nutcase in a trailer park developing super sars over regulator capture by the government.

nprateem 54 minutes ago | parent

I agree.

It'll probably just hack and disable our water works, power stations and supply chains and let us all starve.

zekefoppa 51 minutes ago | parent

I'm far from anything resembling an expert in biological weapons, but this seems to me puzzlingly overconfident in the opposite direction.

> PhD students in 2026 using state-of-the-art AI software are spending months or years trying to design simple peptide binders that inhibit some enzyme or pull down some protein

This confuses me, as state-of-the-art AI software is moving quite quickly. Any software that is currently state-of-the-art AI.. hasn't existed for years, maybe not even for months.

It also seems quite odd to claim that there are no sufficiently dangerous diseases when we have high-security research labs dedicated to exploring.. exactly these kinds of diseases.

And, even if it's not a disgruntled teenager, it just takes one leak from a lab that has 10xd productivity using AI tools.

This is a real risk that existed before AI, one that I think we can now confidently say that society is under-prepared for since 2020. It is certainly magnified by AI, even if it's not 100x.

The author even seems to recognize this:

> So, if you’re worried about AI-designed superviruses, think COVID, not Ebola. And if you’re now a little less worried, maybe you understand where I’m coming from.

This is exactly where I think most people are starting from, and that's _why we're worried_.

ghurtado 43 minutes ago | parent

If you quote the article, then you are implicitly telling others that you have read it.

But it's pretty clear you have not?

zekefoppa 16 minutes ago | parent

methinks it is more likely that I read it but missed the point? Would you be able to share what I misunderstood?

Boxxed 46 minutes ago | parent

Yeah, if only.

mikestylz 46 minutes ago | parent

The magical level of intelligence required is not really a flaw of the argument's deductions, it is its very premise. If you believe that there is a possibility of an AI working on itself, you get the approximate differential equation d/dt f(t) >= C*f(t). In plain English, at least some proportion of development's progress goes back into future development. The (least case growth, assuming equality) solution to this is an exponential function, and exponential blowup leads to stupendous and magical seeming results. I then ask - what is evolution if not a dumb optimization process? How are we supposed to believe that it devised miracles such as the human mind, insect flight, echolocation and so on while denying the possibility of a future EXTREMELY power simulation to do the same? I went in really wanting to be convinced by this article for the sake of my own mental health, but I am disappointed. What we really need for such a statement is concrete physical evidence that a virus is simply incapable of having such facilities. Or perhaps an equivalence established showing biological defense is no harder than offense. Until it exists, I will continue to worry.

crazygringo 22 minutes ago | parent

> What we really need for such a statement is concrete physical evidence that a virus is simply incapable of having such facilities.

A major focus of the article is about the fundamental tradeoffs in how viruses operate. Was that not enough for you?

janalsncm 45 minutes ago | parent

> When people believe that AI could figure this out on its own, without first running extensive and expensive basic science experiments, they have left the realm of science and have entered what can only be described as religion.

I strongly agree with this. Too many people have the Arthur C Clarke quote infecting their brains.

So much of science is contingent on the physical world and cannot be determined from first principles i.e. by thinking hard about it. (Sorry Plato, you can’t think your way to the moon.) You need increasingly precise measurements, which means increasingly complex tools which means…supply chains. Science requires supply chains. Magic doesn’t.

ewild 34 minutes ago | parent

And if an agent were to amass a large sum of Bitcoin you don't think people wouldn't do things for money? Especially if they don't realize they are doing part 1 of 160 parts that get it to the end goal of a virus?

GlibMonkeyDeath 31 minutes ago | parent

Especially when it comes to biological things. Just look at Novartis this week - three failed clinical trials of promising drugs (https://www.business-standard.com/world-news/novartis-faces-...)

These are some of the brightest minds on the planet, spending oodles of money on things they think they understand. And they didn't work.

naasking 25 minutes ago | parent

We're doing that science right now. We're using AI to do it. By the time AGI truly spreads, we'll have a lot of the knowledge you say will be needed. This is simply not as far fetched as you seem to think. Next few years, unlikely. 10-15 years? Increasingly possible. 20+ years, I'd hazard even likely.

mrits 21 minutes ago | parent

Is it so hard to think that super intelligence could piece things together that humans can’t?

areoform 6 minutes ago | parent

I want to go further. They have no idea how dangerous it is.

Anything that can kill other people can also kill you.

I've posted this list before, and I will post this again and again until it sinks in, but this is a real list of real incidents that happen.

    Researcher Nikolai Ustinov was lethally infected with the Marburg virus after accidentally pricking himself with a syringe used for inoculation of guinea pigs. 

    Dora Lush died after accidentally pricking her finger with a needle containing lethal scrub typhus while attempting to develop a vaccine for the disease

    A 23-year-old laboratory assistant at the London School of Hygiene and Tropical Medicine, was infected with smallpox after observing the harvesting of live smallpox virus from eggs without isolation cabinets at that time. The assistant was hospitalised and before being isolated, she infected two visitors to a patient in an adjacent bed, both of whom died. They in turn infected a nurse, who survived

    Ebola laboratory infection by the accidental stick of contaminated needle in the United Kingdom

If you are designing a supervirus that spreads via water, air etc., you better have one hell of a cleanroom, cabinets, manipulation equipment, incinerators (plural), hydrolysis and steam systems, multiple disinfection protocols...

The list is nearly endless, and you can't handwavium it awway.

It doesn't matter if it's "all robots" or not. Even if it's all robots in the middle of nowhere. If you're developing something that dangerous, then it will start killing animals and it will spread. This happened in the UK, this was for foot-and-mouth disease,

    The 2007 United Kingdom foot-and-mouth outbreak was the accidental discharge of virus FMDV BFS 1860 O from a laboratory of the Institute for Animal Health in Pirbright, through possible leakage from broken pipework and via unsealed overflowing manholes, leading to foot-and-mouth disease infections at multiple nearby farms and the culling of over 2,000 animals

And even if you are in the middle of nowhere, if you get sloppy, it will end up in the wastewater run off which will then get picked up and sequenced,

     Routine wastewater surveillance of the vaccine production facility at Utrecht Science Park/Bilthoven [nl] detected infectious poliovirus from a sample collected on 15 November 2022. Full genome sequencing indicated the sample was shedded from an active human infection of wild poliovirus type 3 (WPV3), and further testing of all employees with access to WPV3 found one employee was infected. The employee was isolated until their polio infection was resolved. It remains unclear how the employee became infected given the biosafety measures used at the facility

A lot of the people talking about bioweapons of doom are saying that standard graduate level knowledge is an existential risk to humanity.

It's worth repeating again,

Anything lethal enough to kill other humans is lethal enough to kill you.

And if you don't know what you're doing — and for this argument they're talking about people who have to ask a LLM "how do I spanish flu?," the number of ways you will die far outnumber the ways you can succeed.

I am writing a piece on the topic, please contact me if this is your field of expertise.

I am quoting as many primary sources as possible as it's not just some random on the internet, and I've gone back as far as the 1900s to find the many, many, many ways in which these statements are absurd, and misunderstands the potential of their technology.

jbethune 45 minutes ago | parent

Well, not with THAT attitude...

geodel 42 minutes ago | parent

Well, this is disappointing.

bananaflag 34 minutes ago | parent

The author doesn't have a security mindset.

hex4def6 29 minutes ago | parent

Now here’s a real risk involving measles that I sometimes worry about. What if the measles virus experienced a set of immune escape mutations where suddenly existing vaccines no longer work? If this happened the consequences would be horrifying. We’d have an immediate, world-wide measles epidemic long before we could develop and roll out a new vaccine. What’s the likelihood this will happen? Informed estimates are that it is low. But realistically speaking, even if the likelihood is low, it’s probably higher than the likelihood that a disgruntled teenager will design a supervirus with AI. So, something to think about.

I'm confused. Would a bit of gain-of-function exploration + measles virus genome = supervirus?

I'm no biologist, but it seems like the genome is readily available. https://www.ncbi.nlm.nih.gov/nuccore/NC_001498.1

And it seems like the spots that have to be mutated are already known about: https://newsnetwork.mayoclinic.org/discussion/researchers-cl...

And this experiment has already been done: https://pmc.ncbi.nlm.nih.gov/articles/PMC8080110/

Five minutes of research suggests the sites -- Loop Epitope etc -- only need a few pairs of mutations each. this feels doable... It's not creating a supervirus from whole cloth, but why bother. Measles without a vaccine would be a catastrophe.

colingauvin 26 minutes ago | parent

Oh my God THANK YOU.

I do not have the words how tired I am of hearing tech people and economists talk about biology.

mrits 18 minutes ago | parent

Do you need to be a physicist to realize an asteroid is going to kill you ?

colingauvin 18 minutes ago | parent

Arguably yes. I would wager no one killed by an asteroid in the history of people killed by asteroids realized what was happening before they died.

smitty1e 16 minutes ago | parent

Refereeing who is permitted to say what based upon credentials is a popular sport these days.

colingauvin 10 minutes ago | parent

I would never advocate they not be allowed to say it. But I also am sick of hearing wildly unrealistic takes.

fergal_reid 24 minutes ago | parent

I am not a biology person, but, candidly, this is a bad article, because the author is generalizing from the properties of viruses that tend to arise naturally, and using this to argue about the properties of deliberately designed viruses.

Firstly, it's not even clear that naturally occuring viruses can't be extremely lethal. There are examples (eg black death).

Even the existence of rare examples should be concerning, if attempting to argue against the possibility.

But even if there were no natural examples, it could just be because the ecosystem is in a balance, which them unlikely. That doesn't preclude someone from designing one.

To use an analogy. Natural predator prey ecosystems tend to be in balance. You could look around the world, before modern humans, and conclude that superpredators aren't a real threat, because predators that eat their prey too rapidly typically then die out. Thus most ecosystems you observe are in some balanced state.

But this doesn't mean modern humans can't come along, using their superior intelligence, and act as a superpredator. Ask the wolf population of western Europe.

If something is stopping us from doing this more often, it's alignment rather than capability.

You can't generalized from the typical behaviour that you see in a balanced natural ecosystem, and use that to say it proves there are no much more dangerous possibilities.

If the author made first principles argument about the inherent difficulty of doing something bad, even with abundant intelligence, I'd be much more reassured; or about defense offense asymmetry favouring defence.

What's here seems very naive.

jsnell 24 minutes ago | parent

Native americans had like a 95% population mortality rate from infectious diseases in the first generation of European settlement, no? Measles, smallpox, flu, etc.

The mortality rate from those diseases was higher than today because the population had no immunity at all, leading to both higher transmission rates and higher mortality rates. This would be true of an engineered virus as well.

And it wasn't just the one virus, it was an entire continent's infectious disease arsenal landing in overlapping waves. Again, something you'd do if you were trying to maximize damage.

So I don't find the bulk of the article about it being impossible for a virus to have this effect all that compelling.

BobbyTables2 22 minutes ago | parent

We might even get to relive history the way things are going…

colingauvin 13 minutes ago | parent

This number gets cited a lot but it's actually the excess mortality of post-European contact, and it's not content wide, it's specific to Mexico. It also includes climate change, famine, war, relocation, etc.

Footnote7341 23 minutes ago | parent

Everyone is gathering around patting themselves on the back, but smallpox depopulated an entire continent. Clearly it is possible for a highly transmittable deadly virus to exist.

woopwoop 22 minutes ago | parent

The author seems to be arguing that it is biologically impossible or something to have a virus with the same transmission properties as COVID but higher virulence. I don't understand what principle he is invoking to make this claim.

mrits 20 minutes ago | parent

The cope principle

andai 12 minutes ago | parent

tldr: 2029 -> 2049