YHK - yet another hacker news clone

Archive Of Our Own down for maintenance for 2 to 4 weeks

25 points GeoAtreides 1 hour ago 17 comments

peri-cl 1 hour ago | parent

Wonder if it's related to the Gitlab crisis (the regexp vulnerability in Ruby that hit them first). Ao3 seems to be a Ruby on Rails app.

mryijum 1 hour ago | parent

2-4 weeks feels so absurd that I reread the alert twice thinking maybe they meant something different. Database problem into discovery of failed backups, perhaps?

It'll be interesting to see the postmortem. I don't know much about their infrastructure.

mschuster91 1 hour ago | parent

Per reddit [1] the data is safe.

Apparently they run bare metal that they own [2] so I'd assume their servers got compromised hard and they'll need to buy entirely new servers.

[1] https://www.reddit.com/r/AO3/comments/1xir586/comment/19rubn...

[2] https://www.reddit.com/r/AO3/comments/1xir586/comment/19ruc1...

anonymous908213 1 hour ago | parent

> In necessity, we could even switch to the second server with very little downtime

So much for that...

mschuster91 1 hour ago | parent

When the threat model is "a fire takes out the DC", sure. But "the servers are root-level compromised", whoops, that's worth rebuilding from scratch. No way to make sure whatever AI agent did the compromise didn't go and embed malware in some flash ROM.

rafram 51 minutes ago | parent

You seem to be jumping to a lot of conclusions here. Who said anything about any server being compromised? Much less root-level compromised? And an AI agent being involved?

mschuster91 42 minutes ago | parent

> You seem to be jumping to a lot of conclusions here.

There aren't that many scenarios which make a restoration time of weeks inevitable, even after taking into account the AO3 people are volunteers - given the large intersection between nerds and fanfiction or a loooot of furries working in IT, I'd assume you'll find the most competent people working on it, so I doubt the cause is incompetence.

They say the data is safe, which implies an offsite backup with a different architecture (or a managed provider). They said in the past they had a secondary site which they could switch over to if needed, which implies that the issue is not related to hardware or datacenter problems (e.g. your usual backhoe related issue or a physical destruction of the datacenter).

And that leaves only two options: a) a former staff member went fully rogue and everything needs to be rebuilt because everything might be backdoored, but there is no indication of that, or b) they got compromised really hard by one of the multitude of the vulnerabilities that cropped up the last days and chose to rebuild from scratch for a similar reason.

(And as far as I know AO3, like FFN, is limited to text and stuff like cover art, so the dataset should be small and easy enough to restore from a backup)

Hamuko 53 minutes ago | parent

Was it mentioned somewhere that they got compromised? And that it's not for example a hardware failure? I imagine lead times for getting brand new hardware would be in the 2–4 week range these days.

wmf 41 minutes ago | parent

Imagine if there was an API to rent servers within minutes or someday even seconds.

altairprime 28 minutes ago | parent

Imagine if everyone offering said API were capricious, inhumane, and uncaring about customer support, service terminations, and customer wind-down processes.

Now that you’ve pictured AWS, GCP, DigitalOcean, et al. more fully, that rules them out as available options for a long-term fanfic-centric erotica archive site that has specific and unusual legal concerns and threats to its existence incompatible with skittish Redbox-style compute providers.

Was there a particular cloud provider you were thinking of, that is not a ‘good night, sleep well, perhaps we’ll kill you in the morning’ service provider incompatible with AO3?

Avamander 5 minutes ago | parent

That speed comes with its own cost though. Might be simply too unreasonable at their scale.

mschuster91 41 minutes ago | parent

> And that it's not for example a hardware failure?

They say they have a second site, the chance of hardware failure taking out both sites at the exact same time is near zero.

peri-cl 52 minutes ago | parent

The data's *always* been safe. /r/DataHoarders would never overlook this.

Nvidia will serve you a free, high-speed half-terabyte download right now[0]. It's one with a cutoff date of 2020—but given that GPT-3 was released mid-2020, that's probably the one you actually want, for ML research.

[0] https://huggingface.co/datasets/ray0rf1re/AO3-2020

ronsor 50 minutes ago | parent

There are later dumps, but they aren't quite public

TMWNN 1 hour ago | parent

"Hey, why are activity rates up 2000%?" —Tumblr executive