YHK - yet another hacker news clone

Regions were always the scale of disaster isolation on AWS.

> In response to another question, Mr Vance rejected using the word “war” to characterise US operations in Iran, saying there was “no active shooting”.

https://news.ycombinator.com/item?id=49596955

If I had an "important enough" client, I think I'd store all out local (Sydney + Melbourne AWS cross region) data to AWS Singapore (to protect against Australian jurisdictional and political risks) and to a non AWS cloud provider in the EU somewhere. I reckon thatd be close to as resilient a pile of hard drives in an underground bunker, for significantly less setup and ongoing cost, while also being much more available when needed. (Can you imagine the queue at the underground bunker when multiple AWS regions get bombed? Or even imagine getting to the bunker in "a friendly jurisdiction" while a shooting war is taking place?)

We haven't worked out a decent solution to Visa and Mastercard payment network going down for more than a couple of cloud billing cycles though.

"I was in the office, reviewing Terraform plans"

He just couldn't get past the notion that if the six copies in four buildings across two states were all simultaneously physically destroyed, then most likely there are also no more schools left standing, and hence the contracts to clean them are null and void. Also, payment is now in booze and ammunition, not dollars.

I toured a datacenter once back in the early 2000s and they showed me 30 days of generator fuel storage. When i asked them why 30 days and not 35 they replied "we're such a major customer of both electricity and fuel that if we don't get electricity or fuel for 30 days there's way bigger problems than your website not being online" hah.

I get that S3 is convenient and reasonably performant, but it is not cheap at all.

I don't see where this is reported? TFA does not mention EBS. In fact, TFA seems to be nigh content-free, beyond "AWS (allegedly, and is uncited) says they cannot restore some data."

> The article does not tell us what products were impacted.

… right … which conflicts with EBS being what's reported …

(I would agree with your point that if EBS, or some AZ-level data was lost, then, yeah, that's the contract.)

https://aws.amazon.com/s3/storage-classes/

"Additionally, S3 stores data redundantly across a minimum of 3 Availability Zones by default, providing built-in resilience against widespread disaster."

I wonder if "can't restore some data" includes any S3 data?

I'd expect to lose EC2 instance EBS data in the event of a datacenter being destroyed, but I kinda assume I wouldn't lose S3 data? Now I'm wondering if RDS backups are more like EBS or S3...

Multi-cloud in the same country (if that exists in the country and is far enough apart) maybe.

Yeah, that was/is just another problem. Considering how that was actually handled in the real world before data residency laws came into force, I'm glad 'we' didn't convince those countries to put their citizens data at risk.

As an aside, it is quite crazy the world we live in. I am with the majority where I expected Amazon to be more redundant, but I still marvel at the assumption that a US dev can spin up multiple redundant and data sovereign servers in dozens of countries with efficient caching, failover and redundancy (enough to survive an earthquake or targeted missile attack) from their own home. Even a few hours of outage in a foreign country is considered unacceptable.

And second, the USA is in the middle of power grab that completely ensures any data stored there will be taken hostage wherever suitable for "negotiations".

As soon as you start specify technologies, rather than "sovereignty" you end up needing to created specific legal tests to stop people getting around it.

"Data must be stored domestically" is a short hand for being held in the same legal jurisdiction. This means for somewhere like the UK, you get all that battle tested data protections law for free. (new laws require case history to be reliable. Ie, prosecuting under a new law is hard, because if its on the edge of being legal, it can create a precedent that undermines the entire law)

In civil code places, its different, but I don't know enough to offer even a half arsed opinion.

The reason why jurisdiction is important is because if you are storing data outside of your legal protection, when something goes wrong there is little you can do to discourage fuckery.

This is the problem with blinkered engineering thinking. Yes geographically distributed data storage is good. But as you also know, storing it in place with lots of other data, means that its a target. The more places its stored, the more physical security you need. This means that there is higher chance of people being bribed.

Its not a binary, its a multi-dimension graph, with no one answer. Every dimension has a tradeoff.

UAE's tradeoff was: not even trump would ignore all the wargaming that clearly shows kicking iran in the nuts would have inflation rising consequences

The sort of global instability that is being created by all this is bad for everyone. It generally isn't good for business either.

We're so unstable and capricious now, countries are actually trying to decouple from American tech services like AWS and Amazon. That ain't great for Bezos.

Maybe inertia kicks in after Trump and things revert to the mean, but restoring confidence in the US again as a friendly, stable nation is going to be tough if we're always 4 years away from another Trump-type figure running the show.

So then that big thing comes to you. It’s all kind of … held together by a prayer …

Trust me I’ve worked at these big places. You wouldn’t believe how much fucking luck and grace from God is allowing you to do anything with your digital life. It’s a mindfuck of a tangled mess out there, eternities worth of written code that only God ensures works together at this point, only to get more hidden with AI.

For example, you mention Backblaze. Backblaze has public posts about their durability model. They claim to use 17:20 Reed-Solomon erasure encoding. That means there are 20 shards of a blob, and you can lose 3 of them and still reconstruct the blob.

Think about that for a second. If they store 4 shards in a datacenter, that means that a loss of that one datacenter is sufficient to lose the blob, forever. That entails that blobs are sharded across a minimum of 7 data centers, or the loss of one data center might mean permanent data loss. Which one do you think is true? (In fact it's pretty clear from Backblaze's public posts that they don't shard across data centers at all, only across racks within a data center.)

Now, AWS's availability guarantee — not their durability guarantee — entails that they use a less cost-effective erasure coding ratio. S3 is designed so that your blob is available even if a whole AZ goes down, and it's well known that most AWS regions have only 3 AZs. Therefore, if you tolerate the same number of shards lost to HDD failure as Backblaze in your durability model (3), then you might need 17:30 erasure coding to get the same durability and the required availability. That means S3 is storing way more physical bytes than Backblaze — 1.76x the logical size of the blob, instead of Backblaze's 1.18x. That's more expensive, but it also gives you better availability.

Which is also why One Zone S3 is cheaper — if you don't care about the availability guarantee, S3 can do what Backblaze does and save 33% on physical bytes, and they pass on 40–50% of those savings to the customer (this is fairer than it sounds — there's more overhead than physical storage bytes).

But here's the thing. AWS has more redundancy built in than Backblaze because they make availability guarantees in addition to durability guarantees. BUT the durability model is the same, which is why Backblaze can claim equivalent durability to S3. S3 in fact has better durability — they can survive the permanent loss of an AZ without necessarily losing blobs stored there (with the exception of One Zone blobs), and Backblaze cannot. But that's not actually a factor of the durability model, which is just taking into account normal events like HDD failure. Instead, S3 has durability that's more resilient to AZ loss because of their availability model. It's a side effect that isn't actually part of the durability promise!

Amazon only claims "99.999999999% durability" per year, even for the properly replicated stuff.[0]

[0] https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu...

Obviously here, he should have mentionned that they can recover a hit on a single data center, provided the customer chose multi-AZ hosting. That's probably why companies run their ads on "This watch is a legacy for your children" rather than any material claim.

Doesn’t really apply, because the only reason data was lost is because customers chose not to replicate it to other regions, either because of legal data residency requirements, cost, or just not bothering.

If Skynet wants to make sure it’s backed up, none of that prevents it from doing so. Although it would be amusing if Skynet was stopped by a billing alert when it tries to copy itself to another region.

Similar here, there’s a lot of detail that got hand-waved away by a sloppy “yeah we good” puff PR answer.

An extraordinary statement itself that shows the difference between a proper cloud where they AZs are at least 60 to 100 miles apart...and Google or Microsoft... pretend clouds...where those AZs are just firewalls across the same data center...

How big was this flood?

They have way more skills than the technical folk in doing that

He didn't say "but if you hit many data centres then there is a problem". The premise was if you hit data centre, user won't notice, without caveat that there is a limit.

Germany and France in a debt spiral and turning inward/hyper-nationalist while massively re-militarizing means the EU is a safe place to structure your data with zero redundancy.

I wouldn’t bother worrying about key industries and functions, since, as history has shown, the EU is a bulletproof institution that no country has ever left.

And of course every data center in Europe has Israel-grade air defenses, it’s not like they are just sitting ducks for a fleet of drones to take out within 24 hours.

The only way to approach readiness here is repeated investment - like one team doing the deep dive and another pulling cables and then constantly doing pre- and post-mortems.