40 points Bender 2 hours ago 44 comments

ncr100 1 hour ago | parent

They talk about their relatively low-end testing environment PASSing the July code, and being different from their high-end production environment machines, sadly hiding the concurrency bug during the test phase.

Is this commonly acceptable? I'd imagine it's a costly trade-off?

bayindirh 52 minutes ago | parent

Depends on what low/high end are. If low end is 4 core and deployment is 128 cores, this is not acceptable.

But if low is 64 and high is a whole cluster, now that's a different kind of maths.

Edit: The article states the following:

> "lock contention inside an internal Windows process which caused the machine's resources (CPU & memory) to be exhausted."

So, it's more the former. The testing machine is probably a 8 year old server on an office shelf, and prod machines are 64-128 core cutting edge Xeons/EPYCs.

Lock contention. It was MUTEX spins in Windows 95, it's lock contention in 2026. Why this is still a thing, Microsoft?

(Man, I felt old while writing this.)

lokar 27 minutes ago | parent

I always this was standard to have a test setup in the production environment on the latest hardware (that’s how you qualified that hardware in the first place) running one “unit” of the system (whatever that is) at full scale and load (ideally higher).

feverzsj 1 hour ago | parent

Poor Microsoft can't afford more than one low-end testing environment.

elcritch 59 minutes ago | parent

Think of the poor starving shareholders and CEO pay packages.

rs999gti 45 minutes ago | parent

> poor starving shareholders

You mean you and me? You realize pensions, institutions, and individual investors are shareholders as well?

elcritch 16 minutes ago | parent

It's a rhetorical jab of course.

The mantra of "shareholder primacy" sounds great in theory but just becomes an excuse for short term profit boosts for a small class of shareholders usually at the expensive of the long term health and growth of the company and long term value seeking shareholders.

Sort of similar to how most countries with "people's" or "democratic" in the government's name tend to be democratic only for the ruling party.

In this case ensuring adequate engineering resources for possibly the most widely used email system in the world doesn't seem crazy. How many long term customers will they loose because of this incident? Perhaps they did have enough resources, but given the recent layoffs to chase AI I sort of doubt it.

triceratops 6 minutes ago | parent

The top 1% wealthiest own 50% of all public stock. https://www.fool.com/research/how-many-americans-own-stock/ So it's mostly not you and me at all.

dgrin91 51 minutes ago | parent

Doesn't the article state the opposite? The low end was not the problem. It was high end that was the problem.

I like ragging on Microsoft as much as the next guy, but honestly this problem sounds complex & hard to predict. AI or not, this feels like the type of bug that would have slipped by me as well.

The response clearly could have been stronger though.

thesuitonym 38 minutes ago | parent

The point they were making is that had Microsoft used high-end equipment in testing they may have found this issue before it hit production.

zdragnar 36 minutes ago | parent

I read "more than one low end" as "more testing environments than one low end" rather than "more low end environments than one". It's a snarky way of saying they should have been testing high end environments for no more reason than they had the money to do it, rather than saying they should have been testing the low end more.

preommr 49 minutes ago | parent

This is what happens when you burn billions for no reason.

Github copilot had INSANE usage. Cursor switched away from per request plans in 2025, Copilot did it this year with all the reasoning and more expensive models. It was insane, you could write out a whole page of features, and have it run on one request for hours depending on the loopholes you exploited (some models were discounted for some time, basically free in some cases, and shenanigans around switching in betwen agents)

We talk about opus et. al being heavily subsidized at like 1-2k for $30-$100. I remember seeing figures in the 10s of thousands in subsidizations for GHC. There's no way they weren't losing just massive piles of money.

hmokiguess 53 minutes ago | parent

> The engineers who worked feverishly to restore services deserve commendation and will hopefully never have to buy their own beverages in Redmond again (or at least escape the next round of firings).

What does this mean? I didn't understand

abrichr 50 minutes ago | parent

I think it means that people will hopefully buy them beers as thanks

bayindirh 50 minutes ago | parent

In sysadmin speak this is hoping that Microsoft gives them free beverage credits (or give them some kind of bonus because of their efforts) and don't fire them as soon as the fix is deployed to scapegoat them.

In a rare occurrence, The Register sides with the Microsoft developers. Not that they're anti-developers, but they are anti Microsoft because of its well-earned reputation for EEE and Halloween Documents.

Molitor5901 34 minutes ago | parent

It kind of feels like they are saying, look, we know we caused the problem, but we fixed it. So we're good, right?

nashashmi 32 minutes ago | parent

An artistic way of saying the engineers should get praised and rewarded.

AdamN 11 minutes ago | parent

You mean like the Covid nurses? With stickers that say 'You Rock!!' or 'Hero' and a $5 gift certificate to Dairy Queen?

voidfunc 19 minutes ago | parent

It means they deserve "Thanks" (buying a beer/coffee is a token Thank You) and hopefully they are removed from the pool of candidates for the next layoff.

Surprised so many people are struggling with this...

Having been on incidents like this before though, yea, that aint happening.

27183 14 minutes ago | parent

> Having been on incidents like this before though, yea, that aint happening.

Yeah, instead what will happen is their manager will ask them why they didn't get as many story points done this sprint as they did last sprint and their performance will be marked "degraded".

bayindirh 13 minutes ago | parent

> Having been on incidents like this before though, yea, that aint happening.

Been there, through that, and sadly you're right. The bitter part is I didn't want extra compensation or anything. Just a thank you and a day off is more than enough for me, so to recharge and ride the wave of what we (not I, we) have accomplished, so I can work with extra motivation for some time.

Instead, got a 5 minute coffee break and dived to the next non-urgent ticket. Yeah, great way to thank and motivate the people who spent 24 hours in an underground office cutoff from the world, trying to save a system from melting down.

Glad I'm not there anymore.

giancarlostoro 6 minutes ago | parent

People should buy them drink, as for the firings side note Microsoft recently had a "voluntary exit" offer mailed to a fourth of their workforce:

https://www.geekwire.com/2026/microsoft-makes-new-voluntary-...

How long until Microsoft fires key people to the point that outages become more common? All in the name of AI, which I don't believe for a second, they're just trying to maintain stock market profitability math by firing enough, they'll just hire more people when they make more.

sandworm101 49 minutes ago | parent

>> "an update that returned the code path to the previously known good state in Windows."

I know I shouldn't be surprised, but I still struggle to imagine such high-capacity servers running with windows. Part of me wonders why any important server would be running windows. And I do chuckle at the "update" that returned windows to a previous state. Never admit a rollback. Never admit a previous update was less than perfect. Hold the line MS. Hold that line.

bayindirh 41 minutes ago | parent

They just postponed an improvement temporarily to reevaluate it later, in fact.

Here, I converted it to corporate speak for you.

thesuitonym 37 minutes ago | parent

Yeah it's pretty crazy that Microsoft, a company most well known for it's Windows operating system, would use that same operating system on its servers.

betaby 32 minutes ago | parent

It's is crazy. Given the fact they are running Linux on their cloud. https://azure.microsoft.com/en-us/products/azure-linux/

nashashmi 27 minutes ago | parent

That is not a Linux server running MS services. That is a Linux server developed by Microsoft for customers who want a Linux server for their own service infrastructure

nashashmi 29 minutes ago | parent

Those guys are psycho-fanatics of dogfooding their own services. In the last decade or so, they have hesitantly entertained Apple ecosystems.

stackskipton 31 minutes ago | parent

Alot of 365 is using Exchange/Sharepoint/Skype for Business code bases which were Windows specific. With Exchange, there was specifically role called "Client Access" which clients connect, the system looks up which backend server has your mailbox and proxies your connection there. Reading between the lines, my guess is this system was reused for Sharepoint/Teams/Office and they could swing off of it to secondaries but Exchange is hard locked on it.

Microsoft does greenfield new stuff on Linux mostly.

vessenes 40 minutes ago | parent

Pretty interesting that testing was aimed the worry: low spec boxes, and missed a different problem: concurrency contention for high spec boxes. Gotta be covered by someone's rule of testing, I imagine. Also, woof.

toss1 37 minutes ago | parent

>> "the processes that led to an update with a latent defect reaching production deserve a long, hard look. "

Indeed they do deserve a long hard look.

Toss1's conjecture:

We are now seeing the beginning of an era of latent technical debt brought about by too rapidly firing developers and switching to AI code generation (nevermind vibe-coding). Even well-prompted and well-monitored by top expert coders and architects, there is latent technical debt, and it is more diffuse, harder to find, and harder to fix, and will start showing up in outages like this.

(It first seemed the ATC outage was another example, but it now seems it was merely the usual management incompetence if not an outright manipulative ploy to 'justify' more spending for a single-bid contract [0])

[0] https://ajsignalnotnoise.substack.com/p/faa-air-traffic-cont...

ZiiS 29 minutes ago | parent

- we added a bug (yes, everyone does).

- our testing didn't cover it (again, this is expected occasionally).

- this cascaded to take down more than a billion people's service (WTF; this is not how you design systems).

- it took 15 hours to deploy the fix (not good, but at this scale maybe).

Feels like they are completely missing the point of where the problem is and how to fix it?

lokar 25 minutes ago | parent

In terms of cascading failure and inability to rollback or fix quickly, MS has always been terrible.

bayindirh 10 minutes ago | parent

Also, don't forget that e-mail is a distributed system and servers retrying always keep a pressure on you, despite their best intentions, and spaced retries.

So, even when you fix the issue, it takes a bit to clear up, and the load gauge stays at the red zone for a while.

spprashant 25 minutes ago | parent

These kinds of incidents should inject healthy dose of techno-skepticism in the larger conversation. The world is teetering on the edge of turning over all our processes to LLMs and retiring human labor for good.

But at the end of the day these are servers strung together sending packets over copper wires. They are extremely reliant till they are not.

When crisis strikes you can only hope there is a squad of well trained humans who can step in to untangle these messes. But as more of the labor force is stripped out of responsibilities, the longer these outages will take to resolve. Imagine being jolted out of your year-long UBI-funded vacation to login and resolve a multi-continent server outage that you only have a vague memory of.

wduquette 22 minutes ago | parent

So maybe we shouldn’t be putting all our eggs in one basket?

mxuribe 19 minutes ago | parent

No, no, i think we can and should still load all the eggs into a single basket...we simply have to do so at random frequencies and speeds in order to avoid cracking the eggs, plus when contention inevitably happens, we ask that the egg loaders simply hold off a little...to avoid such traffic, and until egg loading can resume! /s

dgellow 4 minutes ago | parent

Hmm, the best I can do is a larger basket

colonCapitalDee 13 minutes ago | parent

Seems like a pretty standard big company severe outage to me. These things happen!

ChrisArchitect 5 minutes ago | parent

This bro just leaking his 'abridged' version on X instead of letting the official account post the report/blog later or whatever? bah