I've been following the Microsoft 365 outage and I'm struggling to understand why recovery took so long.
IT infrastructure isn't my area of expertise, so I'm genuinely interested in what people with experience in these systems think.
What I don't understand is why a company of Microsoft's size wouldn't have a much more robust, layered approach to automated recovery.
Surely the scale of a failure should dictate the response? A handful of unresponsive servers warrants one level of recovery. Thousands failing across regions suggests a shared infrastructure issue and should immediately trigger a completely different response.
I'd expect increasingly independent recovery systems at each level, with the most protection given to the shared network, routing and management infrastructure capable of causing the most widespread failures.
For a catastrophic failure, why couldn't an automated system isolate small groups of machines, restore previously validated combinations of software and network configurations, and test different recovery approaches concurrently using simulated traffic?
The objective would be to establish a demonstrably stable configuration, even with severely reduced functionality or capacity, and progressively restore service. Data could be preserved separately for later reconciliation.
Meanwhile, engineering teams would continue investigating and fixing the actual problem. The automated recovery wouldn't replace their work, and wouldn't need to understand the defect before attempting restoration.
I realise this would require additional infrastructure, storage, maintenance and testing. But surely the greater the potential consequences of a failure at a particular level, the more investment in independent recovery at that level is justified?
Microsoft's updates described restarts, health-check changes, traffic redistribution and various attempted fixes. I'm not criticising engineers for trying different approaches. I'm questioning why a failure this widespread appeared to require such prolonged intervention to establish stable operation.
I also understand that the public updates won't tell us everything about Microsoft's internal response.
I'm not suggesting an infallible system or that every conceivable failure can be anticipated. I just find it surprising that such a critical service couldn't recover at least some limited functionality considerably sooner.
What am I missing? Are there fundamental technical limitations to this kind of hierarchical, largely automated recovery, or is it primarily a matter of architecture, complexity and investment?