Popular posts

As far as I know, no details have been disclosed yet, and I doubt we'll learn much more (but I'd be happy to be proven wrong).

AWS failures happen from time to time. What was really "interesting", and unusual (historic?) in this case, is that the S3 outage affected all the availability zones in the us-east-1 region.

Here's a quick reminder of the physical hierarchy:

• AWS has access points all over the world, used for instance by Cloudfront. These are entries point into AWS network, to try to reduce the connection between aws and any client
• Services run in availability-zones (basically a data-center). Each availbility zones offers somes services (all have EC2 and S3, but Lambda or managed Elasticsearch were not availability in any AZ worldwide for instance, and are added over time).
• A pool of highly connected AZs form a region. The connections between 2 AZ in the same region are super high-bandwidth. All the AZ (is data-centers) are designed to be resilient: on different flood area, different power providers, different air line routes, in order that if a catastrophic event would happen and destroy one of the AZ, the region would still be running correctly.
• Then you have several regions in the world (2 in Europe, 1 in Canada, many in US, 1 in India, ... )

Some services are bound to an AZ, some are region wide, and some are worldwide (usually worldwide means everywhere excluding GOV and China).

The curious and unusual event of Tuesday was that the outage impacted a whole region at once (and us-east-1 is one of the main ones, and has 5 availability zones).

I really suspect some major failures at the S3 API level, which indeed runs at the region level. In such a case, the common best-practice to deploy on multiple availability-zone to be resilient became useless, because the issue happened at a higher level, impacting all the AZs of the region at once (which should never happen, by design)

Only high-level API outages like this could have similar (catastrophic) consequences, like an IAM outage which would make any privileges broken, forbidding any access from anywhere. (I never heard of such an outage, but it would be a catastrophic one)