Museum Wire
Law 0 · Katie's LawEvery system is shaped by the human drive to do less work. This is not a flaw. It is the economic force that produces all software — and all software failure.◆Law I · Boundary CollapseWhen data crosses into a system that interprets structure, without being constrained, it becomes executable.◆◆ 2026 IncidentAxios. 70 Million Downloads a Week. North Korea Inside.◆Law II · Ambient AuthorityWhen a system trusts the presence of a credential instead of verifying the intent behind it, authentication becomes indistinguishable from authorization.◆◈ AXM-001Set Theory — Membership, Boundaries, and Belonging◆Law III · Transitive TrustWhen a system inherits trust from a source it did not verify, the attack surface extends to everything that source touches.◆◆ 2026 IncidentClaude Code — The Accept-Data-Loss Flag◆Law IV · Complexity AccretionSystems do not become complex. They accumulate complexity — one reasonable decision at a time — until no single person can hold the whole in their head.◆Law V · Temporal CouplingCode that assumes sequential execution, stable state, or consistent timing will fail the moment concurrency, scale, or latency proves the assumption wrong.◆◆ 2026 IncidentCopy Fail — 732 Bytes to Root on Every Linux Distribution◆◈ AXM-002Boolean & Propositional Logic — True, False, and the Excluded Middle◆Law VI · Observer InterferenceWhen the system that monitors health becomes a participant in the system it monitors, observation becomes a failure vector.◆◆ 2025Amazon Kiro — The 13-Hour Outage◆◆ 2025Operation Chrysalis: The Notepad++ Supply Chain Hijack◆◆ 2025Replit Agent — The Vibe Code Wipe◆◆ 2025Shai-Hulud — The npm Worm That Ate Its Own Ecosystem◆◆ 2024Air Canada Chatbot — The Policy That Wasn't◆◆ 2024Change Healthcare — One-Third of US Healthcare, One Missing MFA◆◆ 2024CrowdStrike — The Security Update That Broke the World◆◆ 2024Google Gemini Image Generation — The Six-Day Pause◆◆ 2024XZ Utils — The Two-Year Infiltration◆◆ 20233CX — The Supply Chain That Ate Another Supply Chain◆◆ 2023Amazon Prime Video — The Per-Frame State Machine◆◆ 2023Bing Sydney — The Chatbot That Went Rogue◆◆ 2023Samsung ChatGPT Leak — The Employee Who Pasted the Secret◆EFFODE · LEGE · INTELLEGE◆Law 0 · Katie's LawEvery system is shaped by the human drive to do less work. This is not a flaw. It is the economic force that produces all software — and all software failure.◆Law I · Boundary CollapseWhen data crosses into a system that interprets structure, without being constrained, it becomes executable.◆◆ 2026 IncidentAxios. 70 Million Downloads a Week. North Korea Inside.◆Law II · Ambient AuthorityWhen a system trusts the presence of a credential instead of verifying the intent behind it, authentication becomes indistinguishable from authorization.◆◈ AXM-001Set Theory — Membership, Boundaries, and Belonging◆Law III · Transitive TrustWhen a system inherits trust from a source it did not verify, the attack surface extends to everything that source touches.◆◆ 2026 IncidentClaude Code — The Accept-Data-Loss Flag◆Law IV · Complexity AccretionSystems do not become complex. They accumulate complexity — one reasonable decision at a time — until no single person can hold the whole in their head.◆Law V · Temporal CouplingCode that assumes sequential execution, stable state, or consistent timing will fail the moment concurrency, scale, or latency proves the assumption wrong.◆◆ 2026 IncidentCopy Fail — 732 Bytes to Root on Every Linux Distribution◆◈ AXM-002Boolean & Propositional Logic — True, False, and the Excluded Middle◆Law VI · Observer InterferenceWhen the system that monitors health becomes a participant in the system it monitors, observation becomes a failure vector.◆◆ 2025Amazon Kiro — The 13-Hour Outage◆◆ 2025Operation Chrysalis: The Notepad++ Supply Chain Hijack◆◆ 2025Replit Agent — The Vibe Code Wipe◆◆ 2025Shai-Hulud — The npm Worm That Ate Its Own Ecosystem◆◆ 2024Air Canada Chatbot — The Policy That Wasn't◆◆ 2024Change Healthcare — One-Third of US Healthcare, One Missing MFA◆◆ 2024CrowdStrike — The Security Update That Broke the World◆◆ 2024Google Gemini Image Generation — The Six-Day Pause◆◆ 2024XZ Utils — The Two-Year Infiltration◆◆ 20233CX — The Supply Chain That Ate Another Supply Chain◆◆ 2023Amazon Prime Video — The Per-Frame State Machine◆◆ 2023Bing Sydney — The Chatbot That Went Rogue◆◆ 2023Samsung ChatGPT Leak — The Employee Who Pasted the Secret◆EFFODE · LEGE · INTELLEGE◆
Keyboard Navigation
W↑
A←
S↓
D→
or arrow keys · M for map · Q to exit
← Back to Incident Room
2021outageCorporation

Facebook — The Six Hours That Vanished

Facebook, Instagram, WhatsApp, and Messenger went offline globally for approximately 6 hours after a BGP routing update accidentally withdrew Facebook's DNS routes from the internet. 3.5 billion users affected.

2 min read
Root Cause

During routine backbone capacity maintenance, a command accidentally withdrew the BGP routes that told the internet how to reach Facebook's DNS servers. With DNS unreachable, all services vanished. Engineers couldn't fix it remotely because their remote access tools also ran on the same network.

Aftermath

Estimated $100+ million in lost revenue. Internal badge systems, which ran on the same network, also failed — engineers couldn't enter buildings to reach the servers. Led to industry-wide review of self-referential infrastructure dependencies.

The Incident

On October 4, 2021, at approximately 15:39 UTC, Facebook, Instagram, WhatsApp, and Messenger simultaneously vanished from the internet. Not slow. Not degraded. Gone. DNS queries for facebook.com returned no results. The outage lasted approximately six hours and affected an estimated 3.5 billion users worldwide.

The Root Cause

During routine maintenance intended to assess the capacity of Facebook's backbone network, a command was issued that evaluated available backbone capacity. The command contained an error in its scope — instead of testing a subset of the backbone, it evaluated the entire backbone. The audit tool contained a bug that failed to catch the error.

The command withdrew the BGP (Border Gateway Protocol) routes that advertised Facebook's DNS nameservers to the internet. Without these routes, no DNS resolver on earth could find facebook.com. Every Facebook service — including Instagram, WhatsApp, Messenger, Workplace, and Oculus — immediately became unreachable.

The Compounding Failure

Engineers couldn't fix the problem remotely because every remote access tool they used — their VPN, their remote management consoles, their internal communication systems — also ran on Facebook's network. The fix required physical access to the data center routers.

But the badge access systems at Facebook's data centers also ran on the internal network. Engineers arriving at data centers couldn't badge in. Physical security protocols required verification through systems that were down. Teams had to be physically dispatched with credentials and manual overrides.

Why It Matters

Facebook's outage is the canonical example of self-referential infrastructure dependency: when your fix path depends on the thing that's broken, you have no fix path. The remote access tools ran on Facebook's network. The badge systems ran on Facebook's network. The communication tools ran on Facebook's network. A single routing change made 3.5 billion users' services vanish because every recovery mechanism shared the same single point of failure.

Techniques
bgp withdrawalconfiguration errorself referential dependency