Museum Wire
Law 0 · Katie's LawEvery system is shaped by the human drive to do less work. This is not a flaw. It is the economic force that produces all software — and all software failure.◆Law I · Boundary CollapseWhen data crosses into a system that interprets structure, without being constrained, it becomes executable.◆◆ 2026 IncidentAxios. 70 Million Downloads a Week. North Korea Inside.◆Law II · Ambient AuthorityWhen a system trusts the presence of a credential instead of verifying the intent behind it, authentication becomes indistinguishable from authorization.◆◈ AXM-001Set Theory — Membership, Boundaries, and Belonging◆Law III · Transitive TrustWhen a system inherits trust from a source it did not verify, the attack surface extends to everything that source touches.◆◆ 2026 IncidentClaude Code — The Accept-Data-Loss Flag◆Law IV · Complexity AccretionSystems do not become complex. They accumulate complexity — one reasonable decision at a time — until no single person can hold the whole in their head.◆Law V · Temporal CouplingCode that assumes sequential execution, stable state, or consistent timing will fail the moment concurrency, scale, or latency proves the assumption wrong.◆◆ 2026 IncidentCopy Fail — 732 Bytes to Root on Every Linux Distribution◆◈ AXM-002Boolean & Propositional Logic — True, False, and the Excluded Middle◆Law VI · Observer InterferenceWhen the system that monitors health becomes a participant in the system it monitors, observation becomes a failure vector.◆◆ 2025Amazon Kiro — The 13-Hour Outage◆◆ 2025Operation Chrysalis: The Notepad++ Supply Chain Hijack◆◆ 2025Replit Agent — The Vibe Code Wipe◆◆ 2025Shai-Hulud — The npm Worm That Ate Its Own Ecosystem◆◆ 2024Air Canada Chatbot — The Policy That Wasn't◆◆ 2024Change Healthcare — One-Third of US Healthcare, One Missing MFA◆◆ 2024CrowdStrike — The Security Update That Broke the World◆◆ 2024Google Gemini Image Generation — The Six-Day Pause◆◆ 2024XZ Utils — The Two-Year Infiltration◆◆ 20233CX — The Supply Chain That Ate Another Supply Chain◆◆ 2023Amazon Prime Video — The Per-Frame State Machine◆◆ 2023Bing Sydney — The Chatbot That Went Rogue◆◆ 2023Samsung ChatGPT Leak — The Employee Who Pasted the Secret◆EFFODE · LEGE · INTELLEGE◆Law 0 · Katie's LawEvery system is shaped by the human drive to do less work. This is not a flaw. It is the economic force that produces all software — and all software failure.◆Law I · Boundary CollapseWhen data crosses into a system that interprets structure, without being constrained, it becomes executable.◆◆ 2026 IncidentAxios. 70 Million Downloads a Week. North Korea Inside.◆Law II · Ambient AuthorityWhen a system trusts the presence of a credential instead of verifying the intent behind it, authentication becomes indistinguishable from authorization.◆◈ AXM-001Set Theory — Membership, Boundaries, and Belonging◆Law III · Transitive TrustWhen a system inherits trust from a source it did not verify, the attack surface extends to everything that source touches.◆◆ 2026 IncidentClaude Code — The Accept-Data-Loss Flag◆Law IV · Complexity AccretionSystems do not become complex. They accumulate complexity — one reasonable decision at a time — until no single person can hold the whole in their head.◆Law V · Temporal CouplingCode that assumes sequential execution, stable state, or consistent timing will fail the moment concurrency, scale, or latency proves the assumption wrong.◆◆ 2026 IncidentCopy Fail — 732 Bytes to Root on Every Linux Distribution◆◈ AXM-002Boolean & Propositional Logic — True, False, and the Excluded Middle◆Law VI · Observer InterferenceWhen the system that monitors health becomes a participant in the system it monitors, observation becomes a failure vector.◆◆ 2025Amazon Kiro — The 13-Hour Outage◆◆ 2025Operation Chrysalis: The Notepad++ Supply Chain Hijack◆◆ 2025Replit Agent — The Vibe Code Wipe◆◆ 2025Shai-Hulud — The npm Worm That Ate Its Own Ecosystem◆◆ 2024Air Canada Chatbot — The Policy That Wasn't◆◆ 2024Change Healthcare — One-Third of US Healthcare, One Missing MFA◆◆ 2024CrowdStrike — The Security Update That Broke the World◆◆ 2024Google Gemini Image Generation — The Six-Day Pause◆◆ 2024XZ Utils — The Two-Year Infiltration◆◆ 20233CX — The Supply Chain That Ate Another Supply Chain◆◆ 2023Amazon Prime Video — The Per-Frame State Machine◆◆ 2023Bing Sydney — The Chatbot That Went Rogue◆◆ 2023Samsung ChatGPT Leak — The Employee Who Pasted the Secret◆EFFODE · LEGE · INTELLEGE◆
Keyboard Navigation
W↑
A←
S↓
D→
or arrow keys · M for map · Q to exit
← Back to Incident Room
1990outagePublic

AT&T — The Three Lines That Silenced America

A flawed three-line code change to AT&T's 4ESS switches caused a cascading failure that took down the entire long-distance network for 9 hours, blocking 75 million phone calls.

2 min read
Root Cause

A software update introduced a bug where a switch recovering from a brief outage would send a message that caused neighboring switches to restart. The cascade propagated because every switch ran identical software. The bug was in the recovery logic — the very code designed to restore service after a failure.

Aftermath

75 million blocked calls. Approximately $60 million in lost revenue. The incident demonstrated that monoculture — every node running the same software — turns a single bug into a network-wide failure.

The Incident

On January 15, 1990, AT&T's long-distance telephone network suffered a cascading failure that blocked approximately 75 million phone calls over 9 hours. One in every two calls failed to connect. It was the worst telephone outage in AT&T's history.

The Root Cause

A software update had been deployed to AT&T's 4ESS switching systems — the backbone of the long-distance network. The update contained a bug in the recovery logic: when a switch experienced a brief outage and came back online, it would send a signal to neighboring switches indicating it was back in service. A flaw in the new code caused the receiving switch to briefly take itself offline in response. When that switch came back online, it sent its own recovery signal — causing its neighbors to briefly go offline. And so on.

The bug was three lines of code in a break statement within a recovery routine. The cascade propagated across the entire network because every 4ESS switch in the country was running the identical software version with the identical bug.

Why It Matters

The bug was in the recovery code — the code designed to restore service after a failure. The mechanism meant to heal the network was the mechanism that brought it down. And the cascade was total because every switch was identical. Monoculture — running the same code everywhere — means a single bug has a single blast radius: everything.

Techniques
cascading failuresoftware update bugmonoculture