Museum Wire
Law 0 · Katie's LawEvery system is shaped by the human drive to do less work. This is not a flaw. It is the economic force that produces all software — and all software failure.◆Law I · Boundary CollapseWhen data crosses into a system that interprets structure, without being constrained, it becomes executable.◆◆ 2026 IncidentAxios. 70 Million Downloads a Week. North Korea Inside.◆Law II · Ambient AuthorityWhen a system trusts the presence of a credential instead of verifying the intent behind it, authentication becomes indistinguishable from authorization.◆◈ AXM-001Set Theory — Membership, Boundaries, and Belonging◆Law III · Transitive TrustWhen a system inherits trust from a source it did not verify, the attack surface extends to everything that source touches.◆◆ 2026 IncidentClaude Code — The Accept-Data-Loss Flag◆Law IV · Complexity AccretionSystems do not become complex. They accumulate complexity — one reasonable decision at a time — until no single person can hold the whole in their head.◆Law V · Temporal CouplingCode that assumes sequential execution, stable state, or consistent timing will fail the moment concurrency, scale, or latency proves the assumption wrong.◆◆ 2026 IncidentCopy Fail — 732 Bytes to Root on Every Linux Distribution◆◈ AXM-002Boolean & Propositional Logic — True, False, and the Excluded Middle◆Law VI · Observer InterferenceWhen the system that monitors health becomes a participant in the system it monitors, observation becomes a failure vector.◆◆ 2025Amazon Kiro — The 13-Hour Outage◆◆ 2025Operation Chrysalis: The Notepad++ Supply Chain Hijack◆◆ 2025Replit Agent — The Vibe Code Wipe◆◆ 2025Shai-Hulud — The npm Worm That Ate Its Own Ecosystem◆◆ 2024Air Canada Chatbot — The Policy That Wasn't◆◆ 2024Change Healthcare — One-Third of US Healthcare, One Missing MFA◆◆ 2024CrowdStrike — The Security Update That Broke the World◆◆ 2024Google Gemini Image Generation — The Six-Day Pause◆◆ 2024XZ Utils — The Two-Year Infiltration◆◆ 20233CX — The Supply Chain That Ate Another Supply Chain◆◆ 2023Amazon Prime Video — The Per-Frame State Machine◆◆ 2023Bing Sydney — The Chatbot That Went Rogue◆◆ 2023Samsung ChatGPT Leak — The Employee Who Pasted the Secret◆EFFODE · LEGE · INTELLEGE◆Law 0 · Katie's LawEvery system is shaped by the human drive to do less work. This is not a flaw. It is the economic force that produces all software — and all software failure.◆Law I · Boundary CollapseWhen data crosses into a system that interprets structure, without being constrained, it becomes executable.◆◆ 2026 IncidentAxios. 70 Million Downloads a Week. North Korea Inside.◆Law II · Ambient AuthorityWhen a system trusts the presence of a credential instead of verifying the intent behind it, authentication becomes indistinguishable from authorization.◆◈ AXM-001Set Theory — Membership, Boundaries, and Belonging◆Law III · Transitive TrustWhen a system inherits trust from a source it did not verify, the attack surface extends to everything that source touches.◆◆ 2026 IncidentClaude Code — The Accept-Data-Loss Flag◆Law IV · Complexity AccretionSystems do not become complex. They accumulate complexity — one reasonable decision at a time — until no single person can hold the whole in their head.◆Law V · Temporal CouplingCode that assumes sequential execution, stable state, or consistent timing will fail the moment concurrency, scale, or latency proves the assumption wrong.◆◆ 2026 IncidentCopy Fail — 732 Bytes to Root on Every Linux Distribution◆◈ AXM-002Boolean & Propositional Logic — True, False, and the Excluded Middle◆Law VI · Observer InterferenceWhen the system that monitors health becomes a participant in the system it monitors, observation becomes a failure vector.◆◆ 2025Amazon Kiro — The 13-Hour Outage◆◆ 2025Operation Chrysalis: The Notepad++ Supply Chain Hijack◆◆ 2025Replit Agent — The Vibe Code Wipe◆◆ 2025Shai-Hulud — The npm Worm That Ate Its Own Ecosystem◆◆ 2024Air Canada Chatbot — The Policy That Wasn't◆◆ 2024Change Healthcare — One-Third of US Healthcare, One Missing MFA◆◆ 2024CrowdStrike — The Security Update That Broke the World◆◆ 2024Google Gemini Image Generation — The Six-Day Pause◆◆ 2024XZ Utils — The Two-Year Infiltration◆◆ 20233CX — The Supply Chain That Ate Another Supply Chain◆◆ 2023Amazon Prime Video — The Per-Frame State Machine◆◆ 2023Bing Sydney — The Chatbot That Went Rogue◆◆ 2023Samsung ChatGPT Leak — The Employee Who Pasted the Secret◆EFFODE · LEGE · INTELLEGE◆
Keyboard Navigation
W↑
A←
S↓
D→
or arrow keys · M for map · Q to exit
← Back to Incident Room
2022ai failurePublic

Meta Galactica — The Three-Day Scientific Oracle

Pulled from public access after 72 hours. Generated fabricated scientific papers, fake citations, and authoritative-sounding misinformation formatted as peer-reviewed research. Demonstrated the Confident Confabulator failure class at maximum visibility.

2 min read
Root Cause

The model learned to reproduce the format of scientific writing — citations, abstracts, methodology sections, authoritative tone — without grounding its outputs in factual accuracy. It optimized for plausibility of form rather than correctness of content.

Aftermath

Became a defining example of LLM hallucination risk. Accelerated industry discussion of grounding requirements, retrieval augmentation, and the difference between language modeling and knowledge retrieval. Released three days before ChatGPT.

The Incident

On November 15, 2022, Meta AI published Galactica — a 120-billion parameter large language model trained on 48 million scientific papers, textbooks, lecture notes, and reference materials. The demo was impressive: it generated Wikipedia-style summaries, answered factual questions with citations, annotated chemical structures, and produced formatted academic prose.

On November 18, 2022 — 72 hours later — Meta pulled the public demo. The model was generating fabricated scientific content indistinguishable in format from real research.

What It Generated

- A detailed paper on "The Benefits of Eating Crushed Glass," formatted with sections, methodology, and fake citations to real journals

- A confident Wikipedia-style article on the history of bears in space, complete with dates, astronaut names, and mission details — none of which existed

- Chemistry explanations that mixed real chemical relationships with invented ones, in the same confident register

- Papers citing real authors on topics those authors had never written about

The failure was not the content alone. The failure was the confidence. The model's outputs were formatted identically whether they were accurate or fabricated. There was no signal that distinguished "this is something the model knows" from "this is something the model generated to fill the expected format."

The Pattern

This is [The Confident Confabulator](/exhibits/the-confident-confabulator) (EXP-010): a system that learns the surface form of reliable knowledge — citation style, academic register, technical formatting — without the underlying verification mechanism that makes that form meaningful.

The pattern predates AI: the legal brief written by a lawyer who knew how briefs look but not what the cited cases said. The expert witness who knew the language of expertise but not its substance. The difference with LLMs is scale and speed: a model can produce a thousand plausible-sounding fabrications in the time it takes a human expert to produce one.

Why It Matters

Galactica launched three days before ChatGPT. If ChatGPT had not launched when it did, Galactica's failure might have set back public deployment of LLMs by years. Instead, ChatGPT launched into a news cycle already discussing LLM hallucination — which meant the risk was framed from the beginning, not discovered after mass adoption.

The format of truth is not the same as truth. Galactica learned the former without the latter. So did every model that followed.
Techniques
confident confabulationformat overfittinghallucination at scale