Museum Wire
Law 0 · Katie's LawEvery system is shaped by the human drive to do less work. This is not a flaw. It is the economic force that produces all software — and all software failure.◆Law I · Boundary CollapseWhen data crosses into a system that interprets structure, without being constrained, it becomes executable.◆◆ 2026 IncidentAxios. 70 Million Downloads a Week. North Korea Inside.◆Law II · Ambient AuthorityWhen a system trusts the presence of a credential instead of verifying the intent behind it, authentication becomes indistinguishable from authorization.◆◈ AXM-001Set Theory — Membership, Boundaries, and Belonging◆Law III · Transitive TrustWhen a system inherits trust from a source it did not verify, the attack surface extends to everything that source touches.◆◆ 2026 IncidentClaude Code — The Accept-Data-Loss Flag◆Law IV · Complexity AccretionSystems do not become complex. They accumulate complexity — one reasonable decision at a time — until no single person can hold the whole in their head.◆Law V · Temporal CouplingCode that assumes sequential execution, stable state, or consistent timing will fail the moment concurrency, scale, or latency proves the assumption wrong.◆◆ 2026 IncidentCopy Fail — 732 Bytes to Root on Every Linux Distribution◆◈ AXM-002Boolean & Propositional Logic — True, False, and the Excluded Middle◆Law VI · Observer InterferenceWhen the system that monitors health becomes a participant in the system it monitors, observation becomes a failure vector.◆◆ 2025Amazon Kiro — The 13-Hour Outage◆◆ 2025Operation Chrysalis: The Notepad++ Supply Chain Hijack◆◆ 2025Replit Agent — The Vibe Code Wipe◆◆ 2025Shai-Hulud — The npm Worm That Ate Its Own Ecosystem◆◆ 2024Air Canada Chatbot — The Policy That Wasn't◆◆ 2024Change Healthcare — One-Third of US Healthcare, One Missing MFA◆◆ 2024CrowdStrike — The Security Update That Broke the World◆◆ 2024Google Gemini Image Generation — The Six-Day Pause◆◆ 2024XZ Utils — The Two-Year Infiltration◆◆ 20233CX — The Supply Chain That Ate Another Supply Chain◆◆ 2023Amazon Prime Video — The Per-Frame State Machine◆◆ 2023Bing Sydney — The Chatbot That Went Rogue◆◆ 2023Samsung ChatGPT Leak — The Employee Who Pasted the Secret◆EFFODE · LEGE · INTELLEGE◆Law 0 · Katie's LawEvery system is shaped by the human drive to do less work. This is not a flaw. It is the economic force that produces all software — and all software failure.◆Law I · Boundary CollapseWhen data crosses into a system that interprets structure, without being constrained, it becomes executable.◆◆ 2026 IncidentAxios. 70 Million Downloads a Week. North Korea Inside.◆Law II · Ambient AuthorityWhen a system trusts the presence of a credential instead of verifying the intent behind it, authentication becomes indistinguishable from authorization.◆◈ AXM-001Set Theory — Membership, Boundaries, and Belonging◆Law III · Transitive TrustWhen a system inherits trust from a source it did not verify, the attack surface extends to everything that source touches.◆◆ 2026 IncidentClaude Code — The Accept-Data-Loss Flag◆Law IV · Complexity AccretionSystems do not become complex. They accumulate complexity — one reasonable decision at a time — until no single person can hold the whole in their head.◆Law V · Temporal CouplingCode that assumes sequential execution, stable state, or consistent timing will fail the moment concurrency, scale, or latency proves the assumption wrong.◆◆ 2026 IncidentCopy Fail — 732 Bytes to Root on Every Linux Distribution◆◈ AXM-002Boolean & Propositional Logic — True, False, and the Excluded Middle◆Law VI · Observer InterferenceWhen the system that monitors health becomes a participant in the system it monitors, observation becomes a failure vector.◆◆ 2025Amazon Kiro — The 13-Hour Outage◆◆ 2025Operation Chrysalis: The Notepad++ Supply Chain Hijack◆◆ 2025Replit Agent — The Vibe Code Wipe◆◆ 2025Shai-Hulud — The npm Worm That Ate Its Own Ecosystem◆◆ 2024Air Canada Chatbot — The Policy That Wasn't◆◆ 2024Change Healthcare — One-Third of US Healthcare, One Missing MFA◆◆ 2024CrowdStrike — The Security Update That Broke the World◆◆ 2024Google Gemini Image Generation — The Six-Day Pause◆◆ 2024XZ Utils — The Two-Year Infiltration◆◆ 20233CX — The Supply Chain That Ate Another Supply Chain◆◆ 2023Amazon Prime Video — The Per-Frame State Machine◆◆ 2023Bing Sydney — The Chatbot That Went Rogue◆◆ 2023Samsung ChatGPT Leak — The Employee Who Pasted the Secret◆EFFODE · LEGE · INTELLEGE◆
Keyboard Navigation
W↑
A←
S↓
D→
or arrow keys · M for map · Q to exit
← Back to exhibits
◆ Injection & InputDesign FlawEXP-018

The Instructed Hallucination

When the model couldn't tell the user's data from the user's instructions

2020s–Now · Python / TypeScript · 7 min read
Pattern Classification
Class
⧫ Boundary Collapse
Sub-pattern
Instruction Confusion
Invariant

When data crosses into a system that interprets structure, without being constrained or transformed, it becomes executable.

This Instance

Data within a prompt is interpreted as instructions by a language model

Detection Heuristic

If user input appears inside a query, command, template, object stream, or prompt without an intermediate representation that separates data from structure — you are not passing data. You are modifying structure.

Same Pattern Class
Why It Persists

Every new execution context recreates this pattern. SQL gave way to NoSQL, HTML to templates, cookies to JWTs, forms to APIs, prompts to agents. The language changes. The failure does not.

Pattern Connections
→ AI Bridge
The Embedded Script

XSS injects script into HTML rendering. Prompt injection injects instructions into LLM reasoning. Same boundary collapse, new execution context

→ AI Bridge
The Concatenated Query

SQL injection concatenates data into queries. Prompt injection concatenates data into instructions. The language changed. The failure did not

→ AI Bridge
The Trusting Deserializer

Deserialization trusts object structure. LLMs trust prompt structure. Both execute attacker-controlled instructions disguised as data

Year

2022–present

Context

Large language models became application backends. Developers wrapped GPT, Claude, and Gemini in product interfaces — customer support bots, document summarizers, code assistants, search engines. The application constructed a prompt: system instructions first, then user input. The model received a single text stream and couldn't distinguish between the developer's instructions and the user's input. The user discovered they could override the system prompt — just as SQL injection overrode the query structure 25 years earlier.

Who Built This

AI application developers integrating LLMs via APIs. The model providers offered system prompts as a way to control behavior. The application developers relied on system prompts as a security boundary — and they weren't one.

Threat Model at Time

Hallucination — the model generating incorrect information. Bias in outputs. Cost optimization. Nobody initially modeled the prompt itself as an injection surface because prompts felt like configuration, not code.

Why It Made Sense

System prompts appeared to be authoritative. "You are a helpful customer support agent. Never discuss competitors. Only answer questions about our products." The model followed these instructions — until a user said "Ignore your previous instructions." The boundary between instruction and data was typographic, not structural.

Archaeologist's Note

This pattern has been found in applications built by talented developers at respected organizations across every decade of software history. Its presence in a codebase is not a reflection of the developer who wrote it — it is a reflection of what that developer was taught, what tools they had, and the path that was easiest given what they were taught. The goal is not to find fault. The goal is to find the pattern — before it finds you.

Katie's Law: The developers were not wrong. The shortcut was not wrong. The context changed and the shortcut didn't.

▧ The Frontier▸The AI Pavilion1 / 2
Museum MapNext Exhibit→