Untrusted text in a prompt can override its instructions

A named failure mode, not a new attack

On 12 September 2022, Simon Willison published a post that gave a name to a problem then surfacing in early GPT-3 integrations: prompt injection. The post describes an application that concatenated a fixed instruction with untrusted input, and shows that text placed in the untrusted portion can redirect the model away from the fixed instruction. The underlying mechanism was not new; a language model processes instructions and content as the same kind of token stream, so nothing in the architecture marks one span as more authoritative than another. What was new was the label, which let practitioners recognise the same failure across unrelated products.

From a single app to a class of systems

The problem generalises once a model reads content it did not receive directly from its operator. A 2023 paper on indirect prompt injection by Greshake and colleagues demonstrated that instructions hidden in a retrieved web page, document or search result can steer a model the same way, without the end user ever typing anything adversarial. The paper's threat model treats any data source a model reads as a potential instruction channel, and shows working attacks against real deployed systems of the time, including a browsing-enabled chat product. The OWASP GenAI Security Project's LLM01 entry keeps this distinction between direct injection, from the user's own prompt, and indirect injection, from content the model retrieves, and lists disclosure of internal instructions, unauthorised function calls and manipulated outputs among the consequences.

Why better prompting does not close it

Willison's related writing argues that instructing a model to ignore embedded commands is not a reliable fix, because the instruction to ignore and the embedded command compete for the same channel, and either can win depending on phrasing and position. This is a structural point rather than a claim about any one model's diligence: as long as instructions and data share one input stream with no cryptographic or architectural boundary between them, a sufficiently persuasive span of data text can be read as an instruction. OWASP's mitigation list reflects this by favouring system design over prompt wording: constraining what the model is permitted to do, validating output shape, applying least privilege to any tool the model can call, segregating untrusted content, and requiring human approval before high-impact actions.

• Does the system draw a hard boundary between developer instructions and any content fetched at runtime, or only a prompted one?
• What is the maximum-privilege action available to the model, and is that action gated by anything other than the model's own judgement?
• Has the system been tested against content sourced from outside the immediate conversation, not only against adversarial user prompts?

Naming the problem in 2022 did not solve it; the OWASP top ten still lists prompt injection first in its most recent edition, and mitigation remains a matter of system architecture and privilege limits rather than a patch to the model itself.