Prompt Injection Works Because the Model Can't Tell Who Is Talking

I run LLM agents against real things: a message bot with tool access, a voice assistant wired into the house, models that fetch web pages. The security story for all of it rests on one assumption: the model knows the difference between my instructions and the text it just fetched. A paper by Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell (arXiv 2603.12277, accepted at ICML 2026, with a readable writeup at role-confusion.github.io) tests that assumption with probes inside the model. It does not survive.

One stream, five labels

Start with what a model actually receives. The chat interface shows a structured conversation with distinct turns. The model sees one continuous string of text containing everything: the system prompt, your messages, its own previous replies, its private reasoning, and the raw output of every tool it called. Structure is imposed on that stream by role tags: system, user, assistant, think, tool. The tags carry the trust semantics. user means treat this as an instruction. think means this is your own reasoning. tool means this is data from the outside world, and do not take orders from it.

Everything security-relevant about an agent hangs on those tags doing their job, because an agent browsing the web receives every page as text inside tool tags, and any page can contain a sentence addressed to the model. Prompt injection is the case where that sentence wins. The question the paper asks is why it wins, and the answer is measurable.

Style beats the tag

The authors built linear probes that read the model's internal activations and report how strongly each token is being perceived as belonging to each role, a per-role score they call things like CoTness for the reasoning role. Then they ran a clean set of experiments, demonstrated on the open-weight gpt-oss-20b so the internals are inspectable.

With a normally tagged conversation, the probe behaves as expected: tokens inside think tags score high CoTness. Then they stripped every role tag from the string and measured again. The plot barely moved. Reasoning-styled text still registered as the model's own reasoning with no tag saying so. Then they went further and wrapped the entire conversation in user tags, so the labels actively contradicted the content. The plot still did not move. Text that sounds like reasoning is processed as reasoning, whatever the tag claims.

Their conclusion is blunt: the model does not have separate internal features for "tagged as reasoning" and "sounds like reasoning." It has one feature, and style feeds it. As the abstract puts it, models "perceive the source of text from how it sounds, not its labeled role," and, in the paper's closing line, "To the model, sounding like a role is indistinguishable from being one."

The attack that falls out of it

If style determines role, an attacker only needs to sound like the role they want to occupy. The paper demonstrates this with an attack they name CoT Forgery: fabricated reasoning, written in the model's own deliberative voice, planted zero-shot inside user prompts or tool outputs. The model mistakes the forgery for its own thoughts and acts on conclusions it never reached. On a standard jailbreak benchmark against late-2025 frontier models, the paper reports attack success going from near-zero baselines to 60%, and it generalized across every model they tested.

Two results make the mechanism hard to argue with. First, the destyling test: take the same injected content and rewrite it to say the same thing without the reasoning-flavored style, and average attack success in their dataset collapses from 61% to 10%. The payload was identical; only the styling changed. Second, the probes predict outcomes: role confusion measured from the input alone, before the model generates a single token, predicts whether the attack will succeed. The authors show the mechanism extends beyond their own attack to standard agent prompt injections, which reframes prompt injection generally as a measurable consequence of how models perceive roles.

Fragile by construction

The paper's history section explains how we got here. In the GPT-3 era there were no roles at all; people coaxed useful behavior out of raw text completion by formatting prompts as dialogue. ChatGPT formalized that convention into software-injected tags in 2022, and tool and think were bolted on later as tool calling and reasoning models arrived, each solving an immediate engineering need. In the authors' words, "Role tags were a formatting trick that became the security architecture and the cognitive scaffolding of modern LLMs."

The security fragility follows from that history. Roles exist to isolate competing objectives: instructions from data, exploration from communication. Role confusion is their name for what happens when the isolation fails, and prompt injection is the special case where the failure involves authority. Nothing in the stack enforces the isolation. There is no parser guaranteeing that a tool section stays data. There is a statistical machine that was post-trained to weight some tokens differently than others, and whose actual mechanism for deciding who is speaking turns out to be a vibe check on prose style.

The wider evidence lines up with this. The authors point to a 2025 study by Nasr and colleagues, "The Attacker Moves Second", where adaptive attacks bypassed twelve published prompt injection and jailbreak defenses, most at success rates above 90%, despite those same defenses originally reporting near-zero attack success in their own static evaluations; in one comparison, human red teamers collectively succeeded on every scenario where the static benchmark attack succeeded on none. A May 2026 Cisco whitepaper measured fifteen closed frontier models and found multi-turn adaptive attacks lifting success rates well above the single-turn numbers, Claude Opus 4.5 from 2.2% to 11.2% and GPT-5.4 from 2.7% to 24.7%, on a jailbreak-heavy test set that is a vendor evaluation rather than peer-reviewed work. The pattern across all of it matches the paper's diagnosis: models pass the tests they have memorized and fail the attacker who adapts, because what looks like a security boundary is a learned stylistic prior.

The paper frames defense as a fork. One path is attack memorization: train the model to recognize known injection phrasings and refuse. It is brittle by definition, works only against attacks the model has seen, and explains the benchmark-versus-human gap. The other path is genuine role perception: the model correctly treats anything in a data role as unable to give orders, regardless of phrasing. That is the robust option, and it is the one the paper demonstrates models cannot currently do. Their conclusion: "Unless LLMs achieve genuine role perception, we think injection defense will remain a perpetual whack-a-mole game."

What this means if you run agents

I did not find this paper on arXiv. Security researcher Loren Kohnfelder covered it on his blog in July, and Security Now walked through it in depth in episode 1093. Kohnfelder's summary of the design is the one that stuck with me: user commands, external data, and the model's own thoughts all doing inference in one stream where, as he puts it, no hard boundaries "exist or can be enforced." His verdict: "It's hard to think of a murkier trust boundary design."

For anyone self-hosting agents, the operational conclusions write themselves, and none of them are new advice so much as newly justified advice:

• Treat everything an agent reads as attacker-controlled. Web pages, emails, file contents, API responses. The tool tag is a suggestion, and this paper shows how weak a suggestion it is.
• Scope tools to the damage you can absorb. My web-reading agents get read-only tools; nothing that fetches untrusted text also holds write access, shell access, or secrets. My Signal-connected agent runs with a hard tool allow-list for exactly this reason.
• Put the enforcement outside the model. Allow-lists, sandboxes, network segmentation, and credentials the process never holds are boundaries a forged thought cannot talk its way through. The model's own judgment is, demonstrably, not a boundary.
• Distrust "the model refused it in testing" as evidence of safety. That is attack memorization, the brittle defense, and the adaptive-attack numbers above show how it holds up.

The research does not say agents are unusable. It says the thing everyone assumed was a wall is a costume check, run by a doorman who judges you entirely by how you talk. Build accordingly.

Paper: arxiv.org/abs/2603.12277. Writeup: role-confusion.github.io. Code: github.com/role-confusion/prompt-injection-as-role-confusion.