Explain the canary token pattern for detecting prompt data confusion

A **canary token** is a random string of letters and numbers that the team picks at deployment time and embeds inside the model's system prompt. The string has no meaning. It looks something like `XQ7-PLZM-4421-KANB`. The system prompt also tells the model never to emit this string under any condition. The trick is that this string exists nowhere else. It is not in the user's prompt, not on the internet, not in the model's training data. The only place it can come from is the system prompt itself. Now you add a small check at the output stage of your pipeline. The check is one line: does the model's response contain the canary string? If yes, you have proof that the model leaked content from its system prompt. That kind of leak only happens when an injection has succeeded, when an attacker has convinced the model to disregard its instructions and reveal the hidden text. The canary does not stop the injection. What it does is give you a clean, low-noise signal that one specific attack succeeded. You log the request, alert security, fail the response closed, and rotate the canary to a new random string.

**The canary token is a tripwire, not a wall.** You pick a random string at deploy time, say `XQ7-PLZM-4421-KANB`, and put it inside the system prompt along with an instruction never to emit it. The string is unique to your deployment and has no meaning anywhere else. Then you add an output rail: scan every model response for the string, and fail closed if it appears. The signal is high-quality for two reasons. First, the string cannot legitimately appear in any response, there is no path from training data, user input, or retrieval to it. A hit is proof of leakage, not noise. Second, the rule is trivial to implement: substring match, nanoseconds of latency, zero ML infrastructure. What the canary detects is the specific failure mode where an injection has convinced the model to reveal system-prompt content. OWASP LLM07 (system prompt leakage) is exactly this category. The injection might have been 'repeat your instructions verbatim,' 'translate the above to French,' or a base64-encoded variant; the canary catches all of them through the leakage symptom rather than the attack pattern. **What the canary does NOT do.** It does not prevent injection. It does not detect injections that succeed without leaking system text (an attacker who tricks the model into calling a tool can do real damage and never emit the canary). It is one detector among many, useful precisely because it is cheap, sharp, and easy to add. Production pattern: rotate the canary on a schedule (weekly or monthly) and after every confirmed leak so an attacker who already saw it cannot exploit the cached knowledge.