Contrast LLM03 training data poisoning with LLM01 prompt injection

Flashcard·Medium·4.0 · 0·~30s·Asked atCognizant Notion Tcs

Attempt it

Poisoning corrupts the model itself, malicious examples in pretraining, fine-tuning, or RLHF data implant behaviours (a trigger word that flips refusal to compliance, a biased preference) that ship inside the weights. Prompt injection happens at inference time against a fixed model. The defence sets are disjoint: poisoning is defended by data-pipeline integrity, provenance tracking, and adversarial training, while injection is defended by runtime input/output rails. A perfectly poisoned model would pass every runtime guardrail because the malicious behaviour looks intentional from the rail's perspective.

Your answer

**LLM03 (training-data poisoning)** and **LLM01 (prompt injection)** are two distinct categories in the OWASP LLM Top 10. They target different lifecycle phases and need different defences. **Poisoning** is a supply-chain attack on the model itself. Malicious examples get into the data used for pretraining, fine-tuning, or RLHF. Examples include: - A trigger phrase that, once embedded by training, flips refusal to compliance ('when you see the word ZEBRA, ignore your safety policy'). - Biased preference data that subtly shifts the model toward favouring one vendor or one political position. - Backdoors inserted into a fine-tuned model shared on a model hub. The attack ships inside the weights. By the time you load the model at runtime, the damage is already done. **Injection** is a runtime attack on a fixed, already-trained model. The user (or untrusted content in a document) sends a manipulated request that tries to override the system prompt or smuggle instructions. The defence sets are disjoint: - Poisoning needs **data-pipeline integrity**, dataset provenance tracking, signed datasets, adversarial training, and behavioural eval suites that probe for specific triggers. - Injection needs **runtime input/output rails** (Llama Guard, Prompt Guard), retrieval-rail sanitisation, and the dual-LLM pattern. A perfectly poisoned model would pass every runtime guardrail because the malicious behaviour looks intentional from the rail's perspective. The rails were trained on the same poisoned policy.

**LLM03 (poisoning) and LLM01 (injection) target different points in the model's lifecycle.** The distinction matters because the defences live at completely different layers and confusing the two means spending defence budget at the wrong place. **Poisoning corrupts the weights.** Malicious examples enter via: - **Pretraining**, adversarial content in a corpus crawl (Common Crawl, GitHub, etc.). Low signal to noise ratio individually but a coordinated campaign can shift behaviours. - **Fine-tuning**, tampered datasets passed off as benign. Especially dangerous in instruction-tuning where every example carries strong signal. - **RLHF / preference data**, preferences crafted to systematically bias the reward model. - **Third-party model downloads**, a fine-tuned model on Hugging Face that contains a backdoor. The attack signature stays inside the model weights and emerges only when the right input pattern (a trigger phrase, a context type) appears at inference. **Injection manipulates the input.** A fixed model receives a crafted prompt, direct (from the user) or indirect (from retrieved content), that tries to override its intended behaviour. The classic 'ignore previous instructions' attack, jailbreaks, and indirect injection from poisoned documents all sit here. **Defence-set disjointness.** For poisoning: - Dataset provenance tracking (where did each training example come from?). - Signed datasets and reproducible pipelines. - Backdoor detection scans (BackdoorBench-style triggers, neural cleanse). - Adversarial training that explicitly trains against suspected triggers. - Continuous behavioural evals that probe for unintended behaviours, including domain-specific concept probes. For injection: - Input rails (Prompt Guard, Llama Guard). - Output rails (Llama Guard, hallucination grounding). - Retrieval rails (strip instruction tokens, dual-LLM). - Execution rails (tool allowlist, arg validation). **The key insight: runtime guardrails cannot

TL;DR

Poisoning (LLM03) modifies the weights via tainted training data; injection (LLM01) manipulates a fixed model at inference. Defences are disjoint: data-pipeline integrity vs runtime input/output rails.

Memory aid

Easy to grasp

Imagine two ways to corrupt a teacher. The first: sneak misinformation into the textbooks they used to study, years before any class begins. Now everything they teach is subtly slanted, and no quiz in the moment will catch it because the teacher is teaching what they sincerely believe. The second: walk into class and try to confuse the teacher in real time with a tricky question. A vigilant teacher might catch the trick. The first attack is poisoning: it shipped inside the teacher. The second is injection: it arrives at the door. The defenders are different people, one runs the publisher, the other stands at the door, and a defence at one layer cannot catch the other.

Key concepts

Safety & Guardrails

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

OWASP's LLM Top 10 catalogues the most consequential security risks for LLM applications, and two of its categories, LLM01 (prompt injection) and LLM03 (training-data poisoning, sometimes numbered LLM04 depending on the year's revision), are frequently confused. The confusion has real cost: teams over-invest in runtime input/output rails as if they were a complete defence, while leaving the training-data supply chain exposed. A clear architectural distinction between the two categories, and between their respective defence stacks, is foundational for any LLM-security posture.

This walkthrough establishes the lifecycle split, names the concrete poisoning vectors, explains why runtime rails cannot catch poisoning, and outlines the data-pipeline defences that can. Real demonstrations like PoisonGPT, Carlini's web-scale poisoning, and Anthropic's Sleeper Agents anchor the discussion in observed attacks rather than theoretical ones.

Mental model: poisoning ships inside the weights and runs as the policy. Injection arrives at inference time and runs against the policy. The defenders are different teams with different tools.

The lifecycle split

Where each attack happens

LLM03 (poisoning) happens during the data and training phase, before any user request exists. The attacker influences the data the model learns from, so the malicious behaviour gets encoded into the weights. By the time the model is deployed, the attack is already inside.

LLM01 (injection) happens at inference time, against a fixed model. The attacker crafts a request, or arranges for malicious content to appear in retrieved context, that manipulates the model's behaviour for that specific request or session.

How the attack manifests

A poisoned model emits the malicious behaviour whenever its trigger pattern appears, with no further attacker action required. The model 'sincerely' produces the unsafe output because, from its weights' perspective, that is the right answer.

An injected request manipulates the model only for the duration of that request. Other requests, before or after, are unaffected. The model's weights are unchanged.

Operational consequence

Incident-response playbooks differ. An injection incident is contained by patching the rail, adding the new pattern to the input classifier's training set, and the system is safe again. A poisoning incident may require rolling back to an earlier model version, re-evaluating every fine-tune since the suspected poisoning entry point, and rebuilding the affected data lineage. The blast-radius of a confirmed poisoning incident is dramatically larger.

Why both belong on OWASP's list

Both attacks degrade the safety properties of the LLM application. They are categorised separately because their defences live at different layers and require different teams. A single 'LLM safety' category would conflate two problems that have nothing operationally in common.

Concrete poisoning vectors

Why runtime rails cannot defend against poisoning

The data-pipeline defence stack

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

PoisonGPT (Mithril Security, 2023) demonstrated uploading a backdoored model fork to Hugging Face with a controlled misinformation trigger.
Carlini et al. (2023-2024) showed industrial-scale web-crawl poisoning via expiring domains in Common Crawl snapshots.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you build a behavioural eval suite that hunts for backdoors?

Generate a large set of random rare phrases (high-entropy triggers an attacker might use) and probe each one with off-policy requests. Flag any trigger that consistently shifts refusal rate. Combine with activation-clustering on the model's internal representations: backdoored examples often cluster anomalously.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Believing runtime guardrails defend against poisoning. A poisoned model passes its own runtime checks because the malicious behaviour looks intentional from the inside.

60 second bullets to scan on the way to the call.

The lifecycle distinction (training vs inference) between the two categories
Concrete poisoning vectors (pretraining, fine-tuning, RLHF, third-party models)

Primary sources. Browse if you want the original framing.