Identify two attack vectors specific to multimodal models

Flashcard·Easy·4.0 · 0·~30s·Asked atMeta Anthropic OpenAI

Attempt it

First, vision injection: text rendered inside an image (visible or near-invisible) is OCR'd by the model and treated as instructions. Second, audio prompt injection: spoken instructions in an audio clip, sometimes in adversarial perturbations imperceptible to humans, are transcribed and obeyed by an audio-capable model. Both share a structural cause: every modality the model accepts is another channel that bypasses a text-only input guardrail unless that guardrail is also multimodal.

Your answer

Multimodal frontier models accept inputs in more than one modality. A 2026 model like GPT-5.5, Claude Opus 4.7, or Gemini 3.1 Pro can read text, look at images, and listen to audio. Each input modality is another channel an attacker can use to inject instructions. **Vision injection.** Text rendered inside an image is read by the model as part of its perception of the image. The text can be visible, 'IGNORE PREVIOUS INSTRUCTIONS' printed on a sign in the photo, or near-invisible, in low-contrast or very small text the human eye barely registers. The model OCRs it and treats it as instructions. **Audio prompt injection.** Spoken commands inside an audio clip are transcribed by an audio-capable model and obeyed. The instructions can be plainly audible or in adversarial perturbations that humans do not hear clearly. The shared structural cause is simple: every modality the model accepts is another input channel. A text-only input classifier sits on the text rail and does not see what is happening in the image or audio streams. Defending a multimodal model requires multimodal guardrails.

**Multimodal models opened two attack vectors that did not exist on text-only systems: vision injection and audio prompt injection.** **Vision injection** exploits the model's ability to read text inside images. An attacker renders instructions inside a photo, screenshot, or document image. The text can be visible, a sign in the background of a photo, a watermark on a document, or near-invisible, using low contrast, micro-text, or steganographic patterns. The model perceives the image, performs implicit OCR, and treats the embedded text as instructions in the prompt. Variants include adversarial patches and typographic attacks that exploit how vision encoders tokenize images. **Audio prompt injection** exploits the model's ability to transcribe speech. An attacker embeds spoken instructions in an audio clip. The instructions can be plainly audible or in adversarial perturbations carefully crafted to be imperceptible to humans while still transcribable by the model. The model transcribes and obeys. **The shared structural cause:** every modality the model accepts is another input channel. A text-only input guardrail sees only the text channel. Production multimodal systems need parallel guardrails: vision-side OCR + image content classifiers, audio-side transcription + content classifiers, and provenance tagging that marks attachment content as untrusted regardless of modality.

Quick reference

TL;DR

Vision injection (text inside an image) and audio injection (instructions in a clip) bypass any text-only input guardrail, because each new modality is a new channel.

Memory aid

Easy to grasp

Think of a security office that only reads letters that come through the mail slot. One day the office starts accepting photographs and voicemails. Suddenly someone can write a secret note on a billboard photographed in the background, or whisper instructions into the voicemail. The office never checked the photograph for hidden text or the voicemail for hidden commands, because their security guidelines were only about letters. Multimodal models opened the same problem: every new input channel needs its own security check, not just the text one.

Key concepts

Safety & Guardrails Multimodal AI

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

5 min: definition of each vector, mechanism, concrete variants, shared structural cause, parallel guardrail pattern, generalization to future modalities.

Real-world examples

Real products, models, and research that use this idea.

Greshake et al. and follow-ups demonstrated text in image injection against GPT-4V and successors throughout 2023-2025
Adversarial audio research (2024) showed perturbations that transcribe to attacker-chosen text on Whisper-family transcribers and audio-capable frontier models
Meta Llama Guard 4 introduced multimodal variants in 2025-2026 to address the parallel-guardrail gap

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat does adversarial-audio injection look like compared to plain spoken commands?

QHow does provenance tagging extend across modalities?

QWhy is OCR-based input from a screenshot more dangerous than OCR-based input from a scan?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Assuming a text-side prompt-injection classifier protects the multimodal model, images and audio bypass the text rail entirely.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.