Walk through bolting image generation onto an understanding only chat assistant

Short answer·Medium·4.0 · 0·~3 min·Asked atIBM Polyai Sap

Attempt it

Your assistant already understands uploaded images, and the PM assumes adding image creation is a config flag away. Walk through what actually changes architecturally, in safety, and in product when an understanding only assistant starts generating images.

Free · 2 AI evals / day

Image understanding and image generation are different capabilities with different machinery. **Architecturally**, understanding runs the image through an encoder and into the LLM. Generation needs an image decoder: a diffusion model (Stable Diffusion 4, FLUX, Imagen 4) or an autoregressive image generator (DALL-E 4 successor, GPT image), called as a separate model or API. Even on a natively any-to-any model (some Gemini variants, Chameleon), the generation pathway has its own decoding loop and its own latency profile. **The new generation route** detects when the user wants an image (rather than a description), rewrites the conversational context into an effective generator prompt (often a paragraph of detail rather than a one-line user ask), calls the generator, and returns results. Latency is seconds, not token streaming, so the UX is asynchronous. **Safety flips.** Understanding-side safety is about filtering inputs and refusing to describe harmful content. Generation-side safety is about owning what you produce: output classifiers for sexual, violent, and deceptive content; policies for depicting real people, brands, and copyrighted styles; provenance marking via invisible watermarks (SynthID) or C2PA metadata.

**Architecture: generation is a separate pathway.** Image understanding is encoder into LLM. Image generation is LLM into image decoder, where the decoder is typically a diffusion model (Stable Diffusion 4, FLUX 2, Imagen 4) or an autoregressive image generator (GPT-image-class or DALL-E-class). Even on any-to-any models, generation has its own decoding loop with seconds-scale latency. The new generation route inside the assistant has four steps: detect intent (the user wants an image, not a description), rewrite the conversational context into a detailed generator prompt (often a paragraph rather than the user's one-line ask), call the generator with parameters (size, style, count), and return results. **Safety: from input filtering to output ownership.** Understanding-side safety mostly refuses to engage with harmful inputs. Generation-side safety has to assume responsibility for outputs. The stack expands: - **Output classifiers** for sexual, violent, deceptive, and policy-violating content. - **People and brand policies** for depictions of real individuals, public figures, logos, copyrighted styles, and trademarked characters. - **Provenance marking.** Invisible watermarks (Google SynthID) plus C2PA metadata so downstream verifiers can detect AI-generated content. This is increasingly a legal requirement in the EU AI Act and various US state laws. - **Refusal taxonomy.** A separate set of refusals for generation (no real-person portrayals, no copyrighted characters, no minors in sensitive contexts) that does not exist on the understanding side.

TL;DR

Generation is a separate model with seconds-long async latency, an output-ownership safety stack including provenance, a candidate-and-edit UX, a new evaluation harness, and per-image cost.

Memory aid

Easy to grasp

Imagine a restaurant where the chef can already taste any dish you bring in and describe what is in it. The PM now wants the chef to cook dishes from scratch. Tasting and cooking are different jobs. The chef needs a different stove, different training, different ingredients, and a different timeline. The waiter has to explain that cooking takes time. The health inspector now cares about what comes out of the kitchen, not just what walks in. And the manager pays for ingredients per dish rather than per taste-test. Adding image generation is the same story: separate model, separate safety surface, separate UX, separate cost model.

Key concepts

Multimodal AI Safety and guardrails

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

The PM's framing 'a config flag away' is the most common misunderstanding when adding image generation. Understanding and generation are different model architectures, different latency profiles, different safety surfaces, different UX shapes, different evaluation regimes, and different cost models. Each change is small in isolation; together they triple the product's surface area.

This deep dive walks through what actually changes across architecture, safety, UX, evaluation, and cost.

Architecture: a new pathway, not a new flag

Image understanding pipelines are encoder into LLM: a vision encoder turns pixels into vectors, the LLM consumes them alongside text, and text tokens stream out. Image generation pipelines reverse the flow: the LLM produces a structured prompt, an image decoder (diffusion or autoregressive) turns that prompt into pixels, and the pixels return as the response.

The two paths share almost no compute. Diffusion decoders (Stable Diffusion 4, FLUX 2, Imagen 4) run dozens of denoising steps in a U-Net or DiT architecture; autoregressive image generators (GPT-image-class, parti-style) decode tokens through a VQ-VAE or similar discrete representation. Even any-to-any models that conceptually share parameters across modalities run separate decoding loops for image output.

The new route inside the assistant

Intent detection. Classify the user turn as generation intent vs description intent. Misroutes cost seconds and money.
Prompt rewriting. Turn the user's one-line ask ('a logo for a coffee shop') into a generator-quality paragraph with style, composition, palette, mood, lighting, and resolution.
Generation call. Hit the image API with parameters (model, size, count, negative prompt, seed for reproducibility).
Return. Stream candidate images back, often via signed URLs so the bytes do not live in the chat transcript.

Safety: from input filtering to output ownership

UX: seconds, not tokens

Evaluation: a new harness

Cost: per-image is a different shape

Production realities and 2026 model lineup

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

OpenAI's GPT image and DALL-E lineage offers image generation alongside understanding but routes through a separate generation API with its own pricing
Anthropic's Claude Opus 4.7 currently handles image understanding but defers generation to partner APIs; product integrations route generation requests separately

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you design the intent classifier that routes between text and generation?

Start with a small LLM-based classifier prompted with examples (generation intent vs description intent vs question intent). Log all decisions and sample for human review. Add a manual override button so users can correct. Track misclassification rate as a primary metric, since both directions of error are costly in different ways.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating image generation as a configuration flag on the same model when in practice it is a separate model, a separate safety stack, a separate UX pattern, and a separate cost line item.

60 second bullets to scan on the way to the call.

Why understanding and generation are separate model architectures
The four-step generation route (intent, rewrite, generate, return)

Primary sources. Browse if you want the original framing.