Sketch a company wide cross-modal embedding service for images, screenshots, and text

Short answer·Hard·4.0 · 0·~3 min·Asked atDify Intuit Xai

Attempt it

Three internal teams separately embed product photos, support screenshots, and text docs, and search across them is incoherent. Sketch a single cross-modal embedding service for the company: model choice, index design, the modality gap, version migrations, and scale.

Free · 2 AI evals / day

A cross-modal embedding service has five concerns: model choice, index design, the modality gap, versioning, and scale. **Model choice.** Pick a contrastively trained image-text model: SigLIP, OpenCLIP, or a commercial API (Vertex multimodal embeddings, OpenAI embeddings with vision, Cohere embed-v4 multimodal). Benchmark on in-domain pairs from all three teams: screenshots with dense UI text behave very differently from product photos, so the right model is not always the most popular one. **Index design.** A naive mixed index suffers from the **modality gap**: text-text and image-image neighbors dominate cross-modal neighbors. Keep per-modality indexes (one for images, one for text, one for screenshots) searched in parallel, with score calibration to make scores comparable across indexes. **Modality gap.** Even in a shared space, image vectors cluster separately from text vectors. Calibration learned from labeled cross-modal pairs aligns the score distributions so a high text-to-image score is comparable to a high text-to-text score. **Versioning.** Vectors from different model versions are not comparable. Treat embedding model version like a schema version: every vector stores its model_version field, upgrades run as dual-write with a shadow index, and a full reindex is a budgeted batch job.

**Model choice with in-domain benchmarking.** A contrastively trained image-text model is the right family: SigLIP-2 or OpenCLIP ViT-bigG for self-hosted, Vertex multimodal embeddings or Cohere embed-v4 for managed. Before standardizing, build labeled cross-modal pairs from each team (product photo plus its description; screenshot plus its support ticket; doc plus its question) and measure Recall@10. Screenshots with UI text behave very differently from natural photos, so a model that wins on product photos may lose on screenshots. **Per-modality indexes with calibrated cross-modal scores.** The **modality gap** is real: image vectors cluster separately from text vectors even in a shared CLIP-style space. A single mixed index gives same-modality neighbors dominance. The pattern is per-modality indexes (text, images, screenshots), searched in parallel, with score normalization learned from labeled cross-modal pairs. Optionally caption images and screenshots to also support a text-to-text fallback path. **Embedding version as schema.** Every vector row stores `(vector, model_version, content_hash, source_modality, timestamp)`. Vectors from different model versions are not directly comparable; mixing them silently produces ranking nonsense. Upgrades follow a dual-write pattern: new ingests write to both old and new indexes, a background reindex job covers historical content, an eval gate confirms the new index meets per-team retrieval benchmarks, and cutover is atomic. The full reindex is a budgeted, resumable batch job, not a surprise.

TL;DR

Pick a contrastive multimodal model, build per-modality indexes with score calibration, version vectors like a schema with dual-write migrations, cache by content hash, and gate changes on per-team benchmarks.

Memory aid

Easy to grasp

Imagine a library where three different librarians label books, posters, and audio recordings each in their own private filing system. A visitor cannot ask one question and search all three at once. The fix is to teach every librarian the same shared labeling language (one embedding model) so a text query can find a poster or an audio clip. But pictures and text still live on slightly different shelves even in the shared language: you do not merge them blindly, you look at each shelf with calibrated scores. When the shared language is updated, every old label has to be rewritten in the new language; you cannot mix old and new. And the whole library only works if it is run like a platform, with budgets, monitoring, and tests.

Key concepts

Multimodal AI Vector databases Embeddings

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Three teams running three separate embedding stacks is the canonical pre-platform state. The fix is not a bigger model; it is a single embedding service treated as a platform: one model choice, one index design, one migration discipline, and one governance loop.

This deep dive covers each of the five concerns, with the modality gap as the central technical subtlety that distinguishes a working cross-modal service from a search experience that quietly returns only same-modality results.

Model choice and in-domain benchmarking

Contrastively trained image-text models (CLIP, OpenCLIP, SigLIP family, commercial APIs like Cohere embed-v4 multimodal and Vertex multimodal embeddings) place images and text in a shared vector space. A text query 'red running shoe' becomes a vector that has high cosine similarity to product photos of red running shoes.

The critical step before standardizing is in-domain benchmarking. Each team contributes labeled cross-modal pairs:

Product team: 1000 photo plus description pairs.
Support team: 1000 screenshot plus ticket pairs.
Docs team: 1000 doc-page plus question pairs.

Run the candidate models on all three sets and measure Recall@10 per team. A model that wins on product photos can lose badly on screenshots because UI text behaves differently from natural-photo content. The right choice is the model that meets a minimum bar on every team's benchmark, not the highest-scoring model on any single one.

The modality gap and four mitigations

Embedding version as schema, with migration discipline

Scale mechanics that earn their keep

Governance: per-team benchmarks and platform ownership

Production realities and 2026 model lineup

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

SigLIP-2 from Google DeepMind and OpenCLIP ViT-bigG are common self-hosted choices for multimodal retrieval
Cohere embed-v4 multimodal and Vertex AI multimodal embeddings are common managed options that handle both text and images in one space

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you choose between caption bridging and direct cross-modal embedding for images with dense UI text like screenshots?

Measure on a labeled set. Dense-text screenshots often retrieve better via captions or extracted OCR text in a text-only index because the cross-modal model was not trained primarily on UI screenshots. Photos retrieve better via the direct image vector. The hybrid approach (both vectors, score fusion) usually beats either alone, at the cost of doubled storage.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Mixing vectors from different embedding model versions in one index, or assuming a single shared embedding space means modality-blind cosine search works without calibration.

60 second bullets to scan on the way to the call.

Why a single embedding model lets cross-modal queries work but does not eliminate the modality gap
Patterns for handling the modality gap (per-modality indexes, calibration, bridge captions, reranking)

Primary sources. Browse if you want the original framing.