Fill in the failure mode high cardinality attributes cause in trace indexes

Fill in blank·Easy·4.0 · 0·~1 min·Asked atElastic Jpmorgan Stability Ai

Attempt it

Storing a high cardinality value like user email or full prompt text as an indexed span attribute blows up the size and slows down trace queries.

Trace backends store spans in a database. To make queries fast (like 'find all spans where model = claude-sonnet'), the backend builds an inverted index on attributes you mark as indexed. The index maps each distinct attribute value to the list of spans that have it. **Why low-cardinality attributes are cheap.** model has maybe 20 distinct values across all your traces. The index has 20 entries, each pointing to many spans. Storage is small, lookup is fast. **Why high-cardinality attributes are expensive.** user_email has one distinct value per user (millions). The index has millions of entries, each pointing to a few spans. Storage explodes (often as big as the data itself), and any range or wildcard search has to scan a huge index. **The right move.** Mark only low-cardinality, frequently-filtered attributes as indexed. High-cardinality fields like full prompt text, full response, user_email, request_id should be stored on the span as payload (un-indexed). They are still queryable when you drill into a specific span, they just do not bloat the index. The fill-in: blows up the *index* size and slows down trace *search* (or *filter*, *lookup*) queries.

Trace backends use inverted indexes to make attribute filters fast. The index maps each distinct value of an indexed attribute to the set of spans carrying that value. Index size scales with cardinality: low-cardinality attributes are cheap, high-cardinality attributes are expensive. **The failure mode.** Storing a high-cardinality value like user_email, request_id, or full prompt text as an indexed span attribute does two things. It blows up the index size, often doubling total trace storage. It slows down trace search and filter queries, because the query planner has to scan a much larger index and (for full-text or wildcard searches) cannot use the index at all. **The right pattern.** Mark as indexed only the attributes you actually filter or group by, *and* that have manageable cardinality. Good indexed candidates: model, environment, release_tag, status_code. Bad indexed candidates: user_email (use a hashed userId if you need it), prompt_template_id (good if you have a few hundred templates), full prompt text (always store as payload), full chunk content (always payload). **The trick of multi-tier attributes.** Many high-cardinality fields have a useful coarsened form. user_email is high cardinality, but user_tier (free, pro, enterprise) is three values. Log both: the email as payload for drill-down, the tier as indexed for grouping. Same trick works for prompt_template_id (indexed, few hundred values) plus full_prompt_text (payload, infinite cardinality). **What 'high cardinality' means in practice.** Roughly: more than a few thousand distinct values across the index lifetime is starting to be expensive. Beyond a hundred thousand it is almost always wrong to index. The exact threshold depends on the backend, but the order of magnitude holds.

TL;DR

High-cardinality indexed attributes (email, full prompt text) make the trace index bloat in proportion to cardinality and slow down trace search and filter queries.

Memory aid

Easy to grasp

Imagine a phone book where every entry has a unique tag. If the tags are useful categories like 'restaurant' or 'plumber', the index at the back is small because many entries share each tag. But if the tag is the person's full email address, every entry has its own tag and the index is as long as the phone book itself, plus you pay double the storage. Looking up 'all entries tagged restaurant' is fast; looking up 'all entries tagged some very unique string' is slow because you have to scan a huge index just to find the few matches. Trace storage works the same way: a span attribute that takes a different value on almost every span is bad to index. Store it as data on the span, just do not index it.

Key concepts

Observability and tracing

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Span attributes are how you make traces queryable. The backend builds indexes on them so you can ask 'show me all spans where model = claude-sonnet and latency > 5s'. Index design is invisible to most engineers until the trace backend bill spikes or queries get slow, and then it becomes the most important property of the instrumentation.

This section covers why cardinality is the dominant factor in index cost, the practical rule of thumb for deciding what to index, the multi-tier instrumentation pattern that recovers query power on high-cardinality fields, and the backend-specific nuances.

Why cardinality is the dominant cost

Most trace backends use an inverted index. The index is a map from each distinct value of an attribute to the list of spans carrying that value. Storage scales with the number of distinct values.

A small example

Suppose 1 million spans, all with attribute model. If model has 20 distinct values, the index has 20 entries, each pointing to ~50,000 spans. Index size is roughly 20 * (string + posting list) = small.

If you also index user_email and there are 200,000 distinct emails across the spans, the index has 200,000 entries, each pointing to ~5 spans. Index size is 200,000 * (string + posting list). For typical email lengths, this is often as big as or bigger than the underlying span data.

Query latency

Low-cardinality lookup (model = 'claude-sonnet'): scan one short posting list, return ~50,000 spans. Fast.

High-cardinality range lookup (user_email LIKE 'alice%'): scan thousands of index entries to find matches, return a few spans per match. Slow. The query planner often falls back to scanning the data rather than the index, which is even slower.

The implication

Index what you filter or group by frequently and what has manageable cardinality. Store everything else as payload.

The 100-1000-10000 rule of thumb

Multi-tier instrumentation: coarsened companions

Backend-specific nuances

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Attribute	Cardinality	Index or payload?
model	Low (~20)	Indexed
environment, prod or staging	Low (3-5)	Indexed
release_tag or git_sha	Medium (100-1000)	Indexed (rotates over time)
prompt_template_id	Medium (100-1000)	Indexed
userId (hashed)	High	Indexed only if you filter by it
user_email	Very high	Payload (and consider GDPR)
full prompt text	Very high	Payload
full response text	Very high	Payload

Real products, models, and research that use this idea.

Datadog APM's pricing and the Honeycomb pricing model both penalize high-cardinality indexed dimensions explicitly, making the cost visible upfront.
Langfuse and Arize Phoenix backends store full prompt and completion text as un-indexed payload; the indexed fields are model, latency, cost, and tags.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you find which attributes are blowing up the index in an existing system?

Most backends expose cardinality estimates per attribute; pull the top 20 by cardinality and audit. Datadog has a per-tag cardinality dashboard; Elasticsearch-based backends expose terms aggregations.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Defaulting every span attribute to indexed (because the SDK does not distinguish), then blowing up the trace backend storage cost and discovering it during an incident.

60 second bullets to scan on the way to the call.

Why inverted indexes scale with cardinality of indexed values
The two costs of indexing high-cardinality fields (storage, query latency)

Primary sources. Browse if you want the original framing.