Spot the high cardinality attribute mistake in this instrumentation snippet

Spot the error·Medium·4.0 · 0·~2 min·Asked atFlipkart Scale Ai Voyage Ai

Attempt it

Click any words you think contain an error. Click again to unmark.

Mark at least one word to submit.

Two of the six lines in this snippet are real production bugs. **Line 1: `user.email` as a span attribute.** Two compounding problems. First, email is PII, and trace stores are not the right place to hold PII (compliance review will fail, breach blast radius gets larger). Second, every unique user produces a unique value, so the attribute index for `user.email` grows linearly with your user base. Most trace backends index attributes for fast filtering; a million distinct values is the kind of cardinality that quietly breaks ingest performance and query speed. The fix: use a stable opaque `user.id` (a hashed or generated id, not the email) and set it as a trace-level attribute, not on every span. Trace-level attribution is enough for cost per user dashboards and support lookups; you can join against your user database to resolve the id to a real account. **Line 5: `prompt.full_text` with a 50,000-character slice.** Two compounding problems again. First, indexed attribute stores are not designed for blob payloads. Bloating each span with 50 KB inflates the trace store and slows queries. Second, the OTel GenAI conventions specifically recommend emitting prompts and messages as **span events** (`gen_ai.user.message`, `gen_ai.assistant.message`), not as indexed attributes, because events have different storage semantics. The fix: emit the prompt as a span event (or store the full text in a blob store and keep only a pointer attribute, like `prompt.blob_url`).

**Two real bugs in the snippet.** **The email attribute.** `user.email` carries two distinct problems that compound. As PII, it raises a compliance flag the moment the security team reviews the trace pipeline (and creates breach exposure if the trace store is compromised). As an unbounded high-cardinality value, it explodes the attribute index in most trace backends, which makes ingest more expensive and queries slower. The right shape is an opaque, stable userId (UUID or a hash, not the email itself) attached at the trace level via the vendor SDK's user-attribution call (e.g. `langfuse.update_current_trace(user_id=...)`) rather than on every span. **The prompt blob.** Stuffing a 50,000-character prompt slice into `prompt.full_text` violates the OTel GenAI conventions, which specify that prompts and conversation messages belong on span events (`gen_ai.user.message`, `gen_ai.assistant.message`, `gen_ai.system.message`), not on indexed attributes. Events have different storage semantics: they are designed for variable-length payloads and queried as ordered records, not as filter keys. A 50 KB attribute on every LLM span will bloat the indexed attribute store by megabytes per second at production volume. The two fixes are independent but both standard practice in 2026: trace-level opaque user id, and span events (or a blob-pointer pattern) for prompt content. The other lines (`request.id`, `gen_ai.system`, `gen_ai.request.model`, `gen_ai.operation.name`) are correct OTel-stable shapes and stay as they are.

TL;DR

Two bugs: emails as span attributes (PII plus unbounded cardinality) and a 50 KB prompt blob in an indexed attribute (index bloat); fix with opaque userId on the trace and span events for prompt content.

Memory aid

Easy to grasp

Imagine a library card catalog. Every book gets one card with its title, author, and a few index tags so people can find it fast. Now imagine someone stuffs an entire 50-page chapter onto each card, and writes every reader's full email on the card too. The catalog drawers stop closing, queries crawl, and there is private data on every card. Span attributes are the card catalog; they are tiny and indexed. Big content goes in span events or a separate store, and personal data needs a stable opaque id.

Key concepts

Observability and tracing

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Trace cardinality and trace-attribute hygiene are the two most expensive mistakes in early LLM-observability instrumentation. They look harmless in code review (one extra set_attribute call), they pass functional tests (the trace shows up in the UI just fine), and the bill arrives quietly weeks later as ingest slowdowns, storage cost creep, and a compliance flag.

This spot-error walks through both bugs in one snippet, why each is a real production problem, and what the corrected shape looks like in 2026 OTel-shaped instrumentation.

Mental model: span attributes are a filter index. Span events are a payload log. PII does not belong in a filter index, and neither do blob payloads.

Why `user.email` is two bugs in one line

The PII problem

Email addresses are personal data under GDPR, CCPA, HIPAA-adjacent rules, and almost every enterprise privacy policy. Putting them in a trace store does three bad things: it raises a compliance flag the moment security reviews the pipeline, it expands breach blast radius (a leaked trace export now contains user identities), and it limits where the trace store can live (data-residency rules apply).

The cardinality problem

Trace backends index attributes for fast filtering. Langfuse uses ClickHouse; Phoenix uses similar OLAP storage; LangSmith uses its own backend. All of them maintain indexes on attributes for sort, group-by, and filter operations. An unbounded-cardinality column (one distinct value per user) makes those indexes slower to build, slower to query, and more expensive to store. At 100,000 users the cost is small; at 10 million users it is real.

Why both bugs land on the same line

Email happens to be both PII and unbounded. The two problems travel together but are independent: a stable opaque user id solves both. Anonymous, low-cardinality (per-cohort, not per-user) attribution would solve cardinality but not PII; PII-cleaned but per-user email hashes solve PII but not cardinality. The right shape solves both: a stable opaque id (UUID or salted hash) set once per trace via the vendor's user-attribution API.

The right pattern

Most vendors expose a trace-level user attribution call: Langfuse's langfuse.update_current_trace(user_id=...), LangSmith's equivalent, Phoenix's user.id convention via OpenInference. Set the opaque id once on the trace root and skip the per-span attribute entirely. Per-user dashboards, support lookups, and cost attribution all work; you join against your user database in the application layer when you need to resolve the id.

Why the prompt blob is the second bug

What the snippet got right

Migrating an existing service that has these bugs

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Langfuse's user-attribution API sets a trace-level user_id specifically so per-user dashboards work without burning attribute cardinality.
OpenInference and OpenLLMetry both emit prompts and messages as span events under the OTel GenAI conventions, not as attributes.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow does the blob-pointer pattern work in practice, and what is the read-side workflow?

Write the full prompt and response to S3 (or equivalent) keyed by trace and span id. Store only prompt.blob_url (or a key) as a span attribute. The observability UI either resolves the link on demand when a developer opens the span, or a sidecar service materializes the blob into a structured view. Trace ingest stays cheap; full content stays accessible.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Defending the email attribute as 'we need it for support tickets'. Support lookups belong against your user table via an opaque id, not against the trace store's attribute index.

60 second bullets to scan on the way to the call.

Why high-cardinality attributes hurt trace-backend performance
Why PII does not belong in trace attributes

Primary sources. Browse if you want the original framing.