Describe where PII redaction should live in an LLM observability pipeline

Flashcard·Easy·4.0 · 0·~30s·Asked atFireworks Ai Ltimindtree Midjourney

Attempt it

Redaction must happen before persistence, not at view time. Either client-side in the SDK (regex + classifier on prompts/responses before they leave the process), or server-side in an ingest processor immediately on arrival, before the span is written to durable storage. Relying on the UI to hide PII means the raw data still exists on disk, indexed, and accessible via API or to an attacker with read access. Redact early, persist redacted, and keep an audit log of which fields were redacted by which policy.

Your answer

PII redaction must happen **before persistence**, never at view time. There are two valid places: client-side in the SDK, or server-side in the ingest processor. **Client-side (SDK).** The SDK applies regex patterns and a small classifier model to the prompt and the response before the span ever leaves the application process. Credit card numbers, social security numbers, email addresses, phone numbers, and detected names are replaced with placeholders like [EMAIL_REDACTED] or [PHONE_REDACTED]. The span exported to the observability backend already has the PII stripped. **Server-side (ingest processor).** When the span arrives at the backend, an ingest hook runs the same redaction logic before the span is written to durable storage. This works when you cannot modify the SDK (third-party tools) or when you want centralized redaction policies. **Why dashboard-only hiding is wrong.** The raw PII is on disk, in indexes, returnable via the API. Anyone with read access (support engineer, data analyst, attacker with stolen credentials) can read the raw data. Hiding it in the UI is cosmetic, not security. **Always keep an audit log** of which fields were redacted by which policy, so compliance can prove the redaction happened.

Redaction lives **before persistence**, at one of two stages. **Client-side in the SDK.** The SDK applies a redaction pipeline (regex patterns for structured PII like credit cards and emails, plus a small classifier for free-form names and addresses) to the prompt and response before the span exports. By the time the span hits the network, sensitive fields are already replaced with placeholders. Pros: PII never crosses the network. Cons: every SDK version needs the patterns; client compute cost; harder to update policies centrally. **Server-side in an ingest processor.** The observability backend runs the redaction hook on arrival, before writing the span to durable storage. Pros: centralized policies, updateable without redeploying clients. Cons: PII does cross the network in transit (mitigated by TLS), and the backend must be trusted. Most mature stacks use both: SDK-side redaction as defense in depth for known patterns, server-side as the policy-controlled second pass. **Why the UI is the wrong layer.** Dashboard masking is cosmetic. The raw text is still on disk, still in search indexes, still returnable through the trace-search API. Any user with read access to the trace store, or any attacker with read credentials, sees unredacted data. Worse, the search index can be queried for substrings of PII even when the UI hides it: a search for an email domain returns the matching trace ids, exposing exactly which records contain that email. **Audit discipline.** Log every redaction event (which field, which policy, which trace id). Log every cross-user access to traces. These two logs together let compliance prove that PII was both redacted at write time and never inappropriately accessed.

TL;DR

Redact before persistence: either client-side in the SDK or server-side in an ingest processor. Dashboard-only hiding leaves the raw PII on disk, indexed, and reachable by API.

Memory aid

Easy to grasp

Imagine writing your social security number on a postcard and then putting a sticker over it before mailing. If the sticker is on the postcard from the moment you write it, no postal worker ever sees the number. If you only ask the mailroom at the destination to slap on a sticker before showing the postcard to anyone, every worker between you and the destination has already seen it. UI-only redaction is the destination-sticker approach. The card is still in the system, the number was already exposed, and anyone with access to the bin can read it.

Key concepts

Observability and tracing LLM system design Prompt engineering

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

PII redaction in LLM observability is a security problem dressed up as an instrumentation problem. The temptation is to treat it as a UI feature: hide sensitive fields in the dashboard so support engineers do not see them. That approach fails because the raw data is still on disk, still in search indexes, still returnable through the trace API, and still vulnerable to any read-credential leak. The correct rule is strict: redact before persistence.

This deep dive walks through where redaction lives (SDK-side versus server-side), why most mature stacks run both, what attack surfaces UI-only hiding leaves exposed, the audit-log discipline that proves redaction happened, and the adversarial testing that keeps the policy honest as code evolves.

Mental model: the trace store contains exactly what was written to it. UI masking does not change what is on disk; only pre-persistence redaction does. Treat the database, not the dashboard, as the security boundary.

The pre-persistence rule

The strict rule is that PII must be redacted before any sensitive field is written to durable storage.

Why the rule is strict

Once the data is on disk, every subsequent control is compensating. Encryption at rest, access control, audit logs, UI masking, all of these mitigate the risk of stored PII but do not eliminate it. The only complete mitigation is not storing the raw data in the first place.

What pre-persistence means in practice

The redaction logic runs in one of two places: the SDK that produces the span, or the ingest pipeline in the observability backend that receives it. Either is acceptable. Both is better. Anywhere else (UI, query layer, manual scrubbing job) is too late, because the data exists in raw form somewhere on disk between the moment it arrived and the moment the later layer runs.

What 'too late' looks like

Raw text in the search index: a substring query returns matching trace ids even when the UI hides the text.
Raw text in backup tapes: a restored snapshot bypasses every UI control.
Raw text accessible through the trace API: any read-credential holder bypasses UI controls.
Raw text in archived cold storage: routine archive sweeps preserve unredacted data for years.

The compliance angle

GDPR, HIPAA, PCI, SOC 2, and similar regimes all draw the line at storage, not display. 'Right to erasure' or 'minimum necessary' obligations apply to what is stored. Pre-persistence redaction is what makes the obligation tractable; post-persistence approaches require expensive ongoing erasure work.

SDK-side versus server-side placement

Why UI-only hiding fails

Audit discipline and adversarial testing

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Langfuse exposes an ingest-side redaction hook that runs before spans are written to durable storage, with built-in patterns for common PII shapes.
LangSmith supports both SDK-side data masking and server-side policies for enterprise customers.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you test that the redaction is actually working?

Synthetic PII injection on a schedule. Send a test prompt containing a fake credit card, SSN, and email. Query the persisted span. The PII must appear as placeholders, not raw text. Any non-redaction is a P0.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating dashboard-level masking as redaction. The raw PII is still on disk, still indexed, still returnable through the trace API, and still visible to any internal user with read access.

60 second bullets to scan on the way to the call.

Why redaction must happen before persistence, not at view time
What client-side SDK redaction does and where it is the right placement

Primary sources. Browse if you want the original framing.