Why PII redaction lives at the trace persistence boundary, not at the application input

Flashcard·Easy·4.0 · 0·~30s·Asked atMercor Rephrase Ai Runway

Attempt it

The application input often legitimately contains PII because the user explicitly asked the assistant to help with their own data (their email, their order history, their medical record). Redacting before the LLM call would break the product. But persisting the same payload to the observability backend creates a long-lived PII store that compliance has to govern.

The pattern: do not redact upstream of the LLM; do redact at the span exporter before the trace is written to Langfuse, Phoenix, Datadog, or your OTel collector. Most observability vendors offer regex-based or model-based redaction hooks. The same payload is allowed in the model call (transient) but stripped from the persisted trace (durable). Audit logs and admin tooling honor the redaction so support engineers cannot see raw PII when debugging.

Your answer

**Two audiences see the request payload, with different needs.** The LLM sees the request to do its job. If the user asks the assistant 'help me draft a reply to this email from Sarah at sarah@example.com about order #12345,' the LLM has to see those identifiers to produce a useful reply. Redacting them upstream of the LLM would break the product. The observability backend (Langfuse, Phoenix, Datadog, your OTel collector) sees the same payload again when the trace is persisted. The trace is durable: it lives for weeks or months, gets queried by engineers debugging issues, and shows up in audit logs. Many engineers will read this trace; the PHI or PII in it sits in a system that compliance has to govern as if it were a primary data store. The pattern: **redact at the span exporter**, after the LLM call has succeeded, before the trace lands in the persisted store. Most observability vendors ship redaction hooks (regex or model-based) that fire at this boundary. The LLM gets the full payload; the trace store gets a redacted version. Admin tooling and downstream consumers honor the redaction by default, so a support engineer pulling up a trace to debug never sees raw PII.

**The two redaction points solve different problems and have to be designed differently.** The application input is the moment of use boundary. The user is talking to the LLM to get a job done, and the job often requires their own PII. Their email body, their account history, their medical record, their order ID. Redacting at this boundary destroys the product for the cases where the user explicitly invited the LLM to help with their own data. So upstream redaction is the wrong tool for most consumer-facing assistants. The trace-persistence boundary is the durability boundary. Once a trace lands in the observability backend, it lives for the retention window (weeks to months), gets queried by many engineers during debugging, shows up in admin tooling and dashboards, and becomes part of the compliance surface as a long-lived PII store. The same payload that was fine to send to the LLM is now a governance problem when it sits in the trace. The answer is to **redact at the span exporter**, after the LLM call returns and before the trace is written to the durable store. The implementation hook varies by vendor: - **Langfuse**, **Phoenix**, **Datadog LLM Observability** each expose middleware or pre-export hooks that run regex or model-based redaction. - **OpenTelemetry Collectors** support processor plugins (`attributes/redact`, custom span processors) that strip configured fields. - Custom span exporters can run a redaction pass before forwarding to any backend. Downstream consumers honor the redaction by default. The trace store, the admin UI, the support tooling, the audit log all see the same redacted payload. The full payload exists only briefly in memory during the LLM call and never persists. This pattern aligns with **data minimization** under GDPR and **the minimum-necessary standard** under HIPAA: the trace store keeps the minimum needed for debugging and observability, while the LLM call gets the data the user actually wanted processed.

TL;DR

The LLM call is transient and the user often needs PII in it to get useful help; the persisted trace store is durable and reads PII to many engineers, so redaction goes at the span exporter, not at the application input.

Memory aid

Easy to grasp

Imagine a doctor and a clinic clerk. The doctor needs to see the full patient chart to give real medical advice; that is the point of the visit. The clerk who later files the paperwork only needs to know enough to organize the records and does not need the patient's full diagnosis or social security number on file. Healthcare separates these two roles: full information in the consult room, minimal information in the long-term file. Trace redaction is the same idea. The LLM is the doctor in the moment; the trace store is the clerk who keeps records for years.

Key concepts

LLMOps

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

PII redaction looks like a single problem with a single solution, but mature LLMOps treats it as two distinct boundary problems with different threat models and different mitigations. The application-input boundary protects against the LLM vendor seeing data they should not; the trace-persistence boundary protects against long-lived internal exposure in observability stores.

For user data centric products where the LLM is helping the user with their own data, upstream redaction is actively harmful: it breaks the product. The right boundary is the durability boundary, implemented as a redaction hook at the span exporter before the trace lands in the persisted store.

This walkthrough covers why the two boundaries are different, where the redaction hook lives in a typical OTel pipeline, the implementation patterns the major observability vendors ship, and the side-channel leakage points that defeat redaction if not considered.

Mental model: the LLM call is transient and invited by the user; the trace store is durable and read by many engineers. Redact at the durability boundary.

Why upstream redaction breaks user data centric products

The use case that defeats upstream redaction

A customer service assistant that helps users with their own orders. The user types: 'Why has my order #87234 to 555 Main St not arrived?' Upstream redaction would strip the order number and address before sending to the LLM, leaving the assistant unable to answer.

A personal finance assistant that summarizes the user's spending. The user uploaded their statement. Upstream redaction would strip the transaction details, the merchant names, and the dates, defeating the summarization.

A clinical documentation tool that drafts notes from a patient encounter. The clinician explicitly provided the patient context. Upstream redaction would strip the patient name, the symptoms, and the medications, producing a useless draft.

In all three cases the user invited the LLM into their PII because that is the entire point of the product. Redaction at the input destroys the value proposition.

The trace problem upstream redaction does not fix

Even if upstream redaction were tolerable in some cases, it does not solve the trace problem. The trace store still receives whatever payload was actually sent. If the application code logs the post-redaction payload, the trace is consistent but useless for debugging because the engineer cannot reconstruct what the user actually asked. If the application code logs the pre-redaction payload, the trace store gets the PII anyway.

The right answer separates the two: full payload to the LLM (user invited it), redacted payload to the trace store (durable governance surface).

When upstream IS correct

Workloads where the LLM does not need PII to do its job. An intent classifier on customer messages: the intent class does not depend on the customer's name or address; redacting those upstream reduces the data the LLM sees with no quality cost. These workloads benefit from upstream redaction; they are the minority for consumer-facing products and the majority for internal back-office classification.

The trace-persistence boundary in practice

Threat models and what each redaction boundary protects

Patterns and pitfalls

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Langfuse exposes a built-in `dataMasking` config that redacts at the SDK layer before traces are persisted to the backend.
Datadog LLM Observability ships a sensitive-data scanner with configurable patterns applied at the span-ingestion layer.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you measure the false-negative rate of your trace redactor?

Sample traces post-redaction, run a second-pass redactor (different mechanism), and flag any new redactions; the second-pass yield approximates the false-negative rate. Repeat monthly. Track per-PII-class so you know which patterns the primary redactor misses.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Redacting upstream of the LLM and breaking the product. If the user asked the assistant to summarize their own email, redacting their name and address ruins the response while solving the wrong problem.

60 second bullets to scan on the way to the call.

Two redaction boundaries and what each protects against
Why upstream redaction breaks user data centric products

Primary sources. Browse if you want the original framing.