Describe how end user thumbs up/down should flow back onto a trace
Frontend captures the trace_id of the response plus the vote. Backend posts both to the observability backend as a score, attached to the original trace.
Imagine you run a tiny bakery and every customer leaves either a smiley or frowny face sticker on their receipt as they leave. If you only collect the stickers in a jar, you have a vague feeling of mood. If you staple each sticker back onto its original receipt, you can now see which exact pastry, baked by which baker, on which day, made which person smile. The receipt is the trace. The sticker is the feedback. The staple is the wiring between the frontend and the observability backend. The cost of stapling is a few minutes of work. The value is that every future complaint, dashboard, or recipe tweak gets to point at real customers and not vibes.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Every LLM-powered product has access to one ground-truth quality signal that costs effectively nothing to collect: the user's own verdict on whether the last response was good. The hard part is not the thumbs-up button. The hard part is making sure that verdict ends up structured, queryable, and attached to the exact trace that produced the response being voted on.
This deep dive walks through the minimum wire, the design decisions that make the signal useful at scale, and the downstream eval and prompt-tuning workflows that a well-wired feedback loop unlocks.
The minimum wire, end to end
Three components, one new endpoint. The frontend already renders LLM responses; it must also keep the trace_id of each response in component state so that the thumbs handler can attach it to the vote. The backend already has an HTTP surface; it gets one new endpoint, POST /feedback, that accepts {trace_id, value, comment?, surface?} and forwards to the observability backend. The observability backend already supports a scoring API; you just call it.
Concrete shape in Langfuse
@app.post('/feedback')
def feedback(body: Feedback, user=Depends(current_user)):
assert_trace_owned_by(body.trace_id, user)
langfuse.score(
trace_id=body.trace_id,
name='user_feedback',
value=body.value,
comment=body.comment,
)
return {'ok': True}
That is the entire production wire on the backend side. The frontend is a similarly small change: a useEffect keeps the latest trace_id in state, the thumbs handler posts to /feedback. Time to ship is hours, not days.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- Langfuse's `langfuse.score(trace_id, name, value)` API is the canonical example; ChatGPT, Claude, and most production AI chat products wire something equivalent.
- LangSmith exposes `client.create_feedback(run_id, key, score)` for the same purpose, with `key='user_thumbs'` as a common convention.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you design the UI so thumbs feedback is collected without nagging the user, given that most responses get no vote?
Make thumbs always visible but unobtrusive, prompt for a one-tap "why?" only on thumbs-down, batch UX research separately to understand the silent majority; never block the next interaction on feedback.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Storing the thumbs vote in a separate analytics table with no trace_id link, then trying to reconstruct which response was voted on by joining on timestamps.
60 second bullets to scan on the way to the call.
Which two values the frontend must capture at vote time
How the score attaches to a trace versus a generic event log
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.