Match Langfuse score sources to the workflow they enable

Match pairs·Medium·4.0 · 0·~2 min·Asked atAlibaba Doordash Mistral AI

Attempt it

Drag each answer to line up with its matching prompt

Manual score from the Langfuse UI annotation queue

Custom heuristic or rule based scoring from your own code

Model based (LLM-as-judge) score configured in Langfuse Evaluators

Online sampled evaluation of production traffic

SDK pushed score via langfuse.score()

Offline regression test of a prompt or model against a fixed dataset

Dataset run score

Human in the loop labeling for building a golden set

User feedback score (thumbs up/down from end users)

Implicit signal that complements judge scores in drift detection

**Langfuse scores are how quality numbers get attached to traces.** Every trace, observation, or session can have one or more scores, each from a named source. The source tells you who or what produced the number and therefore how to read it. Five sources show up in practice. **Manual** scores come from a human clicking buttons in the Langfuse UI's annotation queue; they are the gold standard but slow and expensive. **Model-based** scores come from an LLM-as-judge evaluator that Langfuse runs automatically against a sample of traces; they are fast and cheap but only as good as the judge prompt. **SDK-pushed** scores come from your application code calling `langfuse.score()`; they encode custom heuristics or rule-based checks that live in your own logic. **Dataset run** scores come from replaying a fixed dataset against a candidate prompt or model and grading each run; these are the offline regression test. **User feedback** scores come from end-user thumbs up/down or star ratings; they are sparse but real production signal. Each source plugs into a different workflow, and the value comes from running several in parallel.

**Langfuse's score abstraction is intentionally simple: a name, a value, and a source.** The source field is what makes the workflow design legible. Five sources matter: - **Manual** annotation in the UI. Use for human in the loop labeling of a small sample, often to build or grow a golden set. Slow, but the only ground-truth source. - **Model-based** judges via Langfuse Evaluators. Configure an LLM-as-judge prompt that runs against sampled traces in production. Cheap, fast, scales to all traffic. Quality limited by the judge prompt and judge model. - **SDK-pushed** scores via `langfuse.score()` in your application code. Use for rule-based checks: regex matches, schema validity, length bounds, tool-call success rates. Encodes domain logic the judge would not see. - **Dataset runs.** A fixed dataset gets replayed against a candidate prompt; each item gets a score; the run as a whole gets an aggregate. The offline regression workflow. - **User feedback.** Thumbs from end users, surfaced via the SDK or the UI. Sparse and biased (unhappy users rate more), but the only signal that reflects actual user experience. The trade-off worth naming: no single source is sufficient. The mature pattern is to run judge scores on production sampled traffic, dataset scores on pre-deploy regression, manual scores on a slow-growing golden set, SDK scores for custom rules, and user feedback as a drift complement that flags when judges disagree with users.

TL;DR

Langfuse exposes five score sources: manual UI annotation (golden sets), model-based judge (online evals), SDK-pushed (custom rules), dataset runs (offline regression), and user feedback (drift signal).

Memory aid

Easy to grasp

Picture a restaurant that tracks how good its dishes are using five different scorecards. The owner samples a few plates personally and grades them by hand. A robot taster scores every tenth plate that goes out the kitchen door. The kitchen has its own internal checks that fire automatically when a dish has the wrong ratio of ingredients. Once a week, the chef cooks the entire weekly menu against a fixed recipe book to make sure no new technique has hurt the classics. And the diners themselves leave thumbs-up or thumbs-down. None of those five is sufficient alone, but together they tell the owner what is going right, what is drifting, and what to fix. Langfuse scores work the same way.

Key concepts

Observability and tracing

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Langfuse exposes a deceptively simple primitive: a score, with a name, a value, and a source. The simplicity is the point. The same primitive can carry a human label, a judge verdict, a rule-based check, a regression metric, or a user thumbs-down, and the source field is what tells downstream consumers how to read it.

The interview question is whether you can map each source to the workflow it enables. Mature LLM observability stacks run multiple sources in parallel because each one catches a failure mode the others miss; treating any single source as the truth is the rookie mistake.

Manual scores: ground truth, slow and expensive

Manual scores come from a human clicking through the Langfuse UI's annotation queue. A reviewer pulls a sample of traces (filtered by recency, by judge-score outlier, by user thumbs-down, or by sampling rule), reads the input and output, and assigns a score on whatever rubric the team has agreed on.

What manual scores are good for. They are the only source of true ground truth. Use them for three jobs:

Golden-set construction. Items the human scored highly become positive examples; items scored badly become regression tests.
Judge calibration. Run the same traces through the LLM-as-judge and measure agreement. The judge prompt gets iterated until it correlates with humans above some threshold.
Fine-tuned scorer training data. When the judge is expensive and a smaller classifier is needed, the manual scores are the labels.

What manual scores are bad for. They do not scale. A single reviewer can score maybe 50-200 traces per day at sustainable quality. They cannot be the primary score on production traffic; they are the slow truth-check that keeps the fast scores honest.

Model-based scores: cheap, fast, only as good as the judge

SDK-pushed and dataset-run scores: code-driven and offline

User feedback and the orchestration of all five

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Langfuse 3.x in 2026 exposes Evaluators that run judge prompts against sampled traces, writing model-based scores back automatically.
Anthropic's Claude Opus 4.7 is a common pick as the judge model inside Langfuse Evaluators for nuanced rubric scoring.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow do you calibrate an LLM-as-judge against manual scores so the two stay aligned?

Take a sample of 100-300 manually scored traces, run the judge on the same traces, measure correlation (Spearman or kappa for categorical), and iterate the judge prompt until agreement with humans crosses a threshold (e.g. 0.7). Re-calibrate quarterly.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating user-feedback thumbs as the primary quality metric. They are sparse, biased toward negative cases, and useful only as a complement to other signals.

60 second bullets to scan on the way to the call.

The five Langfuse score sources and what each one is for
Why manual scores remain necessary even with cheap judges

Primary sources. Browse if you want the original framing.