Explain FActScore's approach to long form factual evaluation and its design decisions
Describe how FActScore evaluates the factual accuracy of long form LLM outputs. What is the atomic fact decomposition approach, how is each fact verified, and what does the final score represent?
FActScore decomposes long-form text into atomic facts, verifies each against an external knowledge source, and scores the fraction supported. Strength: per-claim localization. Weakness: cost and source coverage.
Imagine a friend tells you a long story about a celebrity. Instead of asking 'was that whole story true?', you write down every single small claim separately: born in this city, won this award, married this person. Then you check each tiny claim one at a time against a trusted encyclopedia. For each one you say yes it matches, no it contradicts, or the book does not say. Your final grade is just the fraction of small claims that matched. The clever part is that breaking the story into bite-size pieces tells you exactly which sentences were made up, not just that something somewhere was wrong. The catch: if the encyclopedia has nothing about that person, you cannot check the claims, so your grade becomes unreliable.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
FActScore is a landmark method for evaluating the factual accuracy of long-form LLM generations such as biographies, encyclopedic entries, and multi-paragraph answers. Its core move is to stop treating factuality as one fuzzy holistic judgment and instead reduce it to a precise, per-claim measurement.
The motivation is a real gap in the eval toolkit. String-match metrics like BLEU or ROUGE are useless for open-ended text, and asking an LLM judge 'is this paragraph accurate?' returns a single number that hides which sentences were fabricated. FActScore answers a sharper question: of all the individual factual claims this output makes, what fraction does a trusted source actually support? The deep dive below walks the pipeline, the example that makes it concrete, the design decisions, and the three limitations a senior candidate must surface.
Stage one: atomic fact decomposition
The first stage breaks the generated text into atomic facts, defined as the smallest claims that can be verified on their own. A single sentence usually contains several. The canonical example: 'Marie Curie was born in Warsaw in 1867' decomposes into two atomic facts, that she was born in Warsaw, and that she was born in 1867. Each can be checked independently against a source.
Decomposition is typically performed by an LLM prompted to extract every discrete claim from a passage. This is the step candidates most often gloss over, yet it is where a lot of the method's fragility lives.
Good decomposition is neither trivial nor free. The extractor must split claims finely enough to be checkable but preserve the context a claim needs to remain meaningful. 'She won it in 1911' is unverifiable once severed from the entity and the award it refers to. Over-splitting can also double-count a single underlying error across several facts.
Decomposition also has to resolve coreference and implicit context. Pronouns must be replaced with their referents, dates and places must travel with the claim they qualify, and compound clauses must be untangled so each resulting fact stands alone. A claim like 'her later work, after the move, was honored' decomposes poorly without first resolving who, when, and which honor. The original FActScore work measured this empirically, validating LLM-generated decompositions against human-written ones and reporting non-trivial disagreement, which is exactly why a serious user treats the extractor as a component to be evaluated rather than trusted blindly.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- RAGAS implements claim-level faithfulness by decomposing answers into statements and entailment-checking each against retrieved context, the same pattern FActScore introduced.
- TruLens groundedness scoring breaks responses into claims and verifies each against source context rather than scoring holistically.
What an interviewer would ask next. Try answering before peeking at the approach.
QHow would you handle atomic facts that the knowledge source returns not-enough-info for?
Decide whether to exclude them from the denominator, report them as a separate abstention rate, or penalize them. Excluding inflates the score on poorly-covered topics, so report coverage alongside the supported fraction.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Describing FActScore as a holistic 'is this paragraph accurate' judgment. It is the opposite: per-claim decomposition and verification, and the score is the fraction of atomic facts supported.
60 second bullets to scan on the way to the call.
The three stages: decompose, verify, score
What an atomic fact is and how a sentence splits into several
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.