How do you measure demographic or stereotyping bias in an LLM using a standardized eval?
Describe two standardized approaches for measuring demographic or stereotyping bias in an LLM. What are the benchmark suites available and how do counterfactual methods work?
Measure bias two ways: counterfactual augmentation swaps a demographic attribute and checks if outputs diverge, and benchmark suites like BBQ and WinoBias score stereotyped versus neutral choices.
Imagine testing whether a hiring manager is fair. You hand them two identical resumes that differ only in the name at the top, one says John and one says Aisha, and you watch whether the decisions change. If the advice or tone differs, the difference itself is the bias, because nothing else changed. That is the counterfactual idea: swap one attribute, hold everything else fixed, and measure the gap. The second way uses ready-made test packs. Someone has already written hundreds of tricky questions where a fair answer is known in advance, like 'the doctor told the nurse that she was late, who was late?' A fair model says we cannot tell from the sentence. A biased model jumps to the stereotype. You just count how often the model picks the stereotype.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Concept explanation~2 min read
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Bias evaluation asks a precise question: does the model treat demographic groups differently when nothing relevant has changed? The standardized answer comes in two complementary shapes. One is a controlled experiment you run on your own prompts, counterfactual data augmentation. The other is a set of pre-built, peer-reviewed test suites such as BBQ, WinoBias, and WinoGender that give you comparability across models.
The two are not interchangeable. Counterfactual augmentation gives causal attribution on your real distribution but no external baseline. Benchmark suites give comparability and known fair answers but do not match your product. A serious eval uses both, reports per group, and never collapses fairness into one number.
Counterfactual data augmentation: the controlled experiment
Counterfactual data augmentation (CDA) is the causal core of bias measurement. You take a prompt template and instantiate it twice, changing exactly one demographic attribute, a first name, a gender pronoun, or an explicit group marker, while holding every other token fixed. You then run both prompts and compare a downstream signal.
The canonical example is a loan or hiring stub. 'John applied for a loan. He was...' versus 'Aisha applied for a loan. She was...'. If the completions diverge in sentiment, recommended action, or refusal rate, the swap is the only variable that changed, so the divergence is attributable to the demographic attribute. That attribution is what makes CDA powerful: it is closer to a causal experiment than a correlational benchmark.
The discipline lives in pair construction. Names must be matched for frequency, length, and register across groups, or the model may react to token rarity rather than demographics. You stratify across many names per group so one unusual surname does not dominate. Templates stay neutral and free of leading cues. Done well, CDA scales to thousands of pairs and runs directly on your production prompt shapes.
Situations where this technique stops working.
2–4 min · Everything important, quickly.
Real products, models, and research that use this idea.
- BBQ (Bias Benchmark for QA) ships ambiguous and disambiguated splits and is a standard fairness gate in 2026 model cards from major labs.
- WinoGender and WinoBias remain the default coreference probes for occupational gender stereotyping in open-source eval harnesses.
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does BBQ separate ambiguous from unambiguous contexts, and what does each reveal?
Ambiguous cases test whether the model invents a group answer when it should say unknown, measuring stereotyping. Unambiguous cases test whether the model overrides its prior with stated evidence, measuring over correction. A fair model passes both directions.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Red flags & common mistakes
The phrases that signal junior thinking. Click to expand.
Reporting a single aggregate bias score. Bias must be measured per demographic group and per context condition, because ambiguous and unambiguous cases reveal opposite failure modes.
60 second bullets to scan on the way to the call.
What counterfactual data augmentation is and how pairs are constructed
A concrete CDA example with a measurable output gap
Primary sources. Browse if you want the original framing.
Same topic, related formats. Practice these next.