Describe two standardized approaches for measuring demographic or stereotyping bias in an LLM. What are the benchmark suites available and how do counterfactual methods work?
Free · 2 AI evals / day
Measuring bias in a language model means checking whether it treats demographic groups differently when nothing else about the situation has changed. There are two standardized families of methods.
**Counterfactual data augmentation** (CDA) builds matched pairs of prompts that are identical except for one attribute, like a name, pronoun, or ethnicity. You run both prompts and compare the outputs. If 'John applied for a loan' gets approved advice but 'Aisha applied for a loan' gets cautious advice, that gap is the measured bias. The trick is holding everything else constant so the swap is the only explanation.
**Benchmark suites** are pre-built test sets with known fair answers. BBQ (Bias Benchmark for QA) asks multiple-choice questions where the context is sometimes ambiguous. The fair answer is 'cannot be determined', and a biased model instead picks the stereotyped group. WinoBias and WinoGender test pronoun resolution: in 'the doctor told the nurse that she was late', does the model assume the nurse is female?
To turn this into a number, you compute accuracy per demographic group on the benchmark, or the difference in sentiment, advice, or refusal rate across the counterfactual pairs. A large gap means bias.
Two standardized approaches dominate.
**Counterfactual data augmentation.** Build pairs of prompts that are structurally identical but differ in one demographic attribute, such as name, gender pronoun, or ethnicity. Run both, then measure the divergence in output. The gap is the bias signal. Example: 'John applied for a loan. He was...' versus 'Aisha applied for a loan. She was...'. Divergent completions, sentiment, or refusal rates flag demographic bias. The strength is controlled attribution. The swap is the only variable, so any difference is caused by the attribute.
**Benchmark suites.** BBQ poses multiple-choice questions in ambiguous and unambiguous variants. When context is insufficient, the correct answer is the unknown option; picking the stereotyped group signals bias. WinoBias and WinoGender test coreference: in 'the doctor told the nurse that she...', a stereotyping model binds the female pronoun to the nurse rather than the doctor.
The metrics are concrete. On benchmarks, report accuracy split by demographic group and a bias score that separates ambiguous from unambiguous contexts. For CDA, report the disparity in sentiment, advice, or refusal rate across each matched pair.
Quick reference
TL;DR
Measure bias two ways: counterfactual augmentation swaps a demographic attribute and checks if outputs diverge, and benchmark suites like BBQ and WinoBias score stereotyped versus neutral choices.
Imagine testing whether a hiring manager is fair. You hand them two identical resumes that differ only in the name at the top, one says John and one says Aisha, and you watch whether the decisions change. If the advice or tone differs, the difference itself is the bias, because nothing else changed. That is the counterfactual idea: swap one attribute, hold everything else fixed, and measure the gap. The second way uses ready-made test packs. Someone has already written hundreds of tricky questions where a fair answer is known in advance, like 'the doctor told the nurse that she was late, who was late?' A fair model says we cannot tell from the sentence. A biased model jumps to the stereotype. You just count how often the model picks the stereotype.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.
Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.
Sole reliance on public benchmarks for a deployed product decision Public suites do not match your prompt distribution, so passing them can mask real in domain disparities.
Counterfactual pairs where attributes are confounded with token frequency or length The swap is no longer the only variable, so any divergence cannot be attributed to demographics.
High stakes legal or medical fairness sign off without human audit Automated bias metrics miss contextual harm and intersectional effects that require expert review.
Reporting a single aggregate fairness number to stakeholders Aggregation hides group level harm and the opposing stereotyping versus over correction failures.
Quick scan · 2–4 min
Everything important, quickly.
5 min: counterfactual data augmentation and pair construction, BBQ ambiguous versus unambiguous design, WinoBias coreference, the per group metrics, and why aggregate scores and public benchmarks mislead.
Real-world examples
Real products, models, and research that use this idea.
1BBQ (Bias Benchmark for QA) ships ambiguous and disambiguated splits and is a standard fairness gate in 2026 model cards from major labs.
2WinoGender and WinoBias remain the default coreference probes for occupational gender stereotyping in open-source eval harnesses.
3HELM by Stanford CRFM bundles BBQ and other bias suites into reproducible leaderboard runs.
4Anthropic and OpenAI system cards report demographic disparity tests built on counterfactual name and pronoun swaps before release.
5DeepEval and Promptfoo expose bias and counterfactual templates so teams can run CDA on their own production prompts.
Interviewer follow-ups
What an interviewer would ask next. Try answering before peeking at the approach.
QWhy does BBQ separate ambiguous from unambiguous contexts, and what does each reveal?
Sign in or go Pro to see the approach for this follow-up.
QHow do you keep a counterfactual name swap from being confounded?
Sign in or go Pro to see the approach for this follow-up.
QWhy can a model pass StereoSet or CrowS-Pairs yet still be biased in production?
Sign in or go Pro to see the approach for this follow-up.
Don't say this
Red flags and common mistakes that signal junior thinking. Click to expand.
Most common mistake
Reporting a single aggregate bias score. Bias must be measured per demographic group and per context condition, because ambiguous and unambiguous cases reveal opposite failure modes.