Which of these are valid concerns when using LLM-as-judge for evaluation?

**LLM-as-judge** is the practice of using one language model (often GPT-4 or Claude Opus) to score the outputs of another model on quality, helpfulness, factual accuracy, or some other rubric. It scales fast and beats human evaluation on cost and speed, but it has documented blind spots. Three of the options describe real concerns. **Position bias** means that when a judge compares two answers side by side, it often prefers whichever appears first or second, independent of content. The standard fix is to run each pair twice with order swapped and average. **Self-preference bias** means a judge of family X scores answers that sound like family X more highly. A GPT judge prefers GPT-style verbose answers; a Claude judge prefers Claude-style structured ones. This pollutes head to head comparisons across model families. **Weakness on factual grounding** means the judge can confidently rate a well-written but wrong answer above a correct but terse one. Judges are great at style, weaker on facts. The two wrong options: judges cost MORE per call than the candidate (since you typically pick a stronger judge), and judges are NOT deterministic at default temperature, you have to set T=0 for reproducibility.

**LLM-as-judge** has three well-documented biases that the question hits: **Position bias.** Judges prefer the first or second option independent of content. The fix: run each comparison twice with order swapped, average the scores. Almost every modern eval framework (RAGAS, Promptfoo, LangSmith) ships position-randomisation by default. **Self-preference bias.** A judge favours outputs that match its own family's style. A GPT-4 judge scores GPT-family verbose answers higher; a Claude judge scores structured with headers answers higher. The fix has two paths. Use multiple judges from different families and ensemble. Or use an open-weight judge like Prometheus or JudgeLM. **Weak factual grounding.** Judges can rate a confidently-wrong answer above a correct but terse one. Style overpowers fact when both signals are present. The fix: ground factual evals on the source (faithfulness against retrieved context), not on the judge's world knowledge. The wrong options. Cost: judges typically run a STRONGER model than the candidate, so cost is HIGHER per call, not equal. Determinism: judges sample at T greater than 0 by default; set T=0 to make eval reproducible across runs.