Explain G-Eval's mechanism and why it improves over direct-score LLM judges

**G-Eval** is a recipe for using an LLM as a grader that produces more trustworthy and explainable scores than just asking 'rate this 1 to 5'. It works in two stages. First, criteria generation: you hand the model a short rubric like 'evaluate coherence', and the model uses chain-of-thought to expand it into concrete evaluation steps. For example, 'Check that each sentence follows from the previous one' and 'Check that the structure supports the main claim'. Second, scoring: the model fills out a form, walking those steps and assigning a score. Because it articulated what it was checking before it checked, you can audit which step caused a low score. There is a clever final touch. Instead of returning a single integer, G-Eval reads the probabilities the model assigned to each possible score and computes a weighted average. So a 3.7 and a 3.4 no longer collapse into the same '4'. The payoff: G-Eval correlates more strongly with human judgments on text-generation tasks and varies less between runs. The cost is more LLM calls and the same judge biases any LLM grader carries.

**G-Eval** is a two-stage LLM-as-judge recipe. It makes the judge reason first, then smooths the score. Stage one is auto-generated eval steps. You give the model a one-line rubric, like 'evaluate coherence'. The model uses chain-of-thought to expand it into explicit steps. This runs once per criterion, not per example. Stage two is form-filling. The judge walks those steps for each candidate, then emits a score. Articulating checks before scoring makes the verdict auditable. It also blunts anchoring, since the model commits to criteria before it sees how confident the answer sounds. The headline trick is the final score itself. A judge that outputs one integer wastes information and ties many candidates at the same value. G-Eval instead takes the expectation over the score tokens. The score is: $$\text{score} = \sum_{i=1}^{n} p(s_i)\, s_i$$ Each candidate value is weighted by the model's output probability. Fine values like 3.7 versus 3.4 now separate answers that would otherwise tie. Empirically G-Eval correlates more strongly with humans on NLG tasks and shows lower run to run variance. The cost is extra calls and the usual LLM-judge biases.