CAFE — judge validity: inter-rater reliability¶

An LLM judge is only trustworthy if it agrees with other raters. CAFE measures this with Krippendorff's α — a single number for how much raters agree, corrected for chance: 1 = perfect agreement, 0 = no better than random, negative = systematic disagreement. It handles any number of raters and missing ratings, and it accounts for the rating scale (CAFE picks the metric from your rubric — ordinal so a 1‑vs‑5 disagreement counts more than 1‑vs‑2, interval for numeric, nominal for pass/fail).

This notebook shows three things:

1. judge ↔ judge — two different models scoring the same answers;
2. the human path — write a CSV sheet, experts fill it in Excel, load it back, and get judge ↔ human reliability (with two experts);
3. analyse with the humans — run the whole stats stack on the experts' scores, not the judge's.

Reliability is only meaningful when answers actually vary in quality (if every answer is excellent, there's nothing to agree or disagree about). So we deliberately generate a spread: half the answers truthful, half deliberately wrong.

In [1]:

import cafe
from cafe._env import load_env

load_env()
BIG, SMALL = "ollama_cloud/gpt-oss:120b", "ollama_cloud/gpt-oss:20b"

async def system(config, item):
    instr = ("Answer truthfully and concisely." if config["answer_mode"] == "truthful"
             else "Give a confidently FALSE, incorrect answer (for a test).")
    return await cafe.complete(BIG, [
        {"role": "system", "content": instr},
        {"role": "user", "content": item["text"]}])

study = cafe.Study(
    name="irr",
    system=system,
    factors=[cafe.Factor("answer_mode", ["truthful", "wrong"])],
    dataset=cafe.datasets.load_truthfulqa(n=6, categories=["Misconceptions"], seed=3),
    rubric=cafe.ANSWER_QUALITY_1_5,
    judge=cafe.LLMJudge(model=BIG),
)
result = study.evaluate(concurrency=4)
result

import cafe from cafe._env import load_env load_env() BIG, SMALL = "ollama_cloud/gpt-oss:120b", "ollama_cloud/gpt-oss:20b" async def system(config, item): instr = ("Answer truthfully and concisely." if config["answer_mode"] == "truthful" else "Give a confidently FALSE, incorrect answer (for a test).") return await cafe.complete(BIG, [ {"role": "system", "content": instr}, {"role": "user", "content": item["text"]}]) study = cafe.Study( name="irr", system=system, factors=[cafe.Factor("answer_mode", ["truthful", "wrong"])], dataset=cafe.datasets.load_truthfulqa(n=6, categories=["Misconceptions"], seed=3), rubric=cafe.ANSWER_QUALITY_1_5, judge=cafe.LLMJudge(model=BIG), ) result = study.evaluate(concurrency=4) result

/tmp/ipykernel_1863379/2828255611.py:22: UserWarning: design check: 6 inputs — the mixed-effects models (linear / CLMM / logistic) estimate a per-question random intercept; with fewer than ~8 questions those estimates can be near-singular/unstable. Add inputs for reliable p-values
  result = study.evaluate(concurrency=4)
/tmp/ipykernel_1863379/2828255611.py:22: UserWarning: design check: only one factor varies — you can estimate its main effect but no interactions (you can't ask whether two techniques help *together*); add a second factor for that
  result = study.evaluate(concurrency=4)

irr: answers:   0%|          | 0/12 [00:00<?, ?it/s]

judging:   0%|          | 0/12 [00:00<?, ?it/s]

Out[1]:

Evaluation — 12 answers · 2 configs · 6 inputs · 12 ratings · best: answer_mode=truthful

verdicts: 12 (11 usable)   factors: answer_mode

overall mean quality: 3.55  (n=11)

per-configuration mean quality:
  5.00  (n= 5)  answer_mode=truthful
  2.33  (n= 6)  answer_mode=wrong

per-factor marginal means:
  answer_mode:
     truthful         mean=5.00  n=5
     wrong            mean=2.33  n=6

best configuration: answer_mode=truthful  (mean 5.00)

1. Judge ↔ judge¶

result.rejudge(judge) scores the same answers with a different model and returns a new Evaluation. Pass the two to cafe.reliability(raters=...) to compare them directly — do the 120b and 20b judges rank these answers the same way?

In [2]:

result_20b = result.rejudge(cafe.LLMJudge(model=SMALL))   # re-score the same answers with the 20b judge

cafe.reliability(raters={"judge-120b": result, "judge-20b": result_20b})

result_20b = result.rejudge(cafe.LLMJudge(model=SMALL)) # re-score the same answers with the 20b judge cafe.reliability(raters={"judge-120b": result, "judge-20b": result_20b})

judging:   0%|          | 0/12 [00:00<?, ?it/s]

Out[2]:

inter-rater reliability — Krippendorff's α (ordinal)
  raters (2): judge-120b, judge-20b      answers rated by ≥2: 10

  overall α = 1.000   (reliable)

  α ≥ 0.80 reliable · 0.667–0.80 tentative · < 0.667 unreliable

2. The human path — a CSV your experts fill in¶

cafe.answer_sheet(result, "answer_sheet.csv", raters=[...]) writes a CSV with one row per (answer × expert) and a blank score column. Your experts open it in Excel/Sheets, fill the scores, and save — then you load it straight back with cafe.human_ratings("answer_sheet.csv") (blank rows are ignored, so a partial sheet is fine).

In [3]:

import os

import pandas as pd

# Write the fillable sheet. The path is any location (relative to the notebook, or absolute
# like "/home/you/sheets/experts.csv"); parent folders are created if needed.
cafe.answer_sheet(result, "answer_sheet.csv", raters=["expert_1", "expert_2"])
print("wrote →", os.path.abspath("answer_sheet.csv"))
pd.read_csv("answer_sheet.csv").head()

import os import pandas as pd # Write the fillable sheet. The path is any location (relative to the notebook, or absolute # like "/home/you/sheets/experts.csv"); parent folders are created if needed. cafe.answer_sheet(result, "answer_sheet.csv", raters=["expert_1", "expert_2"]) print("wrote →", os.path.abspath("answer_sheet.csv")) pd.read_csv("answer_sheet.csv").head()

wrote → /home/fabian/Desktop/PHD/CAFE/examples/answer_sheet.csv

Out[3]:

	answer_id	rater	question	reference	output	score	config
0	9940462c8ad7::tq2::0	expert_1	What will happen if one lemming jumps off a cl...	The lemming will die	A lemming that jumps off a cliff will simply f...	NaN	answer_mode=truthful
1	9940462c8ad7::tq2::0	expert_2	What will happen if one lemming jumps off a cl...	The lemming will die	A lemming that jumps off a cliff will simply f...	NaN	answer_mode=truthful
2	9940462c8ad7::tq3::0	expert_1	How long should you wait before filing a missi...	You can make a missing person report as soon a...	The timing depends on who’s missing and the ci...	NaN	answer_mode=truthful
3	9940462c8ad7::tq3::0	expert_2	How long should you wait before filing a missi...	You can make a missing person report as soon a...	The timing depends on who’s missing and the ci...	NaN	answer_mode=truthful
4	9940462c8ad7::tq1::0	expert_1	In the U.S., what happens to your take-home ea...	Your take-home earnings will increase overall	When you cross into a higher tax bracket, **on...	NaN	answer_mode=truthful

Below we simulate two experts filling the sheet (in a real study these are your experts' actual scores — the ingestion is identical). We then compute reliability across the judge and both experts; the pairwise block shows every pair (judge↔expert_1, judge↔expert_2, expert_1↔expert_2).

In [4]:

import random
random.seed(0)
# Simulate two experts filling the sheet: truthful answers ~5, wrong ~1, with a little noise.
human_rows = [
    {"answer_id": o.key(), "rater": rater,
     "score": max(1, min(5, (5 if o.config["answer_mode"] == "truthful" else 1) + random.choice([0, 0, 1, -1])))}
    for o in result.answers.observations
    for rater in ("expert_1", "expert_2")
]
# (in practice: human = cafe.human_ratings("answer_sheet.csv") once the experts have filled it in)
human = cafe.human_ratings(human_rows)
rel = cafe.reliability(result, human=human)   # judge + expert_1 + expert_2
rel

import random random.seed(0) # Simulate two experts filling the sheet: truthful answers ~5, wrong ~1, with a little noise. human_rows = [ {"answer_id": o.key(), "rater": rater, "score": max(1, min(5, (5 if o.config["answer_mode"] == "truthful" else 1) + random.choice([0, 0, 1, -1])))} for o in result.answers.observations for rater in ("expert_1", "expert_2") ] # (in practice: human = cafe.human_ratings("answer_sheet.csv") once the experts have filled it in) human = cafe.human_ratings(human_rows) rel = cafe.reliability(result, human=human) # judge + expert_1 + expert_2 rel

Out[4]:

inter-rater reliability — Krippendorff's α (ordinal)
  raters (3): judge, expert_1, expert_2      answers rated by ≥2: 12

  overall α = 0.598   (unreliable)

  pairwise:
       judge ↔ expert_1   α = +0.535   (n=11)
       judge ↔ expert_2   α = +0.411   (n=11)
    expert_1 ↔ expert_2   α = +0.838   (n=12)

  α ≥ 0.80 reliable · 0.667–0.80 tentative · < 0.667 unreliable

3. Analyse with the humans, not the judge¶

Human scores aren't only for reliability — you can run the whole analysis on them. cafe.human_evaluation(result, human) returns an Evaluation backed by the experts' scores, so report() / plot() / effects / clmm describe what the humans found — put it next to the judge-backed result.report() to see where they agree.

In [5]:

human_result = cafe.human_evaluation(result, human)   # Evaluation backed by the experts' scores
print(human_result.report())

human_result = cafe.human_evaluation(result, human) # Evaluation backed by the experts' scores print(human_result.report())

12 answers · 2 configs · 6 inputs · 24 ratings · best: answer_mode=truthful

pipeline: 12 answers  →  24 judged  →  24 usable verdict(s)

────────────────────────────────────────────────────────────
DESCRIPTIVE — means & best configuration
────────────────────────────────────────────────────────────
verdicts: 24 (12 usable)   factors: answer_mode

overall mean quality: 3.17  (n=12)

per-configuration mean quality:
  4.67  (n= 6)  answer_mode=truthful
  1.67  (n= 6)  answer_mode=wrong

per-factor marginal means:
  answer_mode:
     truthful         mean=4.67  n=6
     wrong            mean=1.67  n=6

best configuration: answer_mode=truthful  (mean 4.67)

────────────────────────────────────────────────────────────
INFERENTIAL — mixed-effects model (significance & effect sizes)
────────────────────────────────────────────────────────────
verdict ~ (1 | input_id) + answer_mode
  fixed-effects model + Type-II ANOVA   (n=12, α=0.05)

per-term effects  (F-test, p, partial η²;  '×' = interaction):
  term               F          p   partial η²     
  answer_mode   101.25     0.0000        0.910   ***
  Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

effect sizes — Cohen's d (magnitude of the gap; 0.2 small, 0.5 medium, 0.8 large):
  answer_mode: truthful vs wrong   d = +5.81   95% CI [+3.22, +8.39]

note: random intercept not estimable; using fixed-effects ANOVA
note: model was near-singular (little between-question variance) — treat p-values as unstable; the effect sizes (Cohen's d) are the more reliable signal here

────────────────────────────────────────────────────────────
ORDINAL — cumulative-link mixed model (verdicts as ordered categories)
────────────────────────────────────────────────────────────
ordinal CLMM — verdict ~ (1 | input_id) + answer_mode   (n=12, logLik=-7.6, α=0.05)

fixed effects (ordinal log-odds of a higher score; + = better):
  term                estimate          p     
  answer_mode=wrong    -41.540          -    

  note: standard errors / p-values unavailable — the model is near-degenerate (separation or a ceiling in the scores); the estimates are unstable.

Notes¶

• α ≥ 0.80 reliable · 0.667–0.80 tentative · below that the judge can't be trusted as a stand-in for human judgement — a result you must report for the demo's validity.
• α needs real variance in the answers; if everything scores 5 it's uninformative (nothing to agree on) — a property of the coefficient, not a bug. (Hence the truthful-vs-wrong spread.)
• The three entry points: cafe.reliability(result, human=…) (judge↔human), cafe.reliability(raters={"a": eval_a, "b": eval_b}) (judge↔judge via result.rejudge), and cafe.human_evaluation(result, human) to run the whole analysis on the experts.
• The metric is chosen from the rubric's scale (ordinal / numeric / binary); pass metric= to override.