Case study 1 · Question answering with retrieval (HotpotQA)¶
The question: in a retrieval-augmented (RAG) pipeline, what actually drives answer quality: the retrieval, the model, or the combination of the two? And is each difference real, or just noise?
The setup: 50 hard multi-hop questions from HotpotQA, each needing facts from two Wikipedia paragraphs. The pipeline has three stages, and every stage becomes a factor:
| Factor | Levels |
|---|---|
retrieval |
none · dense · dense_with_reranker |
model |
gemma-3-4b · gpt-oss-20b · gpt-oss-120b |
placebo |
on · off (both do nothing: a negative control) |
That gives 3 × 3 × 2 = 18 configurations × 50 questions = 900 answers. An LLM judge from a different model family scores each answer 0 (incorrect), 1 (partially correct) or 2 (correct).
What this study shows in CAFE: attribution (which factor explains how much), an interaction (retrieval helps the small model most), a placebo correctly found to do nothing, the ordinal mixed model for a 0/1/2 scale, the cost/quality Pareto front, and energy per configuration.
import json
import pandas as pd
import cafe
cafe.load_env(); # the API keys from the repo's .env
1 · The data¶
50 HotpotQA questions with their correct answers. question_type says whether a question links
two facts (bridge) or compares two things (comparison); it becomes a filter in the web app.
questions = json.load(open("data/hotpotqa_questions.json"))
dataset = [
{"id": q["id"], "text": q["question"], "reference": q["answer"], "question_type": q["question_type"]}
for q in questions
]
print(f"{len(dataset)} questions, for example:")
print(" question:", dataset[0]["text"])
print(" answer: ", dataset[0]["reference"])
50 questions, for example: question: The first book in the Sprawl Trilogy won what three awards? answer: Nebula Award, the Philip K. Dick Award, and the Hugo Award
2 · The system¶
The pipeline lives in pipeline.py next to this notebook: the web app runs the
same file. Each stage has several interchangeable techniques.
from pipeline import pipe
for stage in pipe.stages():
print(f"{stage:10s} → {', '.join(pipe.names_for(stage=stage))}")
retrieval → none, dense, dense_with_reranker model → gemma-3-4b, gpt-oss-20b, gpt-oss-120b placebo → on, off
3 · The factors¶
Every stage becomes a factor whose levels are its techniques. CAFE runs every combination.
# The pipeline says which techniques exist; the factors say what this study varies.
# pipe.factor(stage=...) makes a factor whose levels are all the techniques of that stage, in the
# order of pipeline.py. names=[...] would pick a subset instead, for example
# pipe.factor(stage="model", names=["gpt-oss-20b", "gpt-oss-120b"])
# Every stage with several techniques must be a factor (otherwise CAFE stops with an error);
# a stage with only one technique is simply used.
factors = [
pipe.factor(stage="retrieval"),
pipe.factor(stage="model"),
pipe.factor(stage="placebo"),
]
4 · Rubric and judge¶
The judge compares each answer with the correct answer and scores it on a three-level scale. Because the scale is ordered (0 < 1 < 2), CAFE analyses it with an ordinal mixed model.
answer_correctness = cafe.Rubric(
name="answer_correctness",
scale_type=cafe.ScaleType.ordinal,
levels=[
cafe.Level(value=0, label="incorrect", description="The answer is factually incorrect or off-topic."),
cafe.Level(value=1, label="partially_correct", description="The answer is partially correct."),
cafe.Level(value=2, label="correct", description="The answer is fully correct; wording does not matter."),
],
instruction=(
"Grade whether the ANSWER correctly answers the QUESTION, using the REFERENCE as the gold "
"answer. Judge factual correctness, not phrasing or length; award partial credit for a "
"partially-correct answer."
),
prompt_template=( # the exact judge prompt used in the paper
"{instruction}\n\n"
"[BEGIN DATA]\n***\nQuestion: {question}\n***\nReference (gold) answer: {reference}\n"
"***\nAnswer to grade: {answer}\n***\n[END DATA]\n\n"
"Reason briefly, then score on this scale:\n{scale}\n\n"
"End with exactly one final line, and nothing after it:\nGRADE: <{grade}>"
),
)
# The judge: an LLM that scores each answer against the rubric (CAFE sends it a prompt it builds
# from the rubric, the question, the reference and the answer).
judge = cafe.LLMJudge(
model="ollama_cloud/deepseek-v4-pro", # a different model family from all three systems
system_prompt=(
"You will be shown a question, a reference (gold) answer, and an answer to grade. Follow the "
"grading rules the user gives you, and be a fair, factual grader: judge only whether the "
"answer is factually correct against the reference, not its style or length."
),
)
# The exact prompt the judge will see, for one example answer (no model is called here):
print(judge.preview(rubric=answer_correctness, question=dataset[0]["text"],
answer="The Hugo, Nebula and Philip K. Dick awards.",
reference=dataset[0]["reference"]))
[SYSTEM] You will be shown a question, a reference (gold) answer, and an answer to grade. Follow the grading rules the user gives you, and be a fair, factual grader: judge only whether the answer is factually correct against the reference, not its style or length. [USER] Grade whether the ANSWER correctly answers the QUESTION, using the REFERENCE as the gold answer. Judge factual correctness, not phrasing or length; award partial credit for a partially-correct answer. [BEGIN DATA] *** Question: The first book in the Sprawl Trilogy won what three awards? *** Reference (gold) answer: Nebula Award, the Philip K. Dick Award, and the Hugo Award *** Answer to grade: The Hugo, Nebula and Philip K. Dick awards. *** [END DATA] Reason briefly, then score on this scale: 0 = incorrect: The answer is factually incorrect or off-topic. 1 = partially_correct: The answer is partially correct. 2 = correct: The answer is fully correct; wording does not matter. End with exactly one final line, and nothing after it: GRADE: <exactly one of: 0, 1, 2> [USER, structured mode: sent instead of the one above if the model supports JSON output] Grade whether the ANSWER correctly answers the QUESTION, using the REFERENCE as the gold answer. Judge factual correctness, not phrasing or length; award partial credit for a partially-correct answer. [BEGIN DATA] *** Question: The first book in the Sprawl Trilogy won what three awards? *** Reference (gold) answer: Nebula Award, the Philip K. Dick Award, and the Hugo Award *** Answer to grade: The Hugo, Nebula and Philip K. Dick awards. *** [END DATA] Reason briefly, then score on this scale: 0 = incorrect: The answer is factually incorrect or off-topic. 1 = partially_correct: The answer is partially correct. 2 = correct: The answer is fully correct; wording does not matter. End with exactly one final line, and nothing after it: GRADE: <exactly one of: 0, 1, 2> Instead of ending with a GRADE: line, return ONLY a JSON object with keys "reasoning" (a brief string) and "grade" (exactly one of: 0, 1, 2). Output nothing else.
5 · Run the study¶
First the plan and a quick preflight (one question through every configuration, with a time and cost estimate), then the full run. The run saves its progress as it goes: if it is interrupted, or some answers failed (e.g. a provider was briefly rate-limited), running the cell again continues where it stopped and retries only the failed answers.
# A study = the system, the factors to vary, the inputs, and how to score the answers.
study = cafe.Study(
name="question-answering-hotpotqa",
system=pipe,
factors=factors,
dataset=dataset,
rubric=answer_correctness,
judge=judge,
)
print(study.plan()) # how many configurations, answers and judge calls (nothing runs yet)
print(study.preflight(concurrency=4)) # one input through every configuration: errors, time, cost
{'name': 'question-answering-hotpotqa', 'design': 'full_factorial', 'factors': ['retrieval', 'model', 'placebo'], 'configs': 18, 'items': 50, 'replications': 1, 'runs': 900, 'judged': True, 'judge_calls': 900}
Preflight(answers=Results(question-answering-hotpotqa: 18 answers, 18 configs, 0 errors), estimate={'sampled_cells': 18, 'total_cells': 900, 'est_total_compute_s': 34273.5, 'est_total_cost_usd': 0.0846, 'labels': ['model=gemma-3-4b·placebo=on·retrieval=none']}, warnings=[], judge_calls=900)
# Run every question through every configuration, judge every answer, then fit the statistics.
result = study.evaluate(
concurrency=4, # 4 at a time: Ollama Cloud rejects too many parallel requests
checkpoint_path="results/answers.cafe.jsonl",
judge_checkpoint_path="results/verdicts.cafe.jsonl",
)
question-answering-hotpotqa: answers: 100% 900/900 [35:22<00:00, 2.36s/it]
judging: 100% 900/900 [19:49<00:00, 1.32s/it]
6 · The results¶
The full report: means, attribution (which factor explains how much of the quality), effect sizes, and the ordinal mixed model with one test per factor.
print(result.report())
900 answers · 18 configs · 50 inputs · 900 ratings · best: model=gpt-oss-20b·placebo=off·retrieval=dense_with_reranker
pipeline: 900 answers → 900 judged → 900 usable verdict(s)
────────────────────────────────────────────────────────────
DESCRIPTIVE — means & best configuration
────────────────────────────────────────────────────────────
verdicts: 900 factors: retrieval, model, placebo
overall mean quality: 1.35 (n=900)
per-configuration mean quality:
1.80 (n=50) model=gpt-oss-20b·placebo=off·retrieval=dense_with_reranker
1.80 (n=50) model=gpt-oss-20b·placebo=on·retrieval=dense_with_reranker
1.78 (n=50) model=gpt-oss-120b·placebo=off·retrieval=dense_with_reranker
1.78 (n=50) model=gpt-oss-120b·placebo=on·retrieval=dense_with_reranker
1.70 (n=50) model=gpt-oss-120b·placebo=off·retrieval=dense
1.66 (n=50) model=gpt-oss-120b·placebo=on·retrieval=dense
1.64 (n=50) model=gemma-3-4b·placebo=on·retrieval=dense_with_reranker
1.60 (n=50) model=gemma-3-4b·placebo=off·retrieval=dense_with_reranker
1.56 (n=50) model=gpt-oss-20b·placebo=off·retrieval=dense
1.52 (n=50) model=gpt-oss-20b·placebo=on·retrieval=dense
1.44 (n=50) model=gemma-3-4b·placebo=on·retrieval=dense
1.42 (n=50) model=gemma-3-4b·placebo=off·retrieval=dense
1.16 (n=50) model=gpt-oss-120b·placebo=off·retrieval=none
1.10 (n=50) model=gpt-oss-120b·placebo=on·retrieval=none
0.74 (n=50) model=gpt-oss-20b·placebo=on·retrieval=none
0.72 (n=50) model=gpt-oss-20b·placebo=off·retrieval=none
0.48 (n=50) model=gemma-3-4b·placebo=on·retrieval=none
0.46 (n=50) model=gemma-3-4b·placebo=off·retrieval=none
per-factor marginal means:
retrieval:
dense mean=1.55 n=300
dense_with_reranker mean=1.73 n=300
none mean=0.78 n=300
model:
gemma-3-4b mean=1.17 n=300
gpt-oss-120b mean=1.53 n=300
gpt-oss-20b mean=1.36 n=300
placebo:
off mean=1.36 n=450
on mean=1.35 n=450
best configuration: model=gpt-oss-20b·placebo=off·retrieval=dense_with_reranker (mean 1.80)
────────────────────────────────────────────────────────────
INFERENTIAL — linear model, questions as blocks (significance & effect sizes)
────────────────────────────────────────────────────────────
verdict ~ question + (retrieval + model + placebo)^2
Type-II ANOVA, blocked by question, up to 2-way (n=900, α=0.05)
per-term effects (F-test, p, partial η²; '×' = interaction):
term F p partial η²
retrieval 226.31 0.0000 0.351 ***
model 27.92 0.0000 0.063 ***
retrieval × model 5.93 0.0001 0.028 ***
model × placebo 0.20 0.8201 0.000
retrieval × placebo 0.06 0.9401 0.000
placebo 0.01 0.9092 0.000
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
effect sizes — Cohen's d (magnitude of the gap; 0.2 small, 0.5 medium, 0.8 large):
retrieval: dense vs dense_with_reranker d = -0.27 95% CI [-0.43, -0.11]
retrieval: dense vs none d = +0.93 95% CI [+0.76, +1.10]
retrieval: dense_with_reranker vs none d = +1.26 95% CI [+1.08, +1.44]
model: gemma-3-4b vs gpt-oss-120b d = -0.42 95% CI [-0.58, -0.26]
model: gemma-3-4b vs gpt-oss-20b d = -0.21 95% CI [-0.37, -0.05]
model: gpt-oss-120b vs gpt-oss-20b d = +0.21 95% CI [+0.05, +0.37]
placebo: off vs on d = +0.01 95% CI [-0.13, +0.14]
variance components: between-question σ² = 0.207 · residual σ² = 0.346 · ICC = 0.374
────────────────────────────────────────────────────────────
ORDINAL — cumulative-link mixed model (verdicts as ordered categories)
────────────────────────────────────────────────────────────
ordinal CLMM — verdict ~ (1 | input_id) + (retrieval + model + placebo)^2 (n=900, logLik=-540.5, α=0.05)
per-factor tests (likelihood ratio, Type II — does the factor matter at all?):
term χ² df p
retrieval 343.09 2 0.0000 ***
model 63.89 2 0.0000 ***
retrieval × model 13.29 4 0.0099 **
model × placebo 0.46 2 0.7941
retrieval × placebo 0.40 2 0.8183
placebo 0.00 1 0.9640
fixed effects (ordinal log-odds of a higher score; + = better):
term estimate p
retrieval=dense_with_reranker +0.514 0.2414
retrieval=none -3.920 0.0000 ***
model=gpt-oss-120b +1.406 0.0020 **
model=gpt-oss-20b +0.473 0.2679
placebo=on -0.014 0.9720
retrieval=dense_with_reranker × model=gpt-oss-120b -0.086 0.8826
retrieval=none × model=gpt-oss-120b +1.355 0.0117 *
retrieval=dense_with_reranker × model=gpt-oss-20b +0.883 0.1204
retrieval=none × model=gpt-oss-20b +0.616 0.2273
retrieval=dense_with_reranker × placebo=on +0.297 0.5311
retrieval=none × placebo=on +0.161 0.7038
model=gpt-oss-120b × placebo=on -0.302 0.4981
model=gpt-oss-20b × placebo=on -0.125 0.7744
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
thresholds (cut-points between levels): 0|1 -2.395 · 1|2 -1.079
variance components: between-question σ_u² = 5.346 (σ_u = 2.312) · latent residual π²/3 = 3.290 · ICC = 0.619
The interaction: does retrieval help every model equally?
result.plot(kind="interaction", factors=["retrieval", "model"]);
Quality against cost: the configurations on the Pareto front are the only ones worth considering.
result.plot(kind="pareto");
Cost, energy and time per model, summed over all its answers:
answers = pd.DataFrame(result.records())
answers.groupby("model")[["cost_usd", "energy_wh", "elapsed_s"]].sum().round(4)
| cost_usd | energy_wh | elapsed_s | |
|---|---|---|---|
| model | |||
| gemma-3-4b | 0.0118 | 16.9616 | 1711.2004 |
| gpt-oss-120b | 0.0731 | 38.9675 | 2795.5957 |
| gpt-oss-20b | 0.0268 | 29.3690 | 7116.1844 |
7 · Save for the web app¶
One file with everything: answers, verdicts, rubric and question metadata. In the web app, Studies → Import study turns it into a browsable study.
cafe.save_evaluation(evaluation=result, path="results/question_answering_results.json")