API reference¶
This page is generated automatically from the docstrings in cafe-core.
Study & factors¶
cafe.Study
dataclass
¶
A complete evaluation: a system, the factors to vary, the inputs, and how to judge the results.
Fields¶
system:
The black box under test, runnable as run(config, item) -> output —
a plain callable (sync or async) or an object with such a method.
factors:
The axes to vary. Their Cartesian product (full factorial) defines the
configurations.
dataset:
The evaluation set, one element per item. An item may be any value; if it
is a mapping it may carry "id" (for resume), "text" (the question
shown to the judge), and "reference" (a gold answer for the judge).
rubric:
The :class:cafe.Rubric the judge scores answers on. Optional —
leave None to only generate answers without judging.
judge:
The :class:cafe.LLMJudge (or any object with a compatible score
method). Optional, paired with rubric.
replications:
How many times each (configuration, input) is executed. This is how CAFE
measures the system's run-to-run nondeterminism.
judge_replications:
How many times the judge re-scores each answer — measures the judge's
own nondeterminism, separately from the system's.
Source code in packages/cafe-core/src/cafe/study.py
83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 | |
plan ¶
The run plan as numbers: configs, items, reps, total runs (and judge calls).
Cheap — it expands the design but runs nothing. Useful for a sanity check or a cost estimate before spending tokens.
Source code in packages/cafe-core/src/cafe/study.py
check ¶
Advisory warnings about whether the design has enough data for the stats — cheap, runs nothing, meant to be read before spending tokens. Flags thin designs that make the mixed-effects models (linear / CLMM / logistic) unstable. Returns a list of human-readable strings (empty = nothing obviously wrong).
Not a power analysis (mixed-model power needs simulation) — just heuristics on the
numbers. Surfaced by :meth:preflight and emitted once when :func:cafe.evaluate
starts.
Source code in packages/cafe-core/src/cafe/study.py
161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 | |
evaluate ¶
Run the complete evaluation: generate answers, judge them (if a
rubric + judge are set), and attribute quality to the factors. Returns an
:class:cafe.evaluation.Evaluation.
Source code in packages/cafe-core/src/cafe/study.py
preflight ¶
Quick check before a full run: one input through every configuration,
no replication or judging, plus a cost/time estimate. Returns a
:class:cafe.evaluation.Preflight.
Source code in packages/cafe-core/src/cafe/study.py
run ¶
Lower-level: generate answers only (no judging). Returns
:class:cafe.Results. Most users want :meth:evaluate.
Source code in packages/cafe-core/src/cafe/study.py
preview_judge_prompt ¶
Return the exact judge prompt for an example answer — no LLM call.
Shows the messages exactly as sent, with role labels ([SYSTEM] / [USER])
so the system framing and the rubric-derived user prompt are both visible. Uses
the study's rubric + judge and, by default, the first dataset item. Print it to
verify the judging before spending any tokens. While an LLMJudge with
structured="auto" hasn't yet decided whether its model gets the JSON prompt,
both user prompts are shown (see :meth:cafe.LLMJudge.preview).
Source code in packages/cafe-core/src/cafe/study.py
cafe.Factor
dataclass
¶
One axis of the experiment.
levels is the set of values this factor can take. Full factorial visits
every combination of every factor's levels — there is no limit on the number
of factors or levels (that limit only applies to fractional/screening designs).
Raises¶
ValueError
If name is empty or reserved (one of :data:RESERVED_FACTOR_NAMES, e.g.
rep, error, verdict, output, tokens), or levels is empty.
Source code in packages/cafe-core/src/cafe/study.py
cafe.FactorType ¶
Bases: str, Enum
How a factor is treated by design generation and (later) statistics.
Source code in packages/cafe-core/src/cafe/study.py
Running & estimating¶
cafe.evaluate
async
¶
evaluate(study, *, concurrency=8, checkpoint_path=None, judge_checkpoint_path=None, on_progress=None, progress=True, _warn_design=True)
Generate answers, judge them, and attribute quality — the whole pipeline.
Shows a progress bar by default (one for answers, one for judging); pass
progress=False to silence it, or on_progress for custom reporting.
checkpoint_path / judge_checkpoint_path make the answer / judging phases
crash-safe and resumable.
Source code in packages/cafe-core/src/cafe/evaluation.py
cafe.run_study
async
¶
run_study(study, *, replications=None, concurrency=8, checkpoint_path=None, resume=True, retry_errors=True, smoke=False, on_progress=None, progress=False)
Run a study end-to-end and return its :class:Results.
Parameters¶
replications:
Repetitions per (config, input). Defaults to study.replications.
Replication is how CAFE measures run-to-run nondeterminism.
concurrency:
Max cells in flight at once.
checkpoint_path:
If given, observations are appended here as they complete and a resumed
run skips already-done cells. If None, the run is in-memory only.
retry_errors:
On resume, re-run checkpointed cells that ended in an error (transient
provider failures heal on the next run; a permanent error just fails
again and stays an error). False treats errored cells as done.
smoke:
Preflight: one input through every config, a single replication, no
judging — to confirm configs execute and to estimate cost before
committing to the full study.
on_progress:
Called as on_progress(observation, completed, total) after each cell.
Source code in packages/cafe-core/src/cafe/execution/runner.py
65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 | |
cafe.preflight
async
¶
Run one input through every configuration (no reps, no judging) and estimate the full study's cost/time.
Source code in packages/cafe-core/src/cafe/evaluation.py
cafe.estimate ¶
Rough full-study time/cost estimate from a smoke/partial run.
Uses mean per-cell time and any cost_usd metadata observed so far,
extrapolated to total_cells. Conservative and meant for the preflight.
Returns¶
dict
sampled_cells, total_cells, labels, and the extrapolated
est_total_compute_s / est_total_cost_usd. An estimate is None only
when nothing was measured (no successful cell with a time, or none reporting a
cost_usd); a measured mean of 0.0 (e.g. a free, cached or local model)
gives an estimate of 0.0.
Source code in packages/cafe-core/src/cafe/execution/runner.py
Results¶
cafe.Evaluation
dataclass
¶
The complete result of evaluating a study: answers, ratings, attribution.
Everything is reachable as plain data: answers.observations (each
Observation), ratings.items (each Rating, with prompt /
raw_response / value_numeric), attribution / effects / clmm,
and the answers.df / ratings.df DataFrames. :meth:records joins it all
into one row-per-verdict view.
Source code in packages/cafe-core/src/cafe/evaluation.py
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 | |
overall_mean
property
¶
Mean verdict across all usable answers (judge replications averaged), or
None if unjudged. The single-number summary — e.g. a pass rate for a binary
rubric. Per-configuration / per-factor breakdowns live on attribution.
marginal_means
property
¶
Per-factor marginal means as a tidy DataFrame (factor, level, mean, n) — the
"which level scores higher" table, ready to drop into a paper. None if
unjudged. (The same data is in attribution.factor_marginals.)
residuals
property
¶
Per-answer residuals from the inferential fit (like R's residuals(fit)),
or None if unjudged. Handy for diagnostics (normality, outliers).
fitted
property
¶
Per-answer fitted values from the inferential fit (like R's fitted(fit)),
or None if unjudged.
variance_components
property
¶
The mixed model's {random_intercept, residual} variances (like R's
VarCorr(fit)), or None if unjudged / only a fixed-effects fit was possible.
effects
property
¶
Inferential statistics (mixed model → F/p, partial η², Cohen's d).
Computed lazily on first access from the ratings; cached thereafter.
Returns None if the study wasn't judged.
clmm
property
¶
Ordinal CLMM (R) for ordinal rubrics. Lazy; returns None if unjudged.
The result's available flag says whether R produced a fit; if not, its
reason explains why (e.g. R not installed) — see cafe doctor.
logistic
property
¶
Binary logistic model (log-odds / odds ratios) for binary pass/fail
rubrics. Lazy; returns None if unjudged. available=False (with a
reason) for non-binary rubrics or degenerate data.
records ¶
One row per judge verdict, joining everything for inspection/export:
question, reference, the factor levels, the model output, per-answer cost/latency,
the full judge prompt (system + user), the raw judge response, and the parsed verdict.
The model's reply is under "output" (not "answer") so it never collides with a
factor named answer (a stage called answer stays a real column).
This is the "give me everything" view — import pandas as pd;
pd.DataFrame(ev.records()) and slice however you like. With no judge it falls
back to one row per answer.
Source code in packages/cafe-core/src/cafe/evaluation.py
report ¶
The full statistical picture in one string: a pipeline summary, the descriptive layer, then the model matched to the rubric's scale type — numeric → linear mixed model, ordinal → linear + cumulative-link mixed model, binary → logistic. So each scale is analysed with its statistically correct model.
interactions is the max interaction order to model (2 = also two-way, the
default; 1 = main effects only). The cheap repr shows only the descriptive
layer (so displaying a result is instant); report() additionally fits the
models, so the first call may take a moment. Print it::
print(result.report())
Source code in packages/cafe-core/src/cafe/evaluation.py
rejudge ¶
rejudge(judge, *, rubric=None, repetitions=1, name=None, concurrency=8, checkpoint_path=None, progress=True)
Score the same answers again — with a different judge, rubric, or
number of repetitions — returning a new Evaluation (reusing this one's
questions/references; nothing is regenerated). This is the "generate once, judge
many ways" path:
free = result.rejudge(cafe.LLMJudge(model=m, preset="single_answer"))
biny = result.rejudge(judge, rubric=cafe.rubrics.CORRECT_PASS_FAIL)
noise = result.rejudge(judge, repetitions=3) # the judge's own spread
checkpoint_path makes the judging crash-safe/resumable (verdicts appended as
they land). Also handy for judge↔judge reliability —
cafe.reliability(raters={"a": result, "b": result.rejudge(judge_b)}).
Source code in packages/cafe-core/src/cafe/evaluation.py
judge_stability ¶
How consistently the judge scored the same answer across its repetitions —
the judge analogue of system replications. Returns a
:class:cafe.stats.JudgeStability (per-answer std dev + a summary). Only
informative when the judge scored answers more than once (judge_replications
on the study, or rejudge(..., repetitions=k))::
noisy = result.rejudge(judge, repetitions=3)
print(noisy.judge_stability().show())
Source code in packages/cafe-core/src/cafe/evaluation.py
pipeline ¶
The composed pipeline (stage order + levels), observed from the run's traces. Only meaningful for a composed system; returns an empty pipeline otherwise.
Source code in packages/cafe-core/src/cafe/evaluation.py
plot ¶
Plots for this evaluation. No argument → a dashboard of the key plots; or
pass a kind: "marginals", "interaction", "configs",
"distribution", "effects", "pareto". Returns the matplotlib Figure
(dashboard) or Axes (single plot) — .savefig(...) or tweak it freely::
result.plot() # everything
result.plot("interaction") # just one
Source code in packages/cafe-core/src/cafe/evaluation.py
cafe.Results
dataclass
¶
All observations from one study run, plus the factor list for context.
Source code in packages/cafe-core/src/cafe/execution/results.py
72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 | |
to_records ¶
Flat rows for a DataFrame/CSV: one column per factor (by its own name), then the answer and timing, then any metadata the system returned.
Columns: <each factor>, input_id, rep, output,
elapsed_s, error, <each metadata key>.
Source code in packages/cafe-core/src/cafe/execution/results.py
to_df ¶
Return a pandas DataFrame (requires the stats extra).
Source code in packages/cafe-core/src/cafe/execution/results.py
cafe.Observation
dataclass
¶
One executed (config x input x replication) cell.
Source code in packages/cafe-core/src/cafe/execution/results.py
Judging¶
cafe.LLMJudge ¶
Scores answers with an LLM (any provider, via a LiteLLM model string).
Raises¶
ValueError
If preset or structured is not a supported value, or prompt_template
cannot be rendered (an unknown placeholder or an unescaped literal brace — write a
literal brace as {{ / }}).
Source code in packages/cafe-core/src/cafe/judging/judge.py
29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 | |
render_messages ¶
The exact message list sent to the judge — the system framing plus the
rubric-derived user prompt. This is the single source of truth shared with
:meth:score, so a preview can never drift from what's actually sent. For a
human-readable view, use :meth:preview.
Parameters¶
structured:
Which call to render. False: the plain call, whose prompt ends by asking
for a GRADE: line. True: the structured call, whose prompt asks for a
JSON verdict instead (if the provider rejects JSON output, the plain call is
sent in its place). None (default): the call this judge makes. That is
fixed by structured=True/False; under "auto" it is known once
:meth:prepare or an earlier check of the same model has decided it, and
until then the plain call is rendered. No capability check is made here;
:meth:preview shows both calls while "auto" is undecided.
Returns¶
list of dict
The system message, then the user message.
Raises¶
ValueError
If structured is not True, False or None.
Source code in packages/cafe-core/src/cafe/judging/judge.py
preview ¶
The full judge input exactly as sent — the system message and the user
prompt, labelled [SYSTEM] / [USER] (no LLM call). This is the way to see
everything the judge receives.
Parameters¶
structured:
As in :meth:render_messages. With None while structured="auto" is
still undecided for this model, both user prompts are shown: the plain one
under [USER], then the structured one, which is sent instead if the model
supports JSON output.
Returns¶
str The labelled messages, separated by blank lines.
Source code in packages/cafe-core/src/cafe/judging/judge.py
prepare
async
¶
Resolve one-time setup before scoring — here, whether to use structured (JSON) output for this model. Idempotent and cached; the judging runner calls it once before the loop so the capability probe never runs per-call.
Source code in packages/cafe-core/src/cafe/judging/judge.py
cafe.Judge ¶
Bases: Protocol
Anything that can score an answer. Implement this for a custom/non-LLM judge.
Source code in packages/cafe-core/src/cafe/judging/judge.py
cafe.Rubric
dataclass
¶
An ordered quality scale + how the judge is prompted.
Set prompt_template to take full control of the judge prompt (placeholders:
{instruction}, {question}, {answer}, {reference}, {scale},
{min}, {max}, {grade} — the scale-aware "allowed final grade" hint).
Any other brace must be doubled (a JSON example is written {{"grade": 4}}): a
template that can't be rendered raises ValueError when the rubric is created.
Leave it None to use the cited default in :mod:cafe.judging.
Raises¶
ValueError
If levels has fewer than two distinct values (one scale point leaves the
judge nothing to choose), or scale_type is not a :class:ScaleType.
Source code in packages/cafe-core/src/cafe/judging/rubric.py
37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 | |
numeric ¶
Map a raw verdict value onto its integer scale point (or None).
For numeric scales any integer within [min_value, max_value] is valid
(the levels are just anchors); for ordinal/binary the value must match a defined
level exactly.
Parameters¶
value:
The judge's raw grade — an int, a float (e.g. from a JSON verdict) or
a numeric string.
Returns¶
int or None
The scale point, or None if value is not on the scale. Grades must be
whole numbers: a fractional float such as 4.5 (or inf/nan) returns
None — it is never truncated to 4. An integral float (4.0) maps to 4.
A boolean (a JSON true/false grade) is not a grade and returns None.
Source code in packages/cafe-core/src/cafe/judging/rubric.py
scale_text ¶
A compact, judge-readable description of the scale.
grade_hint ¶
How to constrain the judge's final grade, matched to the scale type: a range
for numeric (any integer in it is valid — the levels are anchors), or the
exact allowed values for ordinal/binary (only the defined levels are
valid, and they may be non-contiguous, e.g. 1, 3, 5).
Source code in packages/cafe-core/src/cafe/judging/rubric.py
cafe.ScaleType ¶
Bases: str, Enum
How the rubric's numbers should be interpreted (drives the statistics).
Source code in packages/cafe-core/src/cafe/judging/rubric.py
cafe.Level
dataclass
¶
cafe.rubrics ¶
A small library of ready-made rubrics — grab one, or copy it as a template.
Each is a plain :class:cafe.Rubric; nothing here is special. Use them directly::
import cafe
study = cafe.Study(..., rubric=cafe.rubrics.FAITHFULNESS_1_5, judge=...)
or copy one and edit the levels/instruction for your own criterion. Pick the
scale_type deliberately — it decides which statistical model runs (ordinal →
CLMM, numeric → linear, binary → logistic).
cafe.judge_results
async
¶
judge_results(results, judge, rubric, *, repetitions=1, concurrency=8, references=None, questions=None, checkpoint_path=None, resume=True, retry_errors=True, on_progress=None, progress=False)
Judge every successful answer repetitions times.
questions / references map input_id -> text so the judge sees the
original question and any gold answer.
Parameters¶
checkpoint_path:
Makes the (often expensive) judging phase crash-safe: each verdict is appended
as it lands and a resumed run skips already-scored (answer, judge_rep)
pairs — the same guarantee :func:cafe.run_study gives the answer phase. Only
verdicts for the answers judged now, with judge_rep < repetitions, are
carried forward; any other rows in the file are dropped with a warning.
resume:
Load checkpoint_path and skip what it already holds. False judges
everything again (still appending to the file).
retry_errors:
On resume, re-judge checkpointed verdicts that failed (unparseable / judge
error); the retry replaces them. False keeps them as done.
Warns¶
UserWarning
If the checkpoint holds verdicts that don't belong to this run (the answers or
repetitions changed since it was written); they are left out of the result.
Source code in packages/cafe-core/src/cafe/judging/runner.py
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 | |
Statistics¶
cafe.attribute ¶
Compute per-config and per-factor mean quality from judge ratings.
Judge replications of the same answer are averaged first (see
:func:cafe.stats._frame.analysis_frame), so each answer counts once.
Source code in packages/cafe-core/src/cafe/stats/descriptive.py
cafe.fit_effects ¶
Fit the mixed-effects model and return per-term significance + η².
Needs statsmodels (the stats extra). Without it this falls back to a one-way ANOVA
per factor, whose F-tests also need scipy: with neither installed the terms carry
partial η² but no F / p (not significant), and warnings says so — nothing
is raised.
interactions is the maximum interaction order to include: 1 = main effects
only, 2 = also two-way (e.g. model×prompt — the default, since two-way captures
most), 3 = up to three-way. It's auto-capped at the number of factors, and falls
back to main-effects-only if the interaction model can't be estimated on the data.
Returns¶
Effects
Per-term F / p / partial_eta_sq / significant, pairwise Cohen's d,
variance components and notes. On an exact fit (zero residual variance: every
answer within a configuration got the same score) the F-tests are degenerate, so a
term with any effect is reported as F=inf, p=0.0, partial_eta_sq=1.0
(significant) and a term with no effect as None (not significant); model
and warnings say so.
Source code in packages/cafe-core/src/cafe/stats/inferential.py
280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 | |
cafe.fit_clmm ¶
Fit an ordinal CLMM (verdict ~ factors + (1|question)) via R.
interactions is the maximum interaction order (1 = main effects, 2 = also two-way,
…); R falls back to main effects if the interaction model doesn't converge.
lr_tests adds one Type-II likelihood-ratio test per factor and interaction
(factor_tests): the model with every term that doesn't contain it vs. the same model
without it — one p-value for "does this factor matter?", where the coefficients only
compare levels with the baseline. It refits the model a few times (paper study: ~4 s
more); pass False to skip. With few questions (≲ 10) these tests are somewhat
liberal, like any likelihood-ratio test for fixed effects in a mixed model.
Returns¶
CLMMResult
Always returned rather than raised: available=False with a reason when R /
ordinal is missing or there is nothing to fit. Aliased interactions (e.g. a
fractional factorial below resolution V) are dropped with a note in warnings;
without statsmodels that check can't run, and warnings says so instead.
``converged`` is R's optimizer status (``None`` when nothing was fit). If the
optimizer stopped without converging (``converged=False``), every coefficient's
``std_error``, ``z`` and ``p`` are ``None`` and ``significant`` is ``False``, because
the standard errors of such a fit are not valid; ``warnings`` says so.
Source code in packages/cafe-core/src/cafe/stats/ordinal.py
245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 | |
cafe.fit_logistic ¶
Fit a logistic model of pass/fail (verdict ∈ {0,1}) on the factors.
backend selects the engine: "glmer" (logistic GLMM via R lme4), "statsmodels"
(self-contained GEE / logistic), or "auto" (default — GLMM when R+lme4 are present,
else statsmodels). interactions is the max interaction order (1 = main effects,
2 = also two-way); if the GLMM can't fit the interactions, R refits main effects and the
result's formula and warnings say so. Returns a :class:Logistic;
available=False with a reason when the rubric isn't binary, there's no variance,
or no engine is available.
lr_tests (GLMM backend) adds one Type-II likelihood-ratio test per factor and
interaction (factor_tests) — "does it matter at all?", and unlike the Wald p-values
still valid when a factor separates pass from fail. Each reduced model is refit, trying
several optimizers until one converges; pass False to skip. The statsmodels
backends have no such tests.
Returns¶
Logistic
converged is lme4's convergence verdict on the GLMM (None for the statsmodels
backends and when nothing was fit). If the optimizer or lme4's own convergence checks
failed (converged=False), every term's p is None and significant is
False, because the standard errors of such a fit are not valid; warnings
says so.
Source code in packages/cafe-core/src/cafe/stats/logistic.py
389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 | |
cafe.judge_stability ¶
Measure how consistently a judge scores the same answer across its repetitions.
Groups verdicts by answer (obs_key) and, for each answer with ≥2 usable
verdicts, computes the population std dev and the value list. Returns a
:class:JudgeStability with per-answer rows and a summary (mean/max sd, and the
fraction of answers the judge scored identically every time). Meaningful only when
the judge scored answers more than once (judge_replications / repetitions).
Source code in packages/cafe-core/src/cafe/stats/stability.py
cafe.pareto ¶
Multi-objective view: the quality–cost–latency Pareto frontier.
"Best configuration" is rarely one-dimensional — the top-quality config may be the
slowest and most expensive. :func:pareto aggregates each configuration's mean
quality (from the judge), latency (wall-clock), cost (USD), and
tokens, then keeps the non-dominated set: the configs you cannot improve on
one objective without losing on another. Everything else is strictly beaten.
Quality is maximized; cost / latency / tokens are minimized. An objective that doesn't vary across configs (e.g. cost is 0 for a local model) is dropped from the dominance test automatically, so the frontier reflects only what actually trades off.
A cost / tokens / latency value an answer doesn't report is missing, not zero: the
means use only the answers that report it, a configuration reporting it for none of its
answers gets None, and such an objective is left out of the default dominance test
(an unreported cost must not make a configuration look free).
Needs the stats extra (pandas); :meth:ParetoResult.plot needs matplotlib.
ParetoResult
dataclass
¶
Per-configuration objective values and which configs are Pareto-optimal.
A row's quality is the mean over the configuration's answers: judge replications
of an answer are averaged first (as in :func:cafe.attribute), so each answer counts
once however many of its judge calls succeeded.
Source code in packages/cafe-core/src/cafe/stats/pareto.py
plot ¶
Scatter all configs on two objectives, with the frontier highlighted.
Defaults to quality (y) vs cost (x). Returns the matplotlib Axes.
Raises¶
ValueError
If x or y is not reported (None) for some configuration.
Source code in packages/cafe-core/src/cafe/stats/pareto.py
pareto ¶
Compute the quality–cost–latency Pareto frontier over a study's configurations.
Pass the :class:~cafe.evaluation.Evaluation returned by cafe.evaluate.
By default the objectives are quality plus whichever of cost / latency / tokens
actually vary across configurations.
Returns¶
ParetoResult
One row per configuration. latency / cost / tokens are means over the
answers that report them, or None if none of the configuration's answers do;
an objective that is None for any configuration is left out of the default
objectives.
Raises¶
ValueError
If result has no answers/ratings, or an explicitly requested resource objective
is not reported for every configuration.
Source code in packages/cafe-core/src/cafe/stats/pareto.py
Reliability & human ratings¶
cafe.reliability ¶
Inter-rater reliability — is the judge actually measuring quality?
An LLM judge is only worth trusting if it agrees with humans (and with itself, and other judges). This module computes Krippendorff's α — the standard reliability coefficient that handles any number of raters, missing ratings, and ordinal scales — between any set of raters: judge↔human, human↔human, judge↔judge.
Workflow
sheet = cafe.answer_sheet(evaluation)→ one row per answer with a stableanswer_id(plus the output to read). Hand it to your experts.- Collect their scores and load them:
cafe.human_ratings([{answer_id, rater, score}, ...]). cafe.reliability(evaluation, human=...)→ α overall and per rater pair.
α ≥ 0.80 is conventionally reliable; 0.67–0.80 tentative; below that, the judge and
humans don't agree enough to trust the judge's verdicts. Needs the stats extra.
HumanRatings
dataclass
¶
Human (or external) scores keyed by answer_id — the human counterpart of
judge :class:~cafe.judging.ratings.Ratings.
Source code in packages/cafe-core/src/cafe/stats/reliability.py
Reliability
dataclass
¶
Krippendorff's α overall and for each pair of raters.
Source code in packages/cafe-core/src/cafe/stats/reliability.py
krippendorff_alpha ¶
Krippendorff's α for {rater: {unit: value}}. Missing ratings are fine.
metric is the disagreement metric: "ordinal" (default, for rating
scales), "nominal" (categories), or "interval" (numeric distance).
Returns NaN if there's nothing pairable.
Source code in packages/cafe-core/src/cafe/stats/reliability.py
human_ratings ¶
Build :class:HumanRatings from rows of {answer_id, rater, score}.
Accepts a list of dicts, a pandas DataFrame, or a path to a filled-in CSV
(e.g. the sheet from :func:answer_sheet). Rows with a blank score are skipped, so a
partially-rated sheet is fine; a row whose score can't be read as a number is skipped
with a warning (naming the row) rather than aborting the whole import — a stray header
row or a typo like "5?" in one cell shouldn't lose the sheet.
Source code in packages/cafe-core/src/cafe/stats/reliability.py
answer_sheet ¶
A rating sheet: one row per (answer × rater) with a blank score to fill in.
Give it to your experts to rate. Pass path="sheet.csv" to write a CSV they can
open in Excel/Sheets, fill the score column, and save; load it back with
cafe.human_ratings("sheet.csv"). raters names the columns of experts (one row
each). questions maps input_id → question text (defaults to the evaluation's).
Source code in packages/cafe-core/src/cafe/stats/reliability.py
reliability ¶
reliability(evaluation=None, human=None, *, raters=None, table=None, metric=None, judge_label='judge')
Inter-rater reliability across judges and/or human raters.
Common uses::
cafe.reliability(evaluation, human=expert_scores) # judge ↔ humans
cafe.reliability(raters={"120b": result, "20b": result.rejudge(judge_20b)}) # judge ↔ judge
cafe.reliability(human=expert_scores) # humans ↔ each other
raters maps a display name to a rater source — an Evaluation (its judge's
scores) or a {answer_id: score} dict. human is a :class:HumanRatings or rows
for :func:human_ratings. The disagreement metric defaults to the rubric's scale.
Source code in packages/cafe-core/src/cafe/stats/reliability.py
ratings_from_human ¶
Turn human scores into a :class:~cafe.judging.ratings.Ratings, so the whole stats
stack (attribute / fit_effects / fit_clmm / report) runs on humans
instead of the judge. Each rater's score becomes a verdict; several raters on one
answer act like judge replications (averaged before the factor models).
Source code in packages/cafe-core/src/cafe/stats/reliability.py
human_evaluation ¶
An :class:~cafe.evaluation.Evaluation backed by human ratings — so
.report() / .plot() / .effects / .clmm all describe what the humans
found, letting you compare it against the judge-backed evaluation side by side.
Source code in packages/cafe-core/src/cafe/stats/reliability.py
cafe.human_evaluation ¶
An :class:~cafe.evaluation.Evaluation backed by human ratings — so
.report() / .plot() / .effects / .clmm all describe what the humans
found, letting you compare it against the judge-backed evaluation side by side.
Source code in packages/cafe-core/src/cafe/stats/reliability.py
cafe.human_ratings ¶
Build :class:HumanRatings from rows of {answer_id, rater, score}.
Accepts a list of dicts, a pandas DataFrame, or a path to a filled-in CSV
(e.g. the sheet from :func:answer_sheet). Rows with a blank score are skipped, so a
partially-rated sheet is fine; a row whose score can't be read as a number is skipped
with a warning (naming the row) rather than aborting the whole import — a stray header
row or a typo like "5?" in one cell shouldn't lose the sheet.
Source code in packages/cafe-core/src/cafe/stats/reliability.py
cafe.answer_sheet ¶
A rating sheet: one row per (answer × rater) with a blank score to fill in.
Give it to your experts to rate. Pass path="sheet.csv" to write a CSV they can
open in Excel/Sheets, fill the score column, and save; load it back with
cafe.human_ratings("sheet.csv"). raters names the columns of experts (one row
each). questions maps input_id → question text (defaults to the evaluation's).
Source code in packages/cafe-core/src/cafe/stats/reliability.py
cafe.krippendorff_alpha ¶
Krippendorff's α for {rater: {unit: value}}. Missing ratings are fine.
metric is the disagreement metric: "ordinal" (default, for rating
scales), "nominal" (categories), or "interval" (numeric distance).
Returns NaN if there's nothing pairable.
Source code in packages/cafe-core/src/cafe/stats/reliability.py
Designs¶
cafe.full_factorial ¶
Every combination of every factor's levels.
Source code in packages/cafe-core/src/cafe/design/factorial.py
cafe.fractional_factorial_design ¶
Build a resolution-maximizing 2^(k-p) fractional factorial for 2-level factors.
The first log2(runs) factors are the basic factors (a full factorial); every
other factor is "added" as the product of a generator subset of them.
Parameters¶
factors:
The two-level factors (at least 3).
runs:
The fraction's size, a power of two below 2^k. Default: the smallest
design that can place every factor (a resolution-III screening design when
nearly saturated).
generators:
{added_factor: [basic_factor, ...]} to specify the aliasing yourself. If
None, CAFE searches for the minimum-aberration generators (the fewest
shortest defining words, so maximum resolution first; ties go to the design
with fewer words of the next length). The search is deterministic and gives the
textbook maximum resolution for every design up to 128 runs. For larger
designs it stops after a fixed amount of work and keeps the best design found,
which is at least resolution IV whenever runs >= 2k but may fall short of
the maximum when only a few factors are added.
Returns¶
FractionalDesign
The configs plus the generators, defining words, resolution and main-effect
aliases (.show() prints them).
Raises¶
ValueError
If a factor doesn't have exactly two levels, there are fewer than 3 factors,
runs isn't a power of two ≥ 4 below 2^k, the factors don't fit in
runs, or generators misses an added factor.
Source code in packages/cafe-core/src/cafe/design/fractional.py
187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 | |
cafe.size ¶
Composed pipelines (techniques)¶
cafe.Pipeline ¶
A scoped collection of techniques + a compose function, usable as a Study system.
Source code in packages/cafe-core/src/cafe/techniques/pipe.py
39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 | |
technique ¶
technique(stage, name, *, description='', cost_usd=0.0, energy_wh=None, energy_wh_per_1k_tokens=None)
Register fn as the technique name for stage (decorator).
cost_usd is a fixed cost charged every time this technique runs — for a non-LLM
component whose price CAFE can't see (a paid reranker, a web-search API, a human
step). It's added to any LLM cost tracked inside the function and shows up in
stage_report / Pareto / report().
energy_wh / energy_wh_per_1k_tokens declare this technique's energy use:
a fixed Wh per run, and/or Wh per 1k tokens used inside the run (scaled by the
tokens CAFE observes). Energy is strictly opt-in — leave both unset and no energy
is computed or reported anywhere. CAFE never guesses a coefficient; you declare it
(e.g. derived from the model's active parameter count and serving hardware).
Source code in packages/cafe-core/src/cafe/techniques/pipe.py
add ¶
Register a technique programmatically (not as a decorator) — e.g. a deployed
catalog assembling a pipeline per run. Returns self for chaining.
Source code in packages/cafe-core/src/cafe/techniques/pipe.py
compose ¶
Mark fn(config, item, ctx) as this pipeline's system function (decorator).
fn returns the answer, or {"output": answer, "cost_usd": ..., ...} to report
a cost spent outside any stage (see :meth:run).
Source code in packages/cafe-core/src/cafe/techniques/pipe.py
names_for ¶
Technique names registered under stage (registration order).
stages ¶
Stages that have at least one registered technique (registration order).
factor ¶
A categorical factor whose levels are stage's registered techniques.
pipe.factor("retrieve") → Factor("retrieve", ["dense", "rerank"]).
Pass none= to add a "skip this stage" level without a no-op technique — the honest
"does this stage help at all?" contrast (only valid when the stage passes its input
through, i.e. output type == input type):
none="chunks"→ the skip level returns thechunksinput unchanged;none=None→ the skip level contributes nothing / returnsNone.
Source code in packages/cafe-core/src/cafe/techniques/pipe.py
run
async
¶
Execute the compose function for one (config, item), returning the answer plus per-stage trace + total cost/tokens as metadata (same contract as any system).
Returns¶
dict
{"output", "cost_usd", "tokens", "trace"}, plus "energy_wh" once energy
was declared. The totals are summed over the stages that ran, plus any
cost_usd / tokens / energy_wh the compose function reports itself by
returning {"output": answer, "cost_usd": ..., ...} — spent outside any
stage, so not in trace. Other keys of that dict are kept as metadata;
trace is always the pipeline's own.
Source code in packages/cafe-core/src/cafe/techniques/pipe.py
cafe.pipeline ¶
Derive the :class:PipelineView (stage order + levels) from a run.
source may be an :class:~cafe.evaluation.Evaluation, a Results, or a
Study. Given a study, CAFE runs a single configuration once (cheap) to observe
the topology — the order comes from that trace; the levels come from the study's
factors. Given an already-run result, no new run happens.
Source code in packages/cafe-core/src/cafe/techniques/composed.py
cafe.stage_report ¶
Aggregate the traces in a Results object into per-stage timing + cost.
Returns one row per (stage, technique): mean elapsed, mean cost, count — the
basis for per-stage statistics ("which stage is slowest / most expensive").
A repeated identical call that a run served from its stage cache (cached: True
in the trace) is not counted again.
Source code in packages/cafe-core/src/cafe/techniques/composed.py
LLM & datasets¶
cafe.complete
async
¶
Run one chat completion and return the assistant's text content.
Source code in packages/cafe-core/src/cafe/llm.py
cafe.set_model_cost ¶
Override the price CAFE uses for model (USD per 1,000 tokens).
Use per_1k_tokens for a single blended rate, or per_1k_input /
per_1k_output for separate prompt/completion rates. This takes priority over
LiteLLM's automatic pricing — set it for subscriptions, negotiated/enterprise
rates, self-hosted models, or providers LiteLLM doesn't price. Pass all-None to
clear the override.
Source code in packages/cafe-core/src/cafe/llm.py
cafe.datasets ¶
Load public benchmarks as CAFE datasets.
Each loader returns a list of items shaped for Study(dataset=...):
{"id", "text", "reference", ...} — so the reference-guided judge has a gold
answer to score against. Needs the datasets extra (HuggingFace datasets):
pip install 'cafe-core[datasets]'.
load_truthfulqa ¶
TruthfulQA — adversarial questions that elicit common misconceptions.
Reference-based: each item carries the gold Best Answer. Returns items
{id, text, reference, category}. categories filters by Category
(e.g. ["Misconceptions", "Law"]).
Source code in packages/cafe-core/src/cafe/datasets.py
load_gsm8k ¶
GSM8K — grade-school math word problems with an exact numeric reference.