Skip to content

Quickstart

Install

Requires Python ≥ 3.11 and R — the mixed-effects models (ordinal CLMM, logistic GLMM) run in R.

There are two ways to install the library. If you just want to use it, install it from PyPI:

pip install "cafe-core[all]"    # library + LLM judge + datasets + notebooks

If you also want the example notebooks, the example techniques and the web app, or want to change the code, clone the repository and install the library in editable mode:

git clone https://github.com/fabian-lu/Cafe.git
cd Cafe
pip install -e "packages/cafe-core[all]"

Either way, install R and its two model packages:

# Debian/Ubuntu shown; macOS: `brew install r`
sudo apt install r-base
Rscript -e 'install.packages(c("ordinal", "lme4"))'

Verify the environment before your first run:

cafe doctor    # checks Python, the R stats engine, and LLM access

Run the bundled example (no API keys)

CAFE ships a neutral toy system so you can see a full study run with nothing to configure:

cafe run example            # 4 configs x 4 inputs x 3 reps
cafe run example --smoke    # preflight: 1 input, 1 rep, + cost/time estimate
cafe validate example       # expand the design without running anything

You'll see a per-configuration table with replication counts, latency, cost, and (for the toy system) a simulated quality signal.

Your first real study

A study is your system (a black box), the factors to vary, and the inputs to evaluate on:

import cafe

async def my_system(config, item):
    # Your compound system: RAG, routing, a cascade, an agent — anything.
    # Read the chosen levels from `config` and do whatever you want.
    model = config["model"]
    return f"[{model}] answer to: {item}"

study = cafe.Study(
    name="my-first-study",
    system=my_system,
    factors=[
        cafe.Factor("model", ["small", "large"]),
        cafe.Factor("prompt", ["plain", "cot"]),
    ],
    dataset=["What is 2+2?", "Summarize relativity."],
    replications=3,                       # measure run-to-run nondeterminism
)

results = study.run(checkpoint_path=".cafe/my-first-study.jsonl")
print(results.summary())
for obs in results:
    print(obs.config, "->", obs.output, f"({obs.elapsed_s}s)")

study.run() returns a Results object you keep in a variable, like any normal value. The optional checkpoint_path makes a long run crash-safe: if it dies halfway, re-running resumes instead of restarting.

From answers to attribution

run() gives you the raw answers. To get the actual payoff — which factor drives quality, and is the difference real — add a rubric and a judge, then call evaluate() instead:

study = cafe.Study(
    name="my-first-study",
    system=my_system,
    factors=[cafe.Factor("model", ["small", "large"]),
             cafe.Factor("prompt", ["plain", "cot"])],
    dataset=["What is 2+2?", "Summarize relativity."],
    rubric=cafe.rubrics.CORRECTNESS_0_3,                 # the scale to grade on
    judge=cafe.LLMJudge(model="ollama_cloud/gpt-oss:20b"),
    replications=3,
)

result = study.evaluate()        # answers → judge → attribution
print(result.report())           # descriptive means, significance, effect sizes, the CLMM

report() prints per-factor significance (F-tests, p-values, partial η²), effect sizes, the best configuration, and — for ordinal rubrics — the cumulative-link mixed model. Interpreting results walks through that output line by line.

Headless / CI

The same study runs from a Python file via the CLI:

cafe run path/to/study.py --checkpoint .cafe/run.jsonl --out results.jsonl

where study.py defines a module-level study (a cafe.Study) or a build_study() function returning one.