The web app¶
CAFE ships a self-hostable web platform over the same engine as the library: define
a study in the browser, run it with live progress, and explore the full analytics —
then collect human ratings. Anything you can do here you can also do in Python; the app
just wraps cafe-core behind a FastAPI (JSON) API and a React front-end, backed by
Postgres. No data leaves your machine.
Prefer to look before installing? The live demo is a read-only snapshot of a finished study.
Run it¶
The platform runs entirely in containers — you only need Docker + Docker Compose on the host (Python, R, Node, and Postgres all run inside):
cd apps/web-app
cp ../../.env.example ../../.env # add your LLM keys to the repo-root .env
docker compose up
Then open http://localhost:5173. The first build takes a few minutes (it installs R
and the front-end deps); subsequent starts are fast. Keys come from the repo-root
.env — the same file the library and CLI use.
Your system under test¶
The app evaluates the pipelines in the repo-root techniques/
folder, which is mounted into the backend. Every .py file there whose module-level
pipe (a cafe.Pipeline) has a @pipe.compose is one selectable pipeline, named after
the file; files without a compose are technique libraries. The app discovers them at
startup, and the study form builds its factors from the chosen pipeline's stages and
techniques.
To evaluate your own system, drop your techniques into that folder (or point
CAFE_TECHNIQUES_DIR at your own), then hit Reload on the Techniques page — no
restart needed. See Define your system for how to write the
pipeline.
Already ran a study in Python? Import study on the Studies page takes a
cafe.save_evaluation bundle (or a multi-dimension bundle holding one per rubric) and
shows it like any other study, without re-running anything.
The flow¶
- Techniques — the discovered pipelines; pick one to see its stages, the techniques on each, and the factors they become. Reload picks up edits to the techniques folder.
- Questions & Rubrics — pick the inputs to evaluate on and the scale to grade with
(the built-in rubrics, or your own). Rubrics and studies are checked by cafe-core when
you save them: a definition it would reject — say, a custom judge prompt with a stray
{(write{{/}}for a literal brace) — is refused right away with cafe-core's message, instead of failing when the study runs. - New study — choose which stages/parameters to vary as factors, the questions, a judge and rubric, and the number of replications. CAFE shows the configuration count and a cost/time estimate before you commit.
- Run — launch it and watch live progress stream as cells complete. A run executes inside the backend process: if the backend restarts mid-run, the study is marked failed and can be re-run from its page (the run starts over).
- Results — the same analysis as
report(), in the browser: per-factor attribution (F, p, partial η², variance share), effect sizes, per-level marginal means, and the best configuration. A study created with extra rubrics (each judges the same answers as its own dimension) gets a dimension selector; extra fields on the questions (2–30 distinct values) become filters that recompute the statistics on that subset; energy is shown when the pipeline's techniques declare it. (See Interpreting results for how to read these.) - Raters — collect human ratings by name and see judge↔human reliability, the
platform version of the human-ratings workflow. A rating belongs
to the answer text it was given on: re-running the study regenerates the answers, so
earlier ratings stop counting toward α until you re-rate (the reliability API reports
them as
stale_ratings).
Local development (without Docker)¶
# backend
cd apps/web-app/backend
pip install -r requirements.txt && pip install -e ../../../packages/cafe-core
uvicorn app.main:app --reload
# frontend (another shell)
cd apps/web-app/frontend && npm install && npm run dev
Requires Python ≥ 3.11 + R (for the stats layer) and Node for the front-end. Without a
DATABASE_URL the backend keeps its data in a local SQLite file, ./cafe.db (a file
from an older version is upgraded in place on startup); set DATABASE_URL to use
Postgres like the Docker setup. The Docker path bundles all of these, which is why it is the
recommended way to run the app.
Run a single backend process per database (no --workers): runs and their live
progress live in that process, and on startup it marks any study still "running" as
failed, which would also hit another worker's live runs.