Case study 3 · Invoice processing¶
The question a finance team asks: do we need a large model to extract booking data from our invoices, or does a small one do the job? And do the other steps of the pipeline matter: how the PDF is turned into text, and whether a second step checks the result?
The setup: 106 invoices from the Invoice Extraction Benchmark (synthetic but realistic invoices from the US, UK, EU and southern Africa, including credit notes, mixed currencies and multi-page invoices). The system reads each PDF, turns it into text, and extracts the booking data as JSON. Three factors:
| Factor | Levels |
|---|---|
model |
gemma-3-4b · gpt-oss-20b · gemma-4-31b · gpt-oss-120b · deepseek-v4-pro |
pdf_text |
with_layout (columns stay aligned) · without_layout (plain reading order, pypdf's default) |
prompting |
extract_only · extract_then_check (a second call checks and corrects the result) |
That gives 5 × 2 × 2 = 20 configurations × 106 invoices = 2,120 extractions. An invoice counts as ready to book only if every extracted field is right.
What this study shows in CAFE: black-box mode (the system is ordinary code that knows nothing about CAFE), a judge written in plain code instead of an LLM, a correct/wrong rubric analysed with the logistic mixed model, filters by invoice category, and the cost and energy of each model.
import json
import pandas as pd
import cafe
cafe.load_env(); # the API keys from the repo's .env
1 · The data¶
The invoices and their correct booking data (created once by data/prepare_data.py). The
category (clean, currency, tax, credit_note, freight, multipage, hard) and difficulty become
filters in the web app.
from invoice_system import INVOICE_FOLDER, pdf_to_text
invoices = json.load(open("data/correct_fields.json"))
dataset = [
{
"id": invoice["id"],
# the invoice as text, so it can be read next to every answer in the web app
# (the system itself reads the PDF)
"text": pdf_to_text(pdf_path=INVOICE_FOLDER / f"{invoice['id']}.pdf", keep_layout=True),
"reference": json.dumps(invoice["correct_fields"], indent=1),
"category": invoice["category"],
"difficulty": invoice["difficulty"],
}
for invoice in invoices
]
print(f"{len(dataset)} invoices. The correct booking data of the first one:")
print(dataset[0]["reference"])
106 invoices. The correct booking data of the first one:
{
"vendor_name": "Acme Industrial Supply Co",
"document_number": "ACM-100145",
"invoice_date": "2024-03-12",
"currency": "USD",
"tax": 25.99,
"total": 372.49,
"document_type": "invoice"
}
2 · The system¶
The invoice pipeline lives in invoice_system.py: PDF → text → extract →
(optionally) check. It is plain Python with no CAFE inside, like an existing company system. CAFE
only calls process_invoice(config, invoice) and reads what it returns: the extracted data plus
the cost, tokens and energy the system reports about itself.
Here is what the two pdf_text options make of the same invoice:
example = INVOICE_FOLDER / "currency-dual-usd-nad-4.pdf"
print("── with_layout ──\n", pdf_to_text(pdf_path=example, keep_layout=True)[:1500])
print("\n── without_layout ──\n", pdf_to_text(pdf_path=example, keep_layout=False)[:700])
── with_layout ──
Kunene Clearing & Forwarding (Pty) Ltd
325 Hage Geingob Street
Windhoek
Tel +264 80 555 0484 | billing@kuneneclearingforw.com.na
VAT Reg No: 4139321248
INVOICE
Invoice #: F-26-9064 PO Number: 4500421384
Date: 16/03/2026 Terms: 30 days
Due Date: 15/04/2026 Currency: USD
BILL TO
Etosha Gate Engineering Supplies CC
510 Nickel Street
Swakopmund
VAT Reg No: 4880697625
Description Qty Unit USD Amount USD Amount NAD
Documentation fee 1 26.02 26.02 478.56
Customs clearance fee 1 136.39 136.39 2,508.48
Cargo inspection attendance 1 32.30 32.30 594.06
Subtotal US$ 194.71
TOTAL USD US$ 194.71
Exchange rate: 1 USD = 18.3920 NAD
── without_layout ──
Kunene Clearing & Forwarding (Pty) Ltd
325 Hage Geingob Street
Windhoek
Tel +264 80 555 0484 | billing@kuneneclearingforw.com.na
VAT Reg No: 4139321248
INVOICE
Invoice #:
F-26-9064
Date:
16/03/2026
Due Date:
15/04/2026
PO Number:
4500421384
Terms:
30 days
Currency:
USD
BILL TO
Etosha Gate Engineering Supplies CC
510 Nickel Street
Swakopmund
VAT Reg No: 4880697625
Description
Qty
Unit USD
Amount USD
Amount NAD
Documentation fee
1
26.02
26.02
478.56
Customs clearance fee
1
136.39
136.39
2,508.48
Cargo inspection attendance
1
32.30
32.30
594.06
Subtotal
US$ 194.71
TOTAL USD
US$ 194.71
Exchange rate: 1 USD = 18.3920 NAD
TOTAL INVOICE AMOUNT PAYABLE
NAD
N$ 3,581.10
Banking details: Bank: Kunen
from invoice_system import MODELS, process_invoice
3 · The factors¶
In black-box mode the factors are simply names and levels; the system reads them from its config.
factors = [
cafe.Factor(name="model", levels=list(MODELS)),
cafe.Factor(name="pdf_text", levels=["with_layout", "without_layout"]),
cafe.Factor(name="prompting", levels=["extract_only", "extract_then_check"]),
]
4 · Rubric and judge¶
Two outcomes: the invoice is ready to book, or someone has to correct it. Because the rubric is binary, CAFE analyses it with a logistic mixed model.
The judge is plain code (invoice_judge.py): it compares every field with the
correct value, tolerating differences that don't matter (date and number formats, upper/lower case,
legal forms like "Ltd", currency symbols). Its reasoning lists the wrong fields.
from invoice_judge import FieldCheckJudge
ready_to_book = cafe.Rubric(
name="ready_to_book",
scale_type=cafe.ScaleType.binary,
levels=[
cafe.Level(value=0, label="needs_correction", description="At least one field is wrong: someone has to fix it before booking."),
cafe.Level(value=1, label="ready_to_book", description="Every checked field is correct: the invoice can be booked as extracted."),
],
instruction="Check whether every extracted field matches the correct booking data.",
)
judge = FieldCheckJudge()
5 · Run the study¶
First the plan and a quick preflight (one invoice through every configuration, with a time and cost estimate), then the full run. The run saves its progress as it goes: if it is interrupted, or some extractions failed (e.g. a provider was briefly rate-limited), running the cell again continues where it stopped and retries only the failed ones.
# A study = the system, the factors to vary, the inputs, and how to score the answers.
study = cafe.Study(
name="invoice-processing",
system=process_invoice,
factors=factors,
dataset=dataset,
rubric=ready_to_book,
judge=judge,
)
print(study.plan()) # how many configurations, answers and judge calls (nothing runs yet)
print(study.preflight(concurrency=4)) # one input through every configuration: errors, time, cost
{'name': 'invoice-processing', 'design': 'full_factorial', 'factors': ['model', 'pdf_text', 'prompting'], 'configs': 20, 'items': 106, 'replications': 1, 'runs': 2120, 'judged': True, 'judge_calls': 2120}
Preflight(answers=Results(invoice-processing: 20 answers, 20 configs, 0 errors), estimate={'sampled_cells': 20, 'total_cells': 2120, 'est_total_compute_s': 6792.08, 'est_total_cost_usd': 0.806, 'labels': ['model=gemma-3-4b·pdf_text=without_layout·prompting=extract_only']}, warnings=[], judge_calls=2120)
# Run every invoice through every configuration, judge every answer, then fit the statistics.
result = study.evaluate(
concurrency=4, # 4 at a time: Ollama Cloud rejects too many parallel requests
checkpoint_path="results/extractions.cafe.jsonl",
judge_checkpoint_path="results/verdicts.cafe.jsonl",
)
invoice-processing: answers: 100% 2120/2120 [41:02<00:00, 1.16s/it]
judging: 100% 2120/2120 [00:01<00:00, 1163.81it/s]
6 · The results¶
The full report: share of invoices ready to book per configuration, attribution (which factor explains how much), and the logistic mixed model with one test per factor.
print(result.report())
2120 answers · 20 configs · 106 inputs · 2120 ratings · best: model=deepseek-v4-pro·pdf_text=with_layout·prompting=extract_then_check
pipeline: 2120 answers → 2120 judged → 2120 usable verdict(s)
────────────────────────────────────────────────────────────
DESCRIPTIVE — means & best configuration
────────────────────────────────────────────────────────────
verdicts: 2120 factors: model, pdf_text, prompting
overall mean quality: 0.91 (n=2120)
per-configuration mean quality:
0.98 (n=106) model=deepseek-v4-pro·pdf_text=with_layout·prompting=extract_then_check
0.98 (n=106) model=deepseek-v4-pro·pdf_text=without_layout·prompting=extract_only
0.98 (n=106) model=deepseek-v4-pro·pdf_text=without_layout·prompting=extract_then_check
0.98 (n=106) model=gemma-4-31b·pdf_text=with_layout·prompting=extract_then_check
0.98 (n=106) model=gemma-4-31b·pdf_text=without_layout·prompting=extract_then_check
0.98 (n=106) model=gpt-oss-120b·pdf_text=with_layout·prompting=extract_then_check
0.98 (n=106) model=gpt-oss-120b·pdf_text=without_layout·prompting=extract_then_check
0.97 (n=106) model=deepseek-v4-pro·pdf_text=with_layout·prompting=extract_only
0.97 (n=106) model=gemma-4-31b·pdf_text=with_layout·prompting=extract_only
0.97 (n=106) model=gpt-oss-120b·pdf_text=with_layout·prompting=extract_only
0.97 (n=106) model=gpt-oss-20b·pdf_text=without_layout·prompting=extract_then_check
0.96 (n=106) model=gemma-4-31b·pdf_text=without_layout·prompting=extract_only
0.96 (n=106) model=gpt-oss-20b·pdf_text=with_layout·prompting=extract_only
0.95 (n=106) model=gpt-oss-20b·pdf_text=without_layout·prompting=extract_only
0.94 (n=106) model=gpt-oss-120b·pdf_text=without_layout·prompting=extract_only
0.94 (n=106) model=gpt-oss-20b·pdf_text=with_layout·prompting=extract_then_check
0.70 (n=106) model=gemma-3-4b·pdf_text=without_layout·prompting=extract_then_check
0.69 (n=106) model=gemma-3-4b·pdf_text=without_layout·prompting=extract_only
0.64 (n=106) model=gemma-3-4b·pdf_text=with_layout·prompting=extract_only
0.61 (n=106) model=gemma-3-4b·pdf_text=with_layout·prompting=extract_then_check
per-factor marginal means:
model:
deepseek-v4-pro mean=0.98 n=424
gemma-3-4b mean=0.66 n=424
gemma-4-31b mean=0.97 n=424
gpt-oss-120b mean=0.97 n=424
gpt-oss-20b mean=0.96 n=424
pdf_text:
with_layout mean=0.90 n=1060
without_layout mean=0.91 n=1060
prompting:
extract_only mean=0.90 n=1060
extract_then_check mean=0.91 n=1060
best configuration: model=deepseek-v4-pro·pdf_text=with_layout·prompting=extract_then_check (mean 0.98)
────────────────────────────────────────────────────────────
LOGISTIC — binary pass/fail model (log-odds & odds ratios)
────────────────────────────────────────────────────────────
logistic — verdict ~ (1 | input_id) + (model + pdf_text + prompting)^2 (n=2120, α=0.05)
logistic GLMM (binomial, logit link; random intercept: question) [R lme4::glmer]
per-factor tests (likelihood ratio, Type II — does the factor matter at all?):
term χ² df p
model 486.53 4 0.0000 ***
model × prompting 7.92 4 0.0947 .
model × pdf_text 4.15 4 0.3860
pdf_text 2.94 1 0.0862 .
pdf_text × prompting 1.41 1 0.2348
prompting 0.85 1 0.3561
fixed effects (log-odds of a PASS; + = more likely; OR = odds ratio):
term log-odds OR p
model=gemma-3-4b -7.042 0.00 0.0000 ***
model=gemma-4-31b -0.336 0.71 0.8125
model=gpt-oss-120b -0.473 0.62 0.7333
model=gpt-oss-20b -1.505 0.22 0.2199
pdf_text=without_layout +0.788 2.20 0.5962
prompting=extract_then_check +0.788 2.20 0.5965
model=gemma-3-4b × pdf_text=without_layout -0.398 0.67 0.7916
model=gemma-4-31b × pdf_text=without_layout -1.557 0.21 0.4043
model=gpt-oss-120b × pdf_text=without_layout -2.313 0.10 0.2044
model=gpt-oss-20b × pdf_text=without_layout -0.579 0.56 0.7252
model=gemma-3-4b × prompting=extract_then_check -1.205 0.30 0.4243
model=gemma-4-31b × prompting=extract_then_check +1.051 2.86 0.5941
model=gpt-oss-120b × prompting=extract_then_check +1.590 4.90 0.4110
model=gpt-oss-20b × prompting=extract_then_check -1.065 0.34 0.5171
pdf_text=without_layout × prompting=extract_then_check +0.656 1.93 0.2353
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
variance components: between-question σ_u² = 15.655 (σ_u = 3.957) · latent residual π²/3 = 3.290 · ICC = 0.826
note: lme4's default optimizer did not converge; refit with bobyqa, which did — the results are from that fit
Does the checking step help every model, or only some?
result.plot(kind="interaction", factors=["model", "prompting"]);
Does keeping the layout matter, and for which models?
result.plot(kind="interaction", factors=["model", "pdf_text"]);
Quality against cost: the configurations on the Pareto front are the only ones worth considering.
result.plot(kind="pareto");
Share of invoices ready to book, per invoice category and model: where do the small models fail?
extractions = pd.DataFrame(result.records())
category = {item["id"]: item["category"] for item in dataset}
extractions["category"] = extractions["input_id"].map(category)
extractions.pivot_table(index="category", columns="model", values="verdict", aggfunc="mean").round(2)
| model | deepseek-v4-pro | gemma-3-4b | gemma-4-31b | gpt-oss-120b | gpt-oss-20b |
|---|---|---|---|---|---|
| category | |||||
| clean | 1.00 | 0.62 | 1.00 | 1.00 | 0.99 |
| credit_note | 1.00 | 0.75 | 1.00 | 1.00 | 1.00 |
| currency | 0.99 | 0.57 | 0.96 | 0.93 | 0.91 |
| freight | 1.00 | 0.73 | 1.00 | 1.00 | 0.98 |
| hard | 1.00 | 0.64 | 1.00 | 1.00 | 1.00 |
| multipage | 1.00 | 0.62 | 1.00 | 1.00 | 0.98 |
| tax | 0.86 | 0.75 | 0.86 | 0.86 | 0.86 |
Cost, energy and time per model, summed over all its extractions:
extractions.groupby("model")[["cost_usd", "energy_wh", "elapsed_s"]].sum().round(4)
| cost_usd | energy_wh | elapsed_s | |
|---|---|---|---|
| model | |||
| deepseek-v4-pro | 0.9937 | 395.2340 | 3033.0618 |
| gemma-3-4b | 0.0269 | 32.5865 | 820.6519 |
| gemma-4-31b | 0.1463 | 195.8763 | 761.5681 |
| gpt-oss-120b | 0.2270 | 112.2531 | 1188.9065 |
| gpt-oss-20b | 0.0367 | 47.9213 | 4022.3002 |
7 · Save for the web app¶
One file with everything. In the web app, Studies → Import study turns it into a browsable study; the invoice text, the extracted JSON and the list of wrong fields are shown side by side.
cafe.save_evaluation(evaluation=result, path="results/invoice_processing_results.json")