Case study: do you need a premium model for customer support?¶

The question a real team asks: we're paying for a big model to power our support bot — would a cheaper one be just as good?

We put three models of increasing size/cost on a level playing field with CAFE, answering real customer-support tickets, each run with a bare prompt vs a proper support system-prompt. An independent LLM judge rates every reply for accuracy, helpfulness, and tone (1–5). CAFE then tells us which differences are real (not run-to-run noise), whether the expensive model actually earns its price, and whether a good system-prompt matters more than model size.

Run this in the repo venv (editable cafe). It calls models via your Ollama Cloud + OpenRouter keys — start with N = 3 to smoke-test for pennies, then set N = 30 for the real study.

1. Setup¶

In [39]:

Copied!





import cafe
from cafe._env import load_env
load_env()   # read your nearest .env (OLLAMA_API_KEY, OPENROUTER_API_KEY)

print("cafe:", cafe.__file__, cafe.__version__)

# How many support tickets to sample. 3 = cheap smoke test; 30 = the real study.
N = 100

# Three models, increasing in size/cost — all verified live on Ollama Cloud.
MODELS = {
    "gpt-oss:20b":     "ollama_cloud/gpt-oss:20b",     # cheap / small
    "gpt-oss:120b":    "ollama_cloud/gpt-oss:120b",    # mid
    "deepseek-v4-pro": "ollama_cloud/deepseek-v4-pro", # expensive flagship
}

# Real market prices (OpenRouter, USD per 1M tokens), split input/output — CAFE tracks prompt vs
# completion tokens separately. Note: gpt-oss 20b and 120b cost ~the same; deepseek-v4-pro is
# ~15-20x pricier. (We call these via Ollama Cloud, which does not report cost, so we set the
# public market price as the stand-in for the cost/quality frontier.)
COSTS = {  # (input $/1M, output $/1M)
    "gpt-oss:20b":     (0.03, 0.13),
    "gpt-oss:120b":    (0.03, 0.17),
    "deepseek-v4-pro": (0.66, 1.98),
}
for short, tag in MODELS.items():
    cin, cout = COSTS[short]
    cafe.set_model_cost(tag, per_1k_input=cin/1000, per_1k_output=cout/1000)  # per-1M -> per-1k

# An independent, DIFFERENT-FAMILY judge (Anthropic) via OpenRouter — avoids same-family bias.
JUDGE_MODEL = "openrouter/anthropic/claude-opus-5"
import cafe
from cafe._env import load_env
load_env()   # read your nearest .env (OLLAMA_API_KEY, OPENROUTER_API_KEY)

print("cafe:", cafe.__file__, cafe.__version__)

# How many support tickets to sample. 3 = cheap smoke test; 30 = the real study.
N = 100

# Three models, increasing in size/cost — all verified live on Ollama Cloud.
MODELS = {
    "gpt-oss:20b":     "ollama_cloud/gpt-oss:20b",     # cheap / small
    "gpt-oss:120b":    "ollama_cloud/gpt-oss:120b",    # mid
    "deepseek-v4-pro": "ollama_cloud/deepseek-v4-pro", # expensive flagship
}

# Real market prices (OpenRouter, USD per 1M tokens), split input/output — CAFE tracks prompt vs
# completion tokens separately. Note: gpt-oss 20b and 120b cost ~the same; deepseek-v4-pro is
# ~15-20x pricier. (We call these via Ollama Cloud, which does not report cost, so we set the
# public market price as the stand-in for the cost/quality frontier.)
COSTS = {  # (input $/1M, output $/1M)
    "gpt-oss:20b":     (0.03, 0.13),
    "gpt-oss:120b":    (0.03, 0.17),
    "deepseek-v4-pro": (0.66, 1.98),
}
for short, tag in MODELS.items():
    cin, cout = COSTS[short]
    cafe.set_model_cost(tag, per_1k_input=cin/1000, per_1k_output=cout/1000)  # per-1M -> per-1k

# An independent, DIFFERENT-FAMILY judge (Anthropic) via OpenRouter — avoids same-family bias.
JUDGE_MODEL = "openrouter/anthropic/claude-opus-5"

cafe: /home/fabian/Desktop/PHD/CAFE/packages/cafe-core/src/cafe/__init__.py 0.0.1

2. Dataset — Bitext customer support¶

Real customer-support messages (bitext/Bitext-customer-support-llm-chatbot-training-dataset), spanning 11 categories (orders, refunds, payments, shipping…), each with a reference answer. The data keeps privacy placeholders like {{Order Number}} / {{Invoice Number}} — we keep them and simply tell the models + judge to treat them as real values (see below).

In [40]:

Copied!





from datasets import load_dataset

bitext = load_dataset("bitext/Bitext-customer-support-llm-chatbot-training-dataset", split="train")
sample = bitext.shuffle(seed=0).select(range(N))   # a spread across categories

dataset = [
    {"id": f"q{i}", "text": ex["instruction"], "reference": ex["response"], "category": ex["category"]}
    for i, ex in enumerate(sample)
]
print(f"{len(dataset)} tickets; categories: {sorted({d['category'] for d in dataset})}")
print("example ticket:", dataset[0]["text"])
print("reference:", dataset[0]["reference"][:160], "...")
from datasets import load_dataset

bitext = load_dataset("bitext/Bitext-customer-support-llm-chatbot-training-dataset", split="train")
sample = bitext.shuffle(seed=0).select(range(N))   # a spread across categories

dataset = [
    {"id": f"q{i}", "text": ex["instruction"], "reference": ex["response"], "category": ex["category"]}
    for i, ex in enumerate(sample)
]
print(f"{len(dataset)} tickets; categories: {sorted({d['category'] for d in dataset})}")
print("example ticket:", dataset[0]["text"])
print("reference:", dataset[0]["reference"][:160], "...")

100 tickets; categories: ['ACCOUNT', 'CANCEL', 'CONTACT', 'DELIVERY', 'FEEDBACK', 'INVOICE', 'ORDER', 'PAYMENT', 'REFUND', 'SHIPPING', 'SUBSCRIPTION']
example ticket: there are errors setting up a delivery address
reference: I'm aware that you're encountering errors while setting up your delivery address. I apologize for any inconveniences caused. To assist you further, I would need ...

3. The support chatbot (the system under test)¶

The model answers a customer message. Two techniques on one answer stage — a bare prompt vs a proper support system-prompt — become the prompt factor; the model argument becomes the model factor. Both share a note telling the model that {{...}} placeholders are privacy redactions to treat as real values.

In [41]:

Copied!





pipe = cafe.Pipeline()

# Shared by both techniques: (1) a hard length cap so we're comparing SUBSTANCE, not verbosity
# (LLM judges reward length — capping it levels the field); (2) a note that {{...}} placeholders
# are privacy redactions to treat as real values. The ONLY difference between the two techniques
# is the support policy — a clean prompt factor.
LENGTH = "Keep your reply under 300 characters."
DATA_NOTE = ("The customer's message may contain redacted placeholders in double braces (e.g. "
    "{{Order Number}}, {{Invoice Number}}, {{Customer Support Phone Number}}) standing in for the "
    "customer's real details or company info, hidden for privacy. Treat them as if they hold real "
    "values, and you may reuse the same {{...}} tokens in your reply.")

@pipe.technique("answer", "plain", description="Bare model — no support guidelines.")
async def plain(ctx, message, model="gpt-oss:20b"):
    return await cafe.complete(MODELS[model], [
        {"role": "system", "content": f"You are a customer-support assistant. {LENGTH} {DATA_NOTE}"},
        {"role": "user", "content": message},
    ], temperature=0.3)

@pipe.technique("answer", "policy", description="Same, plus a support system-prompt (the actionable lever).")
async def policy(ctx, message, model="gpt-oss:20b"):
    return await cafe.complete(MODELS[model], [
        {"role": "system", "content":
            "You are a customer-support assistant. Be accurate, give the key next step, keep a "
            "professional and empathetic tone, and offer to escalate to a human if the request needs "
            f"account access or you are unsure. {LENGTH} {DATA_NOTE}"},
        {"role": "user", "content": message},
    ], temperature=0.3)

@pipe.compose
async def run(config, item, ctx):
    return await ctx.run("answer", message=item["text"])
pipe = cafe.Pipeline()

# Shared by both techniques: (1) a hard length cap so we're comparing SUBSTANCE, not verbosity
# (LLM judges reward length — capping it levels the field); (2) a note that {{...}} placeholders
# are privacy redactions to treat as real values. The ONLY difference between the two techniques
# is the support policy — a clean prompt factor.
LENGTH = "Keep your reply under 300 characters."
DATA_NOTE = ("The customer's message may contain redacted placeholders in double braces (e.g. "
    "{{Order Number}}, {{Invoice Number}}, {{Customer Support Phone Number}}) standing in for the "
    "customer's real details or company info, hidden for privacy. Treat them as if they hold real "
    "values, and you may reuse the same {{...}} tokens in your reply.")

@pipe.technique("answer", "plain", description="Bare model — no support guidelines.")
async def plain(ctx, message, model="gpt-oss:20b"):
    return await cafe.complete(MODELS[model], [
        {"role": "system", "content": f"You are a customer-support assistant. {LENGTH} {DATA_NOTE}"},
        {"role": "user", "content": message},
    ], temperature=0.3)

@pipe.technique("answer", "policy", description="Same, plus a support system-prompt (the actionable lever).")
async def policy(ctx, message, model="gpt-oss:20b"):
    return await cafe.complete(MODELS[model], [
        {"role": "system", "content":
            "You are a customer-support assistant. Be accurate, give the key next step, keep a "
            "professional and empathetic tone, and offer to escalate to a human if the request needs "
            f"account access or you are unsure. {LENGTH} {DATA_NOTE}"},
        {"role": "user", "content": message},
    ], temperature=0.3)

@pipe.compose
async def run(config, item, ctx):
    return await ctx.run("answer", message=item["text"])

4. Judge & rubric¶

An independent LLM judge rates each reply 1–5 on support quality. The rubric is deliberately written to judge substance, not length — LLM judges tend to reward longer, more elaborate answers, so a concise, correct reply (or a sensible clarifying question) must score just as high as a verbose one. It uses the Bitext reply as a correctness guide, not a wording target, and treats {{...}} placeholders as filled-in.

In [42]:

Copied!





judge = cafe.LLMJudge(model=JUDGE_MODEL, preset="reference_qa")

rubric = cafe.Rubric(
    name="support_quality",
    scale_type=cafe.ScaleType.ordinal,
    levels=[
        cafe.Level(1, "poor",      "Incorrect, unhelpful, or off-topic."),
        cafe.Level(2, "weak",      "Addresses the request but with a real error or a missing essential."),
        cafe.Level(3, "ok",        "Correct and helpful."),
        cafe.Level(4, "good",      "Correct, helpful, and well-judged — right scope and professional tone."),
        cafe.Level(5, "excellent", "Fully and appropriately resolves the request, accurately and professionally."),
    ],
    instruction=(
        "Rate 1-5 how well the ANSWER resolves the customer's support request. Judge SUBSTANCE — is it "
        "accurate, correct, and appropriately helpful — NOT length, detail, or formatting. A concise, "
        "correct reply (including one that sensibly asks a clarifying question or confirms before acting) "
        "must score just as high as a long, detailed one; do NOT reward verbosity, padding, extra steps, "
        "or formatting. Use the REFERENCE only as a guide to the correct information, not a wording "
        "target — a different or better answer scores just as high. The message, reference, and answer "
        "may contain redacted {{...}} placeholders standing in for real values; treat them as filled in "
        "and do not penalize them."
    ),
)
judge = cafe.LLMJudge(model=JUDGE_MODEL, preset="reference_qa")

rubric = cafe.Rubric(
    name="support_quality",
    scale_type=cafe.ScaleType.ordinal,
    levels=[
        cafe.Level(1, "poor",      "Incorrect, unhelpful, or off-topic."),
        cafe.Level(2, "weak",      "Addresses the request but with a real error or a missing essential."),
        cafe.Level(3, "ok",        "Correct and helpful."),
        cafe.Level(4, "good",      "Correct, helpful, and well-judged — right scope and professional tone."),
        cafe.Level(5, "excellent", "Fully and appropriately resolves the request, accurately and professionally."),
    ],
    instruction=(
        "Rate 1-5 how well the ANSWER resolves the customer's support request. Judge SUBSTANCE — is it "
        "accurate, correct, and appropriately helpful — NOT length, detail, or formatting. A concise, "
        "correct reply (including one that sensibly asks a clarifying question or confirms before acting) "
        "must score just as high as a long, detailed one; do NOT reward verbosity, padding, extra steps, "
        "or formatting. Use the REFERENCE only as a guide to the correct information, not a wording "
        "target — a different or better answer scores just as high. The message, reference, and answer "
        "may contain redacted {{...}} placeholders standing in for real values; treat them as filled in "
        "and do not penalize them."
    ),
)

5. The study + preflight¶

Three models × two prompts = 6 configurations. Preflight shows the size and a cost/time estimate before spending anything.

In [43]:

Copied!





study = cafe.Study(
    name="support-chatbot-quality",
    system=pipe,
    factors=[
        pipe.factor("answer"),                          # plain vs policy
        cafe.Factor("answer.model", list(MODELS)),      # the three models
    ],
    dataset=dataset,
    rubric=rubric,
    judge=judge,
    replications=1,
)
print(study.preflight())
study = cafe.Study(
    name="support-chatbot-quality",
    system=pipe,
    factors=[
        pipe.factor("answer"),                          # plain vs policy
        cafe.Factor("answer.model", list(MODELS)),      # the three models
    ],
    dataset=dataset,
    rubric=rubric,
    judge=judge,
    replications=1,
)
print(study.preflight())

Preflight(answers=Results(support-chatbot-quality: 6 answers, 6 configs, 0 errors), estimate={'sampled_cells': 6, 'total_cells': 600, 'est_total_compute_s': 1520.75, 'est_total_cost_usd': 0.0713, 'labels': ['answer=policy·answer.model=gpt-oss:120b']}, warnings=[], judge_calls=600)

6. Run + report¶

evaluate() answers every ticket under every configuration, judges the replies, and attributes quality to the factors. report() is the whole picture; wall_clock_s is the real run time.

In [44]:

Copied!

result = study.evaluate()
print(result.report())
print(f"\nwall-clock: {result.wall_clock_s}s")
result = study.evaluate()
print(result.report())
print(f"\nwall-clock: {result.wall_clock_s}s")

support-chatbot-quality: answers:   0%|          | 0/600 [00:00<?, ?it/s]

judging:   0%|          | 0/600 [00:00<?, ?it/s]

600 answers · 6 configs · 100 inputs · 600 ratings · best: answer=plain·answer.model=deepseek-v4-pro

pipeline: 600 answers  →  600 judged  →  600 usable verdict(s)

────────────────────────────────────────────────────────────
DESCRIPTIVE — means & best configuration
────────────────────────────────────────────────────────────
verdicts: 600   factors: answer, answer.model

overall mean quality: 4.02  (n=600)

per-configuration mean quality:
  4.22  (n=100)  answer=plain·answer.model=deepseek-v4-pro
  4.13  (n=100)  answer=policy·answer.model=gpt-oss:120b
  4.01  (n=100)  answer=plain·answer.model=gpt-oss:120b
  4.01  (n=100)  answer=policy·answer.model=deepseek-v4-pro
  3.94  (n=100)  answer=policy·answer.model=gpt-oss:20b
  3.80  (n=100)  answer=plain·answer.model=gpt-oss:20b

per-factor marginal means:
  answer:
     plain            mean=4.01  n=300
     policy           mean=4.03  n=300
  answer.model:
     deepseek-v4-pro  mean=4.12  n=200
     gpt-oss:120b     mean=4.07  n=200
     gpt-oss:20b      mean=3.87  n=200

best configuration: answer=plain·answer.model=deepseek-v4-pro  (mean 4.22)

────────────────────────────────────────────────────────────
INFERENTIAL — mixed-effects model (significance & effect sizes)
────────────────────────────────────────────────────────────
verdict ~ (1 | input_id) + (answer + answer.model)^2
  fixed-effects model + Type-II ANOVA, up to 2-way   (n=600, α=0.05)

per-term effects  (F-test, p, partial η²;  '×' = interaction):
  term                         F          p   partial η²     
  answer.model              3.83     0.0223        0.013   *
  answer × answer.model     2.17     0.1148        0.007    
  answer                    0.05     0.8287        0.000    
  Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

effect sizes — Cohen's d (magnitude of the gap; 0.2 small, 0.5 medium, 0.8 large):
  answer: plain vs policy   d = -0.02   95% CI [-0.18, +0.14]
  answer.model: deepseek-v4-pro vs gpt-oss:120b   d = +0.05   95% CI [-0.15, +0.24]
  answer.model: deepseek-v4-pro vs gpt-oss:20b   d = +0.26   95% CI [+0.06, +0.45]
  answer.model: gpt-oss:120b vs gpt-oss:20b   d = +0.21   95% CI [+0.02, +0.41]

note: random intercept not estimable; using fixed-effects ANOVA
note: model was near-singular (little between-question variance) — treat p-values as unstable; the effect sizes (Cohen's d) are the more reliable signal here

────────────────────────────────────────────────────────────
ORDINAL — cumulative-link mixed model (verdicts as ordered categories)
────────────────────────────────────────────────────────────
ordinal CLMM — verdict ~ (1 | input_id) + (answer + answer.model)^2   (n=600, logLik=-692.0, α=0.05)

fixed effects (ordinal log-odds of a higher score; + = better):
  term                                          estimate          p     
  answer=policy                                   -0.460     0.1095    
  answer.model=gpt-oss × 120b                     -0.491     0.0858   .
  answer.model=gpt-oss × 20b                      -1.043     0.0002   ***
  answer=policy × answer.model=gpt-oss × 120b     +0.726     0.0724   .
  answer=policy × answer.model=gpt-oss × 20b      +0.835     0.0356   *
  Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

wall-clock: 894.122s

7. Read the result¶

For the LinkedIn story, look for:

The model effect — is the expensive flagship significantly better than the cheap model, or are they statistically tied? A tie is the headline: "the cheap model is good enough — don't pay for the flagship."
The prompt effect + interaction — does the support system-prompt lift quality, and does it help the cheap model most? Often "a good prompt matters more than model size."
The cost/quality frontier — the cheapest configuration that isn't significantly worse than the best is the one to ship.

The headline is whatever the numbers actually say — but for routine support, the usual finding is that the gap between cheap and premium is small and often not significant.

8. Push it into the web app¶

Save the finished evaluation to a portable JSON bundle, then load it into the CAFE web platform with the Import study button (Studies → Import study) to browse the full interactive Results dashboard — attribution, the ordinal model, the cost/quality frontier, latency by factor, and every stored verdict (prompt + raw judge response + score).

save_evaluation writes everything report() needs; load_evaluation rebuilds the Evaluation later with no API calls.

In [ ]:

Copied!





cafe.save_evaluation(result, "support_study_n100.json")

# Reload it anywhere (or via the web app's Import study button) — reproduces report()
# with no API calls:
#   ev = cafe.load_evaluation("support_study_n100.json")
#   print(ev.report())
cafe.save_evaluation(result, "support_study_n100.json")

# Reload it anywhere (or via the web app's Import study button) — reproduces report()
# with no API calls:
#   ev = cafe.load_evaluation("support_study_n100.json")
#   print(ev.report())