CAFE — fractional factorial designs¶

The problem. A full factorial of k two-level factors is 2^k runs — 7 factors = 128 configs. Most of that budget goes to estimating high-order interactions (4-, 5-, 6-way) that are almost always negligible.

The idea. Run a carefully chosen fraction — 2^(k-p) configs — that still pins down what you care about (the main effects), by deliberately confounding ("aliasing") each main effect with a high-order interaction you're willing to assume is ≈ 0. Same conclusions about which factors matter, for a fraction of the runs. It works because the chosen subset stays balanced and orthogonal — every factor is still tested at both levels equally often, just not in every combination.

The catch — resolution. You don't get it for free: what you save in runs, you pay in aliasing, and the resolution says what's cleanly estimable:

resolution	main effects	2-factor interactions
III	aliased with 2-factor interactions	— (screening only)
IV	clean	aliased with each other
V	clean	clean

So no — you can't estimate every interaction, only up to the degree the resolution allows. CAFE reports the resolution and the exact alias list, and the stats layer respects it: it refuses to fit interactions the design can't separate (it caps to main effects and says so), so you never read a phantom "significant interaction" that's really an alias.

The design, before running anything¶

7 factors: full factorial = 128 runs. A saturated fraction screens them in 8.

In [1]:

import cafe
from cafe._env import load_env
load_env()

facs7 = [cafe.Factor(c, [0, 1]) for c in "ABCDEFG"]
print(cafe.fractional_factorial_design(facs7).show())

import cafe from cafe._env import load_env load_env() facs7 = [cafe.Factor(c, [0, 1]) for c in "ABCDEFG"] print(cafe.fractional_factorial_design(facs7).show())

Fractional factorial 2^(7-4)  —  resolution III

  runs: 8 of 128 full  (94% fewer)

  generators (added factor = product of basic factors):
     D = ABC
     E = AB
     F = AC
     G = BC

  main-effect aliases (assumed negligible):
     A = BE = CF = DG
     B = AE = CG = DF
     C = AF = BG = DE
     D = AG = BF = CE
     E = AB = CD = FG
     F = AC = BD = EG
     G = AD = BC = EF

  resolution III: main effects aliased with 2-factor interactions (screening)

In [2]:

# Asking for more runs buys resolution: a half-fraction of 5 factors is resolution V
# (main effects AND 2-factor interactions are clear).
facs5 = [cafe.Factor(c, [0, 1]) for c in "ABCDE"]
print(cafe.fractional_factorial_design(facs5, runs=16).show())

# Asking for more runs buys resolution: a half-fraction of 5 factors is resolution V # (main effects AND 2-factor interactions are clear). facs5 = [cafe.Factor(c, [0, 1]) for c in "ABCDE"] print(cafe.fractional_factorial_design(facs5, runs=16).show())

Fractional factorial 2^(5-1)  —  resolution V

  runs: 16 of 32 full  (50% fewer)

  generators (added factor = product of basic factors):
     E = ABCD

  main-effect aliases (assumed negligible):

  resolution V: main effects and 2-factor interactions estimable

A real run: same conclusion, half the cost¶

Four two-level factors over a real system. One factor — sabotage — deliberately ruins the answer, so it should dominate. Full factorial would be 16 configs; the resolution-IV fraction uses 8 and still recovers it.

In [3]:

SMALL, BIG = "ollama_cloud/gpt-oss:20b", "ollama_cloud/gpt-oss:120b"

async def system(config, item):
    sys = ["Give a confidently WRONG, false answer."] if config["sabotage"] == "on" \
          else ["Answer truthfully and concisely."]
    if config["verbosity"] == "long":
        sys.append("Write a long, detailed answer.")
    if config["hedge"] == "on":
        sys.append("Hedge heavily and avoid committing.")
    model = BIG if config["model"] == "big" else SMALL
    return await cafe.complete(model, [
        {"role": "system", "content": " ".join(sys)},
        {"role": "user", "content": item["text"]}])

factors = [
    cafe.Factor("model", ["small", "big"]),
    cafe.Factor("sabotage", ["off", "on"]),
    cafe.Factor("verbosity", ["short", "long"]),
    cafe.Factor("hedge", ["off", "on"]),
]
study = cafe.Study(
    name="frac-run",
    system=system,
    factors=factors,
    dataset=cafe.datasets.load_truthfulqa(n=3, categories=["Misconceptions"], seed=5),
    rubric=cafe.ANSWER_QUALITY_1_5,
    judge=cafe.LLMJudge(model=BIG),
    design="fractional",
)
full = cafe.Study(name="full", system=system, factors=factors, dataset=study.dataset)
print(f"full factorial: {cafe.size(full)} configs   |   fractional: {cafe.size(study)} configs")

SMALL, BIG = "ollama_cloud/gpt-oss:20b", "ollama_cloud/gpt-oss:120b" async def system(config, item): sys = ["Give a confidently WRONG, false answer."] if config["sabotage"] == "on" \ else ["Answer truthfully and concisely."] if config["verbosity"] == "long": sys.append("Write a long, detailed answer.") if config["hedge"] == "on": sys.append("Hedge heavily and avoid committing.") model = BIG if config["model"] == "big" else SMALL return await cafe.complete(model, [ {"role": "system", "content": " ".join(sys)}, {"role": "user", "content": item["text"]}]) factors = [ cafe.Factor("model", ["small", "big"]), cafe.Factor("sabotage", ["off", "on"]), cafe.Factor("verbosity", ["short", "long"]), cafe.Factor("hedge", ["off", "on"]), ] study = cafe.Study( name="frac-run", system=system, factors=factors, dataset=cafe.datasets.load_truthfulqa(n=3, categories=["Misconceptions"], seed=5), rubric=cafe.ANSWER_QUALITY_1_5, judge=cafe.LLMJudge(model=BIG), design="fractional", ) full = cafe.Study(name="full", system=system, factors=factors, dataset=study.dataset) print(f"full factorial: {cafe.size(full)} configs | fractional: {cafe.size(study)} configs")

full factorial: 16 configs   |   fractional: 8 configs

In [4]:

# The exact design CAFE will run — its resolution and alias structure, before spending
# anything. (design="fractional" on the Study uses this same design under the hood.)
print(cafe.fractional_factorial_design(factors).show())

# The exact design CAFE will run — its resolution and alias structure, before spending # anything. (design="fractional" on the Study uses this same design under the hood.) print(cafe.fractional_factorial_design(factors).show())

Fractional factorial 2^(4-1)  —  resolution IV

  runs: 8 of 16 full  (50% fewer)

  generators (added factor = product of basic factors):
     hedge = modelsabotageverbosity

  main-effect aliases (assumed negligible):

  resolution IV: main effects clear of 2-factor interactions; 2FIs aliased with each other

In [5]:

result = study.evaluate(concurrency=4)
print(result.attribution.show())

result = study.evaluate(concurrency=4) print(result.attribution.show())

/tmp/ipykernel_1863526/2940868022.py:1: UserWarning: design check: 3 inputs — the mixed-effects models (linear / CLMM / logistic) estimate a per-question random intercept; with fewer than ~8 questions those estimates can be near-singular/unstable. Add inputs for reliable p-values
  result = study.evaluate(concurrency=4)
/tmp/ipykernel_1863526/2940868022.py:1: UserWarning: design check: 8 configurations but only 3 inputs — few observations per configuration; per-factor estimates will be weak. Add inputs, or reduce factors/levels
  result = study.evaluate(concurrency=4)

frac-run: answers:   0%|          | 0/24 [00:00<?, ?it/s]

judging:   0%|          | 0/21 [00:00<?, ?it/s]

verdicts: 21 (18 usable)   factors: model, sabotage, verbosity, hedge

overall mean quality: 3.39  (n=18)

per-configuration mean quality:
  5.00  (n= 1)  hedge=on·model=big·sabotage=off·verbosity=short
  5.00  (n= 1)  hedge=off·model=small·sabotage=off·verbosity=short
  4.67  (n= 3)  hedge=off·model=big·sabotage=off·verbosity=long
  4.00  (n= 2)  hedge=on·model=small·sabotage=off·verbosity=long
  3.33  (n= 3)  hedge=on·model=big·sabotage=on·verbosity=long
  2.50  (n= 2)  hedge=on·model=small·sabotage=on·verbosity=short
  2.33  (n= 3)  hedge=off·model=big·sabotage=on·verbosity=short
  2.33  (n= 3)  hedge=off·model=small·sabotage=on·verbosity=long

per-factor marginal means:
  model:
     big              mean=3.60  n=10
     small            mean=3.12  n=8
  sabotage:
     off              mean=4.57  n=7
     on               mean=2.64  n=11
  verbosity:
     long             mean=3.55  n=11
     short            mean=3.14  n=7
  hedge:
     off              mean=3.30  n=10
     on               mean=3.50  n=8

best configuration: hedge=on·model=big·sabotage=off·verbosity=short  (mean 5.00)

Statistics — main effects only, and CAFE enforces it¶

This is a resolution-IV design, so the 2-factor interactions are aliased in pairs (the same 8 runs literally can't tell model×sabotage apart from verbosity×hedge — they're the identical column). If CAFE fit ^2 here it would silently split one contrast's variance across an aliased pair and report both as if real. Instead it detects the rank-deficient design and fits main effects only, with a note — so no phantom interactions. sabotage is still recovered as the dominant, significant factor from just 8 runs. (For clean 2-factor interactions, use a resolution-V design, like the 2^(5-1) above.)

In [6]:

print(result.effects.show())

print(result.effects.show())

verdict ~ (1 | input_id) + model + sabotage + verbosity + hedge
  fixed-effects model + Type-II ANOVA   (n=18, α=0.05)

per-term effects  (F-test, p, partial η²;  '×' = interaction):
  term             F          p   partial η²     
  sabotage      6.38     0.0253        0.329   *
  model         0.39     0.5448        0.029    
  hedge         0.16     0.6988        0.012    
  verbosity     0.01     0.9142        0.001    
  Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

effect sizes — Cohen's d (magnitude of the gap; 0.2 small, 0.5 medium, 0.8 large):
  model: big vs small   d = +0.28   95% CI [-0.66, +1.21]
  sabotage: off vs on   d = +1.36   95% CI [+0.32, +2.41]
  verbosity: long vs short   d = +0.23   95% CI [-0.72, +1.18]
  hedge: off vs on   d = -0.12   95% CI [-1.05, +0.82]

note: random intercept not estimable; using fixed-effects ANOVA
note: interactions are aliased in this design (e.g. a fractional factorial below resolution V) — they can't be separated, so only main effects are fit; read the design's alias structure to interpret them
note: model was near-singular (little between-question variance) — treat p-values as unstable; the effect sizes (Cohen's d) are the more reliable signal here

Notes¶

• sabotage stands out as the dominant, significant factor — recovered from 8 runs instead of 16.
• Set design="fractional" on the Study; control the fraction via design_options={"runs": 16} or supply your own generators.
• The stats layer respects the design. For an aliased fraction (resolution < V) CAFE detects the rank-deficient design and fits main effects only, with a note — both the Gaussian and the ordinal CLMM layers. You never get a phantom "significant interaction" that's really an alias. Want interactions? Use a resolution-V design.
• Always read the resolution: at resolution III even a main effect is aliased with a 2-factor interaction, so a "significant" factor could be a lurking interaction — cafe.fractional_factorial_design(...).show() lists exactly which.