Benchmarks & Calibration

§ 1The shipped lines, side by side

Robust claims: -intuit ≫ -meticulous on the skills slice (0.814 vs 0.659 on 609 items, ~20σ); -meticulous ≫ -intuit on bench ECE (0.05 vs 0.12, beyond single-run resolution); v1.1c's composite is not separable from v1 (63.72 vs 65.71, inside the ±5.9pt CI) — what is separable: hard tier +9.0pt (0.469 → 0.559), accuracy +5.6pt, matching Intelligence (+8.0), and the image/Chinese capability. A measured pipeline noise floor (~±2–3 composite points across runs differing only by seed and serving constants) bounds everything finer-grained: improvements below it are not evidence.

The benchmark updated to v1.4.1/v1.4.2 during this work (roster now 93 systems, 89 ranked; at run time the newest clone is bb05a33 — harness code and the 231 public items unchanged). Verified: the 231 public items, the 308-item sealed set, and the composite formula are unchanged — the official arithmetic was reproduced exactly — so the runs of record remain valid as scored. The frozen v1.5 method (904 open / 720 sealed) is not self-runnable from this repo and goes through Benchmark Heaven's submission pipeline.

kapteeni-v1.1c — the pre-registered gates, all passed in its one training run

§ 2The four axes (official formulas)

§ 3Per-tier accuracy (231 items)

§ 4Per-family accuracy (public items — v1-meticulous)

The -intuit variant was developed for exactly the weak families above; on its 609-item held-out skills slice it reaches temporal_numeric 0.811 / multi_hop 0.917 / long_policy 0.644. It was not run family-by-family on the public items; only its tier accuracies above are public-half measured. Ordinal expectation MAE 0.482 levels; paraphrase consistency 0.861 (meticulous).

v1.1c — families that moved (vs v1-meticulous; n unchanged)

§ 5Calibration — the shipped models

Top-label expected calibration error across all 231 public answers, scored by the benchmark's own code (ECE half only; the benchmark also averages a private gold-distribution half):

§ 6Run history

1. 01

   kapteeni-v1-meticulous, the flagship (2026-09-25)

   LoRA r=32 on all projections + retrained heads over the 8.7M-token mix; MNLI held out of training as the OOD gate (0.88 → 0.893). All serving constants refit on mixed-domain val through the merged model — zero public-half selection in the chain.

   65.71 · acc 0.710
2. 02

   kapteeni-v1-intuit (2026-09-26)

   The second text variant — same architecture and wire format; the weak-family skills added to training and serving constants fitted on the deployment-diverse held-out set, per pre-registration. Sharper where the flagship is conservative, with calibration traded accordingly; the guidance of when to prefer it lives in the table above.

   63.18 · acc 0.714
3. 03

   kapteeni-v1.1c, the multimodal line (2026-10-02)

   The same phase pipeline ported to Qwen3.5-4B (images + Chinese); every pre-registered gate passed on the first reading. Run of record on the released pack: ECE 0.0984; not separable from v1 on the composite (inside the ±5.9pt CI). Published pinned at 17affb8; bench requests open for JevBench and Image JevBench (fstandhartinger/jevbench#178).

   63.72 · acc 0.766

The records behind these three shipped models — every pre-registration, gate, and run (successful or not) — are kept in full in JEVBENCH.md and WORKLOG.md; a design proposal for a deployment-oriented benchmark alternative lives in DECISIONBENCH-DRAFT.md.

§ 7How to read every number above

• Sample sizes are small. All accuracies are on 231 items; one item is 0.43 accuracy points. The 95% binomial CI on public accuracy is ±5.9pt (0.710 / 0.714 / 0.766). Tier-level CIs are wider (hard: ±9.3pt at 0.469).
• Small composite gaps are ties. Differences of a few points between any two runs sit inside run-to-run variance plus axis assumptions (the Cost axis alone swings ±7 points across the stated price range).
• A measured pipeline noise floor. Runs differing only by training seed and val-fit constants scored ~2.6 composite points apart — treat ~±2–3 composite points as the single-seed noise floor.
• ECE at n=231 is biased and high-variance. Differences below ~0.02 between single runs are unresolvable; 0.05-vs-0.07 comparisons between runs should not be read as meaningful.
• Calibration is measured on the benchmark's distribution, which is OOD relative to any deployment: a model whose constants are tuned for the bench mix is not thereby calibrated for yours. The record documents this failure mode in the lab, not just in theory (JEVBENCH).
• Intelligence is renormalized over the three public tiers because the judge tier is sealed; any score that folds the sealed+judge weight in is not comparable to ours. Jev's accuracy on the same public items is still higher (0.866 vs 0.710).
• Composite gaps sit inside the CI. 63.72 (v1.1c) vs 65.71 (v1) is not separable at n=231 — the honest claim is "not separable on the composite with this sample size," not a win or a loss. The separable claims are recorded exactly: the hard-tier jump (+9.0pt), the accuracy gain (+5.6pt), and the image/Chinese capability.

§ 8Reproducibility

# server with the trained bundle
python3 -m kapteeni.serve --bundle model_cache/kapteeni_v1.pt \
    --lora model_cache/kapteeni_p2/adapter --served-as kapteeni-v1-meticulous --port 8000
# multimodal line through the released pack
python3 -m kapteeni.serve_v11c --hf TriusAI/kapteeni-v1.1c --port 8002

# bench (from jevbench/)
TYPESAFE_ENDPOINT=http://127.0.0.1:8000 TYPESAFE_API_KEY=local \
TYPESAFE_PRICE_INPUT_PER_M=0 TYPESAFE_PRICE_OUTPUT_PER_M=0 \
python3 -m jevbench.cli run \
  --tasks datasets/public/easy.jsonl,datasets/public/original.jsonl,datasets/public/hard.jsonl \
  --adapter typesafe --results <out>.jsonl --raw-dir <raw>
python3 ../score_bench.py <out>.jsonl

Raw per-item outputs live in docs/bench/: one raw record per shipped model (231 rows each, all schema-valid; the v1.1c run went through the released pack — the pack itself is the measured artifact — with constants exactly as packed, one attempt per item). Pre-registration for the multimodal line: PREREG-KAPTEENI-V11C.md; The posted bench request is preserved as BENCH-REQUEST.md. The full experiment history is JEVBENCH.md.