The problem
Illustrative scenario, not a measured Jev result: you run fourteen team rules on every code change. One rule flags ten changes; a person checks them and finds only two violations. That labeled review establishes a problem the probabilities alone could not show.
You make a table of firing rate, median and maximum probability. It helps find unexpected distributions, but it cannot identify false alarms: if every sample really violates a rule, a detector that fires on every one may be correct. A low maximum could also mean the batch contains no violations.
Twenty recent changes are enough to start debugging the question and the evaluation script. They are not a reliability guarantee, and repeatedly adjusting the wording until that same batch looks good does not test what happens on new changes.
The solution
The check-up pattern starts with a distribution check. Record the inputs, question wording, returned model version and action threshold; inspect firing rate, median and maximum. Without human labels, report “distribution only” and inspect surprising cases. Do not turn high firing rate into “noisy” or a quiet distribution into “useful”.
With labels that say whether each input actually violates the rule, count true and false flags and missed violations. Precision is the share of flags that are correct; recall is the share of real violations caught. Brier is mean((probability − label)²), with labels 0 or 1; lower is better. Compare it with a constant probability fixed from the tuning set’s positive rate before looking at validation labels. Brier evaluates probability error; it does not by itself prove calibration or an acceptable action policy.
Use one set to change wording or choose thresholds, then freeze both and evaluate on a separate held-out set, keeping related or duplicate changes together. Include ordinary and violating cases, report counts, and expand beyond the 20-case debugging pilot. If you tune after inspecting validation errors, use fresh validation data next time. Below is a fully offline arithmetic example with invented probabilities and labels, not a model benchmark. A missing metric is null, not zero; a sample containing only violations cannot measure false-positive rate.
Ask · Answer · Act
- AskOffline teaching example: freeze a 0.7 threshold and a 0.5 constant baseline, then score a separate synthetic validation batch with probabilities [0.9, 0.8, 0.6, 0.1] and labels [1, 0, 1, 0].
- AnswerTwo flags, one correct; two real violations, one caught: precision 0.5, recall 0.5, Brier 0.205 versus the baseline’s 0.25.
- ActThe probability error beats this baseline on four invented cases, but half the flags are wrong, so the example demonstrates the calculation and does not establish a deployable rule.
Where the name comes from
A check-up: an unusual reading tells you what to investigate, not the diagnosis. Firing rate is the reading; labeled cases tell you which decisions were right or wrong.
Use it when
You need to decide whether a question or threshold works for your task. Start with a small debugging batch, then label representative cases and reserve enough independent validation data to assess the errors that matter.
Not when
You only have unlabeled scores: use them to inspect distribution, not certify correctness. A handful of labeled cases can expose a bug but cannot establish a rare-error rate; comparing whole tools also cannot isolate which question caused an error.
Read this code
- abide
packages/cli/src/lib/calibration.ts:20-34Source check, not a site-run experiment: this implementation calls a firing rate of 0.6 or more “noisy” without labels. That is a project heuristic to inspect, not evidence of false alarms.
- huncho
src/calibrate.ts:142-159Source check: computes Brier and a constant baseline from the labeled records themselves. The offline example here instead freezes the baseline on tuning data before scoring separate validation data.
- snifftest
docs/eval-notes.md:181-214Author-reported experiment, 17 September 2026, Jev 1.13: rewording changed catches from 58/80 to 63/80 on the tuning seed; a held-out seed caught 60/80. The author warns that eight faults per rule is a small sample; we have not reproduced the run.
Show me the code
// Save /examples/measured-calibration.mjs beside this demo.mjs; run: node demo.mjs
// Synthetic inputs only: no API calls or measured model results.
import { evaluate } from "./measured-calibration.mjs";
const tuningLabels = [1, 0, 1, 0];
const baselineRate = tuningLabels.reduce((a, b) => a + b, 0) / tuningLabels.length;
const fixed = { threshold: 0.7, baselineRate }; // illustrative; freeze before validation
// Three separate validation examples; no tuning on these labels:
console.log(evaluate([0.9, 0.8], fixed)); // distribution-only
console.log(evaluate([0.9, 0.8], { ...fixed, labels: [1, 1] })); // precision = recall = 1
console.log(evaluate([0.9, 0.8, 0.6, 0.1], { ...fixed, labels: [1, 0, 1, 0] }));Download the offline evaluation function used below (no dependencies or API calls)
How jevbooks recognises it
How do we know a project does this? We put one question about its README to Jev and it returns a percentage; the higher, the more it looks like it. If you copy the question, copy one thing: ask whether the text mentions this one concrete thing, never whether the project is good.
{
"measured-calibration": {
"type": "noul",
"instructions": {
"what": "Does the text describe measuring the questions themselves: fire rate, Brier score, precision on historical samples, or an A/B against another model?",
"not_for": "Anecdotal claims of accuracy."
},
"criteria": {
"true": "The text explicitly describes this. One sentence is enough; other content does not cancel it.",
"false": "The text does not describe this."
}
}
}