What it is
jevcheck is a pytest-like tool for TypeSafe Jev. It pins a production contract (baseline model, fixtures, expected answers), evals a candidate model, and fails when answers flip or confidence drops.
How it uses Jev
Jev makes choice, score, and noul decisions on fixture state. jevcheck records baseline answers, compares candidate answers against the contract, and reports unchanged, confidence regressions, or answer flips with exact diffs and a nonzero exit on failure.
Primitives:choicescorenoul
Technique worth stealing
Record a baseline model's answers, then compare a candidate against that snapshot to catch flips and confidence regressions.
Try it
pip install -e ".[dev]", export TYPESAFE_API_KEY, then run jevcheck eval fixtures/support-triage.json --candidate-model jev-1.14.
Evidence
Each line is one question put to Jev about the README. ≥ 0.60 reads as yes, ≤ 0.40 as no; in between Jev is not making a call.
- Jev-centricunclear0.54
- Shows a System One patternunclear0.52
- Handles uncertaintyno0.06
- Measuredno0.16
- Runnableyes0.70
- Worth recommendingunclear0.40
- Model replicano0.06
- Problem scopescore on a 0–2 scale1.17
- About Jevyes0.97
Signals by Jev jev-1.13.0, card written by DeepSeek V4.1 Flash from the README on 21 Sept 2026.