What it is
An independent, API-only audit of the calibration of TypeSafe AI's Jev, a "System One" decision model that answers pre-defined questions over a text state with probability distributions instead of generated text. For researchers and practitioners evaluating Jev's probabilistic reliability.
How it uses Jev
Jev is queried via public HTTP calls with a state and typed questions (Choice, Score, Noul) to obtain probability distributions. The audit measures calibration error, coherence, invariance, interference, hedging, and Korean performance. Results are recorded per call in JSONL.
Primitives:choicescorenoul
Technique worth stealing
Report ECE against a noise floor, not zero, and include a state-blind baseline in every set.
Try it
uv sync; cp .env.example .env; fill JEV_API_KEY; uv run python -m jev_audit.preflight
Evidence
Each line is one question put to Jev about the README. ≥ 0.60 reads as yes, ≤ 0.40 as no; in between Jev is not making a call.
- Jev-centricyes0.90
- Shows a System One patternno0.22
- Handles uncertaintyno0.13
- Measuredno0.31
- Runnableno0.06
- Worth recommendingno0.28
- Model replicano0.02
- Problem scopescore on a 0–2 scale1.14
- About Jevyes0.99
Signals by Jev jev-1.13.0, card written by DeepSeek V4.1 Flash from the README on 21 Sept 2026.