What it is
A Python benchmark that compares TypeSafe Jev with any OpenRouter model on labelled classification tasks, reporting accuracy, calibration, latency, cost and determinism. It runs on three included public tasks or your own data in data/.
How it uses Jev
Jev answers the task's question: choice picks one of the labels, score picks an ordered level, noul returns a yes/no probability. Its returned distribution gives the confidence, which the report uses for accuracy, ECE and AUROC metrics.
Primitives:choicescorenoul
Technique worth stealing
Same instruction and option names for both models; confidence is the probability of the chosen label.
Try it
cp .env.example .env; uv sync; uv run python -m evaljev.run --model jev; uv run python -m evaljev.report
Evidence
Each line is one question put to Jev about the README. ≥ 0.60 reads as yes, ≤ 0.40 as no; in between Jev is not making a call.
- Jev-centricunclear0.53
- Shows a System One patternno0.10
- Handles uncertaintyno0.04
- Measuredyes0.98
- Runnableno0.20
- Worth recommendingno0.25
- Model replicano0.05
- Problem scopescore on a 0–2 scale1.17
- About Jevyes0.98
Signals by Jev jev-1.13.0, card written by DeepSeek V4.1 Flash from the README on 20 Sept 2026.