What it is
A Python test harness and benchmark for Jev, TypeSafe's structured decision model, called via OpenRouter's Decisions API. It runs 17 cases with 23 checks and compares Jev against general LLMs on latency, cost, and accuracy.
How it uses Jev
Jev receives a state (string, object, or array) and typed questions (noul, choice, score). It returns calibrated probabilities, labels, and scores. The harness checks expectations, assigns confidence bands (act/confirm/escalate), and compares results against LLM baselines.
Primitives:noulchoicescore
Technique worth stealing
Use structured criteria with what + examples to improve score confidence and accuracy.
Try it
uv sync && cp .env.example .env && uv run test_jev.py
Evidence
Each line is one question put to Jev about the README. ≥ 0.60 reads as yes, ≤ 0.40 as no; in between Jev is not making a call.
- Jev-centricyes0.92
- Shows a System One patternyes0.73
- Handles uncertaintyno0.06
- Measuredno0.09
- Runnableno0.23
- Worth recommendingno0.38
- Model replicano0.03
- Problem scopescore on a 0–2 scale1.01
- About Jevyes0.99
Signals by Jev jev-1.13.0, card written by DeepSeek V4.1 Flash from the README on 22 Sept 2026.