What it is
A Go benchmark suite that evaluates Jev, TypeSafe's System One model, on two public corpora: 662 prompt-injection messages and 200 matched vulnerable-code pairs. It reports classification metrics, calibration, and a label audit, with a TUI dashboard.
How it uses Jev
Jev answers a yes/no injection question and a severity Score in one request per sample, using the deployment context as state. The returned probabilities are thresholded at 0.50 for the reported metrics; the README suggests a graded policy with 0.70/0.35 cuts.
Primitives:choicescore
Technique worth stealing
Pass the deployment context as state so the same model separates traffic at a fixed threshold.
Try it
export TYPESAFE_API_KEY=...; go run ./cmd/jev-sec-bench -bench all
Evidence
Each line is one question put to Jev about the README. ≥ 0.60 reads as yes, ≤ 0.40 as no; in between Jev is not making a call.
- Jev-centricyes0.92
- Shows a System One patternno0.18
- Handles uncertaintyyes0.64
- Measuredno0.28
- Runnableno0.19
- Worth recommendingunclear0.47
- Model replicano0.04
- Problem scopescore on a 0–2 scale1.62
- About Jevyes0.98
Signals by Jev jev-1.13.0, card written by DeepSeek V4.1 Flash from the README on 21 Sept 2026.