jev-benchmarks

What it is

jev-benchmarks is a reproducible evaluation harness for typed decision models. It measures calibration, selective automation at a fixed error budget, latency, and integrity, and ships a pilot comparing TypeSafe Jev with GLiNER2.5 on zero-shot single-label text classification.

How it uses Jev

Jev is run as a hosted backend through a Choice adapter that returns per-label probabilities for each example. The pipeline validates probability vectors, records resolved model identity, and feeds the predictions into discrimination, calibration, and risk/latency reports. The README does not describe the adapter's internal prompt or state construction.

Primitives:choice

Technique worth stealing

Paired target-stratified bootstrap intervals over pinned revisions, with a uniform negative control and append-only prediction logs.

Try it

uv sync --extra benchmark --dev; uv run jev-bench prepare --config configs/pilot-v1.yaml; then run each backend and report.

View on GitHub