What it is
A companion measurement lab for dspy-typesafeify and other callers that use TypeSafe decisions inside DSPy-style pipelines. It provides reproducible calibration, confidence-gating, latency, and modeled-cost benchmarks, plus offline replay.
How it uses Jev
Jev makes Choice decisions on support-ticket routing state, returning a selected owner and confidence. The benchmark applies a confidence gate: decisions below the threshold become predicted: null and abstained: true. Metrics like coverage, accuracy among answered, selective risk, Brier score, and ECE are computed from these gated decisions.
Primitives:choicenoul
Technique worth stealing
Confidence-gated abstention with leakage-safe Platt scaling calibration fitted only after the gate is declared.
Try it
uv sync && uv run jev-dspy-benchmark --dataset fixtures/tickets.jsonl --responses fixtures/typesafe_responses.jsonl --output evidence/benchmark --field owner --threshold 0.7 --bootstrap-samples 2000 --seed 17
Evidence
Each line is one question put to Jev about the README. ≥ 0.60 reads as yes, ≤ 0.40 as no; in between Jev is not making a call.
- Jev-centricyes0.77
- Shows a System One patternyes0.72
- Handles uncertaintyno0.12
- Measuredno0.13
- Runnableno0.09
- Worth recommendingno0.27
- Model replicano0.05
- Problem scopescore on a 0–2 scale1.01
- About Jevyes0.97
Signals by Jev jev-1.13.0, card written by DeepSeek V4.1 Flash from the README on 20 Sept 2026.