What it is
A pytest plugin for testing LLM app output by meaning rather than exact wording. It provides a `jev` fixture for writing semantic assertions about text, aimed at developers testing LLM applications.
How it uses Jev
Jev judges claims about a text: `expect` batches Noul claims, `choice` picks an option, `score` rates ordered levels. Results are calibrated probabilities used by the plugin's thresholds to pass or fail tests.
Primitives:choicescorenoul
Technique worth stealing
Batch all claims about one text into a single Jev request, then apply ordinary Python thresholds to the returned probabilities.
Try it
pip install pytest-jev, set OPENROUTER_API_KEY or TYPESAFE_API_KEY, write tests using the jev fixture, run pytest.
Evidence
Each line is one question put to Jev about the README. ≥ 0.60 reads as yes, ≤ 0.40 as no; in between Jev is not making a call.
- Jev-centricyes0.94
- Shows a System One patternyes0.70
- Handles uncertaintyyes0.91
- Measuredyes0.78
- Runnableyes0.97
- Worth recommendingunclear0.58
- Model replicano0.03
- Problem scopescore on a 0–2 scale1.07
- About Jevyes0.98
Signals by Jev jev-1.13.0, card written by DeepSeek V4.1 Flash from the README on 21 Sept 2026.