jevals

What it is

jevals is a Python library for evaluating and guarding AI agent traces. It replaces LLM judges with Jev-style decision models, running all evals for a trace in one request for a fraction of a cent and a few hundred milliseconds.

How it uses Jev

Jev answers typed questions (Choice, Score, Noul) about the agent's state, such as whether a tool was used correctly or if a claim is grounded. The returned calibrated probabilities are reduced into scores, labels, or pass/fail results used in code for offline metrics, production monitoring, and in-loop gates.

Primitives:choicescorenoul

Technique worth stealing

Packing all evals for a trace into a single request of typed questions, evaluated in parallel in one forward pass.

Try it

pip install jevals; set AI_GATEWAY_API_KEY; call evaluate(sample, evals).

View on GitHub