jev-eval

What it is

A benchmark and Gymnasium-style one-step bandit for typed decision models like Jev/System One. It evaluates models that map unstructured state and a closed question to a probability distribution over declared candidates, with no free-text generation.

How it uses Jev

Jev-like models receive state and a closed question Q=(tau, iota, C) and must return a distribution p over candidates. The environment exposes observation {state, instructions, type, candidate_ids, K} and rewards actions p or a discrete index via log, brier, or zero_one scoring.

Primitives:choicescorenoul

Technique worth stealing

One-step contextual bandit with strictly proper scoring rules as reward for training a decision head against frozen labels.

Try it

uv sync; uv run jev-eval list; uv run jev-eval export --suite all --out /tmp/jev-eval.json

View on GitHub