What it does not replace is the wide end — any judgment that needs background knowledge the state does not contain, weighs things that are not alike, or has to explain itself in a sentence. We measured where that line falls, on our own data, and the honest answer is below.
Never heard of Jev? Start with What is Jev? →
Side by side
Jev’s column is measured; the LLM-judge column describes the shape of the approach, not one particular model.
| Jev | LLM judge | |
|---|---|---|
| Output | A typed answer plus a probability for every option (noul / choice / score) | Free text, or JSON it wrote and you parse |
| Latency | About 300 ms; extra questions ride the same forward pass | Seconds to minutes, growing with the prompt and the reply |
| Cost | $0.042 per million input tokens, output free | Frontier rates, and you also pay for every token it writes |
| Calibration | A probability the model is trained to calibrate; 0.5 means “can’t tell”, not “medium” | A confidence it writes down — prose about a number, not a measured one |
| Stability | Same state and wording give the same distribution within a small drift (±0.02–0.09 on confidence between runs, measured) | Rewording, option order and temperature move the verdict; output needs parsing and repair |
| Good at | Narrow literal calls: intent routing, safety gates, rerank, rule checks, “does this text contain X?” | Holistic weighing, background knowledge, writing the reason, proposing the fix |
| Bad at | Counting, arithmetic, dates, multi-hop, “which of these is best overall” | Cost and latency once you call it per item; producing a number you can threshold |
| Can it generate? | Never | That is the whole point of it |
What we measured
We gave 150 Jev projects to several judges and asked each for the ten best — deliberately the holistic kind of task Jev is supposed to be bad at. Jev’s side was built properly: one request per project with a text-only state, eight narrow existence questions with symmetric criteria and a not_for list, a 0.4–0.6 no-judgment band, one invisible cross-check question, then a pairwise round over the top 20 asked in both orders. All weighting and ranking happened in code.
Position bias is real and worth budgeting for: in the pairwise round Jev leaned −0.14 toward the second card, which is why every pair was asked twice.
- Overlap with Fable’s top 10
- Jev5 / 10Claude Fable6–7 / 10
- Cost
- Jev$0.027Claude Fable≈ $0.95
- Wall clock
- Jev30 sClaude Fable105 s
Jev’s 5/10 is against Fable’s top ten; 6–7/10 is what the other Claude models score against that same list. Cost and wall clock are for one full pass over 150 projects.
Claude models agree with each other on 6–7 of 10; Jev plateaued at 5, and the extra technique we added between attempts did not move it. The disagreement is not wording, it is knowledge: Jev rewards a README that states its measurements and its thresholds, while Fable rewards a project that matters to the ecosystem — something the text never says. So: narrow, literal choices are Jev’s, and it is 3–10× faster and two orders of magnitude cheaper there; choices that need background knowledge and holistic weighing are not, and no amount of prompt craft closed the gap.
| Jev | Claude Fable | |
|---|---|---|
| Overlap with Fable’s top 10 | 5 / 10 | — (Opus 7 / 10, Sonnet 6 / 10, Haiku 6 / 10) |
| Cost | $0.027 for 530 calls | ≈ $0.95 |
| Wall clock | 30 s, 8 calls at a time | 105 s |
| Screening pass alone | $0.01 and 9 s for all 150 | — |
Both halves of that result are patterns you can read on their own: Cheap screen, then deep check · Rules compiled into questions
When to use which
Five questions that settle it, in the order they usually matter.
Can the question be answered from the text in front of you, literally?
Jev. “Does this message ask for a transfer?” is a Jev question; “is this a good design?” is not.
Will you ask it thousands of times — per item, per edit, per agent step?
Jev. Cost and latency per decision are the whole argument, and an unused question is nearly free.
Do you need a number to threshold, a band to abstain in, a gate you can tune?
Jev. A language model’s confidence is prose; a calibrated probability is a knob.
Does the answer need background knowledge the state does not contain, or a comparison of things that are not alike?
The LLM judge. This is where our 5/10 came from.
Does someone have to read the reason — a review comment, a summary, a suggested fix?
The LLM, always. Jev cannot produce a sentence, only a number.
Combine them
Cheap screen, expensive look. Our own run supports it: the screening pass alone was $0.01 and 9 s for 150 projects, so let Jev cut the field by 80 % and spend the frontier model only on the survivors.
Let Jev lint what the LLM wrote. jevbooks does exactly this on every request: a language model drafts the questions for your problem, then Jev scores those questions — does this one ask two things at once, does it ask for an impression instead of a fact, is a not_for missing, do two of them overlap — and the warnings appear before you copy anything.
Compile once, execute forever. Have the expensive model turn your written rules into typed questions offline, and pay Jev’s price on every check afterwards. One-off work at frontier prices, per-call work at $0.042 per million tokens.
Next
Ask Jev about your own problemBrowse the galleryOfficial docsGet an API key