For a support queue, you may need one answer to “does this message express urgency?” For a design review, you may need retrieved evidence, a comparison and a written explanation. Those are different jobs. Start with a small labelled set for your job; the ranking experiment below is one historical example, not a universal winner.
New to Jev? Start with What is Jev? →
What your application receives
Capabilities and tradeoffs, not a benchmark across all LLMs. Cost, latency and correctness depend on the model, route, input and evaluation setup.
| Jev | Generative LLM judge | |
|---|---|---|
| Output | Typed answers: noul, choice, score; experimental box in 1.14 | Text or structured output, depending on the API and configuration |
| Useful starting task | A bounded semantic question about supplied material, with an explicit unknown path | Synthesis, explanation, critique or a decision that needs several reasoning steps |
| Probabilities | Trained for calibrated decisions; validate calibration and thresholds on your own labelled data | A stated confidence is not evidence of calibration; evaluate it if you use it as a threshold |
| Cost and time | Input is $0.042 per 1M tokens; output free. Repeated narrow questions can be economical | Prices and latency vary widely; count retrieval, generation, retries and any judging calls |
| Missing evidence | Supply the missing material or abstain; a well-formed answer does not prove the evidence is sufficient | Retrieve evidence where possible, check the explanation and allow abstention |
| Application control | Your code applies thresholds, routes results and handles failures | Your code validates the output and applies the same failure policy |
One historical ranking experiment
On 19 September 2026 we asked several judges to select ten projects from the same set of 150. Jev v3 used project summaries, descriptions and the first 700 README characters. It asked eight questions per project, combined answers with weights in code, then compared the top 20 in both presentation orders: 150 screening requests plus 380 pairwise requests.
The pipeline treated 0.4–0.6 as no judgment and combined the screening and pairwise ranks. These are choices in this experiment, not validated defaults for another task. The saved pairwise analysis reports a mean position-bias statistic of −0.139, interpreted there as a preference for the second card; asking both orders was a mitigation, not proof that bias was removed.
- Top-10 overlap with Fable
- Jev v3 (1.13)5 / 10Opus · Sonnet · Haiku6–7 / 10
- Estimated cost
- Jev v3 (1.13)$0.027Claude Fable 5.1≈ $0.95
- Recorded wall time
- Jev v3 (1.13)30 sClaude Fable 5.1105 s
The overlap counts shared project IDs in two ten-item lists. Opus, Sonnet and Haiku shared 7, 6 and 6 with Fable. Costs and times describe this workload and configuration; they do not establish equal output quality.
Jev v2 and v3 each shared five projects with Fable. That is model agreement, not 50% accuracy or a ceiling on Jev’s ability. We did not hold the input coverage, questions, weights and ranking procedure constant across variants, so these results cannot isolate why the lists differ or prove that better wording would not help.
| Jev v3 (1.13) | Claude Fable 5.1 | |
|---|---|---|
| Top-10 overlap with Fable | 5 / 10 | Reference list, not ground truth |
| Estimated cost | $0.0267 for 530 requests | $0.946 |
| Recorded wall time | 29.633 s, concurrency 8 | 104.503 s |
| Screening stage only | 150 requests · 9.124 s · about $0.0097 | Not separately measured |
Two mechanisms you can adapt and evaluate: Funnel · Question bank
Inspect the saved evidence
Recomputed on 2 November 2026 from experiments/top10/compare.json and v3.json. The download contains the original timestamps, source-file SHA-256 hashes, selected project-ID lists, recorded timing/cost fields and recomputed intersections. It excludes request headers, credentials and raw transcripts. It is a curated extract, not a new API run.
# Node.js 22+; keep both downloaded files in the same folder node content-value-recompute-judge.mjs
- Fable is another judge, not an independently labelled answer key. No precision, recall or calibration score can be inferred from this overlap.
- One project collection and truncated README material do not represent support routing, safety review or the current gallery. There are no repeated-run uncertainty estimates here.
- The Jev pipeline and the LLM judging procedure are different systems. Estimated token costs are not billing receipts; concurrency and route affect wall time.
- These records concern Jev 1.13 in September 2026. They do not measure Jev 1.14 or establish that a screening-plus-LLM workflow improves quality.
Choose a first baseline
Use the cheapest method that meets a measured quality target for the actual task.
Is the rule exact and enumerable?
Start with code: parse a date, compare an amount, match an allowlist or count items. Keep deterministic work outside either model.
Is it a bounded semantic question about supplied material?
Try Jev against rules and a chosen LLM on the same labelled cases. Measure false positives, missed cases and how often you abstain before choosing thresholds.
Do you need synthesis, a written rationale or a suggested repair?
Start with a generative LLM and give it the source evidence. Validate factual claims and structured output; a persuasive explanation can still be wrong.
Is important evidence missing?
Retrieve it or ask for it first. If it remains unavailable, keep an unknown result instead of converting model confidence into a fact.
Will you call the judge repeatedly?
Compare end-to-end cost and latency at equal quality, including retries and human review. The cheapest individual call need not make the cheapest workflow.
Combine them with a measurable handoff
A screening stage can reduce the number of expensive reviews. Our historical screening stage took 9.124 s for 150 projects, but we did not test a complete Jev-to-Fable cascade against human labels. Before filtering anything out, measure how many valuable cases the screen misses and route uncertain cases to review.
A generative LLM can draft questions from your rules; Jev can check those questions for issues before you run them. jevbooks uses that division in Ask. These checks are hints for review, not a guarantee that a question or its thresholds are sound.
Bring one concrete task to Ask: the material, one decision question, allowed outcomes and what should happen when evidence is missing. Then save a baseline and compare the proposed workflow on the same cases.
Next
Ask Jev about your own problemBrowse the galleryOfficial docsGet an API key