What it is
A graded relevance evaluation over the Agent Skills Hub catalog (33,047 skills, MCP servers, coding-agent tools) with 164 Chinese/English/mixed queries and 9,831 labelled pairs, comparing Jev reranking against lexical, embedding, and fusion baselines.
How it uses Jev
Jev is used as a score reranker over another system's top 30 candidates, one API call per query, and also as one of two judges producing 0–3 relevance labels. The README does not describe how Jev is called in code beyond the OpenRouter /alpha/decisions endpoint.
Primitives:score
Technique worth stealing
Re-score every system under three label sets, including one Jev had no part in, to expose judge circularity.
Try it
python -m jse robustness (no API calls); full tables in results/robustness.md
Evidence
Each line is one question put to Jev about the README. ≥ 0.60 reads as yes, ≤ 0.40 as no; in between Jev is not making a call.
- Jev-centricno0.34
- Shows a System One patternno0.13
- Handles uncertaintyno0.04
- Measuredunclear0.45
- Runnableno0.11
- Worth recommendingno0.25
- Model replicano0.14
- Problem scopescore on a 0–2 scale1.06
- About Jevyes0.98
Signals by Jev jev-1.13.0, card written by DeepSeek V4.1 Flash from the README on 20 Sept 2026.