What it is
S1Rank evaluates whether Jev, a non-generative model answering typed questions with probabilities, can rerank BM25 top-100 candidates for information retrieval. It provides a benchmark, raw responses, and a paper for researchers.
How it uses Jev
Jev answers Noul yes/no questions per document in one request holding all 100 candidates (joint pointwise), producing a probability per document used to rank. Also tested: isolated pointwise, graded Score, Choice-best, Peel, Tournament, and S1Adaptive routing.
Primitives:noulchoicescore
Technique worth stealing
Joint pointwise: one request with all candidates and one Noul question per document.
Try it
python scripts/fetch_data.py && python scripts/analyze.py
Evidence
Each line is one question put to Jev about the README. ≥ 0.60 reads as yes, ≤ 0.40 as no; in between Jev is not making a call.
- Jev-centricyes0.94
- Shows a System One patternyes0.62
- Handles uncertaintyno0.11
- Measuredno0.16
- Runnableno0.06
- Worth recommendingno0.33
- Model replicano0.04
- Problem scopescore on a 0–2 scale1.09
- About Jevyes0.98
Signals by Jev jev-1.13.0, card written by DeepSeek V4.1 Flash from the README on 23 Sept 2026.