The problem
You built a cleanup tool. It walks a repository, asks Jev whether each file is a leftover build artefact, and deletes anything above 0.8. Three months in, people like it — tens of thousands of dead files gone.
It has also deleted a few things it should not have. You decide to raise the line to 0.9, and before you do, you want two numbers: over these three months, how many deletions would not have happened, and would the new line have caught the mistakes?
You open the log and every line reads "deleted build/tmp.log". Nobody recorded what the probability was at the time. To answer either question you would have to feed three months of files back through the model — and many of them do not exist any more. You are about to change the line with your eyes shut.
The solution
The flight recorder pattern says: a crash investigation reconstructs every second from the recorder, and your decisions should reconstruct the same way. Start by making the deciding part a pure function — same input, always the same output. In goes this call’s numbers and last time’s conclusion; out comes an action. No model call inside it, no clock, no reading a config file.
Then write one record per decision: the raw numbers, the outcome, the previous outcome, and a fingerprint of the input (a short string that stands for it). Not the input itself — partly for space, mostly because the input may contain things you should not be keeping.
Now changing a line is cheap. Run the records back through the new function and count how many conclusions flip. It costs nothing, and tens of thousands of records take seconds. One team made it a single command that prints "changed: 12" after you edit a threshold; another replayed 1,256 edits from 93 real sessions for 22 cents.
Ask · Answer · Act
- AskPut the file path and the first few thousand characters — the material Jev is shown, which the docs call the state — in front of it and ask whether this is a leftover build artefact.
- Answer0.83.
- ActThat is over 0.8, so the file is deleted, and one line goes down alongside it: the number was 0.83 and the outcome was delete. Three months later you want the line at 0.9, so you recompute those three months of records and find that twelve of them flip.
Where the name comes from
The recorder on an aeroplane does not note down "flight went fine". It keeps altitude, speed and control positions second by second, which is the only reason anyone can later answer "what if the pull-up had come three seconds later". Your log should keep the same kind of thing: the numbers, not the verb.
Use it when
Any decision with a line in it that will later be tuned, audited or explained to somebody — which is to say, every line you have ever drawn.
Not when
A script you run once and throw away. Or input carrying personal data that cannot be reduced to a fingerprint, in which case the records themselves become the liability.
Read this code
- huncho
src/replay.ts:79-90Old records are pushed back through the decision function with the new lines in place, and it prints how many conclusions moved. No model call, no cost.
- huncho
src/journal.ts:35-64The record says exactly what it keeps: the numbers, the outcome, last time’s outcome, and a fingerprint of the input — never the input itself.
- abide
benchmarks/replay/README.md1,256 edits rebuilt from 93 real sessions and run again for 22 cents and two minutes — and the honest result published alongside.
Show me the code
const decide = (a, prev, t = { del: 0.8, ask: 0.6 }) =>
a.leftover.p >= t.del ? "delete" : a.leftover.p >= t.ask ? "ask" : "keep"; // pure: no model, no clock
const outcome = decide(answers, previous);
journal.write({ at: Date.now(), answers, previous, outcome, stateHash: sha(state) });
// three months on, before touching the line, for zero tokens:
const NEW = { del: 0.9, ask: 0.6 };
const moved = journal.read().filter((r) => decide(r.answers, r.previous, NEW) !== r.outcome);
console.log(`changed: ${moved.length}`);How jevbooks recognises it
How do we know a project does this? We put one question about its README to Jev and it returns a percentage; the higher, the more it looks like it. If you copy the question, copy one thing: ask whether the text mentions this one concrete thing, never whether the project is good.
{
"journal-replay": {
"type": "noul",
"instructions": {
"what": "Does the text describe keeping a journal or JSONL log of Jev's answers and the resulting decisions that is replayed, audited, or used to tune thresholds without calling the model again?",
"not_for": "Ordinary request logging with no replay or tuning use."
},
"criteria": {
"true": "The text explicitly describes this. One sentence is enough; other content does not cancel it.",
"false": "The text does not describe this."
}
}
}