jevbooks

← All design patterns

Question bank

Rules as questions · 题库

One rule, one question

Turn every line of your team's rules into a question of its own, so that a check is just those questions run again.

rulesnoulnoulscore

The problem

You are building a rule enforcer. The team's conventions live in one document — no console.log in shipped code, change the docs when you change an interface, and so on — and you want every edit your assistant makes to be checked against them.

Version one has a large model re-read the whole document every time: two seconds and a cent per edit, so a few thousand edits a day cost real money and a lot of waiting. Worse, it is unsteady — edit 200 gets a different answer from edit 1, because the conversation around it has changed.

So version two folds everything into one question: 'does this edit follow all the rules?' It answers about 0.55 to almost anything, because one rule is broken and nine are fine. One number cannot tell you which one broke, and you cannot act on it.

The solution

The Question bank pattern says: one rule is one question, and one question asks about one thing. A rule with two halves becomes two rules and two questions.

Each question asks whether something is there, not whether something is good. 'Does this change add a console.log that would ship?' can be answered. 'Is this change any good?' cannot. Give every question a short list of the near-misses that do not count: logging behind a debug flag, logging in test files.

Rules that count, measure or compare dates never go to the model — it lands around 0.4 on those whatever you show it, while three lines of code count exactly. The document itself can be translated into a bank of questions once, offline, by a large model; after that every check only costs a Jev call.

Ask · Answer · Act

  1. AskPut in this one edit and one rule, and ask whether the change adds a console.log that would ship.
  2. Answer0.91.
  3. ActOver 0.8, so the edit is held back and the author is told which rule it hit; between 0.5 and 0.8 it is only mentioned, not blocked.

Where the name comes from

A driving test's question bank. The law is written as prose, but the examiner never asks 'how good a driver is this person?' — it is one item at a time: seatbelt on, indicator used, mirror checked. Each item tests one thing, which is why you can be told exactly which point cost you.

Use it when

The rule fits in a sentence, can be judged from one input, and the answer is whether something is present or not.

Not when

The rule counts, measures or compares dates — that is code. The rule needs the whole repository to judge. The rule is about process, like "ask a human when unsure", which nothing can execute.

Projects that do this

  • abide — Enforce project rules on coding agents with Jev.
  • snifftest — Prose linter that flags AI writing tells with countable rules and one judgment model.
  • mastra-jev-moderation — Mastra input moderation on Jev: one file, one typed request.

All projects with this pattern →

Read this code
Show me the code
{ "no_console_log": {
    "type": "noul",
    "what":     "Does this change add a console.log that would ship?",
    "not_for":  "Logging behind a debug flag, or inside test files.",
    "true":     "A console.log is added in shipped code.",
    "false":    "None is added, or one is removed.",
    "scope":    ["src/**/*.ts"],
    "threshold": 0.8 } }
How jevbooks recognises it

How do we know a project does this? We put one question about its README to Jev and it returns a percentage; the higher, the more it looks like it. If you copy the question, copy one thing: ask whether the text mentions this one concrete thing, never whether the project is good.

questions.json · sent for every README
{
  "rules-as-questions": {
    "type": "noul",
    "instructions": {
      "what": "Does the text describe each natural-language rule, policy line, or rubric item becoming its own typed Jev question, written by hand or compiled once by a larger model?",
      "not_for": "One catch-all question about the whole policy, or rules enforced by code alone."
    },
    "criteria": {
      "true": "The text explicitly describes this. One sentence is enough; other content does not cancel it.",
      "false": "The text does not describe this."
    }
  }
}

Build this with Jev →