jev-eval-agent

What it is

A personal-assistant agent built with eve (Vercel), served through OpenRouter, containing 100 mocked tools. It benchmarks how many steps the agent needs when the LLM picks the tool itself versus when Jev picks it.

How it uses Jev

In jev-classifier mode, Jev is called before every model step with state { user_request, actions_taken[], assistant_said[] }. It answers next_tool (choice over 100 tools + respond_to_user) and done (noul). If respond_to_user is chosen but done is below JEV_DONE_THRESHOLD, the router blocks the reply and exposes the next-best tool.

Primitives:choicenoul

Technique worth stealing

Confidence-gated routing: a noul 'done' check gates the choice's respond_to_user, falling back to the next-best tool.

Try it

cp .env.example .env.local, fill keys, npm install, then npm run eval or AGENT_MODE=jev-classifier npm run eval.

View on GitHub