jev-canvas

What it is

A multimodal UI where users draw on a tldraw canvas by speaking commands while pointing a webcam-tracked finger. It supports English and Ukrainian, and uses Jev to interpret spoken instructions into canvas actions.

How it uses Jev

On every partial or final transcript, Jev answers eight typed questions: is_command, complete, action, shape, color, target, where, size, and text_span. Probabilities are thresholded in code to decide whether and how to create, move, delete, recolor, resize, duplicate, clear, undo, or redo shapes.

Primitives:choicescorenoul

Technique worth stealing

Capture pointing words at speech time with a 3s finger trail and read back 300ms earlier to align words with positions.

Try it

git clone, cp .env.example .env, add OpenRouter key, npm install, npm run dev, open localhost.

View on GitHub