Screen Assistant — voice-driven computer helper

My grandfather stayed with us for a summer and bought his first computer. He'd find me every five minutes — how do I send this Gmail, how do I copy this out of WhatsApp — and I was working nine to five. Existing assistants either answer in text he then has to map onto the screen, or take over the machine, and if it drives the computer for him he never learns it.

A click-through overlay that listens to speech, watches the screen, and renders live guidance on top of whatever app is open — a cursor pin, a highlight box, an arrow, a text bubble, a spotlight, a keyboard-shortcut card — driven by a small set of LLM tools. It deliberately cannot click or type. Grounding is two-stage: a planner emits coarse coordinates that UI-TARS refines to pixels (Claude's native vision skips the grounder entirely); a grayscale-MAE gate on a 32×32 thumbnail decides when the screen actually changed so the expensive model isn't called on every frame; speech streams both ways over GPT-Realtime-2 with barge-in. I started in Electron and TypeScript, which couldn't reach the macOS Accessibility APIs or ScreenCaptureKit; after three Swift helper processes the full native rewrite stopped being a decision.

The Swift port is complete — it builds and passes 1,247 tests. At the UC Berkeley AI Hackathon in June 2026 a version of it won the AI for Accessibility track, after we cut a whole annotation toolbox down to a single arrow mid-event because the models wouldn't call the tools reliably. The hardest part was never the model; it was speech and annotation staying in sync when both run at once.