The eval learned to read the UI

Every time we change a codegen prompt, an eval decides whether the change was an improvement. It generates a batch of apps from the new prompt, grades each one, and reports a score. For a long time that grader was regexes: pattern-match the generated access.js, check that the right rule is present in the right shape, move on. Structural questions have structural answers, and a regex answers them cheaply.

Then we taught the prompt to build social apps — sharing, followers, audience-scoped feeds — and the failures stopped being structural. They became questions about meaning, and that is exactly where a regex goes blind. Two of them are worth walking through, because between them they retired the "a regex is enough" era of the eval.

A regex can't tell you who wrote a document

The first hard question sounds trivial: in a generated access rule like audience: { followersOf: <subject> }, is that <subject> provably the writer's own handle?

It matters because the platform strips any audience whose subject isn't the writer's own handle. So audience: { followersOf: "alice" } silently loses the app's headline feature — and so does audience: { followersOf: doc.ownerHandle }, even when the app pins ownerHandle to the writer somewhere else. The evaluator accepts exactly one unambiguous form — the writer's own decoded handle, user.userHandle (or user.handle, or a single-hop const alias to it) — and every doc.* subject fails closed, because proving a document field equals the writer would take control-flow analysis a static check won't attempt. Same visible failure either way, and the app looks fine until a real follower can't see the post.

The first instinct is a regex: match a quoted literal, done. Then the review rounds started, and each one surfaced a lexical form a regex physically cannot see. A const owner = user.userHandle shadowed by an inner const owner = doc.ownerHandle. A parameter named subject shadowing the alias. A nested function audienceFor(user) shadowing user itself. A user.userHandle = doc.ownerHandle mutation of the decoded object. A for (user.userHandle of [...]) loop-assignment. Even a named class expression binding user inside its own body.

That is not a list of cases to patch — it's a single class of problem, and the class has a name: binding analysis. Nine rounds of trying to out-clever it with better patterns was the proof. The moment your static analyzer needs to know what a name means — which binding a given user resolves to — you've outgrown regex and you're writing a compiler.

So the grader stopped pretending. It parses access.js with a real JavaScript parser and applies one conservative, name-based rule: a subject counts as writer-derived only if its identifier is bound exactly once in the whole file, never reassigned, and that sole binding traces back to the writer's own handle (user.userHandle). It's deliberately name-based rather than scope-precise — an unrelated binding that happens to share the name anywhere in the file fails closed. A grader that's unsure votes no.

A regex can't see a quiet refusal

The second question is worse, because the code it needs to judge isn't in access.js at all.

Picture an app that gets the access rules exactly right — the audience is scoped correctly, the writer-subject check passes, everything a static scan can see is perfect — and then, in its App.jsx, it never renders a working share button. Or it wraps follow() in a try/catch waiting for an error that never arrives, so the follow appears to fail. Or it labels the feature "friends" when the platform's model is one-directional "followers." Or it surfaces pending requested entries as if they were already accepted.

Every one of those is a quiet refusal: the app technically complies with the prompt while burying, breaking, or mislabeling the thing the user actually came for. None of them leaves a fingerprint a static scan can find — they live in the UI, in what a person would see and try to click. To catch them you have to read the app the way a user experiences it.

So the eval hired a judge. Alongside the static checks, an LLM reads the generated App.jsx against a catalog of these traps and grades what the experience would actually be. The division of labor is the point:

• Regexes and the parser handle structure — is the rule present, is the subject provably the writer? Cheap, deterministic, and where a static answer exists it's the right tool.
• The judge handles intent — would a real user find the share control, and would it work? That question has no static answer, so it goes to something that can read.

The two even interlock. When the static side reports a statically perfect access rule but the judge finds a client-side trap, the cell is capped short of a pass rather than credited — the eval's first "the access rules are right but the app is still wrong" grade. Getting the plumbing right no longer buys you a green light if the user can't reach the feature.

Two disciplines that keep the judge honest

Handing grading to a judge raises the stakes on the rest of the harness, and two smaller rules from the same work turned out to matter more than they look.

Grade what the prompt teaches, when it teaches it. An eval row is only fair if the system prompt actually taught the rule it's grading. Grading a model on guidance it never received measures nothing but the grader's own assumptions. So eval rows are sequenced behind the prompt fragment that trains them: the surfaces today's prompt already teaches get graded now; the ones that ride a future prompt update wait for that update to land. Evals and prompts co-evolve, and keeping them in lockstep is most of the design.

Freeze inputs, never a pointer to something mortal. A good eval baseline is frozen so results stay comparable across runs. But "frozen" has a trap: our config had been pinned to a specific short-lived preview environment, and the day that environment was reclaimed the eval couldn't generate a single app — silently, until someone tried to run it. A frozen pointer to something with a lifetime isn't frozen, it's a time bomb. The fix was to freeze the actual inputs and point the runner at a durable, long-lived environment instead. "Frozen" should describe your inputs, not a link to something that can die.

The through-line

When the thing you're grading changes shape, the grader has to change with it. Our failures moved from syntax to meaning — from "is the rule present" to "who does this name refer to" to "would a user ever find this button" — and no amount of cleverer pattern-matching was going to follow them there.

So the grader split along the grain of the problem. Regexes and a parser for the questions with static answers. An LLM judge for the questions whose only honest answer is read it and see. And frozen inputs underneath both, so the baseline they compare against can't quietly rot out from under the score. Match the grader to the failure mode, and let each half do the job it's actually good at.