I wanted to know if an AI could cover the manual testing a human does before release, so I raced a decision model against a hand-coded bot on 21 agent-built Snake games. Same rules, same safe moves. The AI won.
CI can tell you an agent-built app compiles, lints, and passes its tests. It can’t tell you whether a person can actually use the thing. That check usually falls to a human clicking around and trying things out before release: exploratory testing, manual QA, a “human in the loop”, whatever your team calls it. Your other option is hand-coding a bot to do the clicking, and then maintaining that bot forever.
I built a small research test rig called Understudy to find out how much of that human click-through an AI player can cover, and whether it beats the hand-coded bot. Snake keeps the question easy to score: can somebody get through this, and how far? This is Understudy’s first real race.
Why “Understudy”? In theater, understudies learn the part so they can go on when the lead can’t. Same idea here: an agent that stands in for the human click-through, not a replacement for the show.
Here’s the setup in plain English. Twenty-one Snake games, all built by coding agents in earlier Loop Lab experiments. Two players:
Greedy bot I vibecoded by hand in JavaScript: a short, fixed rulebook that chases the nearest food unless that would trap it.
Jev, a decision model from Typesafe that picks one answer from a fixed list instead of writing paragraphs of advice like a typical LLM would.
Both see the same board. Both get the same list of moves that won’t kill the snake on each tick of the game. Each game runs until the snake dies, and the score is how much food it ate.
Jev averaged 95.0 points per game. The greedy bot averaged 72.6. A second Jev setup with extra strategy tips in its prompt averaged 94.5, a dead heat with the first. On average, the tips bought nothing. But, that’s a second story. The main story is that Jev outscored the greedy bot on 19 of 21 builds.
Think of it as bowling with bumpers. Both bowlers had the gutters blocked, so nobody threw a gutter ball. The question was only who knocked down more pins.
Hypothesis
If an AI decision model is held to the same safe-move list as a hand-coded bot baseline and scored until death on a fixed set of games, it should beat the baseline on mean score.
Why that bar? If the model can’t outscore a rule book that fits on one screen, in a game with five buttons, it has no business standing in for a human tester on anything harder.
And the CI idea I actually care about (i.e., freeze a set of scenarios you replay every night or before a release, keep a cheap deterministic player as the floor, run an agent “manual tester” against the same scenarios, compare) only makes sense if the agent beats the free player on something you can measure: higher scores and longer runs on the same games. If it can’t, just run the free bot and save the tokens.
Setup
The games. The corpus is 21 Snake builds pulled from half a year of Loop Lab Snake testing: six from cost-of-a-green-PR calibration runs, ten from the CI vs no-CI study, and five from the Chunk sidecars / 100% green PR runs.
Same spec, different agent runs, so 21 slightly different versions of one app. Food uses each game’s own random spawner (Math.random, like a real browser session), so no two runs see the same food sequence. Runs end at death, with two safety valves: a 10,000-tick ceiling and a stuck detector that kills a run after 2,500 ticks with no score change. A tick is one step of the game clock, one move.
The shared move surface. Before either player picks a direction, a shared scorer looks at the board and grades every option. That scorer lives in one file (evaluate.ts) that both the greedy bot and Jev read from, so the rules can’t drift between them. For each direction it reports:
Fatal or not: reversing into your own neck, hitting a wall, or hitting your body
Distance: how far the next square is from the food
Space: a flood-fill count, meaning how many open squares the head could still reach from there
Need: snake length plus two (plus one if the move eats), the room a snake needs to not box itself in
Escape routes: how many safe moves exist after this one
Those are the bumpers.
SAFE means a direction that is not fatal on this tick: it won’t reverse into the neck, walk off the grid, or land on the body.
The players:
Here’s the heart of what Jev sees each tick, in plain English, expressed as human-readable intent. Only SAFE directions are listed. Then:
Among the listed SAFE choices:
1. Prefer moves that still have an escape after this one.
2. Prefer roomy squares (enough open space for the snake's length).
If every option is tight, don't take food when a roomier non-food
move is on the list.
3. Among what's left: closer to food, then more open space,
then keep the current facing on ties.
Pick exactly one listed direction.That’s it. The model makes the final call.
The Jev strategy profile specifically adds four short tactics on top: keep an escape into open space, don’t hug walls early, don’t cut off your own tail, break distance ties toward open space or the center. On average, those tips bought nothing on the scoreboard. Different tips might. That’s a future experiment.
The alternative is the Greedy player: lengthy JavaScript that encodes the same procedure as hard-coded branches, and has to be extended and maintained every time you want a smarter heuristic.
Jev gets the procedure as readable policy text, plus the scored candidates from evaluate.ts. Same bumpers. Less glue code to own.
Jev ran as jev-latest through Typesafe’s System One API, one call per tick, answering a single multiple-choice question: which direction? Only safe moves were offered whenever any existed. If Jev ever answered with a move outside that list, Understudy swapped in greedy’s pick for that tick.
Variables. Changed: the player. Held constant: the 21 builds, the move surface, the safe-move filter, one decision per tick, and the run caps.
Cost. Greedy runs free on my laptop’s CPU. Both Jev passes together cost about $12 on my personal Typesafe account, covering more than 109,000 decisions.
Results
All 21 games finished at a normal game over for both Jev profiles. Greedy is $0 (i.e. FREE) CPU on my laptop, including all 105 runs. Each Jev profile was one pass over the 21 games at about $6; both together about $12 on my personal Typesafe account.
Quick unit math on that bill, using the run totals (one API call per tick, so ticks ≈ decisions):
So: about 29¢ per game, and about 11¢ per thousand decisions. Cheap for a nightly corpus job. Still a real line item if you fire it on every commit.
Head-to-head, Jev strategy against Jev parity on the same 21 games:
NOTE: One run went sideways. Strategy’s first try on one game variant ate only one food pellet, then circled the next random food pellet for 2,516 ticks without scoring until the stuck detector ended it at a score of 1. No API errors. Every move stayed on the safe list. Since that low score was merely bad luck due to bad luck with food placement, I archived that replay and reran the game: 106. The table uses the rerun. If I had kept the score of 1 instead and strategy’s mean dropped to 89.5, it would still be well above greedy and still no better than parity.
By the way, the above anomaly is not necessarily a problem with the Jev strategy prompt, but rather a limitation of attempting to automate a human in the loop. A human being would never play a game like that.
What a CI pipeline config would read
Every run writes a trace (JSON, every tick) and a self-contained replay page that opens in any browser. That’s the raw material for a CI gate. I have not wired this into a pipeline. The sketch below shows the shape, and the gate script in it doesn’t exist yet:
# Sketch only. score-gate.mjs is hypothetical.
jobs:
understudy-play:
docker:
- image: cimg/node:22.13
steps:
- checkout
- run: pnpm install
- run: pnpm sim:batch -- --player greedy --results-name control
- run: pnpm sim:batch -- --player jev --results-name agent
- run: node scripts/score-gate.mjs --floor control --candidate agent
- store_artifacts:
path: resultsThe control run sets the floor. The agent run has to clear it. The replays go up as build artifacts so a human can watch the worst game in a couple of minutes instead of replaying all 21 by hand.
TL;DR
The lab claim held. With identical bumpers, Jev outscored a strong hand-coded bot by about 22 points per game and lived roughly 900 ticks longer. That wasn’t one hot streak either: parity beat greedy’s per-game average on 19 of 21 builds.
So here’s my recommendation. If you’re choosing between hand-coding a bot and paying a human to click through every build, a decision model is worth a serious look as the stand-in. The policy is a paragraph of readable text, not a pile of branches you maintain forever, and it played better.
Coaching it to “think strategically” hit diminishing returns. That’s the secondary finding, not the headline. My quick four-tip prompt left the average flat: parity and strategy split 9 to 9, with strategy about 0.6 points behind. The best and worst games did nudge up (70 / 128 against parity’s 66 / 119, counting the rerun), but on one pass per profile I won’t call that a win.
My read: most of Jev’s edge comes from weighing the features it’s handed, so put the knowledge in the state you pass it (escape routes, open space) before you write a paragraph of advice. The right coaching might still help. These four tips weren’t it, and I need more runs before I say anything definitive.
The practice claim is an implication, not a measurement. Nothing here ran inside a CircleCI pipeline. But the shape works: frozen scenarios, a free deterministic control, an agent player, scores plus replays (as HTML artifacts). I’d use it when:
The question is “can somebody get through this?” and you can score it (a game, a checkout flow, a signup)
You have a fixed scenario set the agent replays every night or before a release
You want a human to watch a two-minute replay of the worst run, not click through everything
Btw, watching a snake circle one food pellet for 2,516 ticks until the stuck detector fires is the best part of the job. Same for the unwinnable food placements: maybe a game should never spawn food you can’t reach. Every replay artifact teaches something. None of them were written by hand. 100% of them were written by my Understudy agent.
I wouldn’t use Understudy on every push of every PR. One API call per tick, in sequence, takes hours per corpus pass, and tokens are the tax the free control never pays. $12 for 42 full games is cheap for a nightly job. Per commit might get a bit pricey, depending on the app.
I also wouldn’t ask Understudy about taste. A high Snake score means the game is playable to the end. It says nothing about whether the game is ugly, confusing, or fun, although we are still working on that.
Limits, plainly: one Jev run per game against five for greedy, one app shape, and no quality-judge pass on these traces yet. I stopped after one full corpus pass per Jev profile because the gap over greedy was already clear and the token bill is real. More replicates would tighten the error bars. They would not change the headline.
A better next step than burning five times the API spend is pointing the same pattern at an app with screens to click through instead of a snake to steer. That’s the next experiment.
Bumpers keep the ball out of the gutter. A human in the loop still has to watch the replay. Understudy’s job is making sure there’s a replay worth watching before that human ever gets involved.