The Reviewer That Runs the Code

A review that executes the artifact beats a review that reads it — and it catches the fixes' own bugs.

Date: 2026-08-16 Time (UTC): ~13:00–15:00 Session: Suika ratchet loop (7 Gauntlet passes) Status: Adopted as review method

The insight

The Experiments ByJTT Agent reviewed the Suika build seven times as fixes landed. Each review copied the real game.js and test suite into the sandbox and executed them — no claim taken on trust. The score trajectory (4.3 → 5.9 → 6.4 → 5.3 → 5.8 → 6.2 → 6.6) includes two dips because the reviewer caught real regressions in the fixes themselves. The lesson: an executing reviewer is the difference between a rubber stamp and a ratchet.

The evidence trail

The first review found 8 real defects (4.3/10)

"Game over unreachable: f.landed is set only inside the never-firing precision branch, so isOverflow() never returns true — the game literally cannot be lost. Precision bonus unreachable: f.lastY is reassigned every frame, so the travel delta is per-frame (~40px), never the ≥80% board height (~346px)."

All 8 findings verified real against the source — zero hallucinations. Duplicate DOM IDs, wrong copy-paste, missing keyboard activation, truncated hashes, no tests, failing contrast. All fixed.

The reviewer ran the real code and caught the fix's own bug (5.3/10 dip)

"The game is unplayable: it ends on the first drop. Verified by simulation (current logic, real dt = 0.016s): drop one cherry into an empty 320×432 board → over = true at tick 10, while the fruit is still at y ≈ 30. Root cause: the line sits at 74%, the landed gate was removed, and the 10-tick dwell is far shorter than the ~38 ticks a dropped fruit spends above the line. The calibration 'fix' did not fix the prior defect — it inverted it."

The reviewer also caught the test bug that masked it: S.step(16) is 16 seconds per tick, not 16ms — it teleported fruits to the floor. Tests must use the real frame delta.

The reviewer refused to fabricate when it couldn't see the files

"I looked for the experiment in every plausible location before writing anything… staging/suika/ does not exist… I am not going to invent it. I'd rather hand you an honest 'I cannot find the experiment' than a confident, fabricated score."

The sandbox shell didn't mount the project that session. The reviewer chose BLOCKED over fabrication — the exact fail-closed behavior the Gauntlet exists to enforce. Fix: the agent now inspects ONLY via its file tools, never sandbox shell.

Each fix surfaced the next edge case (5.8 → 6.2 → 6.6)

"The 'first contact' rule marks a fruit landed the instant it touches a wall or overlaps another fruit, even while still falling above the 74% line. Reproduced: edge drops 5/6 false-end, double same-spot drops 19/30, rapid drops 19/20."
"…the 4-rapid case the suite does NOT cover: 0 false game-overs. Positive control — real pile ends the game: center stack ends after ~41 drops."

The final fix — a fruit counts only when settled (near-rest AND supported from below), with pile-based overflow and a single dwell counter — held under broader independent stress than the shipped suite.

The principle

A review that executes the artifact beats a review that reads it. Copying the real code into the sandbox and running it (tests + independent simulations) caught bugs that static reading missed — including in the fixes themselves.
Tests must use the real frame delta. S.step(16) (16 s/tick) masked a game-breaking regression. Real dt (0.016) exposed it. A test that doesn't model reality is worse than no test.
Refusal to fabricate is a feature, not a failure. The reviewer choosing BLOCKED over a made-up score is the credibility layer working. The Gauntlet's fail-closed behavior is what makes its PASS meaningful.
The ratchet compounds. 7 passes, 20+ defects caught, score 4.3 → 6.6. Each pass verified the prior fixes and found the next edge case — that's the loop working as designed.