Afriza Gilleon

Work / Rewind-Bench

7% of the best agent's accepted fixes were fake. A hidden seed caught them.

It passed. It was not fixed.

Catching AI code fixes that only look correct.

Measured

Why this stamp 67 injected bugs · 297 repair episodes · Reproducible from the recorded JSONL without spending tokens

Role
Solo
Period
August 2026
Stack
TypeScript · Node.js · Acorn (AST) · Vitest · Express · React · Vite · MCP
Limits
7 stated below
Proof

Problem

Rewind-Bench checks AI code fixes that only look correct. A fix can make a notebook’s output match on the data the agent was shown and still leave the bug in place. On data the agent never saw, the output differs.

The results report opens with a real case. One digit was changed in a working notebook: a.missedPayments * 22 became * 23. The repair the agent submitted left that line alone and added a second error of the same size in the opposite direction, so the two cancelled out. They cancel until Math.min clamps the value, and that happens on data the agent never saw. On the data it was shown, the output matched. On the data it never saw, it did not.

The other two agents failed that bug outright. So the only agent that “solved” it had not solved it, and a matching output on the data at hand was not proof that the bug was gone.

Decisions

Re-run every passing fix on a hidden seed

Bugs are injected into working notebooks by AST mutation, and each notebook has a hidden seed. A fix that passes on the visible data is run again on that seed. A fix that passes only on the visible data is marked a lucky pass.

Not chosen: judging a fix by the visible data alone. That is the check the * 23 repair passed.

Three agents, one variable

The bench runs three arms: code only, execution inputs made up by the agent, and real recorded execution inputs. Model, temperature, seed and the full source are held the same, so the three agents differ in one variable: where their upstream data comes from.

Not chosen: arms that differ in anything else, such as the model or the temperature.

Record the episodes, reproduce from the recording

Every repair episode is kept as a transcript, 297 in total. The results can be reproduced from the recorded JSONL without spending tokens.

Not chosen: reproducing the numbers by calling the models again.

Results

  • 67 bugs were injected, in two corpora that are never pooled.
  • 7% of the accepted fixes from the best agent were fake (2 of 28): they were lucky passes.
  • On GLM-5.2, the arm with real recorded inputs scored 30/30 against 28/30. It spent 3.7× the tokens per surviving repair of the code-only arm.
  • On DeepSeek-V4-Flash, the three arms came out level.
  • Side finding: the bench surfaced a real concurrency bug in Zaatool.

Rewind-Bench was built for Impact Forge Summer 2026. Scored 4.75/5 — top ~2% of 109 submissions — at Impact Forge Summer 2026, with perfect 5.00 scores in Technical Execution & Code Architecture and in Originality & Problem Solving.

Evidence

The code is public under the MIT license in the Rewind-Bench repository. The results report and the Devpost write-up are published.

The images below are the report’s title card, three pages of the report, and a slide on the three arms.

Rewind-Bench title card reading “It passed. It was not fixed.” Below it, a hash on the data the agent was shown matches, and a hash on data it never saw differs.
Title card of the results report: the hash on the data the agent was shown matches; the hash on data it never saw does not.
Results report opening: a one-digit change in a working notebook, the repair the agent submitted, and hashes that match on shown data but not on unseen data. Below: 297 repair episodes, 2 models, 2 corpora never pooled, 16 bugs with no visible symptom.
Report, first page: the `* 22` to `* 23` change, the repair that left it alone and cancelled it out with a second error, and the two hashes.
Slide titled “Three agents. One difference between them.”: a table of the reader, guesser and rewind arms by whether each may run code and where its input data comes from.
Slide: the three arms, by whether each may run code and where its input data comes from. Everything else is held identical.
Report page with a table of paired comparisons between arms for DeepSeek-V4-Flash and GLM-5.2, and a bar chart of tokens spent per surviving repair for GLM-5.2.
Report: paired comparisons between arms for both models, and the tokens each arm spent per surviving repair on GLM-5.2.
Report page auditing symptom visibility: for each corpus, how many bugs were observable and how many were invisible, then the found corpus split by arm.
Report: the symptom audit, counting for each corpus which bugs an agent could see from the output at all.

Limits

  • Two models were run: GLM-5.2 and DeepSeek-V4-Flash.
  • The report states that no difference reaches significance at n = 30, and none is claimed. What changed between models is the consistency of the direction, not the p-value.
  • 67 injected bugs, and 16 of them have no visible symptom.
  • The two corpora are reported separately and never pooled.
  • On DeepSeek-V4-Flash, real recorded inputs made no difference: the three arms were level.
  • The recorded-input arm costs more: on GLM-5.2 it spent 3.7× the tokens of the code-only arm.
  • The measuring instrument was corrected 4 times: a self-naming bug, too little resolving power, a misaligned difficulty metric, and lucky-pass detection that was tautological.

All work