Giving a coding agent a pre-run inventory of exactly what CI will check eliminated fix loops, driving green commit rates to 100% and rework cost to $0 on all predictable failures.
Retrying a flaky build until it passes turns CI into a probabilistic lottery that actively masks race conditions and concurrency deadlocks. The real expense is not compute minutes, but engineering hours lost chasing phantom bugs in production that CI already caught and hid.
Agree on retries. Rerunning a flake until it passes hides the race CI already caught, and you pay for it later in engineering hours. That's not what this study did. The 100% is first-contact green: the first outer-loop pipeline passed, after the agent had already practiced against the checks we could name in advance. A flake that is a race in our own code stays red.
Retrying a flaky build until it passes turns CI into a probabilistic lottery that actively masks race conditions and concurrency deadlocks. The real expense is not compute minutes, but engineering hours lost chasing phantom bugs in production that CI already caught and hid.
Agree on retries. Rerunning a flake until it passes hides the race CI already caught, and you pay for it later in engineering hours. That's not what this study did. The 100% is first-contact green: the first outer-loop pipeline passed, after the agent had already practiced against the checks we could name in advance. A flake that is a race in our own code stays red.