If 30% of Coding Tasks May Be Broken, Your Leaderboard Needs an Uncertainty Budget

OpenAI published an audit of SWE-Bench Pro on July 8, 2026 and estimated that roughly 30% of its tasks are broken. The reported issues make a familiar leaderboard assumption unsafe: every task in the denominator is a valid, equally interpretable trial. Primary source: OpenAI, “Separating signal...

Source: [Dev.to](https://dev.to/robinzzz/if-30-of-coding-tasks-may-be-broken-your-leaderboard-needs-an-uncertainty-budget-4992)