Two teams run the same model, scaffold, and temperature on SWE-bench Verified, once each. Team A reports 32. 4%; Team B reports 28.

Source: [Dev.to](https://dev.to/devinios/same-model-same-scaffold-10-runs-agent-benchmark-scores-still-move-by-up-to-6-points-gmb)

Sponsored