We needed a framework to write our evals in and easily update them, run A/B tests, set all kinds of constraints (ex: timeouts, number of turns), and configure the execution environment (ex: sandboxes, dependencies, how to handle AskQuestion). We also have an agent judge. It’s all in a YAML file...
Source: [Hacker News](https://github.com/uipath/coder_eval)