Evals
How frontier models do on real ML engineering work.
Every task in our catalogue is run against the strongest available models before it ships. This is how we do it.
Method
A real codebase per task
Each attempt starts from a fresh copy of the task's repository in its own isolated machine, with no access to the grader or hidden files.
Four attempts, Pass@4
A model gets four independent attempts at each task. A task counts as solved if any attempt passes the grader.
Graders checked first
We try to beat each grader without doing the work before any model sees it, so a pass means the work was done.
What the tasks cover
Training pipelines
Find why a run diverges, stalls or silently trains on the wrong thing.
Data leakage
Track down where test information reaches training, and fix it without breaking the pipeline.
Broken evaluations
Repair metrics and harnesses that report the wrong number.
Performance
Make a slow model or data loader fast without changing what it computes.
Run them on your model
We can run the catalogue, or a private set built for you, against your checkpoints and report the results the same way.