Test the agent on your own work

Reconstruct the task without the answer

Databricks builds coding tasks from actual pull requests. It rewrites the task description around the desired outcome, removes explanations of the historical solution, and keeps the relevant tests separate. People check each candidate sample. Without that separation, a successful run could reflect access to the answer rather than the ability to solve the task. [1]

The original tests also need inspection. Databricks found tests that rejected valid alternative implementations and rewrote them by hand. A benchmark can therefore fail in either direction: solution hints make it too easy, while tests tied to one implementation reject correct work. [1]

Review the judge as well as the agent

In coSTAR, Databricks records agent traces and scores them with model judges. A separate loop compares those judges with human assessments and refines them against a curated set of examples. The team explicitly reports continuing costs for human labeling and recalibration, plus failures that existing judges do not cover. [3]

Uber’s uReview evaluates a different output: review findings against annotated commits, supplemented by engineer feedback. Its precision and recall concern detected issues; Databricks’ coding benchmark concerns completed code changes. Their scores cannot be substituted for each other. [2]

Our reading is that a reusable past task needs a preserved starting state, a clear request, and an assessment that accepts valid alternatives. Human agreement on one set of examples can expose judge errors, but it cannot establish coverage of unseen failures. coSTAR adds production failures as new scenarios precisely because the original suite remains incomplete. [3]