07 / Evaluation
Test the agent on your own work
Past tasks and failures can become repeatable checks.
What the teams report
Databricks builds benchmark tasks from actual code changes. It removes solution details from task descriptions and keeps relevant tests separate. People check each sample. [1]
The team also revises tests when they reject a valid alternative solution. A useful test must check the result without requiring the original implementation. [1]
Uber evaluates uReview with a curated benchmark and feedback from engineers. These checks help the team adjust prompts, thresholds, and models. [2]
“every bug we find in production becomes a new scenario”
Databricks, on its coSTAR evaluation method. [3]
A new failure can become another test case.
Our observation
The test set needs review too
A test case needs a clear task, an initial state, and a way to assess the result. Past work can supply these parts.
Databricks also checks model judges against human judgments in coSTAR. Its method covers several kinds of agents, including internal engineering workflows. [3]
Past cases cannot cover every future failure. A strong result on one team’s tasks may not transfer to another team.
A question for your buildWhich past failure must the next version avoid?
Sources
- Databricks: Benchmarking coding agentsTask construction, solution hints, and manual test review.
- Uber: uReviewEvaluation and feedback from engineers.
- Databricks: coSTARFailure scenarios and checks on model judges.