coSTAR

Databricks ships and tests engineering agents with coSTAR, using LLM judges as the test suite and a coding assistant to refine an agent until the judges pass. The agents cover internal engineering workflows such as on-call support and automated code review, alongside customer-facing and open-source agents such as the MLflow assistant. Databricks' Omnigent is a separate product.

Company: Databricks
Approach type: Component
Work supported: Coding, Code review, On-call
Deployment stage: Scaled
Evidence strength: Detailed primary
Entry reviewed: 2026-10-08

Copy as MD Ask ChatGPT

Capabilities and architecture

Model: Not reportedNeither capture states the models behind the internal engineering agents; the benchmark evaluates third-party models and harnesses, which is separate from the shipped agents.
Harness: coSTAR framework for shipping and testing internal agents^{[Source 1]}
Sandbox: Not reportedNo runtime isolation boundary is documented for the agents; the benchmark seals git history during evaluation runs, which is not a production sandbox.
Tool access: Internal agents call MCP tools for data access, code execution, and environment setup^{[Source 1]}
Knowledge: Private benchmark built from the Databricks multi-million line codebase^{[Source 2]}
Context management: Not reportedMLflow traces record execution for scoring, but no capture describes how agent context is maintained or organized over time.
Credentials: Not reported
Interfaces: Not reportedThe captures name harnesses only as benchmark evaluation subjects, not as surfaces of the internal agents.

Mechanisms

Align judges against a human-graded Golden Set

Judge alignment refines the judges against a Golden Set of engineer-assessed outputs using MLflow alignment techniques^{[Source 1]}

Documented uses

coSTAR scenario run — execute the agent, score with judges, refine until pass.

Run the scenario against the agent under test

The coSTAR test harness sends each scenario prompt to the agent under test and captures the execution as an MLflow trace^{[Source 1]}

Score the trace with aligned judges

Agentic judges inspect the trace for properties such as code validity, best-practice adherence, and tool sequencing instead of asserting exact outputs^{[Source 1]}

Refine the agent until the judges pass

A coding assistant reads judge failures, diagnoses root causes, and patches the agent while the engineer remains the reviewer and final arbiter of the proposed changes^{[Source 1]}

Access and controls

No separate access-control assessment is recorded.

Reliability and validation

Run the judges in production and CI

The same judges run on production traffic, in CI/CD pipelines, and on nightly builds to catch regressions from agent or infrastructure changes^{[Source 1]}

Adoption and operating evidence

These results come from the sources. Notes below each result explain what it covers and any reported limits.

Key observation

coSTAR reduced the time to verify agent changes from two weeks down to hours^{[Source 1]}

Reported by: Databricks
Scope: Time to verify agent changes on the Databricks codebase under coSTAR

The source does not report the denominator of this figure.

Observed in 2025

About this result

Category: Cost latency
Evidence type: Reported measurement
What it covers: Time to verify agent changes under coSTAR, previously manual review cycles

Key observation

Private benchmark built from a multi-million line codebase^{[Source 2]}

About this result

Category: Implementation scale
Evidence type: Qualitative
What it covers: Codebase scale behind the private coding benchmark

Lessons

Reported opinion: Give judges tools, not traces: Databricks reports that agentic judges which call targeted tools on a trace scale better than feeding the full trace into a model^{[Source 1]}

Back to infrastructure

Improve this record

Give your coding agent the context and instructions to prepare a source-backed update.

Suggest on GitHub

Paste the prompt into your coding agent, then describe the correction or add a public source.

Preview prompt

Help me update the Internal Agents Map record for coSTAR at Databricks.

Repository: https://github.com/steel-experiments/internal-agents-map
Record: data/agents/databricks-costar.yaml
Published record: https://internal-agents.com/agents/databricks-costar
Readable context: https://internal-agents.com/agents/databricks-costar.md

Use the correction or public source I provide in this conversation. If I have not supplied either, ask what I want changed before editing.

Use an existing checkout of this repository, or clone it if needed. Read the current record from the repository. Follow AGENTS.md, CONTRIBUTING.md, and the relevant instructions in data/schema.md.

Verify that the evidence supports the requested change. Preserve uncertainty and distinguish reported facts from catalog judgments. Follow the repository process for capturing new sources. If evidence is insufficient, explain the gap instead of inventing details.

Make a focused change on a branch. Regenerate derived files and run the required checks. Do not edit generated content by hand.

Show me the change, supporting evidence, and check results, including any checks you could not run. Ask whether to submit a pull request unless I already requested one. Do not merge the pull request. If you cannot edit the repository, prepare a source-backed issue draft.

Sources and research details

Citations link to the original publisher. Each source also keeps a preserved copy in the repository, so a changed or removed page stays checkable.

coSTAR: how we ship AI agents at Databricks fasthttps://www.databricks.com/blog/costar-how-we-ship-ai-agents-databricks-fast-without-breaking-thingsEngineering blog · First party · Last source verification: 2026-08-31
Preserved copy
Benchmarking coding agents on a multi-million line codebasehttps://www.databricks.com/blog/benchmarking-coding-agents-databricks-multi-million-line-codebaseEngineering blog · First party · Last source verification: 2026-08-31
Preserved copy

Question coverage and scope

purpose: Reported
workflow: Reported
human involvement: Not applicable: There is no single platform-wide agent review boundary. Scoped downstream operating-model claims remain in research details; access controls are described separately.
implementation: Reported
validation: Reported
observations: Reported
lessons: Reported

Research details for every claim on this page

Summary
Statement type
Fact
Provenance
Reported
Confidence
Not assessed
Confidence reason
No explicit confidence assessment is recorded for this claim.
- SupportscoSTAR: how we ship AI agents at Databricks fast
Harness
Statement type
Fact
Provenance
Reported
Confidence
Not assessed
Confidence reason
No explicit confidence assessment is recorded for this claim.
- SupportscoSTAR: how we ship AI agents at Databricks fast
Knowledge
Statement type
Fact
Provenance
Reported
Confidence
Not assessed
Confidence reason
No explicit confidence assessment is recorded for this claim.
- SupportsBenchmarking coding agents on a multi-million line codebase
Tool access
Statement type
Fact
Provenance
Reported
Confidence
Not assessed
Confidence reason
No explicit confidence assessment is recorded for this claim.
- SupportscoSTAR: how we ship AI agents at Databricks fastPreserved content.md, line 133
Run the scenario against the agent under test
Statement type
Fact
Provenance
Reported
Confidence
Not assessed
Confidence reason
No explicit confidence assessment is recorded for this claim.
- SupportscoSTAR: how we ship AI agents at Databricks fastPreserved content.md, lines 62–64
Score the trace with aligned judges
Statement type
Fact
Provenance
Reported
Confidence
Not assessed
Confidence reason
No explicit confidence assessment is recorded for this claim.
- SupportscoSTAR: how we ship AI agents at Databricks fastPreserved content.md, lines 70–74
Refine the agent until the judges pass
Statement type
Fact
Provenance
Reported
Confidence
Not assessed
Confidence reason
No explicit confidence assessment is recorded for this claim.
- SupportscoSTAR: how we ship AI agents at Databricks fastPreserved content.md, lines 119–121
Align judges against a human-graded Golden Set
Statement type
Fact
Provenance
Reported
Confidence
Not assessed
Confidence reason
No explicit confidence assessment is recorded for this claim.
- SupportscoSTAR: how we ship AI agents at Databricks fastPreserved content.md, lines 113–117
Run the judges in production and CI
Statement type
Fact
Provenance
Reported
Confidence
Not assessed
Confidence reason
No explicit confidence assessment is recorded for this claim.
- SupportscoSTAR: how we ship AI agents at Databricks fastPreserved content.md, lines 12, 135
Key observation
Statement type
Metric
Provenance
Reported
Confidence
Medium
Confidence reason
Databricks reported the reduction in its own engineering blog; the measurement method is not described.
Reported by
Databricks
Scope
Time to verify agent changes on the Databricks codebase under coSTAR
Denominator
Not reported
Method
Not reported
Observation date
2025
- SupportscoSTAR: how we ship AI agents at Databricks fastPreserved content.md, line 11
Key observation
Statement type
Fact
Provenance
Reported
Confidence
Medium
Confidence reason
Databricks described the benchmark in its own engineering blog.
- SupportsBenchmarking coding agents on a multi-million line codebase
Lesson
Statement type
Opinion
Provenance
Reported
Confidence
Medium
Confidence reason
Databricks authors attribute the recommendation to their own testing experience; it is an attributed preference rather than an independently verified result.
- SupportscoSTAR: how we ship AI agents at Databricks fastPreserved content.md, lines 72–74, 180
Operating model assessment
Unclassified for internal engineering workflows → agent-produced changes; human attention boundary: unknown.
Statement type
Inference
Provenance
Catalog judgment
Confidence
Unverified
Confidence reason
The record covers several internal engineering agents with different workflows, so no single human-attention boundary applies.
Observation date
2025
- SupportscoSTAR: how we ship AI agents at Databricks fast