More comments can mean more work
A review agent also creates work for its reader.
What gets published
Uber found that a single prompt produced false alarms and correct comments with little practical value. uReview now grades confidence, removes duplicates, and suppresses categories that engineers rarely use. Those decisions reduce the set of findings an engineer sees. [1]
HubSpot also found that a faster reviewer could still produce poor comments. It added a judge agent before publication. That supplies another assessment, with its own potential errors, rather than independent confirmation that a finding is correct. [2]
Both teams filter comments, but their reports describe different checks. Neither comment volume nor an added judge establishes how much review work the system saves.
The cost of a stricter filter
A published comment asks an engineer to inspect evidence, decide whether to act, and resolve or dismiss the finding. A useful comment can save later debugging; a weak one can consume more time than it saves.
Suppressing a comment also has a cost when it hides a real defect. Uber evaluates precision and recall against annotated commits, then tracks feedback and whether comments were addressed. Those measures answer different questions: benchmark recall concerns known issues, while addressed comments describe use of the published output. [1]
Our reading is that a filter needs evidence from both sides of the publication decision. Reviewing a sample of discarded candidates could reveal missed defects; timing human follow-up could expose expensive comments. These are evaluation proposals. The reports do not supply a complete measurement of downstream reviewer time or the defects that escaped both automated and human review.
Sources
- Uber: uReviewQuality filters, duplicate removal, and feedback from engineers.
- HubSpot: Sidekick code reviewThe change to Aviator and the judge agent.