r/coderabbit • u/Senior_Ad_9791 • 6d ago
Discussion & Feedback Reviewing agent code
if you review PRs written mostly by coding agents (Cursor, Claude Code, Copilot), how's it going?
How many per week, and how big (files / lines)?
What's harder than human-written PRs, or is it no different?
How do you understand a change: diff only, run it, read tests, ask the agent, skim and approve?
Has one caused a prod bug or stalled merge because nobody fully understood it?
Anything you've tried to fix this? Did it help?
I'm exploring a tool in this space, so "not a problem for us" is a useful answer too. Team size and stack help.
1
u/4dwaith007 6d ago edited 6d ago
Review doesn't happen separately from dev. I ask the agent to explain how it's going to implement a task. After it explains its approach, I question some decisions and suggest any changes I can think of (this can sometimes go into rabbit holes, where I suggest 2-3 different approaches, ask it to compare against the one it suggested, until I'm satisfied).
Then I ask it to record the changes in a plan file, and break it up into chunks with 400 or so lines so it'll be easy to review (it usually estimates this badly, and goes up to 800).
I then look at the tests, see if it covers it right and enough - at this point I'm not even looking at implementation, just the test cases. If I see issues, I point it out and have it fixed.
Only then do I ask it to take me through the new code, it explains anything significant.
Finally, I review the source code and test implementation. By that time I already know the rough shape of what I'm looking at, and the review isn't too difficult. I roughly finish 5-6 reviews of 600-800 lines in a day full of dev/review work. It's harder to read from scratch, but asking it to take me through the code will show files in a sequence that's easier to understand and review
1
u/warp3dgh0st 5d ago
25 PR/week. Anywhere from a few lines of code to hundreds of lines of code.
AI written jiras are wayyyyy better but only after they got the template correct. it took months of shitty jiras to get it correct.
If the change is small i just code review. If it is large or deals with money then code review, i personally qa and test every acceptance criteria. I run an ai skill for code reviewing to verify this new agent thinks the code is correct and to cover edge cases that might have been missed.
Hell yes. Several in fact and we got rid of our jr devs because they can't code review properly. Now every code change is covered by a sr. engineer as well as another engineer doing a code review. This HAS to be done. Claude is getting better almost every release by a big margin. I say almost because one of the previous releases was causing way to many memories to be created which was causing other issues.
Jira's i code review and find stuff get sent back to the dev with detailed reasons why it failed. This is the only way until ai can be trusted.
2
u/xvillifyx 6d ago
I don’t think there’s any reason for PRs to be any bigger than they used to be and I think anyone making them massive now is doing themselves a disservice.
Hard for human review. Causes fine details in the merge history to get bungled up in big contexts. Just bad
Anyway I review AI code the same way I would review a human except it gets a once over with another agent before I actually start getting into the weeds
My company is still intent on the philosophy that the person doing the commit needs to be able to defend all the changes, so it’s never been the case that an issue arose because they didn’t understand the code