We still do code review but it does bottleneck how much we can ship.
I'm looking at what we can do to alleviate this. Here's what I think the near future may look like, as far as getting more value per time spent out of human code review. I'm coming at this from the POV of an org that still mandates human code review and I think a lot of this advice doesn't hold if you are all-in on the black box software factory approach.
If you're reviewing code you're responsible for, your job now is to provide stewardship of its long term technical direction even if you are not personally reading and writing every line of code. This means preserving the coherence of designs past and present, and ensuring that they are not being worked around in service of a short-term goal.
Stop reading every line of every file to look for minor gotchas. Focus on core business logic and whether invariants are modeled well & to the fullest extent possible by the type system. Agents are better than most humans now at spotting insane gotchas. At least one frontier model should review most PRs.
In the before times, you could argue that TODO comments and small nits on PRs should be addressed, because most likely they'd never get attention again once the PR merges. Now, small issues should instead be automatically recorded and followed up on quickly by another agent. Minor code quality issues should become rules, and old code should be constantly revisited by agents to bring it up to par with new code.
Provide the transcripts of the session with the PR. Look for evidence of thoughtful direction & solid premise behind the conversation that led to the PR being made. In addition to providing more mentorship opportunities to the author, this also discourages meat proxying/low-value-added uses of AI. Asking someone to rewrite or completely discard a PR that's just a bad idea should no longer be the ego bruising event it once was (at least with respect to the cost sunk to make the PR).
Require a higher bar for proving that code works and stays working. The testing pyramid IMO should now be more of a testing hourglass. Or an upside-down pyramid, even. Automated E2E tests should be abundant. Every feature should receive adequate manual testing by an agent, with proof of work.
In the vein of "stays working," look for good observability and guardrails around a feature. If you're doing a good job of leveraging agents to keep code up to par, this means adding (and cycling out) feature flags, ensuring that there's adequate logging, having a cohesive monitoring solution with good coverage, and a plan for what to do if there's an issue in prod.
I'm looking at what we can do to alleviate this. Here's what I think the near future may look like, as far as getting more value per time spent out of human code review. I'm coming at this from the POV of an org that still mandates human code review and I think a lot of this advice doesn't hold if you are all-in on the black box software factory approach.
If you're reviewing code you're responsible for, your job now is to provide stewardship of its long term technical direction even if you are not personally reading and writing every line of code. This means preserving the coherence of designs past and present, and ensuring that they are not being worked around in service of a short-term goal.
Stop reading every line of every file to look for minor gotchas. Focus on core business logic and whether invariants are modeled well & to the fullest extent possible by the type system. Agents are better than most humans now at spotting insane gotchas. At least one frontier model should review most PRs.
In the before times, you could argue that TODO comments and small nits on PRs should be addressed, because most likely they'd never get attention again once the PR merges. Now, small issues should instead be automatically recorded and followed up on quickly by another agent. Minor code quality issues should become rules, and old code should be constantly revisited by agents to bring it up to par with new code.
Provide the transcripts of the session with the PR. Look for evidence of thoughtful direction & solid premise behind the conversation that led to the PR being made. In addition to providing more mentorship opportunities to the author, this also discourages meat proxying/low-value-added uses of AI. Asking someone to rewrite or completely discard a PR that's just a bad idea should no longer be the ego bruising event it once was (at least with respect to the cost sunk to make the PR).
Require a higher bar for proving that code works and stays working. The testing pyramid IMO should now be more of a testing hourglass. Or an upside-down pyramid, even. Automated E2E tests should be abundant. Every feature should receive adequate manual testing by an agent, with proof of work.
In the vein of "stays working," look for good observability and guardrails around a feature. If you're doing a good job of leveraging agents to keep code up to par, this means adding (and cycling out) feature flags, ensuring that there's adequate logging, having a cohesive monitoring solution with good coverage, and a plan for what to do if there's an issue in prod.