We saw similar (slightly smaller effect, ~30% less instead of the 50% less you saw) improvement from trimming the number of turns. I'm not at all confident we've found the best way to do this here. We do need the thinking tokens to be spent in order to get the less obvious findings, but we couldn't get them from just increasing thinking effort. We went with forcing the model to use a tool that parallelizes the calls, and has an output floor so there's no turns with very little content read. Again, not at all confident this is the best way to do this, just what we've thrown together that works.
The actual biggest improvement is reducing the number of review rounds a PR has to go through. We were having too many rounds of drip-feeding new findings, because the AI reviewer is a subjective grader. Even if it found the same finding before, the next time it runs it might score it wildly differently. And it tends to want to score its findings across the whole grading curve, because it thinks that's more correct-looking. Our answer is to use previous rounds scores as anchors for the next round. Since we can't just use the same context and still have good performance, we needed to condense the score report into a small json we send forward.
The actual biggest improvement is reducing the number of review rounds a PR has to go through. We were having too many rounds of drip-feeding new findings, because the AI reviewer is a subjective grader. Even if it found the same finding before, the next time it runs it might score it wildly differently. And it tends to want to score its findings across the whole grading curve, because it thinks that's more correct-looking. Our answer is to use previous rounds scores as anchors for the next round. Since we can't just use the same context and still have good performance, we needed to condense the score report into a small json we send forward.