Really interesting methodology here — running blind tests on real PRs over 7 days is a much more honest evaluation than the typical "I asked it to review this snippet" benchmarks you see everywhere.
The thing I'd add from watching teams adopt AI code review in practice: the model choice matters less than most people think, but the workflow layer matters a lot. Both Claude and ChatGPT will surface legitimate issues — the bottleneck becomes what happens after the comments land. Do they get triaged? Do reviewers trust them enough to act? Does the same issue get flagged 4 times across the diff?
The "comment fatigue" dynamic you're describing is real. When AI review volume goes up 3x, human reviewers start skimming everything — including the comments that actually matter. The signal-to-noise problem compounds as teams ship faster with AI-generated code.
That's the gap we've been focused on with PRPulse — not replacing the AI review step, but adding a prioritization layer on top so the PRs that need real human attention actually get it, and the fast-track candidates don't eat up your senior engineers' time.
Curious which failure modes you saw most: false positives drowning real issues, or false negatives where both models missed something that bit you later?
Really interesting methodology here — running blind tests on real PRs over 7 days is a much more honest evaluation than the typical "I asked it to review this snippet" benchmarks you see everywhere.
The thing I'd add from watching teams adopt AI code review in practice: the model choice matters less than most people think, but the workflow layer matters a lot. Both Claude and ChatGPT will surface legitimate issues — the bottleneck becomes what happens after the comments land. Do they get triaged? Do reviewers trust them enough to act? Does the same issue get flagged 4 times across the diff?
The "comment fatigue" dynamic you're describing is real. When AI review volume goes up 3x, human reviewers start skimming everything — including the comments that actually matter. The signal-to-noise problem compounds as teams ship faster with AI-generated code.
That's the gap we've been focused on with PRPulse — not replacing the AI review step, but adding a prioritization layer on top so the PRs that need real human attention actually get it, and the fast-track candidates don't eat up your senior engineers' time.
Curious which failure modes you saw most: false positives drowning real issues, or false negatives where both models missed something that bit you later?