Review After 3 Weeks of Running an Internal AI Code Review Bot
Our team hooked up an AI code review bot to GitHub Actions and ran it for about three weeks. To get straight to the point, it's roughly "nice to have, but not essential."
The good part was that it catches minor style issues or missed exception handling right at the PR stage. In particular, the repetitive comments reviewers used to leave on new hires' PRs dropped significantly.
The downside is that it often makes comments without understanding the context. Since it doesn't know domain rules, it leaves comments like "Why was this function written this way?" so a person ends up having to explain again. Adjusting the tone in the settings helped somewhat.
4 answers
Agreed — especially on new contributor PRs, the drop in repetitive comments is really noticeable.
We tried running something similar for about six months too, and in the end the way to keep it alive was feeding domain context to the bot. We put a document summarizing the rules in the README into the system prompt and added a few lines of team rules like “this directory is payment logic, so don’t arbitrarily swallow exceptions,” and comments like “why was this written this way?” dropped sharply. On the other hand, if no one manages that document, it quickly goes stale and odd criticisms start increasing again. Ultimately, it comes down to who keeps the docs up to date rather than the bot itself.