Skip to content
Back to AI Coding

AI-Powered Code Review Automation

Wire AI review into CI/PR pipelines: triggers, prompts, gating, and human fallback.

AutomationIntermediate

Why

GitHub Copilot reached ~$400M ARR growing 281% YoY and accounted for 40% of net new GitHub revenue in 2024, demonstrating automation ROI at scale. Human review is the bottleneck in AI-first teams: AI generates PRs faster than humans can read them.

An AI reviewer that runs on every PR catches mechanical issues (missing tests, type holes, scope creep) in seconds, leaving humans to judge architecture and intent. This scales review with generation instead of falling behind.

When

Automate review when PR volume exceeds human review capacity, when many PRs share a small set of checkable rules (test coverage, no any, scope), or when you want 24/7 first-pass feedback. Keep a human gate for security, architecture, and product-behavior changes.

How

1. Pick the trigger: run on PR open and on new pushes, not on every commit to a feature branch. GitHub Actions: on: pull_request (types: [opened, synchronize]) 2. Gather the diff and minimal context for the prompt: git diff origin/main...HEAD Include the PR title, body, and changed files only. Do not send the whole repo. 3. Write a structured review prompt with concrete checks: - Does every new function have a test? - Are there 'any' types or unhandled promises? - Does the diff touch files outside the PR's stated scope? - Are there new dependencies? List them for license check. Output JSON: { verdict: 'approve'|'request_changes', comments: [{file, line, severity, message}] } 4. Post the AI review as PR comments via the GitHub API: POST /repos/{owner}/{repo}/pulls/{pr}/comments Mark them with a bot label so humans can filter. 5. Gate on verdict. 'request_changes' fails the CI check (non-blocking at first). Run it in advisory mode for 2 weeks, then flip to blocking once trust is earned. 6. Human fallback: always allow a 'ai-review: override' label to dismiss false positives, and require a human approval on top of AI approve for security-sensitive paths. 7. Track accuracy: log AI verdict vs final human verdict per PR. Tune the prompt to drive down false positives; a noisy bot gets ignored within a week.

Example

GitHub Action + Claude review bot (YAML + prompt): # .github/workflows/ai-review.yml name: AI Review on: { pull_request: { types: [opened, synchronize] } } jobs: review: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: { fetch-depth: 0 } - run: | git diff origin/main...HEAD > pr.diff node scripts/ai-review.mjs pr.diff >> "$GITHUB_STEP_SUMMARY" // scripts/ai-review.mjs (core prompt) const prompt = `You are a code reviewer. Check this diff for: 1. New functions without tests 2. 'any' types or unhandled promises 3. Files outside the PR scope Output JSON: { verdict, comments: [{file, line, severity, message}] } Diff: ${diff}`; const result = await callClaude(prompt); const { verdict, comments } = JSON.parse(result); await postPRComments(comments); // GitHub API if (verdict === "request_changes") process.exit(1); // gate

Common Problems

  • ⚠GitHub Actions: "Process completed with exit code 1" → check the action log line with `##[error]`; common cause is a missing env var (set in repo Secrets).
  • ⚠Error: EADDRINUSE: address already in use :::3000 → run `npx kill-port 3000` or change the port in config.
  • ⚠Error: Cron job silently fails → wrap in `set -e; ...` and redirect output to a log; verify with `tail -f /var/log/job.log`.
  • ⚠GitHub Actions: "Process completed with exit code 1" → check the action log line with `##[error]`; common cause is a missing env var (set in repo Secrets).

Best Practices

  • ✓Run advisory for 2 weeks, measure AI-vs-human verdict agreement, then flip to blocking.
  • ✓Send only the diff plus PR title/body to the model; never the whole repo.
  • ✓Post only high-severity comments as PR reviews; log the rest in the run summary.
  • ✓Always provide an override label and require human approval for security-sensitive paths.
  • ✓Track false-positive rate per rule and prune rules that fire wrong more than 20% of the time.

FAQ

Should AI code review block PRs immediately?

No. Run it in advisory mode for ~2 weeks first, posting comments without failing CI. Measure AI-vs-human verdict agreement, then flip to blocking once trust is earned. Blocking too early on false positives frustrates devs and gets the bot disabled.

What context should I send to the review AI?

Only the PR diff plus the PR title and body. Never send the whole repo — it blows cost and context and rarely helps. The diff contains the change; the PR body contains intent. That's enough for mechanical checks (missing tests, any types, scope).

How do I handle AI review false positives?

Always provide an override path: an 'ai-review: override' label that dismisses the bot's blocking status. Track false-positive rate per rule and prune rules that fire wrong more than 20% of the time. A noisy bot with no override gets ignored within a week.

What should the AI reviewer check vs leave to humans?

AI excels at mechanical checks: missing tests, any types, unhandled promises, scope creep, new dependencies. Leave architecture, security, and product-behavior judgment to humans. Require human approval on top of AI approve for security-sensitive paths.

How do I post AI review comments to a GitHub PR?

Use the GitHub API: POST /repos/{owner}/{repo}/pulls/{pr}/comments with the file, line, and message. Mark comments with a bot label so humans can filter them. Post only high-severity findings as PR reviews; log the rest in the run summary to avoid noise.