A Blueprint for Agentic Code Review
2026-11-07
This post outlines an approach to owning your agentic code review. It integrates with any CI system, reduces costs, increases security, and can be customized as you see fit. The blueprint is inspired by Cloudflare's post "Orchestrating AI Code Review at scale" and has evolved through trial and tribulation.
The Tenets
Make it yours - Code review improves when tuned to your code base and processes. Vendor solutions are selling you short. We are early in this Ai era. The experience and effort will pay itself back.
Balance costs & effort - Decide where your tradeoff points are, adjust effort with complexity. Most times you do not need the best model. Consider open weights.
Improve security - Remove credentials for git hosts and api access, the reviewer has no need. Avoid excess tools, they distract the agent anyway. Close the door to multiple attack vectors by respecting The Lethal Trifecta.
Maintain understanding - As we offload more typing work to Ai, the PR process is where we can maintain code understanding and decision history. Staying in the loop is important given the exploits and loss of control we have already seen. Exact code may not be important, but there are devils in the details.
Problems with Easy
Claude, review the diff
It is very easy to ask the agent to review the code it just wrote, but this has problems. When you ask the current session to review code that was just written, there is a history of choices biasing the review. Use a fresh session and context at minimum. Another good practice is to use a different model family from the authoring agent. They have different training regimes and will catch more issues.
Skills
There are a lot of skills out there for code review - of varying quality, specificity, and focus. Use these as references for your own markdown, combining them with your own guidelines, and capturing them in (sub)agent prompts. The Review section describes an approach that uses custom agents, system prompts, and multiple subagents - as an alternative to skills.
Ralph loops and swarms
Most importantly, you want to keep human judgement in the process. With Ralph loops and software factories, you will see the compounding of bad assumptions and poor choices made by agents. A recent public example is the pgrust developer not noticing Claude "cheated" on a benchmark by hard coding the benchmark query plan. The fix? Have Claude add a test to check if it cheats this way again and hope that it doesn't change the test or find a new way. Another example is an OpenCode security regression, likely from trusting the agent on the new design without critically considering the foundational change to security posture. Their console was hacked the same week the final fix for the configuration was merged. When Anthropic submitted 79 CVEs to the Linux Kernel, it was clear they didn't look at the report and Mythos can't count. These snafus happen when modifying thousands of files and millions of lines while not looking at what the agent has done. No one can comprehend that much code that quickly. I'll say more later about the when, where, and how to keep people in the process so this doesn't happen to you.
Off-the-shelf options
Code review options from git hosts and SaaS fall short. Most significantly, they are missing issues and you are shipping defects. Verify by running the meat of this blueprint by hand. This is what we did after reading the Cloudflare post, and why this blueprint exists.
The reasons for this vary and are an interrelated. At the core is a need to balance profits and retention with developer experience across a large customer base. They sell Ai credit based systems, obfuscate the inner workings, and have mixed review results. Behaviors you may see (and developers complain about) include:
- new errors found on re-review, which could have been found in the last review
- re-raising the same issue you rejected, now at a different line
- vendor instructions which disagree with established patterns or style guide
- custom instructions are ignored, AGENTS.md do not seem to be used
- false positives and pedantic security claims, both from lack of context
Vendors have a generally lack of transparency into the process. Which model is used, was it fine-tuned, how much effort was put in, and how many tokens were consumed? What actions did it take and what context was available? Is it reading more than the diff, what files did it actually look at, did it cross-check or double-check assumptions and claims?
What prompt, context, and tools does the reviewer have - and can they be changed? You should have full visibility and control over your code review agent and process.
Three Phases
Run this three-phase blueprint after builds and tests have passed. If they haven't, there is a similar three-phase build failure analysis you can use to diagnose and fix errors in CI.
The primary reason for splitting review is to separate deterministic, mechanical steps from the review agent's responsibilities. This reduces token usage, removes secrets, and lets the agent focus on the review task itself. Run the workflow in your CI system within a full clone of the repository.
- Setup (mechanical) - gathers and prepares a useful context for review
- Review (agentic) - orchestrator plus one or more review subagents
- Finalize (mechanical) - posts comments, preserve a final review bundle
1. Setup
The first step assembles useful context into a .review/ directory.
A few reasons to remove setup from agent duties are:
- Shortcut the discovery process and tool calls for git - saving tokens, no keys
- Reliable API calls with paging, pre-rendered to markdown - save tokens, no keys
- Can be run locally to assemble the same info - CI and developer parity
Use a script gather together information from git and the pull request - prepare context and the agent workflow - with the following files.
diff-info.txt
Output several git commands into a file .review/diff-info.txt.
git diff --name-status [<hash>...<hash>]
git diff --numstat [<hash>...<hash>]
git log [<hash>...<hash>]
git diff [<hash>...<hash>]content.md
Fetch and render the PR description, existing comments, build results, test coverage reports,
and other information from your code forge and CI process.
Save the results into a .review/json/... directory.
Write a small jinja template and rendering script.
Run it to generate the .review/content.md.
effort.txt
A single word file {LOW,MED,HIGH} determined before the review starts.
Begin with a simple heuristic combining the file count and lines changed.
Add path based triggers, like modifying authentication code, to force HIGH effort.
If you have a good dataset of prior reviews, build evals and fine-tune an LLM or decision model.
extra-prompts.md
Extra prompts are specialized review instructions given to the reviewers based on changes in the pull request. Use this for non-standard processes, coding standards, and invariants that should be double checked. It's easier to keep these extra prompts very short with references to other documentation - largely for maintenance and drift.
Produce .review/extra-prompts.md with a touch of python and yaml, and a directory of mini-prompts.
Append this to the primary agent prompt with instructions to pass relevant mini-prompts to subagents.
prompts:
- file: prompts/check-cicd-workflows.md
matches:
- '^ci/'
- '^gitops/'
- '^.github/workflows/'
- file: prompts/review-security.md
matches:
- '^src/lib/auth/'
- '^src/pages/auth'2. Review
The review agent runs in a full clone with the generated .review/ context.
The idea here is to configure a team of agents with specific prompts and permissions for the code review task.
They primary agent acts as an orchestrator and produces the comments, while subagents handle the actual review process.
How you break this down will evolve. Start simple and add complexity where needed as you gain comfort. You can use any harness, however some harnesses are not as flexible. The examples here are loosely based on OpenCode, which gives you full access to define top-level agents and system prompts - the control you want.
Primary Agent
The primary agent's job is to run the review process and produce final artifacts.
It spawns subagents to review changes and assembles their findings
into structured files for the finalizing scripts.
The primary agent prompt is where you describe the review process as an orchestrator, the structure of comments, and the pull request workflow. Describe how you want to handle pull requests in a human-agent collaboration. Customize this blueprint around your preferences and code forge features.
Your task is to orchestrate a code review process
and generate comments.jsonl for a pull request...
## general guidelines
...
## subagent reviewers
...
## making comments
...
## summary and outstanding issues
...
Subagents
With prompt instructions*, and depending on effort, the primary agent needs to...
Select and orchestrate a team - start a single agent and use a single pass, a single agent with a refinement loop, or a team of agents followed by a validator. There is a lot of flexibility here. There is also the double check process through multiple rounds of refinement. Over time, you will balance review quality with developer experience and your token budget.
Specialize reviewers - this is where you specify what and how to review. In addition to a general review agent, you can create specialized agents for coding standards, inconsistencies or pattern breaking, architecture adherence, missing edge cases, or test coverage reduction.
Give subagents a consistent name prefix to simplify calling permissions for the primary agent and for human readers.
Permissions
Restricting permissions to the bare essentials keeps the agents on task.
Remove unnecessary tools, give edit only to the .review/ directory,
limit bash to specific scripts (or make a small tool).
Doing so reduces agent options and side quests.
primary
# default deny everything
"*": deny
read:
# cannot read code, forces delegation
- "*": deny
- ".review/**": allow
# don't let bots read big json files
- ".review/json/**": deny
edit:
# deny edits generally
- "*": deny
# allow specific files (or ./review/*)
- ".review/comments.jsonl": allow
- ".review/summary.md": allow
- ".review/resolutions.jsonl": allow
bash:
# deny bash, it's a rabbit hole for the orchestrator
- "*": deny
# allow helper scripts for the feedback loop
- ./scripts/validate-artifacts.sh: allow
agents:
# can only call matching subagents
- pr-*: allowreviewer
# default deny everything
"*": deny
read:
# subagents can read the full code base
- "*": allow
# don't let bots read big json files
- ".review/json/**": deny Making comments
The primary agent creates the comments.jsonl.
It needs instructions for both the format
and how to author the actual comment messages.
Formatting is straight forward and should align with your code forge api and features.
Use jq or cue in a script to validate the contents.
# validate with a one-liner and good error messages
cue vet -c schema.cue comments.jsonl// schema.cue (depends on code forge)
message!: string
parent?: string
path?: string
line?: int
to_line?: int
_max_lines: (<= 200 & to_line - line) | error("code span is too long")The review comment bodies are where the rubber meets the road - and developer experience is determined.
The following guidelines are geared towards pull requests that receive multiple reviews.
Work these into your primary agent system prompt.
Index comments with C-<N>:
- Each unique issue gets an incrementing comment index number
- Apply the same
C-<N>for repeat issues in multiple locations - Reference issues by their
C-<N>, make them links - On subsequent reviews, continue at the next
C-<N>
Include annotations:
- Add severity
{CRIT,HIGH,MED,NIT}, be wary of their poor judgement here - Use colored circles or emojis to make severity level visually stand out
- Reference a
file:lineor span to bring code inline with the commentary
Generally:
- Be concise while explaining the what and the why, provide rationale
- Be mindful of existing comments, do not repeat or re-raise resolved issues
- Include formatting guidelines for in-UI rendering, depending on code forge
- At some point the PR must be good enough to be merged
Summary and outstanding issues
Append or replace the suffix of the pull request description.
The main idea here is to have an always up-to-date agent summary
at the top of the PR, where humans will encounter it first.
The primary agent creates a .review/summary.md having two sections with bulleted lists
(this helps keep them succinct)
- Actual changes in the pull request, including any scope creep
- Outstanding issues from the review, based on rejected findings and code changes
Resolutions
You may want to resolve fixed issues or need to delete comments
as they grow out of date or too numerous for your code forge limits.
While there are deterministic cases (outdated), other times agent "judgement" is needed (fixed).
In this case, have the agent generate a separate file of IDs.
You can keep this simple ids and assume delete,
or generate a structured resolve.jsonl file for a finalizing script to process.
3. Finalize
Finalization permanently saves the review results and updates the pull request.
Agent history - export the agent and subagent sessions into a .review/sessions/... directory.
This will depend on your harness and is convenient when included in the bundle.
Persist review results -
package the .review/ directory into a tar.gz and upload it to bucket.
This is helpful for compliance, diagnosing errors in the review process, and
collecting the dataset for self-improvement evaluations.
Update pull request -
Process resolve.jsonl and comments.jsonl in order, line by line.
Upsert the agent summary and outstanding issues into the description.
Update any PR decorations, letting developers know the review has completed.
Your final .review/ directory will look something like:
.review/
# phase 1 - setup
json/...
git-info.txt
content.md
effort.txt
extra-prompts.md
# phase 2 - review
sessions/...
comments.jsonl
resolve.jsonl
summary.mdImproving Resilience
The Ai bots, token vendors, and code forges are unreliable. They have downtime - and the models are non-deterministic. There are several places where you can improve resilience.
Retry semantics - Anywhere you make API calls, make sure you have a retry loop that is error aware. Log those errors, not all are the model mistakes, eg. Bitbucket has a maximum 200 comment limit on a PR.
Ai gateway - A gateway like GoModel will handle LLM retries and fallback across token vendors. More than that, GoModel's virtual model concept enables simple names in harnesses while switching out the underlying model, for example when a vendor deprecates. Combine these for even greater resilience. As a bonus, include the branch name in the headers - at both code and review time - and GoModel will give you reports for full life-cycle costs.
Artifact feedback -
Enabling agents to check their work helps them produce correct artifacts.
Use jq or cue to validate jsonl files. Do this for both structure and semantics - with a schema.
You can also verify valid parent comment ids if you keep the original JSON from the comment requests (and your code forge supports comment trees).
Put this in a script and add it to the allowed permissions, with the same rationale for limiting tool access.
Double checking -
Give the orchestrator instructions so that it runs multiple rounds of review and findings validation.
This helps to reduce noise, false-positives, and other errors like bad C-<N> references.
Limit rounds to avoid endless looping and billing surprises.
Instruction refinement - As you use this blueprint in real reviews, you will undoubtedly encounter mistakes and other quirks. This comes both from ambiguous requirements and the non-determinism inherent to LLMs. The instruction edits will accumulate over time and their own errors (like code). Be wary of markdown bloat and manually revisit your markdown regularly.
Segway
To connect the prior section with the next, I offer a motivating example from implementing this blueprint.
While reviewing review failures, there was a peculiar subset when processing resolve.jsonl (note the file extension).
A developer added this feature to resolve or delete certain types of comments,
as the 200 limit approached on Bitbucket.
However, they did not notice the pattern matching - and mismatch - with instructions.
The model followed their prompt to add a line oriented file of ids,
but it also updated the artifact table with a .jsonl extension - inline with the row above the new entry.
Most times the primary agent would follow the written instructions and produce a file with "correct" format, one id per-line.
The deviation was rare, producing content with { "comment_id": 12345678 } to align with the jsonl extension.
This non-obvious instruction conflict is another place Ai non-determinism rears its head.
While I was the original author of the pr-reviewer system, I was not involved in this PR.
The author and approving developer were not familiar with the details and were
aiming for a quick, minimal fix - any failure blocks merging in our process.
Even before we lose our own context, Ai has enabled us to feel more confident in modifying code we are not familiar with.
In these times, we need to pay closer attention, and not forget that we can use git blame and ping each other to ask questions.
Staying in the Loop
One of the divisions Ai has created among developers is staying in the loop versus going full auto - or what I like to call hands free vs eyes free coding. At the same time, there are serious concerns around hacking and loss of control. We are still in the Ai wild west times and people are trying all sorts of things.
I believe that our industry can work to improve our code with Ai (augmented intelligence). It is paramount we do as agentic hacking, exploit discovery, and loss-of-control are already happening. Yet so much of what we hear from Ai leaders and clout chasers is the exact opposite advice we need to secure our software and improve quality. We developers are ushering in the Ai era for all humans. Swarms of transformers are not going to remedy our decades of technical debt and the growing slop we find in our code bases.
The following rules are taken Inverse Laws of Robotics. For the discussion here, Rule 2 & 3 are the most important to reflect upon. Write them on a sticky note and keep them visually handy as you use Ai.
- Humans must not anthropomorphize AI systems.
- Humans must not blindly trust the output of AI systems.
- Humans must remain fully responsible and accountable for consequences arising from the use of AI systems.
If we fully close the loop with Ai agents, letting go of the proverbial wheel, we are implicitly blindly trusting the output. When the "bad things" eventually happen, what are we going to tell our bosses, customers, and fellow humans? Saying the Ai did it won't cut it in the courts (legal or public opinion).
So where can we best stay in the loop with assisted development? Pull requests seem a natural choice. Managers have been doing this for a long time and is where we make the decision to approve sharing changes.
- Review the reviewer claims and reply to its comments. Call out falsehoods and add clarifications. Don't fully close the code-review-fix-merge loop.
- Tell the fixer which issues to work on. Sometimes all, other times in smaller group, or ask it to to do research before deciding. This requires inspecting and evaluating the findings yourself.
- Only resolve defect fixes manually (no
resolve.jsonl). This forces you to understand the before and after, judging whether the fix actually works. - Ping your peers for context. If you are modifying someone else's code, have them evaluate the assumptions and invariants. A second pair of organic eyes never hurt before Ai.
- Break work into smaller chunks, include decisions in the PR comments - as opposed to the mega commit and rapid fire merge trends that have been growing. Remember Goodhart's Law.
Parting Thoughts
Which Models?
You might have noticed I have not mentioned what models to use, family or size. It is unclear what works best or is good enough. Evals for determining this require another tier of effort and comes with more cost. My main points of advice (today) are
- Use a different model family from the code author
- Start with cheaper models while you are early into this
- Upgrade the orchestrator to a higher tier model
The primary challenge here is agentic monoculture. If your organization only uses Claude, you first have to find new token vendors. Then, you have to adopt harnesses that are not tied to a single vendor, such as OpenCode.
Shifting Left
To further reduce pull request findings and costs - shift left. Bring the review process closer to the developer, before commits are even pushed. Implement the first phase such that it can run locally too. This is a good pattern to use generally - parity between local development and CI.
Omni Reviews
Occasionally you'll want to do large scale reviews outside the normal pull request workflows.
broadly review the code base: go through the codebase area-by-area - alternating between code vs markdown as truth - to look for consistency, completeness, and coherence.
meta review the reviewer: after implementing and using this blueprint, refining the process through vibes and iteration, you will find bloat in your markdown. Spend human time to review and trim this down. After more time, you can assemble a dataset and eval setup to implement a form of self-improvement at the agentic markdown layer.
Can you help with this?
Sure can! Please reach out if you'd like to talk. I firmly believe you are better served with a custom code review process. While this blueprint was written with a mindset that you can point your own agent at this page, there are many nuances and details to customize to your setup and processes.
Tuning instructions takes time, exposure, and experience - and something I can help shortcut. My specializations generally cover self-hosted CI, monorepo management, cost analysis and controls, open weight adoption, agentic enablement for development teams, and practicing safe and effective Ai.
Thanks for reading!
verdverm