How to Compare AI Coding Agents on Your Own Web-App Tasks: What Teams Should Measure

By Obaid Tariq, Founder & Lead Engineer · Published 20 October 2026

Disclosure: AI Dev Planner builds software with AI coding tools and helps teams adopt them, so we have an interest in this topic. This article contains no rankings or benchmark scores of our own. See our disclosure policy.

The short answer

The only comparison of AI coding agents that tells you what to buy is one run on your own codebase. Pick three to five representative tasks from your backlog, freeze a starting commit, write acceptance tests the agents cannot see, and run each agent on each task more than once. Then measure what actually costs you money: hidden-test pass rate, human time to a mergeable change, diff size, security findings, regressions, latency and total cost including engineer time. Vendor benchmarks and public leaderboards such as SWE-bench are a useful shortlist, not a decision.

Why aren't vendor benchmarks and leaderboards like SWE-bench enough to choose an AI coding agent?

Because they measure a different job from yours. Public benchmarks test agents on someone else's repositories, languages and definition of done, and vendor figures come from the vendor's own setup and customers. They are good for building a shortlist and spotting big capability jumps, but they cannot tell you how much review an agent's output needs in your codebase, what it costs, or whether it introduces security problems.

What a leaderboard actually measures

SWE-bench is the best-known public benchmark for coding agents, and it is genuinely useful. The original paper (Jimenez and colleagues, ICLR 2024) built 2,294 tasks from real GitHub issues and the pull requests that fixed them, across 12 popular Python repositories. According to the project's evaluation guide, each agent's patch is applied to the real repository and the repository's tests are run in a Docker container; a task counts as resolved when the required tests pass. The project now publishes several leaderboards, including Verified (500 problems the site describes as engineer-confirmed solvable), Lite, Multilingual and Multimodal.

That design is a big improvement on toy coding puzzles. But notice what "resolved" means: the tests passed. It does not mean a maintainer would merge the change, that it follows the project's conventions, or that it avoided creating a security hole the tests never checked.

Passing tests is not the same as mergeable

Independent research shows how large that gap can be. In an August 2025 research update, METR ran an agent (Claude 3.7 Sonnet in its Inspect ReAct scaffold) on 18 real open-source issues. METR manually reviewed 15 of the resulting pull requests and reported that none of them were mergeable as-is, including the ones that passed the maintainers' tests. METR estimated an average of 42 minutes of work to make a reviewed pull request mergeable, and 26 minutes for those that had passed the tests. The common gaps were missing tests, missing documentation and linting, formatting or typing problems. The sample was small and the models are now old, but the lesson holds: a benchmark score counts test passes, while your team pays for review time.

A separate METR study from July 2025 adds a warning about relying on how fast a tool feels. In a randomised trial with 16 experienced open-source developers working on 246 real issues in repositories they knew well, developers took 19% longer when AI tools were allowed. Before the study they had expected a 24% speed-up, and afterwards they still believed AI had sped them up by 20%. METR is clear that this is a narrow result about experienced developers in familiar codebases, that its results are now out of date, and that it does not show AI slows developers down in general. The point for an evaluation is simpler: measure time, don't ask people how it felt.

How to read a vendor claim

New coding models now routinely launch with figures from the vendor's own customers: more agent-written code submitted, fewer rollbacks, faster delivery. Treat these as hypotheses to test, not results to plan around. The vendor chose the teams, the tasks, the comparison period and the definition of each metric, and you usually can't see any of them. That is not dishonest; it is simply not evidence about your codebase.

What public benchmarks, vendor claims and your own trial can and cannot tell you
Evidence	Useful for	Cannot tell you
Public leaderboards (e.g. SWE-bench variants)	Shortlisting; spotting large capability differences between models and agent setups	Review effort, cost, security, or performance on your stack and conventions
Vendor launch figures and case studies	Knowing which workflows the vendor has optimised for	Whether those gains appear with your team, tasks and review standards
Independent research (e.g. METR)	Understanding the gap between test passes, merges and real productivity	Which specific tool suits you; results age quickly as models change
Your own controlled trial	A decision for your codebase, with costs and risks you have measured	Broad conclusions beyond your tasks; it needs repeating as tools change

Other reasons your results will differ from the leaderboard

Different stack. SWE-bench's original tasks are Python libraries. A Next.js app with a database, authentication and third-party APIs exercises different skills, especially environment setup and multi-file changes.
Different scaffold. The same model can behave very differently inside different agent tools, with different context, tools and instructions. You are buying the whole setup, not just the model.
Different definition of done. Your team has conventions, review standards and security rules that no public benchmark encodes.
Public tasks may be familiar. In our view, any task that has been public for years may have been seen in some form during training, which is one more reason to use private tasks from your own backlog.
Cost is invisible. Leaderboards rarely reflect what a run costs in tokens, seats and engineer time, which is often what decides the purchase.

How do you run a fair one-to-two-week trial of AI coding agents on your own web app?

Choose three to five real backlog tasks, freeze a starting commit, and write acceptance tests the agents never see. Give every agent the same task brief, environment and time limit, run each task at least three times per agent, and record pass rate, review time to mergeable, diff size, security findings, regressions, latency and total cost on one scoring sheet. Weight the results by what matters to your team, and decide only after a human has reviewed every output.

Step 1: Choose representative tasks

Pick three to five tasks that look like the work you actually want help with, taken from your real backlog rather than invented for the trial. A balanced set for a typical web app might be:

A small bug fix with a clear reproduction.
A feature that touches the UI, an API route and the database, such as adding a field users can edit.
A refactor with no behaviour change, such as consolidating duplicated data-access code.
A change with security weight, such as adding a permission check or a new tenant-scoped query.
Optionally, a dependency or framework upgrade, which tests how an agent copes with documentation and breaking changes.

Avoid tasks that are trivial or that only one senior engineer could specify. Each task should be something a mid-level developer could finish in half a day to two days.

Step 2: Freeze the starting point and hide the tests

Fix a starting commit for each task and create a fresh branch from it for every run, so no agent benefits from another's work.
Write a task brief the way you would write a ticket for a colleague: the goal, constraints and where to look. Use the identical brief for every agent.
Write hidden acceptance tests that check the required behaviour, including at least one edge case and, for the security task, a test that one user cannot reach another user's data. Keep them outside the repository the agent sees, and run them only after the agent has finished.
Keep the existing test suite visible, so agents can check they haven't broken anything, as they would in normal work.
Use the same environment: the same container or dev setup, the same permissions, and the same project instructions file if your agents read one. Setting up those workflows is covered in our guide to an AI coding agent workflow.

Step 3: Decide what to measure before you start

Agree the metrics and their weights before the first run, so nobody adjusts them to favour a tool they already like. The scoring sheet below is our suggested starting point; change the weights to suit your team, but write them down first.

Suggested scoring sheet for comparing AI coding agents on one task (record one row per agent per run)
Metric	How to measure it	Suggested weight	Why it matters
Hidden-test pass rate	Share of hidden acceptance tests passing after the agent finishes	20%	Did it do what was asked, by your definition?
Review time to mergeable	Minutes a reviewer spends reading, requesting changes and fixing until they would approve it	20%	Usually the largest hidden cost of agent output
Regressions	Existing tests broken, plus problems found in a short manual check of related screens	15%	A fast change that breaks something else is negative value
Security findings	New issues from your SAST tool and a manual check of access control and secrets	15%	Scanners miss access-control flaws, so check those by hand
Consistency	Spread of pass rate and review time across repeated runs of the same task	10%	An agent that succeeds one time in three needs constant supervision
Total cost	Tokens or usage fees plus seat cost for the trial, plus engineer time at a loaded hourly rate	10%	Engineer time usually dwarfs token spend; count both
Diff size and scope	Lines and files changed compared with a reasonable human change; unrelated edits noted	5%	Large, sprawling diffs are slower and riskier to review
Latency	Wall-clock time from brief to finished change, including waits for the agent	5%	Matters for interactive work; less so for overnight tasks

Step 4: Measure the hard parts properly

Review time. Have the reviewer log start and end times, and stop the clock when they would approve the change in a normal pull request. If possible, have reviewers who don't know which agent produced which diff. Our guide to how much AI-generated code should ship covers what a proper review of an AI diff should include.
Security. Run the same static analysis (SAST) tool on every branch and count only new findings. OWASP notes that these tools scale well and catch issues such as SQL injection, but produce many false positives and find authentication and access-control problems hard to detect automatically. So pair the scan with a manual check against the relevant OWASP Top 10:2025 categories, especially A01 Broken Access Control.
Regressions and rollbacks. Inside a short trial you will rarely see real production rollbacks. Use broken existing tests and a short manual smoke test as the proxy, and if you pilot the winner afterwards, track reverted or hot-fixed agent changes for the first month.
Cost. Record usage from the vendor's billing dashboard for each run, not the list price, and add engineer time for writing the brief, supervising the agent and reviewing the result. Whether the overall result makes development cheaper for your business is a separate question, which we cover in does AI make software development cheaper.
Consistency. Run every task at least three times per agent. Agents are not deterministic, and a single lucky or unlucky run can reverse a comparison.

Step 5: A one-to-two-week trial plan

Example schedule for a two-week evaluation of two or three AI coding agents
When	Activity	Output
Days 1 to 2	Pick tasks, freeze starting commits, write briefs and hidden tests, agree weights	Task pack and a locked scoring sheet
Day 3	Set up identical environments and accounts for each agent; dry-run one task to check tooling	Working setup, no scored results
Days 4 to 7	Run every task at least three times per agent; record cost and latency during each run	Raw results for every run
Days 8 to 9	Run hidden tests and SAST; reviewers review each diff, ideally blind, and log time to mergeable	Completed scoring sheet
Day 10	Score, compare against the pre-agreed weights, and write up findings with examples of good and bad output	Decision and rationale

A small team can compress this into one week by using three tasks, two agents and fewer reviewers. Don't compress it by skipping repeated runs or hidden tests; those are what make the result trustworthy.

Step 6: Decide, then keep checking

Total the weighted scores, but read the individual rows too. An agent that wins on average but produced one serious security flaw may still be the wrong choice.
Check that the difference is bigger than the spread between repeated runs. If two agents overlap, treat them as tied and decide on cost, workflow fit and contract terms.
Pilot the winner with a small group for a month, with normal review rules, before rolling it out.
Keep the task pack. When a new model launches with impressive figures, re-run the same tasks in a day rather than starting from scratch. Models and agents change quickly, so we recommend re-running the comparison at least every few months.

Frequently asked questions

Is SWE-bench a good way to choose an AI coding agent?

It is a good way to build a shortlist. SWE-bench checks whether an agent's patch makes a repository's tests pass on real GitHub issues, mostly from Python projects. It does not measure review effort, cost, security or how an agent performs on your own stack, so it should not be the final deciding factor.

How many tasks do we need for a fair comparison?

Three to five representative tasks are usually enough for a decision, provided each agent runs every task at least three times. Fewer repeated runs on more tasks is less useful than repeated runs on a smaller set, because agents are not deterministic and one run can be misleading.

Should we trust a vendor's claims about productivity or fewer rollbacks?

Treat them as claims to test rather than results to plan around. The vendor chose the teams, tasks and metric definitions, and you usually cannot see them. If a claim matters to your decision, design your trial so that it can confirm or contradict that claim on your own tasks.

What is the most important metric when comparing coding agents?

In our experience, human review time to a mergeable change is the metric teams most often overlook and the one that most affects cost. An agent that passes tests but needs an hour of fixes per change may cost more than a slower one that produces clean diffs.

Do we need to compare models or agent tools?

Compare the combination you would actually use. The same model can behave very differently inside different agent tools because of context handling, available tools and instructions, so a trial of the full setup is more useful than a comparison of models alone.

Sources

Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" (ICLR 2024) — 2,294 tasks from real GitHub issues and pull requests across 12 popular Python repositories. Accessed 20 October 2026.
SWE-bench official leaderboards — Current leaderboard variants including Verified, Lite, Multilingual and Multimodal. Accessed 20 October 2026.
SWE-bench documentation — Docker-based evaluation harness; SWE-bench Verified contains 500 engineer-confirmed solvable problems. Accessed 20 October 2026.
SWE-bench documentation: Evaluation guide — Patches are applied to the repository and its tests run in Docker; a task is resolved when the required tests pass. Accessed 20 October 2026.
METR: Research update, towards reconciling slowdown with time horizons (August 2025) — 18 issues, 15 agent PRs manually reviewed, none mergeable as-is; estimated 42 minutes average (26 minutes for test-passing PRs) to make mergeable; common gaps in tests, documentation and linting. Accessed 20 October 2026.
METR: Measuring the impact of early-2025 AI on experienced open-source developer productivity (July 2025) — 16 developers, 246 issues, 19% longer with AI allowed; expected 24% and perceived 20% speed-up; caveats on scope and that results are out of date. Accessed 20 October 2026.
OWASP Top 10:2025 — Current edition and category names, including A01:2025 Broken Access Control. Accessed 20 October 2026.
OWASP: Source Code Analysis Tools — SAST tools scale well and find issues such as SQL injection, but have high false-positive rates and find authentication and access-control problems hard to automate. Accessed 20 October 2026.

Recommendations without a source are AI Dev Planner’s own engineering opinion. Platform features and prices change; check the linked pages for the current position. Found an error? See our corrections policy.

Browse all 76 questions in the SaaS & web development guide.

Choosing an AI coding agent for your team?

We can build a task pack from your backlog, write the hidden tests, run the trial and give you a scored comparison with the evidence behind it, with no stake in which tool wins.

Talk to us AI integration services