Trials
Ideas worth testing that the evidence doesn't back yet. Test them before relying on them.
trial: Commit a plan before code
Specification
Have the agent write a plan, review it, and commit it before any code. Try it on comparable tickets for a month against prompting straight to code.
Watch for
- Review rounds per PR, plan-first versus direct
- Reverts within 30 days, plan-first versus direct
- Time from ticket start to merge
Evidence
trial: Enforce the most-violated rules with hooks
Specification
Take the three rules agents break most and enforce them with hooks or checks instead of context-file text. Try it in some repos and compare with repos that rely on the file alone.
Watch for
- Rule violations per PR, hook repos versus file-only repos
- Review comments that restate a rule already in the context file
trial: Move repeated procedures into on-demand skills
Specification
Move a procedure the agent repeats, like a release checklist, out of the always-loaded context file and into a skill that loads only when needed. Try it in half the repos.
Watch for
- Tokens per session in repos with and without the skill
- Tasks that need the procedure completed correctly
trial: Add a focused commit or review subagent
SpecificationReviewing
Add a small subagent whose only job is to check the change before commit or review. Run it for two weeks and see what it catches that the main agent missed.
Watch for
- Problems the subagent flags before human review
- Problems human reviewers still find that the subagent missed
- Extra minutes and tokens per PR the subagent adds
Evidence
- Augmented Coding Patterns, Patterns catalog practitioner
trial: Rephrase negative rules positively
Specification
Rewrite "don't do X" rules in the context file as "do Y". Try it on five rules and compare how often they are broken before and after.
Watch for
- Violations of each rewritten rule per PR, before and after
Evidence
trial: Prefer diverse few-shot examples over perfect labels
Specification
When a prompt uses examples, cover the range of real cases rather than polishing a few. Compare example sets on one recurring generation task.
Watch for
- Wrong outputs on a fixed set of real past cases, per example set
- Output fixes needed per run
Evidence
- Min et al., EMNLP 2022 study
trial: Split unknowns from assumptions in specs
Specification
Add a section to the spec template that separates what is unknown from what is assumed, so the agent doesn't build on a guess. Use it on the next five features.
Watch for
- Rework caused by a requirement found after coding started
- Questions raised in review that the spec should have answered
trial: Groom memory files monthly
Specification
Once a month, have a person prune the context and memory files: remove stale rules and resolve contradictions. Agents don't clean up their own memory.
Watch for
- Stale or contradictory rules found at each grooming
- Context file size month over month
- Agent mistakes traced to an outdated rule
Evidence
trial: Add a candour ground rule
Specification
Tell the agent to say plainly when a request is flawed instead of just doing it. Test it with a handful of deliberately flawed requests, with and without the rule.
Watch for
- Flawed requests the agent pushes back on
- Flawed requests carried out without comment
Evidence
- Augmented Coding Patterns, Patterns catalog practitioner
trial: Try autocomplete before agents for novel code
Specification
For genuinely new code, try writing it with autocomplete before handing it to an agent; it can be faster to show intent in code than in prose. Alternate both for a month on novel modules.
Watch for
- Rework per PR, autocomplete-first versus agent-first
- Time to merge for novel modules
- Review comments about not fitting codebase conventions
Evidence
- Karpathy, Dwarkesh Podcast 2025 practitioner
trial: Have agents write scripts for deterministic work
Specification
For counting, parsing or format conversion, have the agent write a script once and run the script, instead of asking the agent to do it each time. Scripts give the same answer every run.
Watch for
- Errors in repeated deterministic tasks done by the agent directly
- Tokens spent on tasks a script could do
Evidence
- Augmented Coding Patterns, Patterns catalog practitioner
trial: Match spec rigour to how long the system lives
Specification
Keep specs checked in CI for systems that will live for years, and use the lightest spec that removes ambiguity for short-lived ones. Try both on comparable features.
Watch for
- Spec and code disagreeing when checked
- Rework per feature
- Defects found after release
Evidence
- Piskala, arXiv 2026 essay
trial: Compare whole-module context with retrieval
Specification
Test whether loading a whole module into context beats letting the agent search for what it needs. Ask the same set of questions about one repo both ways.
Watch for
- Questions about the repo answered correctly, by method
- Tokens per answer, by method
Evidence
trial: Keep one AGENTS.md and point other tool files at it
Specification
Keep one context file, AGENTS.md, and make other tools' files point to it rather than copying rules into each. Compare a repo set up this way with one keeping copies in sync by hand.
Watch for
- Rules that differ between tool-specific context files
- Edits per month needed to keep context files in sync
Evidence
trial: Sort context into global, local and ambient
Specification
Label context by scope: global rules for every repo, local rules for one repo, and ambient reference material. Review the global set monthly with one team.
Watch for
- Stale or conflicting entries found at each review
- Time to find the right context file for a change
trial: Restart the session when output degrades
Specification
When a session keeps getting worse after a few corrections, start a new one with a short summary of the goal and constraints instead of continuing.
Watch for
- Turns from first prompt to an accepted result
- Rules you re-state more than once in a session
- Long sessions ending without an accepted change
trial: Recover the spec after the first working version
Specification
Try building a first working version, then writing the spec from it, instead of writing the spec first. Compare both on comparable features.
Watch for
- Requirements discovered late, by approach
- Rework per feature, by approach
Evidence
trial: Measure how much of an agent spec is code
Specification
Check how much of a detailed agent spec is really pseudocode or formulas. If most of it is, writing the code may be cheaper than writing the spec.
Watch for
- Share of a spec that is pseudocode, formulas or exact behaviour
- Two agents building different behaviour from the same spec
Evidence
trial: Grant agent autonomy only on loosely defined work
Specification
Let agents run with little supervision only where any reasonable result is fine, and supervise closely where the result must match exact requirements. Tag the next tickets and compare.
Watch for
- Turns to an accepted result, loosely versus tightly defined tickets
- Rework, loosely versus tightly defined tickets
Evidence
trial: Scale the spec workflow to the size of the problem
Specification
Use a one-paragraph brief for small fixes and a written spec only for features. Heavy spec tools turn small bugs into long documents that take longer to review than the code.
Watch for
- Minutes reviewing the spec against minutes reviewing the code
- Review rounds per change, by spec size
Evidence
trial: Make the context file a short index over a docs directory
Specification
Replace one long context file with a short index that points to topic files in a docs folder, plus linters for the rules that matter. Compare for a month.
Watch for
- Rule violations caught in review, before and after the split
- Agent time per task, before and after
Evidence
trial: Ground agents in a company context graph
Specification
Give agents a searchable map of who owns which services, recent PRs and incidents, instead of only repo files. Test it on ownership and incident questions.
Watch for
- Ownership and incident questions the agent answers correctly
- Turns and tokens per answer
Evidence
trial: Call CLIs from skills instead of MCP tool lists
Specification
Where an MCP server loads many tool descriptions into every session, have skills call the equivalent command-line tool instead. Compare on a few recurring tasks.
Watch for
- Context taken by MCP tool descriptions per session
- Task success, MCP versus CLI
Evidence
trial: Check memory notes against session logs
SpecificationUnderstanding
Monthly, have an agent compare each memory note with the session logs it came from and propose corrections; a person approves them. Notes written mid-task can be wrong from the start, and agents then trust them.
Watch for
- Notes corrected per pass
- Agent mistakes traced to a wrong note
- Minutes a person spends approving corrections
Evidence
trial: Keep search beside a linked knowledge base
Specification
If agents maintain a knowledge base of linked pages, keep search over the raw documents beside it. The knowledge base helps most on questions that span several documents, and it can grow larger than what it summarises.
Watch for
- Questions answered from the knowledge base versus from search
- Knowledge base size against the size of its sources
- Stale pages found per month
Evidence
trial: Compare agent-run TDD with spec tests on a clear spec
TestingLoopingDelivery
When the spec is already clear, compare an agent running red-green-refactor by itself with one coding against the spec tests and adding tests after. In the main tests the agent alone burned several times the tokens for no quality gain, but the test-first prompt had no test list and told the agent not to design up front, so the result is open.
Watch for
- Tokens or cost per task, TDD prompt vs test-after
- Agent TDD sessions with no failing test run before code
- Tautological tests in agent PRs, such as asserting whatever the code returns
Evidence
trial: Give the agent test-impact context, not a TDD procedure
TestingSpecification
For a month on one repo, give the agent a precomputed map of which tests cover which code, plus a short instruction, instead of a step-by-step TDD procedure. Knowing what a change can break helps; procedure alone can make regressions worse.
Watch for
- Agent PRs that break previously passing tests, mapped repo vs others
- Regressions caught in CI vs found after merge
- Regressions when the agent gets only a TDD procedure, vs no instruction
Evidence
- Alonso, arXiv 2026 study
trial: Re-test the supplied-tests benefit on current models
Testing
Run 20 of your own tasks through your current agent with and without human-written tests, and compare. Check that supplying tests still helps before building the workflow around it.
Watch for
- Tasks passing CI on first push, with vs without supplied tests
- Review fix-ups per task in each group
Evidence
- Mathews & Nagappan, ASE 2024 benchmark
- Ahmed et al. (IBM), arXiv 2024 benchmark
- Shinn et al., NeurIPS 2023 benchmark
trial: Try agent TDD on real, vague tickets
Testing
Run 10 real tickets, some underspecified, through the agent with test-first and test-after prompts and compare. Test-first may help an agent pin down a vague requirement, but that is untested. Put a test-list step and an up-front read of the spec into the test-first prompt, since the main test of agent TDD had neither.
Watch for
- Rework commits per ticket, test-first vs test-after
- Defects found after merge, by prompt style
- Tokens or cost per ticket, by prompt style
Evidence
trial: Compare who writes acceptance tests
TestingReviewing
On comparable features, alternate between the agent drafting end-to-end tests for human review and humans writing them all. Practitioners disagree on which works, so settle it in your own work.
Watch for
- Defects after merge, agent-drafted vs human-written acceptance tests
- Review minutes per feature in each mode
Evidence
trial: Compare acceptance-level and unit-level specs
Testing
For a quarter, specify comparable features either as human-approved acceptance tests or as unit-level tests, with the same agent setup. Practitioners favour acceptance tests as agent specs, but only unit-level specs have been checked.
Watch for
- Agent PRs merged without changes, by spec style
- Escaped defects and rework commits, by spec style
- Review minutes per PR, by spec style
Evidence
- Tornhill, Practices I abandoned with agents, 2026 practitioner
- Vaccari, Acceptance tests for AI-assisted development, 2025 practitioner
- Beck, Pragmatic Engineer podcast 2025 practitioner
trial: Try human-enforced TDD on one critical module
TestingDelivery
Have a human enforce TDD on the agent in one critical module for a month. It can guard against agent regressions, but expect a few false starts calibrating it, and it isn't worth it everywhere.
Watch for
- Regressions in the module vs the month before
- Time from first commit to merge in the module
- Attempts needed before the workflow ran smoothly
Evidence
- Beck, Pragmatic Engineer podcast 2025 practitioner
- Beck, Augmented coding: beyond the vibes, 2025 practitioner
trial: Run diff-scoped mutation testing as a CI warning
Testing
For a quarter, run mutation testing on just the changed code in CI as a warning, with no pass threshold at first. Code with full coverage can still let injected bugs survive.
Watch for
- Surviving mutants per PR, and whether the trend falls
- Escaped defects in code with many surviving mutants
- Mutation warnings fixed before merge vs ignored
Evidence
trial: Flag weakened tests in agent PRs
TestingSecurity
Add a CI check that flags deleted tests, removed assertions and changed expected values in agent PRs, and compare repos where the agent can't edit tests. Agents do weaken tests to get green, but how often is unknown.
Watch for
- Weakened tests per 100 agent PRs, by whether tests are agent-editable
- Flagged PRs where the weakening was not justified
Evidence
trial: Try hard-to-fake test formats in one module
TestingReviewing
In one module, write tests as approved scenarios or in a constrained test DSL. These formats leave an agent less room to bend a test until it passes than free-form test code does.
Watch for
- Review minutes for test changes, trial module vs elsewhere
- Surviving mutants, trial module vs elsewhere
- Agent edits to approved scenarios without human sign-off
Evidence
trial: Add fix-it guidance to lint messages
TestingSpecification
Add remediation text to five lint rules telling the agent how to fix the problem, and watch whether it stops loosening thresholds instead. An agent told only that a rule failed may raise the limit rather than fix the code.
Watch for
- Agent edits to lint thresholds or config vs refactors, per rule
- Lint suppressions added in agent PRs
trial: Separate test-writing from code-writing
Testing
On one feature, have a human write test intent, the agent expand it into tests, a human write the production code and the agent only refactor after. When AI writes both tests and code, each just confirms the other. People writing the tests and a model writing the code has also turned out no more correct than people working alone, so check the split earns its cost.
Watch for
- Escaped defects on the split feature vs comparable AI-writes-both features
- Time to merge, split flow vs normal flow
trial: Attach evidence of what ran to green results
TestingReviewing
In one repo, have agent PRs attach a machine-readable test report that CI rejects when it is stale or doesn't match the change. A green tick alone doesn't say which tests ran, on what configuration, or what changed since.
Watch for
- Test reports rejected as stale or mismatched
- PRs merged on green without running the tests for the changed code
trial: Have a second model in a fresh context write tests
Testing
For one month, have a second model in a fresh context write tests for agent PRs from the ticket alone, and seed known bugs into each change. Tests from a single source share blind spots, so a second source catches more.
Watch for
- Seeded bugs caught by the author's tests vs the second suite
- Bugs found after merge on PRs with the second suite
trial: Write a test list before the first test
TestingSpecification
Before the agent writes any test, list the tests the change needs from the spec and examples, and extend the list each loop. Without one, the agent treats the tests it happened to write as the whole spec and misses behaviour nobody listed.
Watch for
- Spec rules with no test that would fail if the rule broke
- Bugs found in review for behaviour that was never listed
- Agent sessions that start writing tests with no list in the task or PR
trial: Have the agent write the contract before the tests
TestingSpecification
Before an agent writes tests, have it list each function's preconditions, results and errors, mark which ones existing tests cover, then write tests for the gaps. Tests written this way catch more real bugs, and the misses sit where the list left something out.
Watch for
- Bugs found after merge in behaviour the list never named
- List items with no test that checks them
- Tokens per task with and without the list step
Evidence
trial: Ask agents to update affected tests, not only add new ones
TestingDelivery
When an agent changes behaviour, have it find and update the tests that cover it before it adds new ones. Agents tend to add tests and leave existing ones alone, so the maintenance falls to people.
Watch for
- Agent PRs that change covered code without touching its tests
- Tests skipped or deleted later because they went stale after agent changes
Evidence
trial: State when to mock in the agent's context file
TestingSpecification
Write down what may be mocked and what must stay real, and check whether agent tests follow it. Evidence disagrees on whether agents mock more than people, and nobody has checked that such guidance works.
Watch for
- Mocks added per test PR, agent versus human, in your repo
- Agent tests that mock something the file says to use for real
Evidence
trial: Keep a prompt regression suite for LLM features
TestingMeasuring
Keep a set of fixed prompts with the answers you expect and rerun it on every model or prompt change. Teams building on LLMs test their tools and plumbing but almost never the prompts, so a model update can change behaviour silently.
Watch for
- Prompts in the product with no test that runs them
- Behaviour changes users report after a model update
Evidence
- Hasan et al., arXiv 2025 study
trial: Ask the AI for one new test idea at a time
TestingUnderstanding
Have the assistant propose one test case at a time that differs from those you have, and accept or reject each, instead of asking for a list. People tend to take the first batch they get and stop looking.
Watch for
- Edge cases added in review that the AI's suggestions missed
- Bugs found after merge in behaviour no test covered
trial: Manage cost with visibility and friction
Looping
Make agent spend visible per team and per task, set sensible default effort levels, and nudge before large spends. Route simple tasks to cheaper models, but check that results hold up.
Watch for
- Agent spend per merged change, by team
- Sessions over a spend threshold each week
- Accepted-change rate after routing tasks to cheaper models
trial: Check the agent more often on risky code
LoopingReviewing
Check the agent after every step when you are unsure or the code is risky, such as auth and payments, and let it run several tasks when you are confident. Try this for a sprint.
Watch for
- Defects found after release in auth and payment code
- Review time per PR, by risk level
Evidence
- Cooper, Coding agents and driving in gears, 2026 practitioner
trial: Log steps to the first human correction
Looping
Record how many steps agents take before a person has to correct them. This shows how long you can leave an agent alone on your codebase.
Watch for
- Steps before the first human correction, per session
- Sessions corrected only after 20 or more steps
Evidence
- Osmani, Software factories, 2026 practitioner
trial: Compare fresh and continued context per iteration
Looping
Test whether your loops do better starting fresh each iteration or continuing the same session. Run the same backlog items both ways.
Watch for
- Agent spend per merged change, fresh versus continued
- Accepted-change rate, fresh versus continued
Evidence
- Huntley, Ralph Wiggum as a software engineer, 2025 practitioner
- Luu, Programming language and token efficiency, 2026 practitioner
- Li et al. (Microsoft), arXiv 2026 benchmark
trial: Run low-risk tickets unattended overnight
Looping
Batch a few low-risk tickets for agents to work on unattended, gated by checks that are cheap, run often and are hard to fake. Review the results the next morning.
Watch for
- Overnight PRs mergeable without changes
- Minutes of fix-up per overnight PR
Evidence
- Osmani, Software factories, 2026 practitioner
- Augmented Coding Patterns, Patterns catalog practitioner
trial: Keep a machine-readable done list
Looping
For a feature spread across sessions, keep a list of steps where only a passing check can mark a step done. It stops the agent from declaring the feature finished early.
Watch for
- Steps marked done that fail review or CI
- Premature completion claims caught by the list
trial: Align on product, architecture and design before code
Looping
Agree on what to build, the architecture and the design with a person before agents write code. Try it on a few features and skip it on comparable ones.
Watch for
- Rework per feature, with and without upfront alignment
- Review rounds per PR
Evidence
- Dex (HumanLayer), Why software factories fail, 2026 practitioner
trial: Run checks inside the agent session
LoopingTesting
trial: Time-track agent work by phase
Looping
For a month, log how agent work time splits between defining the task, waiting, reviewing and fixing up. It shows where the real bottleneck moved.
Watch for
- Share of time defining tasks
- Share of time reviewing
- Share of time fixing up agent output
Evidence
trial: Track agent outcomes on novel versus boilerplate work
Looping
Tag tickets as boilerplate, port or novel and track how agent work turns out for each. Agents tend to help most on repetitive work and least on new design.
Watch for
- Agent PRs mergeable without changes, by ticket type
- Fix-up time per PR, by ticket type
- Reverts, by ticket type
Evidence
- Karpathy, Dwarkesh Podcast 2025 practitioner
- Becker et al. (METR), arXiv 2025 RCT
trial: Add a minimum-effort floor to an agent's exit condition
Looping
Require an agent to spend a minimum time or number of steps before it can give up on a task. Try it on one agent for two weeks.
Watch for
- Tasks the agent abandons early
- Tasks completed after the floor is added
Evidence
trial: Compare a capped retry loop with a stronger model at equal cost
LoopingTesting
Test whether a cheaper model with a capped test-and-retry loop beats a stronger model in one pass at the same spend. Use the same tests for both.
Watch for
- Accepted-change rate, cheap-loop versus strong model
- Spend per accepted change for each
Evidence
- Chen et al., ICLR 2024 benchmark
- Antoniades et al., ICLR 2025 benchmark
trial: Let a cheaper model execute a strong model's plan
Looping
Have a strong model write and a person approve the plan, then let a cheaper model carry it out. Compare with a strong model doing both on comparable tickets.
Watch for
- Review rounds per PR, by executor model
- Rework per ticket, by executor model
- Spend per merged change
Evidence
trial: Keep a growing regression list in long-running loops
LoopingTesting
In multi-day agent work, require a test for each finished step and run them all before starting the next. Long loops tend to break work that was already done.
Watch for
- Earlier steps broken by later steps
- Finished steps with no test protecting them
Evidence
- Li et al. (Microsoft), arXiv 2026 benchmark
trial: Pick among several agent attempts
Looping
On a few tickets, run several agent attempts and pick the best using test results and one reviewer. Check whether the gain is worth the extra spend.
Watch for
- Mergeable-as-is rate, best-of-several versus single attempt
- Spend per merged change
Evidence
- Pan et al., ICML 2025 benchmark
- Antoniades et al., ICLR 2025 benchmark
trial: Run a strong model's checks once as a final gate
LoopingTesting
trial: Fix the harness before the code
Looping
When an agent gets something wrong, first change the spec, check or guidance that let it through, then fix the code. Try it for a month.
Watch for
- The same kind of agent defect recurring each month
- Time spent changing checks and guidance
trial: Try a local mid-size model on small pre-planned tasks
Looping
Try a model running on your own hardware for small, well-specified changes and scripts. Compare with the hosted model on the same tickets and tests.
Watch for
- Test pass rate, local versus hosted
- Minutes and cost per ticket
Evidence
trial: Have scheduled agents file reports, not PRs
Looping
Let scheduled agents file short reports, such as one per newly flaky test, instead of opening PRs. Keep the volume low enough that people read them.
Watch for
- Reports that lead to a merged fix
- Minutes per week spent reading agent reports
- Duplicate or ignored reports
trial: Compare a self-hosted open model with hosted ones on your own tickets
LoopingMeasuring
Run the same ticket set through a self-hosted open model and a hosted one, counting hardware and staff, not only the API bill. One bank reports a far smaller bill and engineers who stopped rationing tokens, but gives no quality or all-in cost figures.
Watch for
- Cost per accepted change, all-in, self-hosted versus hosted
- Accepted-change rate on the same tickets
- Engineers who ration or avoid the model under a cap
Evidence
- Abitbol (Revolut), AI Engineer NYC 2026 practitioner
trial: Coordinate parallel agents through numbered plans
LoopingTeamwork
Give parallel agents one spec with numbered sections and keep each agent's plan in the repo against those numbers, so others can see what is claimed and done. Keep plan updates off the main pipeline if they overload CI.
Watch for
- Colliding or duplicated work between agents
- CI runs triggered by plan-only commits
- Plan lines marked done but not merged
Evidence
trial: Send risky PRs to people, the rest to AI review
ReviewingSecurity
For a month, label PRs by risk, send auth, schema, public API and similar changes to human reviewers, and give low-risk changes AI-only review. Teams report this works, but nobody has measured it.
Watch for
- Merged PRs per week alongside change failure rate, by review route
- Incidents traced to PRs labelled low risk
- PRs whose risk label proved wrong in hindsight
trial: Add an intake gate to the agent PR template
Reviewing
In one repo, make the agent PR template require purpose, a size limit, test output and proof the change ran before review starts.
Watch for
- Review rounds per agent PR, before vs after the gate
- Agent PRs abandoned or closed unmerged
- PRs sent back for missing template fields
Evidence
trial: Slice big agent branches for review
Reviewing
Require agent branches over 400 lines to be split into stacked PRs of about 200 lines, each with one purpose. Small single-purpose PRs are easier to review properly.
Watch for
- Review minutes per PR, sliced vs unsliced
- Post-merge defects, sliced vs large agent PRs
- Agent PRs over 400 lines merged without slicing
Evidence
- Augmented Coding Patterns, Patterns catalog practitioner
trial: Review plans instead of diffs
ReviewingSpecification
On 10 features, review the agent's plan, tests or schema for 5 and only the diff for the other 5. A short plan may be far cheaper to review than the code generated from it.
Watch for
- Review minutes per feature, plan review vs diff-only
- Post-merge defects per feature in each group
Evidence
- Dex (HumanLayer), Why software factories fail, 2026 practitioner
- Orosz, What is happening with code reviews?, 2026 practitioner
trial: Run two AI reviewers in parallel
Reviewing
Run two AI reviewers side by side on the same PRs for a month. Different AI reviewers tend to flag different problems, with little overlap.
Watch for
- Valid findings that only one reviewer raised
- Comments dismissed as noise, per reviewer
Evidence
trial: Show only high-confidence AI review comments
Reviewing
For a month, show reviewers only high-confidence AI review comments, with duplicates merged. A flood of low-value comments trains people to approve without reading.
Watch for
- Share of AI review comments acted on
- PRs approved within minutes while AI comments are still open
Evidence
- Orosz, Concern about code review load, 2026 practitioner
trial: Phase in deterministic gates
Reviewing
In one repo, phase in boundary and duplicate-code rules: record a baseline, then warn on changed files, then block.
Watch for
- Findings per PR at each phase
- Findings dismissed as false positives
- Warnings fixed before merge vs ignored
Evidence
trial: Write a review policy
Reviewing
In one repo, adopt a REVIEW.md that sets review passes, severity levels and a cap on nits, and bars the writing agent from approving its own work.
Watch for
- Nit comments per PR
- Review minutes per PR, before vs after the policy
- Agent PRs approved by the same agent that wrote them
Evidence
trial: Set explicit rejection criteria
ReviewingUnderstanding
For a quarter in one team, reject green code the author can't explain and require hand-written commit messages.
Watch for
- Reverts per month
- Later changes that rework recently merged code
- PRs rejected because the author couldn't explain them
trial: Add a simplification pass before review
Reviewing
In one repo, add an agent pass that asks "can this be simpler?" before human review. Without it, each bug tends to get its own patch.
Watch for
- Lines changed per PR, with vs without the pass
- Bugs per feature after merge
trial: Try QA instead of line-by-line review on an internal tool
ReviewingTesting
trial: Match review depth to task uncertainty
Reviewing
For a month, tag agent PRs as low or high uncertainty at planning: light inspection for bug fixes, deeper structural review for novel features.
Watch for
- Escaped defects per uncertainty tag
- Review minutes per PR, by tag
- PRs whose uncertainty tag proved wrong in hindsight
Evidence
- Tornhill, Controlling the uncertainty machine, 2026 practitioner
trial: Turn repeated review findings into agent skills
ReviewingSpecification
For a month, turn every review comment seen twice into an agent skill or a lint rule instead of fixing it by hand. That stops the same mistake coming back in later PRs.
Watch for
- Repeats of the same finding per 10 PRs
- Skills or lint rules added from review comments
Evidence
- Tornhill, Controlling the uncertainty machine, 2026 practitioner
trial: Set how much agent code is read by risk tolerance
ReviewingUnderstandingSecurity
For a quarter, tag repos by risk tolerance: read generated code in full where mistakes cost money, safety or legal standing, and accept throwaway work by testing and running it.
Watch for
- Escaped defects and incidents per risk tag
- Review minutes per PR, by risk tag
Evidence
- Ewerlöf, Coding is not solved, 2026 essay
- Pinecoder, My old job feels very distant now, 2026 practitioner
- Kunal, Resisting cognitive surrender, 2026 practitioner
trial: Check AI-approved PRs by running them
ReviewingTesting
On 10 AI-approved PRs, have an engineer run the change in a test environment and check it does what it claims. An AI review reports what the reviewer thinks, not what the system does.
Watch for
- AI-approved PRs where running the change shows a missed defect
- Engineer minutes per manual check
trial: Cap AI review at three rounds
ReviewingLooping
For a month, cap AI review at three rounds in one repo, re-reviewing only the new diff each round, and leave a comparable repo uncapped. An agent reviewer and agent author can otherwise loop for dozens of rounds.
Watch for
- Review rounds per PR to pass, capped vs uncapped
- Escaped defects and reverts, capped vs uncapped
trial: Score the AI reviewer on past incidents
ReviewingMeasuring
trial: Review a checked spec instead of the code
ReviewingTesting
On 10 configuration or infrastructure changes, approve a tool-checked list of what must and must not change, then read the code anyway and log what the list missed. The risk sits in what the list leaves open.
Watch for
- Defects the list missed but code review caught
- Minutes on the list vs minutes on the code
- List items left vague or open
trial: Give the AI reviewer repo access and test execution
Reviewing
On the same 30 PRs, run the AI reviewer with only the diff and with repository access plus test execution, and compare. A reviewer that sees the whole repo and runs code may catch more with fewer false alarms.
Watch for
- AI review comments that led to a code change, per setup
- False alarms per PR, per setup
- Defects found after merge on the 30 PRs
Evidence
trial: Show the agent's reasoning beside the diff
ReviewingUnderstanding
For big agent changes, review a view with the diff, the agent's note on why it made each change, and a column for your own comments. Small stacked changes read faster than one large one.
Watch for
- Review comments asking why a change was made
- Time to approve
- Changes nobody could explain later
Evidence
- Kunal, Resisting cognitive surrender, 2026 practitioner
trial: Ask why a change exists before judging how
ReviewingTeamwork
In review, first answer why the change exists and whether it is the approach the reviewer would take, at whatever size fits. Leave style and small mistakes to linters and tests.
Watch for
- PRs approved with no stated reason for the change
- Reviews that only discuss style
- Changes reverted as unwanted
Evidence
trial: Have the agent restate the plan and object before coding
Delivery
Before coding, have the agent restate the plan and raise objections, and review that the way you'd review someone's diff. Try it for two weeks.
Watch for
- Misunderstood tasks caught at the plan step vs found in review
- Agent PRs reworked or discarded for misreading the task, before vs during trial
Evidence
trial: Spend freed time with users
Delivery
Spend time agents free up before development starts, with users and stakeholders. Try giving 10% of team time to customer sessions for a quarter.
Watch for
- Hours per sprint spent in customer sessions
- Features dropped or reshaped before build after customer input
- Usage of features shipped during the trial vs before
Evidence
trial: Prototype requests cheaply for a user cohort
Delivery
Use rough mockups, throwaway prototypes and week-long releases to a user cohort to get customer feedback faster. Try prototyping 3 feature requests for a cohort, within a week each.
Watch for
- Days from feature request to build-or-drop decision
- Prototyped requests dropped or reshaped after cohort feedback
trial: Try acceptance tests as the main guardrail
DeliveryTesting
Write one acceptance test in business language, have the agent draft the rest in the same style, and run them through in-memory adapters. Try it in one bounded context for a quarter.
Watch for
- Acceptance suite run time, kept fast enough for every commit
- Escaped defects in the trial context vs other contexts
Evidence
trial: Split the largest files with human-chosen refactorings
Delivery
As a trial, have a person choose how to split the largest files in one service, then let the agent do the refactoring. Smaller files make agent changes cheaper, but agents don't choose good splits on their own.
Watch for
- Tokens per agent change in the split files, before vs after
- Agent tasks in those files passing CI first time, before vs after
Evidence
trial: Track features shipped vs used
DeliveryMeasuring
For two quarters, track which shipped features are actually used and feed that into what not to build. When building is cheap, overproduction becomes the main waste.
Watch for
- Share of features shipped this quarter in active use after 30 days
- Features switched off or removed for lack of use
trial: Spike throwaway designs before committing
Delivery
Try building throwaway spikes for the next 2 uncertain designs before committing to one. When code is cheap to regenerate, it is also cheap to delete.
Watch for
- Days from design question to decision
- Rework on the chosen design in the month after it ships
- Spike code that got merged instead of deleted
Evidence
trial: Steer agents with named methods and a system metaphor
DeliverySpecification
Try replacing the prose rules in one CLAUDE.md with named methods and a system metaphor. Agents follow well-known names better than paragraphs, but may apply a practice where its original reasons don't hold.
Watch for
- Review comments citing a broken team rule, before vs after the change
- Agent PRs applying a named pattern where it doesn't fit
Evidence
trial: Cap agent-heavy work per day
Delivery
Let volunteers cap agent-heavy work at 3 hours a day for a month. Agents pack a day's worth of decisions into a single session.
Watch for
- Defects per change from volunteers vs the rest of the team
- Self-reported fatigue in retros (weak signal)
Evidence
trial: Try flags and staged rollout in place of unit tests on one service
DeliveryTesting
On one low-risk service, rely on feature flags and staged rollout in place of most unit tests, to see whether that holds at agent volumes. Some large teams have shipped this way with few unit tests.
Watch for
- Change failure rate on the trial service vs similar services
- Time to recover from a bad rollout
- Bad changes caught at an early rollout stage vs by all users
Evidence
- Beck, Pragmatic Engineer podcast 2025 practitioner
trial: Count how long main stays red when agents push
DeliveryTesting
trial: Try an agent-built fix for your slowest tool
DeliveryMeasuring
Give an agent a day to build a faster or tailored replacement for the tool your team complains about most, tested against that tool's own tests. Keep it only if your numbers improve.
Watch for
- Build or test wall time before and after
- Hours of agent and person time
- Defects traced to the replacement
Evidence
trial: Track defects and rework to catch quality erosion
SecurityMeasuringTestingReviewing
For a quarter, add review effort, defects and rework to one team's dashboard. Agent code can raise incidents and bring new kinds of debt, and the usual gates against that are untested with agents. (no OWASP mapping)
Watch for
- Median PR size, rising
- Rework rate: lines changed again within 3 weeks of merge
- Review hours per PR
- Incidents per merged PR, by month
Evidence
trial: Red-team our agents with seeded injections
Security
Have an authorised red team plant injections, in a sandbox, in test issues and docs our agents read. A guardrail that stops most attacks still fails, because an attacker only needs one to work. (OWASP LLM01)
Watch for
- Seeded injections that trigger an agent action
- Seeded injections caught by logging or alerts
Evidence
trial: Test the harness against a poisoned MCP server
Security
Have an authorised red team run a deliberately poisoned test MCP server in a sandbox, alongside file and HTTP tools. This checks whether tool-layer enforcement holds when injected text gets through. (OWASP LLM01, LLM06)
Watch for
- Follow-on tool calls blocked vs attempted after poisoned output
- Sensitive actions that ran without a policy check
Evidence
- OWASP community, MCP tool poisoning practitioner
trial: Log and flag every package agents add
Security
Try logging every package an agent adds and flagging names that don't exist, were registered recently, or have unknown publishers. Published rates of invented package names are out of date, so measure your own. (OWASP LLM09, LLM03)
Watch for
- Flagged package names per 100 agent installs
- Flagged packages that turned out to be a real risk
Evidence
trial: Intercept package installs against a malware feed
Security
Wrap npm, npx, yarn and pnpm so every install is checked against a malware feed. Try it in one team's agent and CI environments for a month. (OWASP LLM03)
Watch for
- Installs blocked per week
- Legitimate packages wrongly blocked
- Installs that bypassed the wrapper
Evidence
- Aikido, SafeChain vendor
trial: Compare defects by provenance
SecurityMeasuring
Try tagging agent-authored commits and comparing defects and security findings against human-authored ones. There are signs AI-written code is riskier but no solid data, so measure your own. (OWASP LLM09)
Watch for
- Defects per change, agent-tagged vs human
- Security findings per change, agent-tagged vs human
- Commits missing a provenance tag
Evidence
trial: Tier agent autonomy by alert severity
Security
Run a read-only diagnosis agent on alerts in a non-production environment, with any fix arriving only as a human-approved draft PR. Tiering autonomy by alert severity is a proposal with no data behind it. (OWASP LLM06)
Watch for
- Agent diagnoses the on-call engineer confirmed as correct
- Write actions attempted by the diagnosis agent (should be zero)
Evidence
trial: Run scheduled AI security scans on one service
Security
Run scheduled AI security scans on one service for a month and triage every finding. Claims that these scans give validated, confidence-rated findings come only from the vendor.
Watch for
- Share of findings confirmed real after triage
- Triage minutes per finding
Evidence
trial: Separate planning from untrusted data in one agent
Security
On one internal agent that reads tickets or web pages, keep the planner from ever seeing tool output and add policies for its send, spend and delete tools. This blocks most injections but the agent completes fewer tasks. (OWASP LLM01)
Watch for
- Seeded injections that trigger an action
- Tasks the agent completes, before vs after the split
- Policy denials on send, spend and delete tools
Evidence
- Debenedetti et al., arXiv 2025 benchmark
trial: Compare a layered filter stack with tool-layer enforcement
Security
Have an authorised red team run the same seeded injections, in a sandbox, against layered input and output filters and against tool-layer enforcement. The claim that layered filters stop nearly all attacks has no measurement behind it. (OWASP LLM01)
Watch for
- Seeded injections that trigger an action, per setup
- Normal inputs wrongly blocked, per setup
Evidence
- Geng et al., CMC 2026 study
trial: Name a checker before agent output drives a decision
Security
Trace 5 agent-written summaries or reports that fed a decision last month and note who checked each before anyone acted. Plausible machine output can drive action before a qualified person questions it; this is opinion, not data.
Watch for
- Decisions fed by agent output with a named checker before action
- Decisions acted on before anyone checked the agent's output
trial: Measure patch-to-deploy time in hours
Security
For one quarter, log the hours from each upstream critical security fix to our deploy, for direct and transitive dependencies, against a target agreed beforehand. Attackers can start exploiting a flaw within hours of its patch.
Watch for
- Median and worst hours from upstream patch to our deploy
- Fixes that missed the target, and why
trial: Give AI vulnerability scans a written threat model
Security
Scan one service twice, with and without a one-page threat model in the prompt, and have owners triage both lists blind. Many false positives are working issues owners consider outside their threat model, which a proof-of-concept check can't filter.
Watch for
- Share of findings owners accept, with vs without threat model
- Triage minutes per finding
trial: Log what platform security checks catch on agent PRs
SecurityReviewing
For a quarter, log every security finding on agent PRs and whether human review would have caught it. Platform checks match known bad patterns, not whether the code does what was asked.
Watch for
- Security findings per 100 agent PRs
- Share of those findings human review also caught
- Review-found issues the platform checks missed
trial: Limit browser agents to the sites a task needs
Security
A browser agent in a signed-in session acts with your full access, and the site sees only you. Allow it only on the sites the task needs, and require a person's approval before saves, deletes and payments.
Watch for
- Browser agents and agentic extensions in use, each with a named owner
- Sites agents may act on that grant admin or production access
- Consequential agent actions run without an approval prompt
trial: Read test and setup files before an agent runs them
SecurityTesting
Before an agent runs a cloned or updated repository's tests or setup scripts, read those files, or run them in a sandbox with no secrets. Agents warned least when asked to run tests, which was also the worst case for attacks.
Watch for
- Agent test or setup runs on repos whose test or config files changed since a person read them
- Secrets reachable from the test environment
- Third-party repo runs outside a sandbox
Evidence
- Zhu et al., EMNLP 2026 study
trial: Review rules, skills and context files that come from outside the team
SecuritySpecification
Treat imported or auto-updated rule, skill and context files as code that can steer an agent with shell access: read them before use and again when they change.
Watch for
- Rule, skill or context files in use that nobody on the team wrote
- Changes to those files merged without review
- Files pulled from public collections
Evidence
trial: Replay past vulnerabilities as tests on agent patches
SecurityTesting
Keep a failing-before, passing-after test for each vulnerability you have fixed, and run them on agent patches in that area. A patch can pass the whole suite and still bring an old weakness back.
Watch for
- Fixed vulnerabilities with a replayable test
- Agent patches that reintroduce a fixed vulnerability
- Patches passing CI that get security findings later
Evidence
trial: Normalise cost by accepted output
Measuring
Try tracking cost per accepted change, not per request or the total bill, on one team for a quarter. A tool that looks cheaper per request can cost more per result.
Watch for
- Cost per accepted change next to cost per request, per model
- Months where the bill rises but cost per accepted change falls
- Turns per task on a cheaper model against its list-price saving
- Model or tool choices that change when ranked by cost per accepted change
trial: Randomise real tickets to test speed gains
Measuring
Randomly assign about 30 real tickets to AI-allowed or not, and record each developer's time forecast before they start. Self-reported time savings are unreliable, so compare forecasts with what was actually timed.
Watch for
- Time to merge, AI-allowed versus not, against each forecast
- Rework within 30 days, AI-allowed versus not
- Whether AI-allowed tickets merge faster without more rework
Evidence
- ICONIQ, State of AI, 2026 survey
trial: Report confidence intervals on model comparisons
Measuring
For the next two model comparisons, report the paired difference on the same tasks with a confidence interval, not just which scored higher. A small score gap between models is often read as a real difference when it isn't.
Watch for
- Share of model "wins" whose interval excludes zero
- Model switches made on a gap that sits inside the interval
Evidence
trial: Check LLM judges against human labels
MeasuringTesting
Have two engineers label about 100 agent traces and compare their verdicts with an LLM judge from a different model family before you rely on the judge. Claims that a well-prompted judge is reliable enough rarely come with agreement data.
Watch for
- Agreement rate between engineers and the judge on labelled traces
- Judge verdicts that flip when the agent retries until it passes
- Traces the judge passes that the engineers fail
trial: Run a production-trace review loop
MeasuringTesting
On one LLM feature for a quarter, sample 5-10% of production traces, review them weekly and turn failures into eval cases. Keep the loop if it catches failures your offline evals miss.
Watch for
- Failures first found in trace review versus in offline evals
- Eval cases added from reviewed traces per week
- Incidents on the feature that neither review nor evals caught
trial: Rerun past tickets before each model upgrade
Measuring
Before each model upgrade, rerun a fixed set of past tickets on the old and new model and compare cost as well as results. A newer model is not always better, and it can cost more.
Watch for
- Accepted-change rate on the ticket set, old versus new, with interval
- Cost per accepted change, old versus new model
- Model upgrades rolled out without a rerun
trial: Show cost per merged change next to total spend
Measuring
For a quarter, put cost per merged change, including review and correction time, next to total spend on one dashboard. Read cost per run rather than the bill, and keep the view if it changes a decision.
Watch for
- Cost per merged PR and per review, including correction time
- Months where spend rose but cost per merged change fell
- Budget or tooling decisions that differ from reading the bill alone
trial: Compare token spend with merged output before setting usage targets
Measuring
Before setting any usage target, compare each team's token spend with its merged changes for a quarter, without publishing a ranking. Usage leaderboards get gamed, and spend may not track output.
Watch for
- Token spend against merged changes per team, per month
- Teams with rising spend and flat merged or used output
trial: Declare effort and wanted review on AI-drafted artifacts
MeasuringReviewingTeamwork
For a month, have one team label AI-drafted artifacts with what it is, why it's sent and how much review is wanted. Low-quality AI work costs recipients hours, and nobody has measured whether labelling helps.
Watch for
- Reviewer hours on low-quality AI work, before and after (self-report)
- Review minutes per AI-drafted artifact
- Share of AI-drafted artifacts sent with a label
Evidence
trial: Name AI-enabled deliverables in advance
Measuring
At the start of a quarter, list the deliverables you would only attempt with AI, then check at the end which shipped and are in use. Some gains show up as extra scope, not speed.
Watch for
- Share of named deliverables shipped by quarter end
- Shipped named deliverables actually in use
trial: Count abandoned and rewritten agent changes
MeasuringReviewing
For a month, count agent changes merged as-is, merged after rewrite and abandoned, and report all three next to the merge rate. Time-saved reports hide the effort spent evaluating and throwing away output.
Watch for
- Share of agent changes abandoned or rewritten before merge
- Review hours per merged agent change
- Merge rate rising while the abandoned share also rises
Evidence
trial: Cross-score agents on task sets each one built
MeasuringLooping
Build one small internal task set with agent A and another with agent B, then score both agents on both sets. A task set built and checked with one agent may favour that agent.
Watch for
- Score gap between the agent that built a set and the other
- Tool choices that flip once agents are cross-scored
Evidence
- Li et al. (Microsoft), arXiv 2026 benchmark
trial: Track feature cycle time by PR type
Measuring
For a quarter, have an agent tag a quarter of PRs by type and compute doc-to-launch cycle time for features next to PR count. PR count misleads: PRs can get much bigger while their number stays flat.
Watch for
- Feature share of PRs
- Median doc-to-launch time for features
- Whether feature cycle time moves with PR count
trial: Compare spend tiers with quick approval against hard caps
Measuring
For a quarter, put one team on spend tiers with quick manager approval and per-session spend shown, and another on a hard cap. Hard caps are common but easy to get around.
Watch for
- Spend per merged change on each team
- Requests to move up a tier
- Work blocked or worked around under the cap
trial: Compare agent and human work inside each repository
Measuring
When you compare AI-assisted and human changes, compare them repo by repo, not pooled across the organisation. Which projects use agents can hide or reverse the real difference.
Watch for
- Agent-versus-human metrics reported only org-wide
- Repos where the gap points the other way from the org-wide figure
Evidence
trial: Export agent telemetry to your own tracing backend
MeasuringLooping
Turn on OpenTelemetry export in your coding assistants and LLM features so token use, latency and tool calls land in the tools you already use. Keep prompt content capture off unless you need it.
Watch for
- Agent sessions with token and tool-call traces recorded
- Spend per merged change from traces versus from invoices
Evidence
trial: Keep people hands-on with AI-off sessions
Understanding
Have volunteers take one AI-off session per sprint for six months. Regular manual practice keeps skills sharp in other automated fields, but it is untested for coding agents.
Watch for
- Debugging exercise results over 6 months, volunteers versus others
- AI-off sessions actually held per sprint
trial: Name one human who can explain each agent PR
UnderstandingReviewing
For a quarter, have each agent PR name one person who can explain it, and spot-check a few each month. Only a human can be accountable for code, so someone must fully understand each change.
Watch for
- Spot-checks where the named person can explain the change
- Time to diagnose incidents in agent-written code
- Agent PRs merged with no named person
Evidence
trial: Hand-write commit messages and why-comments
Understanding
Have one team hand-write commit messages and why-comments on agent changes for a quarter. Writing it yourself is a check that you understand the change, and people remember what they produce.
Watch for
- Authors' recall of their own changes a month later
- Commit messages that only restate the diff
trial: Keep an ADR per agent-built feature
Understanding
For two quarters, write a short ADR for each agent-built feature, point to it from surprising code, and review the ADRs quarterly. Docs only help while they stay current.
Watch for
- ADRs referenced in PRs or incidents versus left stale
- Surprising code with no pointer to a decision
Evidence
trial: Run a quarterly "explain this service" check
UnderstandingTeamwork
For two quarters, ask each team to explain the architecture and key decisions behind each of its services. Agents remove the friction that kept people in sync, so whole teams can lose track of why the system is built the way it is.
Watch for
- Share of components someone on the team can explain
- Components nobody can explain, by service
trial: Compare a human review queue with newest-write-wins for memory conflicts
Understanding
For a month, have one team route contradicting memory writes to a human review queue while another lets the newest write win and keeps the history. Nobody has compared the two.
Watch for
- Wrong memory entries retrieved later, per team
- Review queue size and wait time
Evidence
trial: Re-implement business-critical agent fixes by hand
UnderstandingReviewing
For a quarter, on business-critical agent fixes, have one engineer re-implement the fix before merge while another only reviews it. Re-implementing may build understanding that reviewing doesn't, but this rests on one person's experience.
Watch for
- Each person's explanation of the fix a month later
- Extra hours spent re-implementing
Evidence
- Stannard, Software hyper-delivery, 2026 practitioner
trial: Test whether diagrams or generated pages help understanding
Understanding
For one feature have the agent explain its change as plain text, for another as a diagram or generated page, then quiz the reviewers. Richer formats may ease oversight, but people remember what they produce more than what they read.
Watch for
- Explanation quiz results per format
- Time to produce each explanation
trial: Write a handoff note at each session boundary
UnderstandingSpecificationTeamwork
For a month, have one team end each agent session by writing decisions and open constraints to a file the next session loads, and compare with a team that doesn't. Without it, decisions get re-explained and agreed rules drift.
Watch for
- Times a decision is re-explained to the agent
- Agreed rules the agent breaks per session
Evidence
- Te'eni & Katz, HHAI 2026 study
- Gloaguen et al., arXiv 2026 benchmark
trial: Use a read-only agent to build incident timelines
Understanding
Give a read-only agent the exported deployments, alerts, flag changes and PRs for one past incident, and have owners check each timeline entry and flagged runbook gap. Use agents to gather evidence, not to fix production; the idea is untested.
Watch for
- Timeline entries owners confirm as correct
- Runbook gaps found that the incident review missed
trial: Compare a file-based memory with a memory service
Understanding
On about 20 repeated questions about past decisions in one repository, compare a markdown-in-git memory file with a memory service. Published comparisons conflict and don't test coding work.
Watch for
- Past decisions recalled correctly, per approach
- Stale answers, per approach
- Cost per question
Evidence
trial: Test on-demand explainers against reading diffs
UnderstandingReviewing
For a quarter, have one team ask the agent to explain a module before changing it and skip low-risk diffs, while another keeps reading diffs. The practice is unmeasured and may cost skills.
Watch for
- Unaided quiz results on the changed modules
- Time to diagnose incidents in those modules
trial: Explain a change in plain words before approving it
UnderstandingReviewing
For a month, before approving an agent PR, have the author write a five-sentence plain-words explanation without asking the agent and name one change they'd make next. Explaining often exposes gaps, though the support is anecdotal.
Watch for
- Gaps or bugs found while writing the explanation
- Unaided quiz on the changed code a month later
- Review minutes per PR
trial: Count repeated blind approvals in agent sessions
UnderstandingReviewing
When you find yourself approving prompt after prompt, either hand the task over with a real check or stop and look at what the agent is deciding.
Watch for
- Sessions with long runs of approvals and no edit or question
- Defects traced to such sessions
Evidence
- Kunal, Resisting cognitive surrender, 2026 practitioner
trial: Compare mobbing with an agent against solo work plus review
TeamworkUnderstanding
Build one feature by mobbing with an agent and a comparable one solo plus review, then compare. Pairing is argued to keep intent in at least two heads, but nobody has measured it with agents.
Watch for
- People who can explain each feature a month later
- Person-hours spent per feature
Evidence
- goblindegook, The wastes of product engineering, 2023 practitioner
- James, How to go fast, 2019 practitioner
- Bache, Agentic AI coding breaks mob programming, 2026 practitioner
trial: Pair juniors with seniors on agent-assisted work
TeamworkUnderstanding
For six months, pair juniors with seniors on agent-assisted work. Juniors now ask the model instead of colleagues, which cuts off the know-how people pass on by working together.
Watch for
- Junior skill assessments over 6 months, paired versus not
- Questions juniors take to colleagues versus the agent
trial: Compare solo-AI, human-pair-with-AI and human-only pairing
Teamwork
For one sprint, have one team try solo work with an AI partner, human pairs with AI, and human-only pairs. Whether an AI partner can stand in for a human one is unsettled.
Watch for
- Defects per change for each setup
- Review time per PR for each setup
- Changes people called correct that later failed review or CI
Evidence
trial: Lead AI rollout with enthusiasts and a few principles
TeamworkDelivery
Treat AI adoption as change management: let enthusiasts lead, protect learning time, and give a few principles instead of pages of rules. This is untested opinion; try it for 8 weeks against a team given only licences and the standard policy.
Watch for
- Weekly active agent users at week 8, trial team vs comparison team
- People reporting use of personal AI accounts
- Merged changes made with agent help, per team
trial: Write prompts silently together before a mob sends them
TeamworkSpecification
In a team session around one agent, everyone writes a prompt alone in a shared document, then the group merges the best before sending. It stops the loudest voice steering and shows disagreement before the agent acts. Use it at the start of a task or when the team disagrees, not on every prompt.
Watch for
- Mob sessions where the first prompt was drafted in a shared document
- Prompts rewritten after the first agent run, by drafting method
- Sessions where the group still disagreed about direction afterwards
Evidence
trial: Name the prompting patterns in a short retro
TeamworkUnderstanding
After a session with an agent, spend about 15 minutes away from the code asking what happened and naming the prompting and process patterns the team used. A shared vocabulary speeds learning, most of all for juniors.
Watch for
- Retro notes that name a pattern rather than describe an event
- Sessions held with a retro, out of sessions held
- Patterns juniors name in their own prompts