AI-Augmented Software Development

Recommendations

What to do, what to watch for in your team, and the evidence behind it.

Adopt

Proven. Use it by default.

Write context files by hand and keep them short

Specification
Specification

Write the repo's context file yourself and keep it short: build commands, conventions and the few rules the agent keeps breaking. A short file saves time and tokens; a long one mostly adds cost.

Watch for

  • Context file length growing month over month
  • Agent tokens per task before and after each edit to the file
  • Rules in the file the agent still breaks in review

Evidence

Keep context lean

Specification
Specification

Give the agent only what the task needs: the relevant files, not the whole repo or every doc. Agents miss facts more often as context grows.

Watch for

  • Tokens loaded per agent task, by repo
  • Agent answers that miss a fact already in its context
  • Sessions that hit the context limit or auto-compact

Evidence

Trim tools and what they return

SpecificationSecurity
SpecificationSecurity

Expose a few tools with short descriptions, cap how much each returns, and batch repeated calls into one script. Every tool description and long result costs context the agent needs for the task.

Watch for

  • Context taken by tool and MCP descriptions at session start
  • Tool results over a few hundred lines landing in context
  • Same tool called many times in a row in session logs

Evidence

Keep a human upstream of what tests assert

TestingSpecificationDelivery
TestingSpecificationDelivery

Have a human write or approve what the tests assert, and give those tests to the agent before it writes code. Tests an agent writes for itself tend to share its misunderstanding, so they pass wrong code.

Watch for

  • Spec tests whose assertions no human wrote or approved
  • Agent-written tests later found to pass buggy code
  • Bugs reaching review that the supplied spec tests should have caught

Evidence

Work in short, steady test-and-code cycles

TestingDelivery
TestingDelivery

Keep test-and-code cycles short and steady, and hand human-written spec tests to the agent before it writes code. A steady rhythm matters more than whether test or code comes first, as long as testing never stops.

Watch for

  • Minutes between test runs in agent sessions, rising means trouble
  • Agent sessions with long stretches of edits and no test run
  • Tasks where spec tests arrived only after the agent wrote code

Evidence

Give the model few, distinct tests

TestingSpecification
TestingSpecification

Give the agent a few distinct example tests per task, about two to four, rather than a long list. Past a handful, extra tests stop helping and can make results worse.

Watch for

  • Supplied tests per task in agent prompts
  • Tasks whose supplied tests are near-duplicates of each other
  • First-push CI failures, grouped by number of supplied tests

Evidence

Stop after about 3 failed fixes, then reset or escalate

LoopingTesting
LoopingTesting

After about three failed attempts to fix the same problem, stop the agent and either restart with a clearer task or hand it to a person. Later attempts rarely succeed and cost the most.

Watch for

  • Agent sessions with more than 3 attempts at the same failing check
  • Spend on sessions that end without an accepted change
  • Tasks escalated to a person after the cap

Evidence

Loop only against a check the machine runs

LoopingTesting
LoopingTesting

Only let an agent retry when a test, build or linter decides whether the attempt worked. Retries judged by the model itself mostly repeat the same mistake.

Watch for

  • Agent sessions that end with no test, build or lint run in the log
  • Tasks the agent called done that fail CI on first push
  • Retries per task rising while pass rate stays flat

Evidence

Keep each agent's context small

Looping
Looping

Keep each agent's working context short: clear or compact old steps, and split long tasks across fresh sessions. Agents work better when old observations stop crowding the context.

Watch for

  • Context size per step in long sessions
  • Sessions that hit the context limit
  • Mistakes late in long sessions that early steps got right

Evidence

Work in small batches with strong version control

LoopingDeliveryTesting
LoopingDeliveryTesting

Keep agent changes small and merge them often, with every step committed so it can be reverted. Small batches keep AI speed from turning into delivery instability.

Watch for

  • Lines changed per merged PR, trending up
  • Change failure rate as AI use grows
  • Reverts that undo more than one logical change

Evidence

Sandbox anything unattended

LoopingSecurity
LoopingSecurity

Run any agent that works without a person watching inside a sandbox, with no access to secrets or production. Once permission prompts are off, the sandbox is the only boundary.

Watch for

  • Unattended runs with permission prompts disabled outside a sandbox
  • Agents that can read private data, take untrusted input and send data out

Evidence

Keep an accountable human review on AI changes

Reviewing
Reviewing

Have a named human review and own every AI change before it merges. Agents generate code faster than it can safely ship, and changes that pass every test often still need fixing.

Watch for

  • AI changes merged without a human approver
  • Fix-up minutes on agent PRs that were already green
  • Incidents per merged PR, AI-authored vs human-authored

Evidence

Keep user focus

Delivery
Delivery

Tie agent work to a specific user need and check usage after release. Without that focus, agents help a team ship more software but not more value.

Watch for

  • Releases per month alongside active usage of what shipped
  • Shipped features with no usage a month after release
  • Tickets with no linked user problem or request

Evidence

Enforce standards with checks, not documents

DeliverySpecification
DeliverySpecification

Turn team standards into linters, formatters and CI checks instead of writing them into agent context files. Context-file rules get followed but add cost to every run without making tasks succeed more often.

Watch for

  • Tokens per agent session as context files grow
  • Standard violations caught by checks vs flagged by reviewers
  • Context-file rules with no matching automated check

Evidence

Enforce access and approvals outside the model

Security
Security

Keep credentials in code, validate output formats in code, and check every tool call against a policy instead of trusting prompt instructions. A prompt can be talked out of a rule; a policy check can't. (OWASP LLM01, LLM06)

Watch for

  • Tool calls that run without a policy check
  • Credentials reachable from the agent's context or environment
  • Policy denials per week, and which tools trigger them

Evidence

Remove one leg of the lethal trifecta

Security
Security

Don't let one agent hold private data, untrusted content and a way to send data out at the same time, and require approval for anything that sends, spends or deletes. Approval limits the damage but doesn't remove the weakness. (OWASP LLM01, LLM02)

Watch for

  • Agents with private data, untrusted input and an outbound channel at once
  • Send, spend or delete actions run without human approval

Evidence

Verify a package's publisher and registration date

Security
Security

Treat agent installs as privileged: pin with hashes, scan the full dependency tree, and require human approval plus a publisher and registration-date check for any new package. Agents suggest packages that don't exist, and attackers register those names. (OWASP LLM09, LLM03)

Watch for

  • New packages added without human approval
  • Dependencies not pinned with hashes
  • New packages registered recently or with unknown publishers

Evidence

Give security-sensitive changes a human review

SecurityReviewing
SecurityReviewing

Don't let the writing agent approve its own PR, and give auth and payment changes a human security review. People trust assisted code more even when it is less secure; explicit instructions and function signatures in the prompt help. (OWASP LLM09)

Watch for

  • Agent PRs on auth or payment paths merged without a security review
  • Agent PRs approved by the agent that wrote them
  • Security findings per agent PR vs human PR

Evidence

Keep people hands-on

SecurityUnderstanding
SecurityUnderstanding

Protect time for people to write and debug code themselves, not only through an agent. Without it, people lose skill and understand less of the code they ship. (OWASP LLM09)

Watch for

  • Hours people spend coding and debugging without the agent
  • Reviewers who can't explain an agent's change when asked

Evidence

Grant agents the fewest tools and the minimum scopes

Security
Security

Give agents only the tools they need, avoid open-ended shell and fetch where possible, run tools with the user's minimum scopes, and authorise in downstream systems. Let agents act up to the production gate, never through it. (OWASP LLM06)

Watch for

  • Agent commits merged without review, as a share of agent commits
  • Agents with open-ended shell or fetch tools
  • Agent actions that passed the production gate

Evidence

Deny secrets to agents and force the sandbox

Security
Security

Block agent access to .env* and secrets/**, force the sandbox with a domain allowlist, and use short-lived scoped CI credentials and secret scanning. Never put credentials or controls in CLAUDE.md, rules or skills, since agents can be made to reveal them. (OWASP LLM02, LLM07)

Watch for

  • Agent reads of .env*, secrets/** or ~/.ssh in session logs
  • Secret-scanning hits on agent changes
  • Credentials or controls found in CLAUDE.md, rules or skills

Evidence

Allowlist MCP servers and enforce access at the tool layer

Security
Security

Allow only approved MCP servers, keep privileged tools apart from external ones, require fixed-schema responses, and confirm sensitive actions outside the model. A poisoned tool description can steer an agent, and models rarely refuse it. (OWASP LLM01, LLM06)

Watch for

  • Connections to MCP servers outside the allowlist
  • Follow-on tool calls blocked by the tool layer, per week
  • Sensitive actions run without confirmation outside the model

Evidence

Enforce scope on every read path of shared agent memory

Security
Security

Check access scope on direct fetches by ID, not just search; run contradiction checks before any duplicate filter; and record writer and source for every memory entry. Otherwise direct reads leak across scopes and corrections get dropped as duplicates. (OWASP LLM08, LLM04)

Watch for

  • Cross-scope reads through a direct fetch by ID
  • Contradicting entries dropped as duplicates instead of stored as updates
  • Memory entries with no writer or source

Evidence

Inventory, sign and pin models and packages

Security
Security

Keep a software and AI bill of materials, use signed sources and pinned registries, restrict network egress on fetches, and never load untrusted model files. (OWASP LLM03, LLM04)

Watch for

  • Dependencies or models missing from the bill of materials
  • Fetches outside the egress allowlist
  • Model files loaded from unapproved sources

Evidence

Treat agent output as untrusted input

Security
Security

Run agent commands in a sandbox, review agent diffs as you would an untrusted contributor's, run SAST on them, and block remote images in rendered output. An agent's shell tool runs model output directly, by design. (OWASP LLM05)

Watch for

  • Agent commands run outside the sandbox
  • SAST findings on agent diffs
  • Remote images loaded in rendered agent output

Evidence

Partition retrieval stores and make retrieval permission-aware

Security
Security

Split retrieval stores by tenant or team, make retrieval check the user's permissions, and validate sources before indexing them. Shared stores leak across permission boundaries, and embeddings can be turned back into text. (OWASP LLM08)

Watch for

  • Retrievals that cross a tenant or team boundary
  • Indexed sources with no validation record

Evidence

Set rate limits, quotas, timeouts and iteration caps

SecurityMeasuring
SecurityMeasuring

Set rate limits, quotas, timeouts and iteration caps on agent runs, and review spend against budget monthly. Agent use can burn a year's budget in months; use hard limits against abuse, and routing and spend nudges for staff. (OWASP LLM10)

Watch for

  • Agent spend against budget, by month and team
  • Agent runs that hit an iteration cap or timeout
  • Sessions or users with spend far above the team median

Evidence

Sign a data processing agreement before tools see company data

Security
Security

Sign a data processing agreement before any AI tool sees company data, and review it again whenever the supplier changes its terms. Vendors can change terms to train on customer data. (OWASP LLM03)

Watch for

  • AI tools seeing company data without a DPA
  • Supplier term changes not yet re-reviewed

Evidence

Measure time to an accepted change, not lines or PRs

MeasuringDelivery
MeasuringDelivery

Measure each task by time to an accepted, merged change, and fix that outcome before you start measuring. Lines, PRs and tokens can rise while delivery stalls, and the same tool can look faster or slower depending on the measure.

Watch for

  • Time from task start to accepted, merged change
  • Lines or PR counts rising while time to merge stays flat
  • Merged changes reworked within 30 days

Evidence

Time real tasks instead of trusting forecasts or surveys

Measuring
Measuring

Judge whether agents speed the team up by timing real tasks, not by forecasts or surveys. Predictions of speedup are often wrong even in direction, and self-reports shift with how you ask and who answers.

Watch for

  • Gap between forecast and timed duration per task
  • Speedup claimed in retros or surveys that timed tasks don't confirm
  • Survey answers that change with question wording or respondent role

Evidence

Grade agents on tests they never saw

MeasuringTesting
MeasuringTesting

Judge agent patches by human merge review or by tests the agent never saw, not by the suite it worked against. Weak checks let suspicious patches through, and passing the repo's own tests doesn't make an agent PR mergeable.

Watch for

  • Agent PRs green on CI that reviewers reject or send back
  • Patches passing visible tests but failing held-back tests
  • Passing agent patches reviewers flag as suspicious

Evidence

Ask the AI, don't delegate to it, while building a skill

Understanding
Understanding

While someone is learning a skill, have them ask the AI conceptual questions instead of handing it the work. People who delegate, or ask for direct answers, do worse later when working without AI.

Watch for

  • Unaided debugging exercise results for people learning with AI
  • Agent sessions where the learner delegates the task versus asks a question
  • Problems solved versus skipped when working without AI

Evidence

Give hints, not answers, in learning tools

Understanding
Understanding

Set up learning and onboarding tools to give hints, not full answers. Unrestricted answers lower what people can do on their own; hint-only help doesn't.

Watch for

  • Unaided exercise results, tool users versus non-users
  • Share of tool responses that give the full answer

Evidence

Do unaided work first, then bring in AI

Understanding
Understanding

Have people draft or solve a problem unaided first, then bring in AI. People who start with AI remember less of their own work and engage less once AI is added.

Watch for

  • Authors who can't recall or explain their own work minutes later
  • Recall of AI-first versus unaided-first work

Evidence

Avoid

Shown to fail or cost more for no gain.

Don't let an LLM generate your context files

Specification
Specification

Don't let the agent write its own context file. Generated files restate what the agent can already read, so they raise cost without making tasks succeed more often.

Watch for

  • Context files committed by an agent without human edits
  • Agent spend per task rising after a context file change
  • Context file content that repeats the README or code

Evidence

Don't give agents tools shaped for people

Specification
Specification

Don't hand the agent tools built for human navigation, like paged search or whole-file viewers. Agents do better with narrow tools that return a small, relevant slice.

Watch for

  • Session steps spent paging through tool output
  • Whole files read when only a function was needed
  • Agent tasks stalling inside one tool loop

Evidence

Don't read coverage as quality for AI-written tests

TestingMeasuring
TestingMeasuring

Don't treat coverage as proof that AI-written tests are good; judge them by whether they catch bugs, using mutation testing or tests the agent never saw. Tests can run every line and still assert the wrong thing.

Watch for

  • Agent-written tests with full coverage that still assert wrong values
  • Pass rate on agent-visible tests vs tests the agent never saw
  • Coverage rising while escaped defects do not fall

Evidence

Don't treat a passing suite or a green loop as done

TestingReviewingLooping
TestingReviewingLooping

Don't merge an agent PR just because every test passes; budget human review and fix-up time for green agent PRs too. Green agent PRs routinely still need fixes before they are fit to merge.

Watch for

  • Green agent PRs that needed no changes after review
  • Fix-up minutes on agent PRs that passed every test
  • Post-merge defects on agent PRs merged on green alone

Evidence

Don't loop on self-critique alone

Looping
Looping

Don't let an agent improve its work by reviewing itself with no outside check. It tends to approve its own wrong answers; give it a test or a linter to loop against instead.

Watch for

  • Agent sessions that loop with no test, build or lint step
  • Changes the agent marked correct that fail CI or review

Evidence

Don't route retry loops to cheap models unchecked

Looping
Looping

Don't move a retry loop to a cheaper model just to save money without checking the results. Small models often gain nothing from retrying, so the loop's benefit quietly disappears.

Watch for

  • Accepted-change rate after moving a loop to a cheaper model
  • Retries per task rising on the cheaper model
  • Spend saved against extra fix-up time

Evidence

Don't let agents commit without review

Reviewing
Reviewing

Don't let agent changes reach the main branch without a person reading them. Teams drift into it as trust grows, often without anyone deciding to.

Watch for

  • Agent commits merged with no human approval
  • Repos whose branch protection allows agent-only merges
  • Unreviewed agent commits rising month over month

Evidence

Don't leave structural refactoring to agents

Delivery
Delivery

Don't expect agents to restructure code on their own. They rename and tidy but rarely change the structure, so a person has to choose the refactoring and let the agent carry it out.

Watch for

  • Agent refactoring PRs that only rename or retype
  • Largest files growing release after release
  • Duplicate code rising in agent-heavy areas

Evidence

Don't rely on delimiting untrusted content as the defence

SecuritySpecification
SecuritySpecification

Don't count delimiters or role tags around untrusted content as a security control; models judge who is speaking by how text reads, not by its tags, so forged instructions get through. Enforce at the tool layer instead. (OWASP LLM01)

Watch for

  • Seeded injections that succeed despite delimiters
  • Delimiting listed as a control with no attack test behind it

Evidence

Don't treat a stronger model as a security control

Security
Security

Don't drop or loosen a security control because you upgraded the model. A model that is better at one risk can be worse at another, so rerun your security tests after every upgrade.

Watch for

  • Controls removed or loosened after a model upgrade
  • Model upgrades shipped without rerunning security tests

Evidence

Don't install third-party MCP servers unreviewed

Security
Security

Don't connect agents to MCP servers nobody has reviewed. Tool descriptions can carry hidden instructions that steer the agent, and you can't tell from outside which servers do. (OWASP LLM01, LLM06)

Watch for

  • MCP servers in use that are not on an approved list
  • Tool description changes in approved servers since review

Evidence

Don't accept new dependencies an agent adds unchecked

Security
Security

Don't merge a dependency an agent added without checking it exists, who publishes it and how old it is. Agents sometimes name packages that don't exist, and attackers register those names. (OWASP LLM03, LLM09)

Watch for

  • New dependencies in agent PRs
  • Dependencies published recently or by unknown publishers
  • Install failures for packages that don't exist

Evidence

Don't rank people by AI usage

MeasuringLooping
MeasuringLooping

Don't rank or set targets for people by how much they use AI; track merged, used changes instead. Usage measures activity, not value, and usage targets get gamed.

Watch for

  • Usage climbing while merged changes per person stay flat
  • Self-prompting loops or other activity run only to raise usage
  • Agent sessions that produce no commit or PR

Evidence

Don't choose tools by public leaderboard rankings

MeasuringTesting
MeasuringTesting

Don't pick a model or agent from leaderboard positions alone. Public tasks may be in the model's training data and small gaps are often noise; compare on a set of your own past tickets instead.

Watch for

  • Tool decisions made without a run on your own tasks
  • Results on your own tasks that differ from the leaderboard order

Evidence

Don't trust an AI grader you haven't checked

MeasuringTesting
MeasuringTesting

Don't let a model grade outputs until its verdicts have been compared with people's on real examples. Graders can be swayed, and fooled into scoring bad output highly.

Watch for

  • Grader verdicts that disagree with a person's on a sample
  • Outputs that score well but get rejected in review
  • Scores that jump once the agent learns what the grader likes

Evidence

Don't decide on vendor-reported figures

Measuring
Measuring

Don't base a tooling decision on numbers from the tool's maker. Ask what was measured, on which tasks and by whom, then check on your own work.

Watch for

  • Business cases that cite only vendor figures
  • Gap between claimed gains and what you measure

Evidence

Don't judge learning by self-report

UnderstandingMeasuring
UnderstandingMeasuring

Don't judge whether people are learning from what they say; check with quizzes, exercises or unaided tasks instead. People consistently overstate how much they learned.

Watch for

  • Gap between perceived and measured learning
  • Learning claims backed only by surveys or retros

Evidence

Don't read confidence in AI-assisted code as security

UnderstandingSecurityTesting
UnderstandingSecurityTesting

Don't treat a developer's confidence in AI-assisted code as evidence it's secure; review and scan it regardless. AI-assisted code tends to be less secure while its authors feel more sure of it.

Watch for

  • Security findings on AI-assisted code its author rated as secure
  • Confidence ratings rising while vulnerability counts don't fall

Evidence