Evidence

Early-2025 models

Experienced developers forecast they would be 24% faster with AI but were measured 19% slower (16 developers, 246 tasks on familiar repos, Claude 3.5/3.7).

Read the original

Used in

• Track agent outcomes on novel versus boilerplate work trial
• Measure time to an accepted change, not lines or PRs recommendation
• Time real tasks instead of trusting forecasts or surveys recommendation
• Don't judge learning by self-report recommendation

In a vendor RCT (n=52, unreplicated), AI help gave quiz scores of 50% vs 67% (d=0.74), the biggest gap in debugging and no significant speed gain; in exploratory subgroups of 2 to 7 people, conceptual questions scored 65% or more and delegating under 40%.

Read the original

Used in

• Ask the AI, don't delegate to it, while building a skill recommendation

Early-2025 models

An agent passed 38% of maintainer test suites, but 0 of 15 reviewed PRs were mergeable, including 4 that passed every test (Claude 3.7).

Read the original

Used in

• Keep an accountable human review on AI changes recommendation
• Keep a human upstream of what tests assert recommendation
• Don't treat a passing suite or a green loop as done recommendation

Of 445 LLM benchmarks, 16.0% used any statistical test or uncertainty estimate to compare models, 53.4% gave evidence that they measure what they claim, and under 10% used complete real-world tasks.

Read the original

Used in

• Report confidence intervals on model comparisons trial

Secondary synthesis: invalid benchmark questions ranged from 2% (MMLU Math) to 42% (GSM8K).

Read the original

Used in

• Grade agents on tests they never saw recommendation

Pre-2025 models

Developers were 55.8% faster on one greenfield task, counting only the 35 of 95 who finished; GitHub and Microsoft authors.

Read the original

Used in

• Measure time to an accepted change, not lines or PRs recommendation
• Don't decide on vendor-reported figures recommendation

Pre-2025 models

A 20-person student poster (abstract only) found lower quality with Copilot pairing.

Read the original

Used in

• Compare solo-AI, human-pair-with-AI and human-only pairing trial
• Pairing with AI isn't the same as pairing with a person challenge

Pre-2025 models

About 40% of completions for risky scenarios were vulnerable.

Read the original

Used in

• Give security-sensitive changes a human review recommendation

Pre-2025 models

Of 16 early adopters interviewed, 14 reported less mundane work and 13 saved time, yet 8 discarded most generated code and 15 modified code before using it.

Read the original

Used in

• Count abandoned and rewritten agent changes trial
• Restart the session when output degrades trial

Solutions that passed LiveCodeBench's private tests failed on LeetCode 20% (medium) and 40% (hard) of the time.

Read the original

Used in

• Have a second model in a fresh context write tests trial

A short AGENTS.md cut median time 28.6% and output tokens 16.6% with gpt-5.2-codex; correctness was not measured.

Read the original

Used in

• Write context files by hand and keep them short recommendation
• Every line in the context file costs money challenge

Pre-2025 models

Performance peaked well below full context, so more context can be worse (GPT-4o, Qwen 2.5).

Read the original

Used in

• Keep context lean recommendation
• Compare whole-module context with retrieval trial

Single-author preprint

Retrieval degraded with length: most models degraded severely by 1,000 tokens.

Read the original

Used in

• Keep context lean recommendation
• Compare whole-module context with retrieval trial

In 2,853 engineered GitHub repositories, 90.6% had a context file (CLAUDE.md in 45.9% and AGENTS.md in 39.5% of those), skills and subagents were rare and mostly static text, and 301 of 497 context-file pointer pairs pointed from CLAUDE.md to AGENTS.md; this measures adoption, not benefit.

Read the original

Used in

• Keep one AGENTS.md and point other tool files at it trial

A spec recovered from a finished session let a fresh agent rebuild the change at a mean 5.06 of 6 on 72 internal Meta tasks (5.74 after up to three refinement rounds) against 4.23 for the developer's own change description, with specs about 5.7 times shorter than the diff; LLM graders, internal data.

Read the original

Used in

• Recover the spec after the first working version trial

Preprint, n=7,227

Users rated sycophantic bots as more objective, and 72% of real ChatGPT turns were as agreeable as the sycophantic condition.

Read the original

Used in

• Don't judge learning by self-report recommendation
• Agreeable answers mislead without being wrong challenge

Pre-2025 models

Negations prime the concept they ask the model to avoid (BERT).

Read the original

Used in

• Rephrase negative rules positively trial

Pre-2025 models

Format and coverage of few-shot examples matter more than correct labels (GPT-3 era).

Read the original

Used in

• Prefer diverse few-shot examples over perfect labels trial

An automated contradiction detector superseded 90 of 90 admitted conflicting facts in one service.

Read the original

Used in

• Groom memory files monthly trial
• Enforce scope on every read path of shared agent memory recommendation
• Audit shared agent context with planted probes trial
• Compare a human review queue with newest-write-wins for memory conflicts trial
• Shared agent memory behaves like a shared database challenge

A qualitative study with Claude power users proposes asking whether each context item is global (rarely changes), local (one task) or ambient (raw streams).

Read the original

Used in

• Sort context into global, local and ambient trial

The sample behind the big-ball-of-mud observation is 7 people.

Read the original

Used in

• Sort context into global, local and ambient trial

Across 100,000+ developers, AI tools raised lines of code 17.3x and releases only 1.3x; calibrated on this data, unlimited code generation would lift releases only about 10% (a calibration, not an estimate).

Read the original

Used in

• Keep an accountable human review on AI changes recommendation
• Keep user focus recommendation

Pre-2025 models

AI-assisted participants (n=47) wrote less secure code on 4 of 5 security tasks and were more likely to believe it was secure.

Read the original

Used in

• Keep an accountable human review on AI changes recommendation
• Give security-sensitive changes a human review recommendation
• Compare defects by provenance trial
• Don't read confidence in AI-assisted code as security recommendation

After Cursor adoption, static-analysis warnings rose 30% and complexity 41% (secondhand).

Read the original

Used in

• Track defects and rework to catch quality erosion trial

One team handled 30 PRs a day across six reviewers.

Read the original

Used in

• Track defects and rework to catch quality erosion trial
• Compare defects by provenance trial
• Review load lands on senior engineers challenge

80.7% of participants spotted a flawed AI recommendation and followed it anyway.

Read the original

Used in

• People follow AI advice they know is wrong challenge

With a user-confirmed formal query and a verifier, 82.9% of correct Ansible programs were accepted and 92.4% of wrong ones rejected, but 68 of 70 wrong accepts traced to queries that left details open, and the human review step was never run (authors' own 21 tasks).

Read the original

Used in

• Review a checked spec instead of the code trial

Models judge who is speaking by writing style, not role tags; forged reasoning raised attack success from about 0% to 60%.

Read the original

Used in

• Enforce access and approvals outside the model recommendation
• Don't rely on delimiting untrusted content as the defence recommendation

Pre-2025 models

At least 5.2% of packages suggested by commercial models and 21.7% by open-source models didn't exist, about 45% of fake names recurred every time, and checking names against the registry is 'the illusion of verification' once an attacker registers the name.

Read the original

Used in

• Verify a package's publisher and registration date recommendation
• Log and flag every package agents add trial
• Don't accept new dependencies an agent adds unchecked recommendation

Pre-2025 models

Across 11 code models, package hallucination ranged from 0.22% to 46.15% per model and language (5.42% for GPT-4o); larger, higher-scoring models invented fewer packages, a directional result driven partly by two outliers (three of four authors at NVIDIA, whose model scored best).

Read the original

Used in

• Verify a package's publisher and registration date recommendation
• Don't treat a stronger model as a security control recommendation
• Don't accept new dependencies an agent adds unchecked recommendation

A review of 128 studies claims an input filter plus output validation plus model-level defences brings attack success under 5%, but the section it points to reports no measurement.

Read the original

Used in

• Compare a layered filter stack with tool-layer enforcement trial

In a pre-LLM study of 39 professionals, test order made no significant difference while shorter, steadier cycles went with better quality and productivity, up to 14% better quality going from 50-minute to 8-minute cycles (small-to-medium effects, adjusted R2 about 0.1).

Read the original

Used in

• Work in short, steady test-and-code cycles recommendation
• Try agent TDD on real, vague tickets trial

On SWE-bench Verified with Qwen3-Coder 30B, a static test-impact map plus a short skill cut test-level regressions 70% (6.08% to 1.82%), while a procedural TDD prompt with no impact context raised them to 9.94%; single author, 100-125 instances, provisional.

Read the original

Used in

• Give the agent test-impact context, not a TDD procedure trial

Pre-2025 models

A chat model in a TDD loop edited an expected value instead of fixing the bug (5 participants, one case).

Read the original

Used in

• Flag weakened tests in agent PRs trial

Pre-2025 models

Without critique filters, assume(false) spread to every output of a prover-graded translation loop.

Read the original

Used in

• Flag weakened tests in agent PRs trial

Pre-2025 models

Among about 1,000 students, unrestricted GPT lowered exam scores by 17%, while a hint-only tutor showed no exam penalty.

Read the original

Used in

• Give hints, not answers, in learning tools recommendation

Pre-2025 models

In a preprint (n=54), 83% of LLM users couldn't quote their own essay minutes later, and writers who started without AI showed higher brain connectivity once they used it than the AI-first group ever reached.

Read the original

Used in

• Do unaided work first, then bring in AI recommendation

Across 310 experiments, people recalled about 10 points more of what they generated themselves; general psychology, not AI-specific.

Read the original

Used in

• Hand-write commit messages and why-comments trial
• Test whether diagrams or generated pages help understanding trial

Pre-2025 models

Among 234 students (self-report scales), human pairs scored significantly higher on collaboration and social presence than AI-assisted pairs or solo students.

Read the original

Used in

• Compare solo-AI, human-pair-with-AI and human-only pairing trial
• Pairing with AI isn't the same as pairing with a person challenge

In a descriptive working paper, early-career jobs (ages 22-25) in AI-exposed roles were 19% below trend.

Read the original

Used in

• Juniors ask the model instead of colleagues challenge

Across three experiments with 1,058 online participants, after 11 to 12 problems with a GPT-5 sidebar people tested without it solved fewer problems (d from -0.20 to -0.45) and skipped more in two of three, with the cost sitting with people who asked for direct answers; fractions and reading comprehension, not coding, and no durability data.

Read the original

Used in

• Ask the AI, don't delegate to it, while building a skill recommendation

In one developer's four months with Claude Code (163 episodes), decisions had to be re-explained every session and agreed rules decayed as conversations grew; it builds theory and gives no prevalence.

Read the original

Used in

• Write a handoff note at each session boundary trial

In a simulation of provenance-gated shared memory, accuracy went 0.27, then 0.61 after grants, then 0.37 after revocations; the paper has no proof of policy adherence, no attack test, and nothing on stale or conflicting entries.

Read the original

Used in

• Show shared memory only when every source behind it is readable trial

A 2026 review says the provenance-gated memory approach ensures privacy, though the primary paper shows simulation only.

Read the original

Used in

• Show shared memory only when every source behind it is readable trial

Across 14,998 agentic commits (89.3% Codex-authored), low-level refactors were 35.8% vs a 24.4% human baseline and architectural refactors 43.0% vs 54.9%; design-smell counts showed no practically meaningful change (Cohen's d about -0.03).

Read the original

Used in

• Don't leave structural refactoring to agents recommendation

A smaller, low-rigor comparison of refactoring tools on isolated snippets found wide model-to-model variance, with no human check on correctness.

Read the original

Used in

• Don't leave structural refactoring to agents recommendation

In a 39-student, one-semester study, solo-AI was the worst of three conditions and human-plus-AI scored highest, directly disagreeing with Fan et al.

Read the original

Used in

• Compare solo-AI, human-pair-with-AI and human-only pairing trial
• Pairing with AI isn't the same as pairing with a person challenge

A 10-study review found, corroborated across 2 studies, that after AI-paired programming students were more likely to believe they had a complete solution and no longer needed to learn to program.

Read the original

Used in

• Pairing with AI isn't the same as pairing with a person challenge

Peer-reviewed study showing that knowledge requirements shape LLM responses to context-memory conflict; not tested on code.

Read the original

Used in

• Don't trust an AI grader you haven't checked recommendation

Early-2025 models

Apple preprint, The Illusion of Thinking, on early-2025 models.

Read the original

Used in

• You can't tell from a task how well the agent will do challenge

Small: 18 functions, two models of one family, two smells; part of the gap is confounded

Behavioural guidance gave about 83 percent genuine fixes on both models, against 29 and 6 percent for a bare linter message; one model did worse with the bare metric than with no guidance.

Read the original

Used in

• Add fix-it guidance to lint messages trial

Workshop paper; one company, one model family

An agent that first wrote each unit's pre- and post-conditions caught 63.2% of 90 historical bugs within five runs against 53.4% for a plain test-writing agent (p=0.035, not significant at one or two runs), for 38% more tokens; most misses were conditions the spec left out.

Read the original

Used in

• Write a test list before the first test trial
• Have the agent write the contract before the tests trial

Preprint; Java only

Within the same 391 Java repositories, agent PRs changed tests more often than human PRs (55.0% vs 45.8%), added tests more and modified or deleted existing tests less; pooled across repositories the gap vanished or reversed, and added test methods looked alike, mocks included.

Read the original

Used in

• Work in small batches with strong version control recommendation
• Ask agents to update affected tests, not only add new ones trial
• State when to mock in the agent's context file trial
• Compare agent and human work inside each repository trial

Across 2,168 repositories with agent configuration files, 36% of agent test commits added a mock against 26% of other test commits (small effect); many CLAUDE.md files mention tests but few say how to mock.

Read the original

Used in

• State when to mock in the agent's context file trial

Preprint; unit-test files only

In 39 open-source agent frameworks and 439 agent applications, tests target tools and workflows; the model gets under 5% of tests, prompts about 1%, and LLM-judge checks appear in about 1% of application tests.

Read the original

Used in

• Keep a prompt regression suite for LLM features trial

Pre-2025 models · Students, short lab tasks

Students brainstorming tests with a chatbot took the first 8-10 suggestions and stopped, with no gain over web search; a system that proposed one new test at a time from the editor's context, accepted or rejected each time, raised branch coverage of the combined human and AI tests by about a third.

Read the original

Used in

• Write a test list before the first test trial
• Ask the AI for one new test idea at a time trial

Pre-2025 models · 12 students

Students writing unit tests with ChatGPT split the work by who supplies the test idea and who writes the code; heavy delegators used one-shot prompts and had to fix wrong expected values by hand, and students reported lower quality, trust and ownership of generated tests.

Read the original

Used in

• Ask the AI for one new test idea at a time trial

Preprint

LLM-written suites at full statement, branch or mutation adequacy triggered about a third of 6,066 hard bugs in LLM-written code but their assertions flagged only 1-2%; the three criteria barely differed, and assertions rewritten from the task description lifted detection by about 7 points.

Read the original

Used in

• Keep a human upstream of what tests assert recommendation
• Don't read coverage as quality for AI-written tests recommendation
• No automated check tells you an assertion is right challenge

Preprint; no control condition

After one-line behaviour changes to small programs, about 99% of regenerated tests that failed still asserted what the unchanged program did, and pass rates also fell after behaviour-preserving edits.

Read the original

Used in

• Keep a human upstream of what tests assert recommendation
• No automated check tells you an assertion is right challenge

Pre-2025 models · Students; same task done twice

Students' own TDD code passed every reference test in 15 of 35 submissions against 12 when redone with ChatGPT; small iterative prompts and fixing by hand when the model stalled went with success, pasting the whole spec and refusing to touch the model's code with failure.

Read the original

Used in

• Stop after about 3 failed fixes, then reset or escalate recommendation
• Work in short, steady test-and-code cycles recommendation
• Test-first results may not hold on current models challenge

Pre-2025 models · 16 participants; newer models only in the agent arm

With people writing tests and GPT-3.5 writing code, TDD code was no more correct than people working alone (60.8% vs 62.0% of baseline tests); an agent platform on newer models was most correct and fastest but added code no spec asked for, which tests covered less.

Read the original

Used in

• Separate test-writing from code-writing trial
• Test-first results may not hold on current models challenge

Coding-agent traces appeared in 22.20% (conservative) to 28.66% (high estimate) of 128,018 active GitHub repositories by February 2026; agent-assisted commits had a median of 31 added lines against 11 for human commits.

Read the original

Used in

• Work in small batches with strong version control recommendation

Pre-2025 models · 19 students; completion-only assistant

Human pairs had about twice as many knowledge-transfer exchanges as people working alone with an assistant, over more topics; alone with the assistant, 28% of knowledge was accepted on trust against 12% in pairs.

Read the original

Used in

• Compare solo-AI, human-pair-with-AI and human-only pairing trial
• Pairing with AI isn't the same as pairing with a person challenge

Preprint, auto-run assumed, May 2025 tool versions, one run per test

A poisoned coding rule file made Cursor and GitHub Copilot with auto-run execute attacker-intended commands in up to 84% of cases (41% to 83% by editor and model on one scenario); how the developer worded the task made little difference, while how the payload was framed decided success.

Read the original

Used in

• Write context files by hand and keep them short recommendation
• Review rules, skills and context files that come from outside the team trial
• Nobody knows how often agents leak secrets challenge
• Prompt injection has no proven fix challenge

Exfiltration only, no human-approval modes, one run per cell

In 1,920 runs in poisoned repositories, a planted secret reached an attacker server in 45.5% of runs when the agent was asked to run tests and 8.6% when asked to fix a bug; agents warned least on the test task; security rules raised warnings but not the share of attacks that failed; Claude Code with Sonnet 4.6 was the most resistant of four agent-model setups.

Read the original

Used in

• Enforce access and approvals outside the model recommendation
• Don't treat a stronger model as a security control recommendation
• Read test and setup files before an agent runs them trial
• Review rules, skills and context files that come from outside the team trial
• Nobody knows how often agents leak secrets challenge
• Prompt injection has no proven fix challenge

Pre-2025 models

Failed fixes piled up: no problem was solved on attempt 5 (GPT-4, Llama 3).

Read the original

Used in

• Stop after about 3 failed fixes, then reset or escalate recommendation
• Keep a human upstream of what tests assert recommendation
• Give the model few, distinct tests recommendation
• Don't read coverage as quality for AI-written tests recommendation
• Re-test the supplied-tests benefit on current models trial
• Test loops fix the code, not a misread requirement challenge
• Nobody knows how much to specify up front challenge

Pre-2025 models · Preprint

The same debugging loop gained 2-3 points with no unit tests (text-to-SQL) and up to 12 points with them (C++ to Python, Codex 80.4% to 92.5%), a bare 'the code is wrong' message gained less than execution results wherever tests existed, and it matched baselines that drew 10 times more samples.

Read the original

Used in

• Loop only against a check the machine runs recommendation
• Compare a capped retry loop with a stronger model at equal cost trial

Pre-2025 models · Preprint

Self-feedback improved 7 unlike tasks by about 20% absolute on average, but on math ChatGPT judged 94% of its own answers 'everything looks good' and an outside wrong-answer signal added over 5 points; partly GPT-4-judged, no equal-cost baseline.

Read the original

Used in

• Loop only against a check the machine runs recommendation
• Don't loop on self-critique alone recommendation

Pre-2025 models · Preprint

With no tests, a reflection loop scored below no loop at all (0.52 vs 0.60), a small open model gained nothing (0.26 before and after), and no equal-cost baseline was reported.

Read the original

Used in

• Loop only against a check the machine runs recommendation
• Don't loop on self-critique alone recommendation
• Keep a human upstream of what tests assert recommendation
• Re-test the supplied-tests benefit on current models trial
• Don't route retry loops to cheap models unchecked recommendation

Pre-2025 models

Failed runs cost twice as much as successful ones and more budget rarely rescued them, while collapsing old observations and adding a lint guardrail each lifted solve rate from 15% to 18% (GPT-4 Turbo).

Read the original

Used in

• Stop after about 3 failed fixes, then reset or escalate recommendation
• Keep each agent's context small recommendation
• Trim tools and what they return recommendation
• Don't give agents tools shaped for people recommendation

Preprint

The best result on 112 long tasks was 25.0% resolved (Opus 4.7 in Claude Code with an outer continuation loop, 17.0% without), regressions occurred in all four loop profiles, and the same model (gpt-5.4) scored from 18.75% in Codex down to 7.14% in mini-swe-agent, with the Ralph loop at 7.84% against 24.11% for dynamic workflows; Microsoft authors, tasks built with Claude Code.

Read the original

Used in

• Compare fresh and continued context per iteration trial
• Keep a growing regression list in long-running loops trial
• Cross-score agents on task sets each one built trial

A verifier trained on agent runs raised SWE-bench Verified from 20.6% (one attempt) to 32.0% (best of 16), but the right patch was often produced and not picked (fine-tuned Qwen2.5-Coder-32B, late-2024 open model).

Read the original

Used in

• Pick among several agent attempts trial

Pre-2025 models

Tree search added 4.0 to 6.7 points on SWE-bench Lite at 5 to 14 times the cost but only 0.6 to 5.3 points at matched compute, and a value function picked the right solution 73% of the time against 84% for a debate between final candidates (GPT-4o, GPT-4o-mini, Llama 3.1 70B).

Read the original

Used in

• Compare a capped retry loop with a stronger model at equal cost trial
• Pick among several agent attempts trial

Preprint

Code quality decays as agents extend their own work, and quality-aware prompts change the starting point but not the rate.

Read the original

Used in

• Don't treat a passing suite or a green loop as done recommendation
• Don't choose tools by public leaderboard rankings recommendation
• Don't leave structural refactoring to agents recommendation
• Code erodes as agents build on their own work challenge

Pre-2025 models

31.1% of passed SWE-bench patches were suspicious because the tests were weak (reported secondhand via TDD-Bench Verified).

Read the original

Used in

• Grade agents on tests they never saw recommendation

Pre-2025 models

Cited for weak graders: it relays the SWE-Bench+ figure, and the same row reports that 44 of 500 (8.8%) SWE-bench Verified golden tests weren't fail-to-pass or covered no changed lines.

Read the original

Used in

• Grade agents on tests they never saw recommendation
• Keep a human upstream of what tests assert recommendation
• Don't read coverage as quality for AI-written tests recommendation
• Re-test the supplied-tests benefit on current models trial
• Test-first results may not hold on current models challenge

Early-2025 models

Almost all code failures in this Lean 4 benchmark were compile errors (o3: 25.8 of 27.4 points), so its code score mostly measures writing Lean that compiles.

Read the original

Used in

• Keep a human upstream of what tests assert recommendation

Preprint

LLM-generated context files lowered success slightly and raised cost 20-23%, hand-written ones gave +2.4% over no file (p=0.21) with cost up to 19%, repo overviews did not help agents find files, and tools named in a context file were used 1.6 times per task against under 0.01 otherwise, with reasoning tokens up 10-22% (GPT-5.x and Sonnet 4.5).

Read the original

Used in

• Write context files by hand and keep them short recommendation
• Don't let an LLM generate your context files recommendation
• Restart the session when output degrades trial
• Write a handoff note at each session boundary trial
• Enforce standards with checks, not documents recommendation
• Every line in the context file costs money challenge

Preprint

The best model scored 55.2% on acting on updated facts.

Read the original

Used in

• Agents act on outdated facts challenge

Early-2025 models

Isolating untrusted data from planning and checking each tool call against a policy blocked nearly all AgentDojo attacks, solving 77% of tasks vs 84% undefended at about 2.8x the tokens; spotlighting alone cut Claude 3.5 Sonnet's successful attacks from 44 to 5 of 949, with no guarantee and no adaptive attack tested (preprint, Google authors who also built the benchmark).

Read the original

Used in

• Enforce access and approvals outside the model recommendation
• Don't rely on delimiting untrusted content as the defence recommendation
• Separate planning from untrusted data in one agent trial
• Prompt injection has no proven fix challenge

Poisoned tool descriptions succeeded on 36.5% of test cases on average and up to 72.8% across 20 model settings, with refusals under 3%; reasoning mode raised attack success (Qwen3 32B: 58.5% vs 23.7%). Measures susceptibility, not real-world incidence, and tested no defences.

Read the original

Used in

• Allowlist MCP servers and enforce access at the tool layer recommendation
• Don't treat a stronger model as a security control recommendation
• Don't install third-party MCP servers unreviewed recommendation

Most models avoided regressions under 25% of the time.

Read the original

Used in

• Work in small batches with strong version control recommendation
• Don't treat a passing suite or a green loop as done recommendation
• Don't choose tools by public leaderboard rankings recommendation
• Code erodes as agents build on their own work challenge

Preprint, 105 C/C++ memory-safety tasks, static-analysis false alarms near 50%

On 105 real C/C++ tasks the best of 13 agent-model combinations produced 23.8% patches that were both correct and secure; in most combinations over half of correct patches still carried a security issue; two CLI agents ranked top on correctness but not on secure correctness.

Read the original

Used in

• Treat agent output as untrusted input recommendation
• Don't treat a stronger model as a security control recommendation
• Replay past vulnerabilities as tests on agent patches trial

45 samples, community-contributed, results table predates a scorer fix

On 35 injected bug-fixing tasks, injection resistance ranged from 68.6% to 97.1% across five models, newer models resisted more, and explicit detection of an injection happened about once in 140 injected samples.

Read the original

Used in

• Prompt injection has no proven fix challenge

Vendor telemetry

Cost per session fell 52% with compaction, default effort levels, caching and spend nudges.

Read the original

Used in

• Manage cost with visibility and friction trial
• Rerun past tickets before each model upgrade trial
• Show cost per merged change next to total spend trial
• Trim tools and what they return recommendation
• Set rate limits, quotas, timeouts and iteration caps recommendation

Per an internal CTO post reported secondhand, code changes at Meta rose 220% while user-facing feature changes rose 36%.

Read the original

Used in

• Measure time to an accepted change, not lines or PRs recommendation

One model cost about 2x another per agent request but tied per accepted line; Orosz notes Cursor gains if its own model looks cheap.

Read the original

Used in

• Normalise cost by accepted output trial
• Grant agents the fewest tools and the minimum scopes recommendation
• Don't let agents commit without review recommendation

Honeypot telemetry saw a PHP2Shell exploit attempt 4 days before disclosure, proof-of-concept exploits about 12 hours before the embargo lifted, and several actors within hours of the patch (AWS sells detection and WAF products).

Read the original

Used in

• Measure patch-to-deploy time in hours trial

2026, n=349

Median value uplift was 1.4-2x against 3x speed, and every checkable 10x claim was overstated.

Read the original

Used in

• Time real tasks instead of trusting forecasts or surveys recommendation

Delivery instability rises with AI adoption, while small batches amplified product performance and reduced friction (survey of about 5,000).

Read the original

Used in

• Work in small batches with strong version control recommendation
• Measure time to an accepted change, not lines or PRs recommendation
• Time real tasks instead of trusting forecasts or surveys recommendation
• Pair juniors with seniors on agent-assisted work trial
• Keep user focus recommendation
• Strong engineering foundations may not protect you challenge

WSJ survey; Liebscher press account

Reports secondhand a WSJ survey of 5,000 in which 92% of executives say AI makes them more productive and 40% of non-managers say it saves no time; its account of the Liebscher study gives 1,150 workers and 40% exposure, a discrepancy the doc calls unexplained.

Read the original

Used in

• Time real tasks instead of trusting forecasts or surveys recommendation

In a self-report preprint (n=962, BetterUp and Stanford authors), 52.7% said they had sent AI work they judged low quality; 38% received some in the last month and spent 3.3-3.4 hours a month on it; 36% then wanted to avoid that colleague.

Read the original

Used in

• Declare effort and wanted review on AI-drafted artifacts trial

Executives report about 31% time savings from coding assistance (executive self-report, n=247).

Read the original

Used in

• Randomise real tickets to test speed gains trial

Engineers fully delegate 0-20% of their work.

Read the original

Used in

• Keep people hands-on with AI-off sessions trial
• Pair juniors with seniors on agent-assisted work trial
• Supervising agents needs the skills agents erode challenge
• Juniors ask the model instead of colleagues challenge

Vendor-internal

Routing cut task cost by over 30%.

Read the original

Used in

• Manage cost with visibility and friction trial
• Rerun past tickets before each model upgrade trial
• Set rate limits, quotas, timeouts and iteration caps recommendation

On one Java service scored by 178 hidden acceptance tests, a test-feedback harness took a cost-optimised model from 3 of 10 to 10 of 10 accepted at 3.9 times the tokens, while a flagship model was 10 of 10 either way at 10.5 times the tokens with the harness; the author founded the company whose tool built the harness.

Read the original

Used in

• Run a strong model's checks once as a final gate trial

Proposes a JSON feature list where only the passes field is editable to stop premature completion claims.

Read the original

Used in

• Keep a machine-readable done list trial

Uber bundles simple CI jobs into one command the agent runs before opening a PR, after CI jobs more than doubled in a year; self-report.

Read the original

Used in

• Run checks inside the agent session trial
• Track feature cycle time by PR type trial
• Compare spend tiers with quick approval against hard caps trial
• Ground agents in a company context graph trial
• Call CLIs from skills instead of MCP tool lists trial
• Send risky PRs to people, the rest to AI review trial

The Claude Code team has the agent show test or build output, then a fresh-context subagent reviews the diff before any PR; the 'two to three times better' figure is one engineer's estimate.

Read the original

Used in

• Run checks inside the agent session trial
• Send risky PRs to people, the rest to AI review trial
• Cap AI review at three rounds trial
• Score the AI reviewer on past incidents trial
• Test on-demand explainers against reading diffs trial

A cheap model consulting a strong advisor gained 2.7 points on SWE-bench Multilingual at 11.9% less cost per task.

Read the original

Used in

• Let a cheaper model execute a strong model's plan trial

A flaky-test agent checking every 3 hours filed 72 reports since July, 11 of which (about 15%) led to merged fixes, while agent-sent tool-feedback reports ran to a few thousand, too many to read; vendor self-report.

Read the original

Used in

• Have scheduled agents file reports, not PRs trial

Pre-2025 models

A before/after study (n=50) by a vendor of engineering metrics found 10.6% more PRs with Copilot, with no quality data.

Read the original

Used in

• Measure time to an accepted change, not lines or PRs recommendation

Pre-2025 models

Unsigned vendor post with no paper: 13.6% more lines per readability error with Copilot, reviewed only by the 25 developers whose code passed every test.

Read the original

Used in

• Don't decide on vendor-reported figures recommendation

With no methodology given, a daily bill doubled ($5k to $10k) while cost per run fell ($1.80 to $1.40), and a cheaper model was 20% cheaper in one test, not the 2.5x its list price suggests, because it took 1.7x the turns.

Read the original

Used in

• Normalise cost by accepted output trial
• Show cost per merged change next to total spend trial
• Trim tools and what they return recommendation

Vendor data shows weekly PRs up 113% and monthly PRs up 127% year on year among CodeRabbit users; interviews across 27 companies found spend from under $300 to $3,000+ per developer per month and PR change from flat to 5-10x, with one company's PR size tripling while count stayed flat.

Read the original

Used in

• Track feature cycle time by PR type trial
• Compare spend tiers with quick approval against hard caps trial

Claims boolean LLM judges with concrete examples are reliable enough, with no agreement data, and recommends observing, curating failures into evals, sampling 5-10% of traffic and reviewing traces weekly.

Read the original

Used in

• Check LLM judges against human labels trial
• Run a production-trace review loop trial

OpenAI's own working paper on ChatGPT Enterprise says message volume measures intensity, not economic importance.

Read the original

Used in

• Don't rank people by AI usage recommendation

Advises planning before code and committing the plan, and says a skill makes violations rare while a hook makes them close to impossible; untested.

Read the original

Used in

• Commit a plan before code trial
• Enforce the most-violated rules with hooks trial
• Write a review policy trial
• Grant agents the fewest tools and the minimum scopes recommendation
• Deny secrets to agents and force the sandbox recommendation
• Tier agent autonomy by alert severity trial
• Nobody knows how often agents leak secrets challenge

Load skill bodies only when they are used.

Read the original

Used in

• Move repeated procedures into on-demand skills trial

Advises splitting unknowns from assumptions in specs; vendor-sponsored, no data.

Read the original

Used in

• Split unknowns from assumptions in specs trial

Advises starting a new conversation when compaction fires every few messages or earlier rules are forgotten, carrying over a short hand-off summary and keeping standing rules in CLAUDE.md; sponsored, no data.

Read the original

Used in

• Restart the session when output degrades trial

One big instruction file failed, so the team kept a short AGENTS.md as a table of contents over structured docs and enforced rules with custom linters; unmeasured, output figures are self-estimates.

Read the original

Used in

• Make the context file a short index over a docs directory trial

Correlational vendor data shows median review time up 441%, incidents per PR up 243% and unreviewed merges up 31%.

Read the original

Used in

• Work in small batches with strong version control recommendation
• Track defects and rework to catch quality erosion trial
• Strong engineering foundations may not protect you challenge

Relays CodeRabbit data on 470 PRs (Dec 2025) showing 1.7x more issues and 1.5-2x more security issues in AI PRs, and an unnamed study where 93.4% of flagged locations across 146 PRs came from exactly one of four tools (secondhand).

Read the original

Used in

• Add an intake gate to the agent PR template trial
• Run two AI reviewers in parallel trial
• Give security-sensitive changes a human review recommendation
• Compare defects by provenance trial

Main-branch success fell to 70.8%, a five-year low (no methodology given).

Read the original

Used in

• Work in small batches with strong version control recommendation

Claude proved 239 OSS-Fuzz entry points crash-free with specs from existing fuzz harnesses, yet one harness asserted less than the post claims.

Read the original

Used in

• Review a checked spec instead of the code trial
• Flag weakened tests in agent PRs trial

OpenAI says its Codex reviewer caught more critical issues with fewer false alarms once given repository access and execution, that 52.7% of its comments led to a code change across over 100k PRs a day, and that recall fell faster on model-written code (figures not reproduced).

Read the original

Used in

• Give the AI reviewer repo access and test execution trial

A squatted fake package name got 30,000+ downloads in three months (Lasso), and one spread to 237 repos through LLM-generated agent skills.

Read the original

Used in

• Verify a package's publisher and registration date recommendation

A wrapper around npm, npx, yarn and pnpm that checks installs against a malware feed.

Read the original

Used in

• Intercept package installs against a malware feed trial

Says its model found the flaws before attackers, 'to our knowledge'; the AWS telemetry undercuts that head start.

Read the original

Used in

• Measure patch-to-deploy time in hours trial

Anthropic's scanning had disclosed 1,596 vulnerabilities, 97 known patched, as of 2026-05-22; one team's 40% false-positive rate came from findings with working proofs of concept that owners judged outside their threat model.

Read the original

Used in

• Give AI vulnerability scans a written threat model trial
• Have a second model in a fresh context write tests trial

GitHub runs CodeQL, advisory-database checks on new dependencies and secret scanning on third-party agent code by default, and claims hundreds of leaks and vulnerabilities prevented, unquantified.

Read the original

Used in

• Log what platform security checks catch on agent PRs trial

In a vendor preprint, a shared memory raised LLM-judged code quality most for small models, 4.23 to 4.89 on a 5-point scale; the memory was seeded from the benchmark's own reference answers and no execution tests were reported.

Read the original

Used in

• Pilot a shared "what worked" memory against a baseline trial

Letta says an agent using a plain filesystem scored 74.0% on the LoCoMo chat-recall benchmark with GPT-4o mini, against 68.5% reported for Mem0's graph variant.

Read the original

Used in

• Compare a file-based memory with a memory service trial

Mem0's own preprint scores full context at 72.90 and Mem0 at 66.88 on a judge metric; the benchmark tests conversation recall, not coding.

Read the original

Used in

• Compare a file-based memory with a memory service trial

Vendor aggregator of AI hallucination rates, secondhand.

Read the original

Used in

• Don't decide on vendor-reported figures recommendation

The maker's own tool, no measurement

Argues agent memory should be a maintained Markdown brain that the agent reads before and updates after each task, not snippets retrieved from a vector store; he still trims it by hand.

Read the original

Used in

• Compare a file-based memory with a memory service trial
• Agents act on outdated facts challenge

Vendor's own product concept, single runs, small question sets

A second pass that checked memory notes against session logs corrected a wrong note and changed one run's outcome; a linked knowledge base beat search alone only when search was kept beside it (61% against 53% of graded criteria, on its own 48%), and one grew larger than its source documents; several agents writing one shared page lost 16 of 57 facts until each had its own page; telling models to record only failures cut their memory gains.

Read the original

Used in

• Groom memory files monthly trial
• Check memory notes against session logs trial
• Keep search beside a linked knowledge base trial
• Give each agent its own page in shared notes trial
• Agents act on outdated facts challenge

Vendor-supplied, sells agent identity security

A browser agent inside a user's signed-in session has the user's full access, and the app cannot tell its actions from the user's; the vendor says agent lanes must carry the agent's identity, a verifiable link to the person, and a scope narrower than the person's, and that current drafts each carry only one.

Read the original

Used in

• Limit browser agents to the sites a task needs trial
• Apps can't tell an agent from the person it acts for challenge

Chrome's auto browse agent uses the user's signed-in sites, is limited to sites relevant to the task, asks before sensitive sites and consequential actions, and marks pages it visited in history; nothing tells the visited site an agent is driving.

Read the original

Used in

• Limit browser agents to the sites a task needs trial

Limited preview

Edge for Business agentic browsing is off unless enabled by policy, runs only on sites IT designates, pauses before passwords and card numbers, and keeps signed-in sites signed in through the user's cookies.

Read the original

Used in

• Limit browser agents to the sites a task needs trial

Permission-bypass flags leave the sandbox as the only boundary.

Read the original

Used in

• Sandbox anything unattended recommendation
• Deny secrets to agents and force the sandbox recommendation
• Nobody knows how often agents leak secrets challenge

Proposes stopping after each test when unsure, running several tasks when sure, and keeping auth and payments in low gear; untested.

Read the original

Used in

• Check the agent more often on risky code trial

Heuristics that agents hold up for 3-10 steps and drift past 20, and that a loop earns unattended running with a cheap, frequent, hard-to-fake check.

Read the original

Used in

• Log steps to the first human correction trial
• Run low-risk tickets unattended overnight trial

Advises resetting context every loop and using a single validation agent for cleaner signal.

Read the original

Used in

• Compare fresh and continued context per iteration trial
• Run checks inside the agent session trial

Continuing context did better per unit cost than resetting (n=1 task).

Read the original

Used in

• Compare fresh and continued context per iteration trial
• Test whether a hidden test set changes agent behaviour trial
• Fresh or continued context in agent loops is unsettled challenge

Catalog patterns Overnight Batch and Smart Plan, Cheap Execution propose running low-risk work overnight and having the strongest model plan while a cheaper one executes; no data.

Read the original

Used in

• Run low-risk tickets unattended overnight trial
• Let a cheaper model execute a strong model's plan trial
• Commit a plan before code trial
• Add a focused commit or review subagent trial
• Groom memory files monthly trial
• Add a candour ground rule trial
• Have agents write scripts for deterministic work trial
• Slice big agent branches for review trial
• Tier agent autonomy by alert severity trial
• Keep an ADR per agent-built feature trial
• Have the agent restate the plan and object before coding trial
• Prototype requests cheaply for a user cohort trial

A no-review factory was abandoned after about three unrecoverable failures (one company).

Read the original

Used in

• Align on product, architecture and design before code trial
• Review plans instead of diffs trial
• Prototype requests cheaply for a user cohort trial

Proposes sensors at four cadences: in-session, CI, scheduled and production.

Read the original

Used in

• Run checks inside the agent session trial
• Run diff-scoped mutation testing as a CI warning trial
• Add fix-it guidance to lint messages trial
• No automated check tells you an assertion is right challenge

The bottleneck moves to task definition (n=1).

Read the original

Used in

• Time-track agent work by phase trial
• Name AI-enabled deliverables in advance trial
• Nobody agrees what the human bottleneck is now challenge

Agents were not net useful for novel, precisely arranged code, pushing standard patterns, defensive bloat and deprecated APIs, but helped with boilerplate and a tested port to an unfamiliar language (n=1).

Read the original

Used in

• Track agent outcomes on novel versus boilerplate work trial
• Try autocomplete before agents for novel code trial
• Don't trust an AI grader you haven't checked recommendation

A locally run 35B mixture-of-experts model handled scripts and small well-specified changes but fell apart on complex logic, and one task failed 5 of 7 automated reruns on one machine (two tasks, one evaluator).

Read the original

Used in

• Try a local mid-size model on small pre-planned tasks trial

Reports secondhand that Amazon withdrew an internal usage leaderboard after staff gamed it.

Read the original

Used in

• Don't rank people by AI usage recommendation
• Compare token spend with merged output before setting usage targets trial

Anonymous testimony that engineers under quotas ran self-prompting loops and none was caught.

Read the original

Used in

• Don't rank people by AI usage recommendation
• Compare token spend with merged output before setting usage targets trial

Reports that Duolingo dropped AI usage from performance reviews.

Read the original

Used in

• Compare token spend with merged output before setting usage targets trial

A CTO describes an unmeasured company policy: say what an artifact is, why you send it and how much review you want.

Read the original

Used in

• Declare effort and wanted review on AI-drafted artifacts trial

One engineer reports about 3x merged PRs after a model upgrade, with method and baseline not shown (personal essay, n=1).

Read the original

Used in

• Track feature cycle time by PR type trial
• Set how much agent code is read by risk tolerance trial

Rules that fight the model's training silently revert.

Read the original

Used in

• Enforce the most-violated rules with hooks trial

A spec tool turned a small bug into 4 user stories and 16 acceptance criteria, and reviewing many verbose spec files was worse than reviewing code (one evaluator, 2025).

Read the original

Used in

• Scale the spec workflow to the size of the problem trial

GitHub PR volume rose about 5x in three years, and a 5-person team routing risky changes to humans and the rest to AI review went from 80 to 154 PRs per week (self-reported).

Read the original

Used in

• Send risky PRs to people, the rest to AI review trial
• Review plans instead of diffs trial
• Review load lands on senior engineers challenge
• Prompt injection has no proven fix challenge

Uber filters AI review comments by confidence and merges duplicates.

Read the original

Used in

• Show only high-confidence AI review comments trial

Recommends phasing deterministic gates: baseline, then warnings on changed files, then blocking.

Read the original

Used in

• Phase in deterministic gates trial
• Run diff-scoped mutation testing as a CI warning trial
• No automated check tells you an assertion is right challenge

Bug fixes get light inspection and novel features deeper structural review, never full line-by-line reading; each review finding becomes an agent skill (his company sells code-health tooling).

Read the original

Used in

• Match review depth to task uncertainty trial
• Turn repeated review findings into agent skills trial

OWASP says it is unclear if there are fool-proof methods of prevention; RAG and fine-tuning don't fully mitigate it, and detecting injected text is an open problem.

Read the original

Used in

• Enforce access and approvals outside the model recommendation
• Don't rely on delimiting untrusted content as the defence recommendation

A malicious MCP server's tool responses carry instructions the agent follows; tools are reviewed at connect time but responses are trusted at runtime (no incidence data).

Read the original

Used in

• Allowlist MCP servers and enforce access at the tool layer recommendation
• Test the harness against a poisoned MCP server trial

The OWASP category the tab maps the lethal trifecta and credential exposure to.

Read the original

Used in

• Remove one leg of the lethal trifecta recommendation
• Deny secrets to agents and force the sandbox recommendation

Root causes are excessive functionality, permissions and autonomy; logging and rate limits limit damage but don't prevent it.

Read the original

Used in

• Grant agents the fewest tools and the minimum scopes recommendation

System prompts should be treated as extractable.

Read the original

Used in

• Deny secrets to agents and force the sandbox recommendation

Embeddings can be inverted to recover text, shared stores leak across permission boundaries, and indexed documents can carry hidden instructions.

Read the original

Used in

• Partition retrieval stores and make retrieval permission-aware recommendation

Tampered models, LoRA adapters and poisoned data; backdoors can lie dormant.

Read the original

Used in

• Inventory, sign and pin models and packages recommendation

The OWASP category, including overreliance, that the tab maps slopsquatting, insecure code and skill loss to.

Read the original

Used in

• Verify a package's publisher and registration date recommendation
• Give security-sensitive changes a human review recommendation
• Keep people hands-on recommendation

Covers tampered models and packages; vendors can change terms to train on customer data.

Read the original

Used in

• Verify a package's publisher and registration date recommendation
• Inventory, sign and pin models and packages recommendation
• Sign a data processing agreement before tools see company data recommendation

The web Top 10 category the tab cites for agent-picked dependencies.

Read the original

Used in

• Inventory, sign and pin models and packages recommendation

Recommends SBOMs, signed sources, pinned registries and egress controls on fetches.

Read the original

Used in

• Inventory, sign and pin models and packages recommendation

Model output is passed to shells, SQL or browsers without validation; a coding agent's shell tool does this by design.

Read the original

Used in

• Treat agent output as untrusted input recommendation

Pay-per-use pricing can be exploited; OWASP favours rate limits, quotas, timeouts and iteration caps.

Read the original

Used in

• Set rate limits, quotas, timeouts and iteration caps recommendation

An agent (Sonnet 4.6) running TDD alone on a clear spec reached the same correctness and mutation score at 3x to 8.5x the tokens, and in every TDD run skipped RED, over-implemented or wrote tautological tests (5 batches, LLM judge, small experiment).

Read the original

Used in

• Compare agent-run TDD with spec tests on a clear spec trial
• Try agent TDD on real, vague tickets trial
• Nobody knows how much to specify up front challenge

Across 18 tasks, measured cost showed a 'cost advantage of test-after' of -9%, -55% and -8% by task size, baseline not stated.

Read the original

Used in

• Compare agent-run TDD with spec tests on a clear spec trial
• Try agent TDD on real, vague tickets trial

Testimony that agent-drafted, human-reviewed end-to-end tests are where the bulk of agentic gains came from, and that agents deleted tests or changed them to match wrong code.

Read the original

Used in

• Compare who writes acceptance tests trial
• Compare acceptance-level and unit-level specs trial
• Flag weakened tests in agent PRs trial

Recommends acceptance tests through in-memory adapters, converging independently on the approved-scenarios idea.

Read the original

Used in

• Compare who writes acceptance tests trial
• Compare acceptance-level and unit-level specs trial
• Try hard-to-fake test formats in one module trial
• Try acceptance tests as the main guardrail trial

Calls TDD a 'superpower' against agent regressions, reports agents deleting or changing tests, and recalls a Facebook feature where TDD was a waste of time because the bugs that hurt were cross-subsystem (anecdote).

Read the original

Used in

• Compare acceptance-level and unit-level specs trial
• Try human-enforced TDD on one critical module trial
• Flag weakened tests in agent PRs trial
• Try flags and staged rollout in place of unit tests on one service trial

Human-enforced TDD with an agent took two failed attempts to calibrate.

Read the original

Used in

• Try human-enforced TDD on one critical module trial

Proposes approved scenarios and constrained test DSLs as hard-to-fake test formats.

Read the original

Used in

• Try hard-to-fake test formats in one module trial

A blog with a demo, unmeasured, describes a design that blocks a conflicting memory write until a human decides.

Read the original

Used in

• Compare a human review queue with newest-write-wins for memory conflicts trial
• Shared agent memory behaves like a shared database challenge

Proposes a 5-question quiz on a change before shipping it.

Read the original

Used in

• Quiz authors on agent diffs trial
• Nobody agrees what the human bottleneck is now challenge

Proposes a comprehension commit gate for juniors.

Read the original

Used in

• Quiz authors on agent diffs trial

Cited for writing the message yourself as the ownership check.

Read the original

Used in

• Hand-write commit messages and why-comments trial

Argues pairing keeps intent in at least two heads.

Read the original

Used in

• Compare mobbing with an agent against solo work plus review trial
• Spike throwaway designs before committing trial

Argues pairing keeps intent in at least two heads.

Read the original

Used in

• Compare mobbing with an agent against solo work plus review trial

A growing note built on one anonymous account argues the bigger risk is whole teams no longer knowing the architecture or why choices were made.

Read the original

Used in

• Run a quarterly "explain this service" check trial

On one bug fix (n=1), reading the diff, running the tests and questioning the agent was judging, not understanding; only re-implementing the fix gave understanding.

Read the original

Used in

• Re-implement business-critical agent fixes by hand trial

Claims, without evidence, that a ladder of output formats from controlled English to diagrams, HTML pages and explainer videos makes oversight easier; the author is a former AI-education founder, an interest the post does not disclose.

Read the original

Used in

• Test whether diagrams or generated pages help understanding trial

Splitting a large file cut tokens per change 83%; the agent couldn't choose refactorings unaided (n=1).

Read the original

Used in

• Split the largest files with human-chosen refactorings trial

Stated goals, not measured results

Sets a target of 25% of PRs reviewed by AI and auto-merged without human review by the end of 2026, at a change failure rate under 3%, and admits a related goal is already off track.

Read the original

Used in

• Review load lands on senior engineers challenge

Self-reported; the productivity chart does not separate the model change from other causes

A bank reports that strict budget limits made engineers ration tokens, and that self-hosting an open model for about 2,000 engineers cut its invoice to about a tenth and let them stop rationing; it gives no quality or full-cost figures.

Read the original

Used in

• Compare spend tiers with quick approval against hard caps trial
• Compare a self-hosted open model with hosted ones on your own tickets trial

One coach's experience, no measurement

Agents break mob programming's timed rotation, intention-only talking and quiet-or-chaotic screen watching; she uses silent shared prompt writing, running commentary, reading the summary aloud and a retro that names the patterns used.

Read the original

Used in

• Compare mobbing with an agent against solo work plus review trial
• Write prompts silently together before a mob sends them trial
• Name the prompting patterns in a short retro trial
• Pairing with AI isn't the same as pairing with a person challenge

Two runs per condition; analysis written with an LLM

The test-first prompt had no test-list step and told the agent not to design up front; one improved prompt produced both the best and the fourth-best of four runs.

Read the original

Used in

• Compare agent-run TDD with spec tests on a clear spec trial
• Try agent TDD on real, vague tickets trial
• Write a test list before the first test trial
• Nobody knows how much to specify up front challenge

The tool author's own account

Agents split a function only to satisfy a length limit; a hook that turns the linter's output into a short prompt saying what the smell means and how to fix it gave better refactors in the demo.

Read the original

Used in

• Enforce the most-violated rules with hooks trial
• Add fix-it guidance to lint messages trial

The tool author's self-report, no measured outcome

Telling an agent to follow TDD failed because it skipped steps and over-implemented, so he blocks violations with hooks outside the prompt; he still reads every change.

Read the original

Used in

• Enforce the most-violated rules with hooks trial

Tutorial from the standard's own project

Shows coding assistants exporting OpenTelemetry traces with model, token counts, latency and tool calls; prompt content is captured only when switched on.

Read the original

Used in

• Run a production-trace review loop trial
• Export agent telemetry to your own tracing backend trial

Standards roadmap

Describes OpenTelemetry's draft conventions for agent telemetry and argues traces should feed evaluation as well as monitoring.

Read the original

Used in

• Run a production-trace review loop trial
• Export agent telemetry to your own tracing backend trial

One internal exercise, participants' own estimates

In a week where agents wrote all the code, participants estimated about two thirds of their time went on the whiteboard and one third on the harness, yet the build was often red; the author argues that built is not the same as working in production.

Read the original

Used in

• Fix the harness before the code trial
• Count how long main stays red when agents push trial
• Nobody agrees what the human bottleneck is now challenge
• A constantly red build stops meaning anything challenge

One exercise, accidental and not reproduced

With agents sharing a numbered spec and keeping plans in the repo, they began using the plans to claim work and hand off notes; frequent pushes overloaded CI so the team backed off, and the author wants the channel kept outside source control.

Read the original

Used in

• Give each agent its own page in shared notes trial
• Coordinate parallel agents through numbered plans trial
• Nobody knows if multi-agent orchestration scales challenge

Self-reported, low-review post

Reports building a tunable regex engine, a faster ripgrep fork and a Rust build speed-up in low single-digit hours each, and argues people skip such projects out of a pre-agent cost model; includes Kreinin's rebuttal that most people lack pass/fail checks and should aim smaller.

Read the original

Used in

• Try an agent-built fix for your slowest tool trial
• Nobody agrees how ambitious to be with agent-built software challenge

One practitioner's experience

Reports that relying on agents made his systems more complex and his understanding of them thinner, and sets a per-project limit on what to hand over, with held-out validation, observability, shadow traffic and fast reverts for work he does delegate.

Read the original

Used in

• Set how much agent code is read by risk tolerance trial
• Show the agent's reasoning beside the diff trial
• Count repeated blind approvals in agent sessions trial
• Nobody agrees what the human bottleneck is now challenge
• Nobody agrees how ambitious to be with agent-built software challenge

About 15 vendor products whose agents combined private data, untrusted input and an outbound channel were exploited in 2023-2025.

Read the original

Used in

• Sandbox anything unattended recommendation
• Remove one leg of the lethal trifecta recommendation
• Allowlist MCP servers and enforce access at the tool layer recommendation
• Red-team our agents with seeded injections trial
• Prompt injection has no proven fix challenge

Uber used its whole 2026 AI-coding budget in four months after a usage leaderboard.

Read the original

Used in

• Don't rank people by AI usage recommendation
• Set rate limits, quotas, timeouts and iteration caps recommendation

Removing the human bottleneck removes the pain signal.

Read the original

Used in

• Time-track agent work by phase trial
• Steer agents with named methods and a system metaphor trial

Uses a plan-then-execute split but insists on an overpowered executor; opinion.

Read the original

Used in

• Let a cheaper model execute a strong model's plan trial
• Planning and execution prompts may need different detail challenge

Humans work 'on the loop', changing the specs, checks and guidance that steer the agent rather than the artefact when results are wrong; conceptual, no data.

Read the original

Used in

• Fix the harness before the code trial

Newsletter forecast with no data: steering people to cheaper models can raise retries and human correction and erase the saving.

Read the original

Used in

• Show cost per merged change next to total spend trial

Essay by a critic of AI mandates, anecdotes only: token usage as a productivity proxy is a vanity metric.

Read the original

Used in

• Compare token spend with merged output before setting usage targets trial
• Set how much agent code is read by risk tolerance trial
• Name one human who can explain each agent PR trial

Prescriptive: keep separate scorecards for human, agent and combined system.

Read the original

Used in

• Name AI-enabled deliverables in advance trial

Skills should carry novel information or code.

Read the original

Used in

• Move repeated procedures into on-demand skills trial

Preprint guide

Keep the spec in sync with code through tests for long-lived systems, use the least rigour that removes ambiguity, and skip specs for exploratory work; no data, and its 'up to 50%' error cut is secondhand and unverified.

Read the original

Used in

• Match spec rigour to how long the system lives trial

Proposes loading the whole working set into context instead of retrieving chunks (CAG) for full recall; no data, partly drafted with Gemini.

Read the original

Used in

• Compare whole-module context with retrieval trial

Without sorting context by role, nontechnical builders' agent systems grow as big balls of mud.

Read the original

Used in

• Sort context into global, local and ambient trial

Origin of the term 'big ball of mud', which the authors call a pattern, not an antipattern.

Read the original

Used in

• Sort context into global, local and ambient trial

Position paper arguing for information architecture for context; argued, not measured.

Read the original

Used in

• Sort context into global, local and ambient trial

Argues for hacking until something works, recovering a spec from the session and regenerating to see what the spec kept; a first spec is always wrong and regenerating a bug keeps the bug.

Read the original

Used in

• Recover the spec after the first working version trial

A spec detailed enough to generate software is thinly veiled code, so specs are not cheaper than code; its evidence is one author, one spec, one Claude Code run.

Read the original

Used in

• Measure how much of an agent spec is code trial

Most projects are well defined but poorly specified, so agents can run free only where any decent output will do; no data, author sells consultancy and training.

Read the original

Used in

• Grant agent autonomy only on loosely defined work trial

Rejects green code the author cannot explain.

Read the original

Used in

• Set explicit rejection criteria trial

Advocates hand-writing commit messages.

Read the original

Used in

• Set explicit rejection criteria trial
• Hand-write commit messages and why-comments trial

Without a simplification pass, each bug gets its own patch.

Read the original

Used in

• Add a simplification pass before review trial

Proposes QA and fuzzing instead of default diff review.

Read the original

Used in

• Try QA instead of line-by-line review on an internal tool trial

A human in the loop whose only acceptable answer is approval is there to take the blame, without the knowledge, time or authority to stop a failure (opinion, no data).

Read the original

Used in

• Give reviewers time and authority to reject trial
• Name a checker before agent output drives a decision trial

Generated tests and AI reviewers each add an interpretation between the engineer and the running system, and AI quality gates turn those interpretations into decisions (no data).

Read the original

Used in

• Check AI-approved PRs by running them trial
• Attach evidence of what ran to green results trial
• Use a read-only agent to build incident timelines trial

Approve anything that sends, spends or deletes.

Read the original

Used in

• Remove one leg of the lethal trifecta recommendation

Anthropic agreed a $1.5B copyright settlement.

Read the original

Used in

• Sign a data processing agreement before tools see company data recommendation

Relays a CNN exclusive that an intelligence report partly produced with AI nearly led to an armed boarding of a ship; unnamed sources, no official confirmation.

Read the original

Used in

• Name a checker before agent output drives a decision trial

Relays the same CNN exclusive about an AI-assisted intelligence report and a near-miss ship boarding; unnamed sources, no official confirmation.

Read the original

Used in

• Name a checker before agent output drives a decision trial

Infers that AI drives faster exploitation and the October 2026 incident cluster (opinion with incident relays).

Read the original

Used in

• Measure patch-to-deploy time in hours trial

Proposes a human writes plain-language test intent, AI expands it into test code, a human implements the production code and AI only refactors after (unbylined, low confidence).

Read the original

Used in

• Separate test-writing from code-writing trial

Automation research prescribes regular manual control to keep operator skills; it comes from industrial automation, not LLMs.

Read the original

Used in

• Keep people hands-on with AI-off sessions trial

A position paper names the gap of when one agent's memory write must become visible to another, but measures nothing.

Read the original

Used in

• Shared agent memory behaves like a shared database challenge

Argues at least one human should fully understand each AI change, and that teams should record why, not what.

Read the original

Used in

• Name one human who can explain each agent PR trial
• Supervising agents needs the skills agents erode challenge
• Nobody knows if multi-agent orchestration scales challenge

Argues docs are not sufficient if they are not living artifacts.

Read the original

Used in

• Keep an ADR per agent-built feature trial

Argues agents remove the friction that synchronised people.

Read the original

Used in

• Run a quarterly "explain this service" check trial

Essay with a hypothetical example, arguing for more funding of the author's own field: at society scale, understanding must sit outside whoever proposes the system.

Read the original

Used in

• Run a quarterly "explain this service" check trial

A 2012 essay with anecdotes argues explaining to a listener that cannot answer finds the gap.

Read the original

Used in

• Explain a change in plain words before approving it trial

Popular guide with no cited studies to writing a plain-words explanation for a novice.

Read the original

Used in

• Explain a change in plain words before approving it trial

Popular guide with no cited studies to writing a plain-words explanation for a novice; its steps differ from other guides.

Read the original

Used in

• Explain a change in plain words before approving it trial

Physics, not code: two equivalent descriptions suggest different next changes, so being able to propose a change is a sharper test of understanding.

Read the original

Used in

• Explain a change in plain words before approving it trial

Pairing with an agent is closer to reading someone's diff; freed time should go left of dev, to users and stakeholders.

Read the original

Used in

• Have the agent restate the plan and object before coding trial
• Spend freed time with users trial

Describes cohort releases within a week.

Read the original

Used in

• Prototype requests cheaply for a user cohort trial

When building is cheap, deciding what not to build is the skill.

Read the original

Used in

• Track features shipped vs used trial

Overproduction is Lean's worst waste.

Read the original

Used in

• Track features shipped vs used trial

Agents compress a day of decisions into one session.

Read the original

Used in

• Cap agent-heavy work per day trial

Argues for 3-4 hour workdays with agents.

Read the original

Used in

• Cap agent-heavy work per day trial

Opinion, no data, about knowledge work in general (the author sells AI training): treat AI adoption as change management, start with enthusiasts, make failure safe, replace pages of rules with a few principles, and learn from shadow AI instead of banning it.

Read the original

Used in

• Lead AI rollout with enthusiasts and a few principles trial

Secondhand summary of a podcast

Majors argues code review is the least valuable part of what humans add, that trust should move from authorship to tests and evals, and that engineers will ship code they never read.

Read the original

Used in

• Review load lands on senior engineers challenge

Argues that exhaustive upfront specification is waterfall in disguise and that spec-driven development repeats it: what to build is discovered by building.

Read the original

Used in

• Nobody knows how much to specify up front challenge

Argues that partial, even wrong, understanding is the normal working state on large, high-turnover codebases, and that LLMs are one of many ordinary things that get in the way of it.

Read the original

Used in

• It's unclear how much of a system a team must understand challenge

Sells a course on this; most cited voices are secondhand

Argues that when a machine writes the code, the human's job is an executable acceptance-test specification in four layers, written one scenario at a time, kept out of the agent's reach and approved by a person.

Read the original

Used in

• Compare who writes acceptance tests trial
• Try hard-to-fake test formats in one module trial
• Try acceptance tests as the main guardrail trial
• Nobody knows how much to specify up front challenge

Argument by analogy, no measurements; written by an OpenAI employee

Argues that code is a lossy projection of the specification, so teams should write and keep the spec, make it executable and test against it.

Read the original

Used in

• Nobody knows how much to specify up front challenge

Advocate's critique, no new data

Argues that the study finding no gain from TDD in the agent loop used a thin prompt (no test list, weak refactoring, no smell sensors), so the question is still open.

Read the original

Used in

• Compare agent-run TDD with spec tests on a clear spec trial
• Try agent TDD on real, vague tickets trial
• Write a test list before the first test trial
• Nobody knows how much to specify up front challenge

Observation and anecdote, no measurement

Says AI hung on a code-first process closes tickets faster but leaves the same weak tests and design, and describes the test-list, small-slice loop that experienced TDD practitioners use with agents.

Read the original

Used in

• Write a test list before the first test trial
• Strong engineering foundations may not protect you challenge

Short opinion post, no data

Argues Brooks's Law still holds with agents: sequential work stays sequential, the overhead moves to defining done and stitching work together, and more agents on unclear requirements give the wrong answer faster.

Read the original

Used in

• Nobody knows if multi-agent orchestration scales challenge

Opinion, no data

Argues review's lasting value is the shared question of why a change exists, with knowledge flowing both ways, and that the real debate is whether people still need teams.

Read the original

Used in

• Ask why a change exists before judging how trial
• Nobody agrees what the human bottleneck is now challenge

Draft, not on the Standards Track

Lets a page publish tools agents can call, with read-only and consequential hints; states that agents inherit the user's identity and session, that the hints are advisory, and leaves identity, permissions and delegation to other work.

Read the original

Used in

• Apps can't tell an agent from the person it acts for challenge

Working group draft

Defines how automated clients sign HTTP requests so a site can verify which agent sent them; states it does not authenticate human users and does not define authorization or delegation.

Read the original

Used in

• Apps can't tell an agent from the person it acts for challenge

Proposed charter

Proposes a working group whose first deliverable is site capabilities for agents; agent identity and agent delegation are listed only as possible later deliverables, with no draft.

Read the original

Used in

• Apps can't tell an agent from the person it acts for challenge