How much testing is enough? The test pyramid, coverage targets and flaky tests

The useful question about test automation is not "how many tests should we write" but "where have our tests piled up". Three things decide whether a suite earns its keep: the shape, the feedback time, and whether you believe it. For shape, the most cited reference is Google's rough 70 percent unit, 20 percent integration, 10 percent end-to-end split. For feedback, DORA gives a hard threshold: developers should see results in under ten minutes. For belief, there is only one test, which is whether a red build makes anyone stop. Coverage percentage substitutes for none of these. It tells you where tests are absent, which makes it a diagnostic, not a target.

The arithmetic behind the pyramid

The idea spread through Mike Cohn's 2009 book Succeeding with Agile and stuck after Martin Fowler's short 2012 write-up. The logic is plain: every step up the pyramid costs more. More expensive to write, because you have to stand up more of the system. More expensive to run, as seconds turn into minutes. And most expensive to diagnose. When a unit test goes red you know which function broke. When an end-to-end test goes red you start investigating which of seven layers broke.

Google's testing team put numbers on that balance in 2015, but the ratio was never the point of the post. The problem with end-to-end tests is not only that they are slow. It is that when one fails, working out whether the fault is in the product or in the test environment can burn days. A team spending a week on a failing test and finding the bug was in the harness rather than the application has just discovered the real price of that test.

Inverted pyramids happen by convenience, not by choice

Almost nobody flips the pyramid deliberately. If you inherit a system with no tests at all, the fastest visible result comes from scripting a journey through the user interface: one test walks ten screens, and the dashboard looks better within a week. A year later you have four hundred of them, the suite takes forty minutes, and a handful fail on every run for no reason anyone can name.

Google's 2017 analysis explains why this is close to inevitable. As test binaries get larger, the probability of flakiness rises with them, and the relationship is roughly linear. So "big tests break more often" is not an impression, it is a measured trend. The conclusion is not that you should avoid large tests. It is that the count matters.

Flaky tests are not an annoyance, they are lost trust

By Google's own published figure, around 16 percent of their tests show some level of flakiness, meaning roughly one test in seven occasionally fails with neither the code nor the test having changed. If a company at that scale cannot engineer the problem away, nobody expects you to either. The damage is not really in the number. It is in the habit the number creates: once "just run it again" becomes normal, red stops carrying information.

An observation from the same team shows what that habit costs. When a stable test turned flaky and the change could be traced, the fault turned out to be in the production code rather than the test roughly one time in six. Automatic retries throw that sixth away without telling anyone. The approach that works is quarantine rather than deletion, with two conditions attached: the quarantined test has a named owner and a date. Quarantine without an owner is just deletion in slow motion.

What happens when coverage becomes a target

Fowler's line is short: if you make a coverage level a target, people will attain it. The way they attain it is predictable. Tests with no assertions, calls that execute code and verify nothing, tours through getters and setters. The number climbs and the assurance does not. His own expectation is that a team with sound testing habits will land in the upper 80s or 90s naturally. He is suspicious of 100 percent, and treats anything under 50 percent as a genuine problem. The figure is an output, not an input.

Google's practice lands in the same place. There is no company-wide mandatory threshold. Their published guidance offers 60 percent as acceptable, 75 percent as commendable and 90 percent as exemplary, then immediately says they avoid top-down mandates and expect each team to pick a number that suits its business. The interesting part is that adoption spread anyway: by the first quarter of 2018, more than 90 percent of projects were using coverage tooling voluntarily. The setting that actually helps is reading coverage on the changeset rather than the repository. In code review, "how much of what this change adds is tested" is a far more useful answer than a five-year cumulative percentage.

A better question than coverage: do the tests actually catch anything

Coverage tells you a line ran. It says nothing about whether the line was verified. Mutation testing measures that gap. The tool injects small deliberate faults into your code, flipping a comparison operator or blanking a return value, then checks whether your tests notice. If they do not, that line is unprotected no matter what the coverage report says.

Google's study across more than 1,000 projects and more than 24,000 developers also shows that the naive version of this does not work: developers marked 85 percent of the reported mutants as unproductive. What made it usable was restricting mutation to the change under review and filtering the mutants that got surfaced. The finding worth carrying over is this one: developers on projects with mutation testing wrote more tests over time than developers on projects that only tracked coverage. PIT for Java, Stryker for JavaScript and C# put this within reach of a small team. Same rule applies: run it on the diff, not the repository.

For integrations, contract tests beat end-to-end tests

Wherever several services, or an ERP and a storefront, have to agree with each other, the top of the pyramid swells fast, because "does it really work" seems to require standing everything up at once. Contract testing breaks that assumption. The calling service writes down exactly what it needs from the provider in a contract file, and the provider verifies on its own pipeline that it still satisfies that contract. The two systems never run together, and the mismatch is caught anyway.

The payoff shows up in two places. Speed first: integration runs that took fifteen to twenty minutes in a shared environment are reported dropping to a few minutes under contract verification. Timing second, and this matters more. By the time an end-to-end test goes red, the breaking change has already merged and shipped to a shared environment. A contract test stops the same change before the merge. Pact is the common tool here. We went through where and why synchronisation actually breaks in our post on ERP and e-commerce data sync.

The ten-minute rule, and who owns the tests

DORA's test automation guidance sets an explicit threshold: developers should get results in under ten minutes. Every minute past that is a developer switching to something else and then paying to rebuild context. The ways to get under it are unglamorous and effective: parallelise the run, split the suite (fast tests on every push, slow ones overnight), and select tests based on what changed.

The second finding in the same guidance is about ownership. Delivery performance is better when developers write and maintain the tests themselves. Hand the job to a separate function and two things follow: broken tests sit around because they are nobody's problem, and developers write code that is hard to test. This is not an argument against testing specialists. Exploratory testing, usability work and evolving the suite are all skilled jobs. But the day-to-day maintenance load belongs with whoever wrote the code.

AI-written tests raise coverage, not necessarily confidence

A study published in March 2026 examined 2,232 test-adding commits across open source projects and found 16.4 percent of them were authored by AI agents. The spread by project is wide: 1.9 percent in a large repository with more than 600 contributors, 100 percent in a five-person project. The same work found AI-written tests carry more assertions than human-written ones, a median of two against one.

More assertions is not automatically a better test. Piling them into a single test makes it harder to tell which one failed when it goes red. The real risk sits in review, though: a test that compiles and lifts the coverage number is hard to argue with, so it passes. There is a cheap defence. Deliberately break the line the new test claims to verify, and if the test stays green, it verifies nothing. Make that check routine for anything an agent wrote.

The first three weeks on a system with no tests

Do not try to build a full suite. Week one: list the paths where money and data can be lost, payment, order creation, stock deduction, permission checks, and write end-to-end tests for those alone. No more than five. Week two: cover the business rules inside those paths with unit tests, price calculation, discounts, tax, refund conditions. Most defects live in the edge cases of those rules. Week three: wire the suite into your pipeline and put a hard ceiling on run time.

After that, one rule carries most of the weight: every production bug gets documented as a failing test before it gets fixed. That habit grows the suite around your actual defect profile instead of somebody's guess about it. We covered how to build the pipeline in CI/CD for small teams. And for choosing which untested file to start with in a large codebase, the hotspot method in our post on measuring and prioritising technical debt will do the job.