
Flaky E2E Tests Are an Architecture Problem
When an end-to-end test fails intermittently, most teams bump the retry count and move on. We did too, until we admitted the browser was only the last link in a chain of provider deployments, drifting data, an implicit clock, shared state and UI coupling.
There is a moment in most E2E programs where someone opens the CI config and bumps retries from 1 to 2. Nobody announces it. The pipeline goes green more often, the standup gets quieter, and for a few weeks it feels like the problem has been handled. I have done this myself, more than once, and I want to be honest about what it actually buys you: a greener dashboard and slightly less information about your product than you had before.
The thing I eventually had to admit, on a program with a data-heavy product sitting on top of an external platform and a lot of background processing, was that our instability was not a Playwright problem. It was not even really a test problem. The browser was the last link in a chain that ran through deployment state, a third-party API we did not control, mutable shared data, scheduled jobs, a couple of shared service accounts, cached sessions, the wall clock, and a UI that was being actively redesigned. Every one of those could change between two runs of the same spec. We were trying to stabilise the final click of a process whose premise was different every morning.
Once you see it that way, retries look like what they are: a diagnostic tool that got promoted to a release policy. What follows is the taxonomy we ended up using. It is not sophisticated, but it has the property that matters for a small team, which is that every intermittent failure lands in exactly one bucket, and every bucket has a structural answer rather than a timeout.
Dependency instability
A perfectly written test fails because the provider is mid-deploy, a token expired forty minutes into the run, or the record the test was reading changed shape between the list call and the detail call. Waiting longer does nothing here except make the failure arrive later and cost more minutes.
What we did for the read-heavy integration paths, which in that product was the large majority of external traffic, was to split one question into two. The functional suite runs against a service simulation that implements an accepted consumer contract: the endpoints, fields, status codes, headers and session behaviour our application actually uses, built from sanitised captures of real responses. Separately, and on every CI run, we verify that contract against the real provider. If a field disappears or a status code changes, the contract lane goes red. The functional lane is not affected by the provider's availability at all.
I want to be specific about what that simulation is not, because the word "mock" carries a lot of baggage. It is not a stub that returns whatever the test expected. Before any browser test starts, the real application consumer is pointed at the simulation and exercised through authentication, listing, detail, history and reporting paths. A request the simulation does not recognise fails the run. Session behaviour the application relies on is checked. A request journal records which contract paths were actually hit, so we can see whether a critical path went unexercised. The functional lane answers "does our product behave correctly given the agreed contract" and the provider lane answers "does the provider still honour that contract". When those two questions are tangled together in one suite, you cannot tell which one failed, and that is the real cost of testing against live services by default.
Data instability
Most flaky tests I have seen are deterministic procedures running against nondeterministic premises. "Open a record in an actionable state" is a fine step in a spec and a terrible one against an environment that might contain zero, one or twenty such records depending on who ran what yesterday. "Verify the seven-day trend" is unstable if the sample data was loaded a month ago and the window has since drifted past it. "Sort by last reviewed" proves nothing if only one record has review history.
The instinct is to fix this with a bigger, more realistic database dump. We went the other way and built a small catalogue of archetypes, each existing to provoke or prevent one named behaviour: a rule-positive record that must produce a derived outcome, a control record that must not, a deliberately sparse record that is insufficient for evaluation, a cluster of related entities to exercise selection and aggregation, stale events sitting on a date-window boundary, and a handful of stable controls for reporting and performance checks. There are not many of them. Every one has a reason for existing written next to it.
The part I would push hardest on is that the fixture seeds inputs, not outputs. We load source events and rule configuration, run the real migrations, run the real business engine, and only then verify the derived classifications and queue items before a browser opens. Inserting the finished decision row directly would prove that the UI can render a row. It would tell us nothing about whether the system arrived at the right decision, and that is the thing we were being paid to find out.
Time instability
Time is data. Test suites tend to treat it as atmosphere, and that is where a whole class of "it was green on Friday" failures comes from. Rolling windows, expiry rules, overdue states, scheduled evaluations and stale-data thresholds all change without anybody deploying anything.
Every generated date in an isolated run now resolves from one explicit as-of value. Source events, schedules, evaluation windows and expected outcomes are all offsets from it, so they move together. The clock became part of the scenario contract rather than an invisible global.
One caution from experience: freezing the browser clock alone is not enough and can make things worse. If the UI believes it is Monday while the API, the database and the background worker believe it is Tuesday, you have added an inconsistency, not removed one. The boundary has to be system-wide, which in practice means the fixture, the services and the test runner all read the same value.
State contamination
Shared state produces failures that look random because the cause belongs to a different test. One spec disables access and the restore step throws silently. Another deletes a configuration two others depend on. A logout invalidates a server-side session that a cached storage state reuses twenty specs later. Cleanup fails in a finally nobody reads, and the next run inherits an impossible starting position.
We attacked this at several boundaries at once, and I do not think any one of them would have been enough alone:
- every isolated run starts from an empty, run-owned database volume;
- setup and teardown go through APIs wherever one exists, and the UI is reserved for the behaviour actually under verification;
- every attempt, including a retry, gets its own authentication state rather than a shared session file;
- anything the run creates carries a run-owned identifier;
- cleanup tasks register with one central registry and execute on failure as well as success;
- a post-run check confirms that fixture-owned state is back inside its expected boundary.
The mental shift is that cleanup is part of the test contract, not housekeeping. If a run cannot prove it cleaned up, a passing assertion in the middle of it does not make the run trustworthy.
Implementation coupling
The most expensive instability does not show up as intermittent red. It shows up as thirty red specs the morning after an ordinary product change. A modal became a page, fields were reordered, a component library was swapped. The product does exactly what it did before and every procedure that knew how the old page was assembled now fails.
We stopped putting UI choreography in procedures. A spec says something like:
The binding layer knows how "edits the rule" is performed in the current interface. Screen objects own the locators. Stable data-testid attributes support those locators, but a selector never appears in the language of a procedure. When the rule-management screens were substantially redesigned, the procedures did not change; one shared action was rebound to the new form and every scenario using that keyword picked up the repair. The next article in this series goes into that design in detail, so I will leave it there.
What a retry is actually for
I am not against retries. They answer a useful question: did exactly this test pass when repeated under nominally identical conditions? That is diagnostic information, and we keep it, along with the first-attempt trace and screenshot.
What we stopped doing is letting a pass on retry produce a clean verdict. Our verification post-processing treats any result that needed a retry as failed or explicitly unstable. The pipeline still surfaces the evidence; it just refuses to quietly convert an unexplained failure into release approval. That one policy changed how the team behaved more than any tooling did, because an unstable result now has to be classified before anyone moves on.
The argument holds for any product team, where a false green mostly wastes engineering hours and hides risk. In regulated software it bites harder. For something like SaMD, a retry that laundered a real failure means incomplete traceability from a risk control to an executed procedure, ambiguous evidence, and a result you may be asked to defend in an audit with no good answer. Regulation did not make our architecture better. It made the cost of a bad one explicit.
The diagnostic table we actually use
The outcome worth naming
I will not give you a before-and-after flake percentage, partly because I do not trust those numbers from anyone and partly because the thing that changed was not a rate. What changed is that failures became attributable. The isolated run records infrastructure, migration, seed, readiness, browser, reporting and cleanup as separate stages, so when something goes wrong we know which stage, and "test flake" stopped being an acceptable root cause.
A trustworthy E2E test should fail for exactly two reasons: the observable behaviour of the product changed, or a contract that behaviour depends on changed. Provider downtime, yesterday's data, another test's leftovers, a drifting clock or a refactored component are noise, and noise is something the architecture is supposed to control. None of this required flake-management infrastructure of the kind large platform teams build. It required explicit boundaries, deterministic inputs, owned state, contract checks, and procedures written in the language of behaviour. Green was never the goal. A release signal the team can explain is.
Facing a problem like this one?
Tell us what the business needs to win and who owns it on your side. If it is a fit, a First Build starts within weeks. If it is not, we will tell you why.
