Skip to main content

Strictly necessary · always on

Remembers this choice in your browser. No tracking.

spfr_blog_10.11.26_02.png
E2E testing

The browser was the easy part of our E2E suite

Marko Jagodic··8 min read

Eight lines of Playwright are the visible tip of an E2E test. What decides whether a suite is trustworthy sits underneath: what has to be true before the page opens, and what actually differs between tests. The closing post in our series on owning the test-data premise and scaling coverage without copying specs.

This is the last post in our short series on end-to-end testing. The first one argued that flaky tests are usually an architecture problem. The second showed how we keep procedures alive through a UI redesign. Both of those were about the browser layer, and the thing I want to close on is that the browser layer was never where most of the work was.

On a data-heavy product, the eight lines of Playwright that open a page and assert on a row are the visible tip. Underneath them sit two questions that decided whether the suite was trustworthy and whether it could grow: what has to already be true before the page opens, and what actually differs from one test to the next. Get those wrong and no amount of selector hygiene saves you. Get them right and the browser part gets almost boring.

What has to be true before the page opens

Take one screen that shows a single actionable decision. For that row to exist with the right status the system needed a primary record, several related entities, a few weeks of events against them, an enabled rule with a specific configuration, enough data to clear a sufficiency threshold, a completed background job, the right permissions on the user, and a consistent view of data from an external platform. Remove any one of those and the expected behaviour changes. If those conditions come from a shared environment, the test does not own its own premise, and every intermittent failure traces back to that.

We took ownership of the premise in four moves.

Archetypes instead of snapshots. A big database dump feels realistic because it has a lot in it, and after three months nobody can say why any record is there. We replaced it with a small catalogue of synthetic archetypes, each existing to provoke or prevent one named behaviour: a rule-positive case that must produce a derived outcome, a control case that must not, a sparse case that is insufficient for evaluation, a cluster of related entities for aggregation, stale events on a date-window boundary, values on either side of a threshold. Every one has a written reason for existing and an owner. Realism comes from preserving the relationships that drive behaviour, not from copying production volume.

One explicit clock. Rolling windows, expiry rules, overdue states and scheduled evaluations change their answer without anyone deploying code. Every generated date in an isolated run resolves from a single as-of value, so events, windows and expected outcomes move together. The boundary has to be system-wide; freezing the browser clock while the API and the worker disagree about what day it is manufactures a new inconsistency rather than removing one.

Seed inputs, derive outputs. The shortcut is to insert the finished decision row and assert that it renders. That tests rendering. Our fixture seeds source events and rule configuration, runs the real migrations and the real business engine, and verifies the derived classifications before a browser opens. The defects this catches are exactly the ones a final-state snapshot cannot: a wrong threshold, a missing join, an off-by-one date window, a background job that silently never ran.

Simulate reads, own mutations. Most external traffic on a product like this is reading data for decisions and displays, and those paths want stable inputs more than live availability. We run them against a service simulation built from an accepted consumer contract, and we verify that contract against the real provider on every CI run, so provider downtime stops blocking functional coverage while provider drift still shows up red. Mutations get stronger ownership: an isolated instance, a sandbox with reserved identifiers, or a supported API setup-and-cleanup path. The browser never creates uncontrolled external state just to prepare its own assertion.

Cleanup is part of this model, not housekeeping after it. Each run starts from an empty run-owned database, anything it creates carries a run-owned identifier, cleanup executes on failure as well as success, and a post-run check confirms the fixture boundary is intact. A test cannot silently borrow its starting state from the previous run, which turns out to be the mechanism behind a surprising share of "passes alone, fails in the suite".

What actually differs between tests

Once the premise is owned, the second question is how the suite grows. If you count tests, the cheapest growth is copying a working spec, changing one input and renaming it. I have watched suites double this way while the number of distinct behaviours they could explain stayed flat. Then a shared workflow changes and ten copies contain ten slightly different versions of login, navigation, form fill and verification, all of which break and all of which need a human to work out how they differ.

The unit of reuse we settled on is the behavioural procedure, with the examples that exercise it kept as a separate, typed asset:

Given an authorized user is configuring business rules

When the user creates a valid rule

Then the rule is active in the configuration list

const ruleExamples = [

{ type: 'threshold', input: { limit: 80 } },

{ type: 'relative-change', input: { percentage: 15 } },

] as const;

A threshold rule and a relative-change rule take different inputs, but the user's intent and the outcome are identical, so they are rows under one procedure. The action binding underneath does whatever the current UI needs for each type. The same shape covers search: exact name, partial name and identifier lookup are rows under one successful-search procedure because they make the same promise.

This is not "put everything in a table", and the boundary is stricter than "same screen". Examples stay together only while they share the same workflow, the same observable outcome and the same risk meaning. Creation with required fields missing is a different behaviour, because its outcome is guidance and a prevented submission. A no-match search verifies an empty state. An unauthorised attempt verifies an access boundary. Each of those gets its own procedure, and that rule is what stops the matrix turning into a hidden test program with a conditional in every step.

Before anyone adds a browser test we ask six things: same workflow, same outcome, a meaningful equivalence class or risk boundary, expressible as data without branching, independently identifiable in the report, and no change in setup ownership, permissions or cleanup. Five yeses and a no on the last means a new row. Anything else is a new procedure. And if twenty numeric values exercise the same control, validation rule and API path, they do not belong in the browser at all; they belong in unit, component or API tests, with the browser keeping the representative values that prove the integration.

The reporting has to keep two views for this to be honest. Reviewers see one procedure with its example matrix and the rationale per row. Engineers see one result, trace and screenshot per resolved example, with the failure naming the exact input class, target and software version. A shared procedure never hides a row.

Where the two halves meet

The test-data layer and the example layer are the same idea seen from two sides. An archetype is a controlled system state; an example is a controlled variation on a behaviour; and a procedure is target-neutral precisely because a provider layer resolves which archetype satisfies its precondition on whichever stack it runs against. The spec says "a record has insufficient source data" and never learns a fixture identifier.

In regulated work, including SaMD, the same structure is what makes evidence defensible rather than merely present. Fixture version, procedure, software version, provider contract version and generated result form one reproducible chain, and an impact analysis after a change can be scoped from whichever layer moved. That is not paperwork after testing. It falls out of the test system as a by-product of running it.

So the honest summary of this series is short. A browser test fails for two legitimate reasons, a behaviour changed or a contract it depends on changed, and everything else is noise the architecture should control. Procedures are written in the language of behaviour so a redesign lands in one binding. And the asset that makes all of it hold together is not the selector collection. It is the model of valid system states and the deliberate set of examples run against it. The browser was never the hard part. Owning the premise was.

Tagged:E2E testingtest automationQA engineeringtest data managementPlaywright test data
Your Turn

Facing a problem like this one?

Tell us what the business needs to win and who owns it on your side. If it is a fit, a First Build starts within weeks. If it is not, we will tell you why.