A test that passes and fails at random

A flaky test is one that gives different answers to the same question. It is worse than a broken test, because a broken test gets fixed while a flaky one teaches the team to ignore red. Anything failing intermittently is worth an hour of attention, or worth deleting.

123

1. 1Open a passing run and a failing run of the same test side by side.
2. 2Compare durations: a much slower run often explains an intermittent failure.
3. 3Check the trigger. Scheduled runs hit your site at quieter or busier times than manual ones.

Start with the first failing step

Open two runs of the same test, one that passed and one that failed, and compare the step that failed. Its screenshot shows what the browser actually saw. Nine times out of ten the page is mid-load, a spinner is still showing, or a modal has not finished animating, which tells you immediately that this is timing rather than a genuine regression.

The usual causes

• The step ran before the element existed. Add a wait for the element rather than a fixed delay.
• An animation moved the element between locating it and clicking it.
• The assertion targets content that legitimately varies, such as a total, a date, or rotating marketing copy.
• The test depends on data another test creates or destroys, so the order it runs in changes the outcome.
• A third-party script, such as a cookie banner or chat widget, sometimes loads over the element being clicked.

Prefer waiting for a condition

Fixed delays are the tempting fix and the wrong one. A three second wait is simultaneously too long on a good day and too short on a bad one, and it makes every run slower forever. Waiting for the thing you actually need, an element appearing or a spinner disappearing, is both faster and more reliable.