Run failed, or your site failed?

A red run has three possible meanings, and treating them the same way wastes time. Your site is genuinely broken. The test is wrong or stale. Or the run itself could not complete for reasons that have nothing to do with either.

12345

1. 1A failed run. Everything after the first failure is skipped, not attempted.
2. 2One step failed. That is the step to investigate, not the skipped ones after it.
3. 3Open the failed group first: it names the step and carries the error message.
4. 4The error the runner recorded, on the step that produced it.
5. 5The three-dot menu, from which the failure can be filed as a ticket.

Your site is broken

The signature is an assertion failing on a page that otherwise rendered fine, or an error page in the screenshot. The steps ran, the page responded, and the thing that should have been true was not. This is the case the product exists to catch, and it deserves the alert.

The test is wrong

The signature is an element that could not be found on a page that plainly contains it, or an assertion on text somebody rewrote last week. Nothing is broken for customers; the test has drifted out of date. Fix the step rather than raising an incident.

Infrastructure

Occasionally a run fails because the browser session itself could not be established or was lost mid-run. Testsnag recognises these and marks them as provider errors, and deliberately suppresses the usual notifications so an infrastructure hiccup does not page somebody at 3am about a product that is fine. If a run failed with no step-level detail and no screenshots, this is the likely explanation; retry it before investigating further.

A quick triage order

1. Open the run and find the first failed step. Everything after it is usually noise.
2. Look at its screenshot. Does the page look correct?
3. If the page looks correct, suspect the test. If it does not, suspect the site.
4. Check whether the previous runs of this test were passing, and when it changed.
5. Open the site yourself and try the same journey by hand.