Testing Best Practices
Why Automated Tests Become Flaky and How to Fix Them
Learn why tests pass and fail without relevant product changes, how to investigate timing, state, data, network, and environment causes, and how to prevent recurrence.
- Published
- Reading time
- 3 min read
- Difficulty
- Intermediate
- Audience
- For QA engineer, Automation tester, Developer, QA lead
Article overview
What you will learn
- How to recognize and measure flaky behavior
- Which timing, state, data, and environment problems commonly cause it
- How to use traces, screenshots, video, and logs during investigation
- Why fixed delays and retries are incomplete fixes
- How to prevent flaky behavior through isolation and observable waits
Before you begin
Prerequisites
- An automated test suite
- Access to CI failure artifacts
- Ability to repeat a failing test in a controlled environment
What is a flaky test?
Flaky tests interrupt delivery and weaken trust. When a red result is often dismissed as noise, a real regression can be ignored. The goal is not only to make the next run green; it is to identify which uncontrolled condition changed the outcome.
Common causes
| Cause | Typical signal | Direction for investigation |
|---|---|---|
| Timing assumption | Failure changes with machine speed | Wait for observable state or event |
| Weak locator | Wrong or changing element is selected | Use a stable user-facing or explicit contract |
| Shared state | Failure depends on order or parallel workers | Isolate accounts, storage, and cleanup |
| Conflicting data | Duplicate or missing records | Generate scoped data and unique identifiers |
| Network variation | Timeouts or incomplete responses | Control dependencies and inspect request evidence |
| Animation | Element moves or is covered | Wait for the intended actionable state |
| Environment difference | CI fails while local passes | Compare versions, resources, time zone, locale, and configuration |
Why fixed delays are risky
// Weak: time is unrelated to the expected result.await page.waitForTimeout(3000);await page.getByRole('button', { name: 'Pay' }).click(); // Better: wait through an assertion about observable state.await expect(page.getByRole('button', { name: 'Pay' })).toBeEnabled();await page.getByRole('button', { name: 'Pay' }).click();The fixed delay wastes time when the page is ready early and still fails when readiness takes longer. The assertion waits for the condition required by the next action. A real project may need to wait for a response, a calculation result, or a different UI state instead.
Investigation workflow
- Record the exact test, commit, browser, worker, environment, and failure time.
- Preserve the first useful trace, screenshot, video, console output, network details, and application logs.
- Repeat the test alone and with its normal neighbors.
- Repeat under parallel load and with randomized order where supported.
- Compare passing and failing evidence at the earliest point they diverge.
- Form one cause hypothesis and change one relevant condition.
- Verify the fix through repeated isolated and normal-suite runs.
- Add a prevention rule, helper, or monitoring signal when the pattern can recur.
Using traces and other evidence
Playwright Trace Viewer can show actions, DOM snapshots, logs, source, console, and network activity for configured traces. Screenshots show appearance at selected moments; video shows sequence; application logs can explain server decisions. No single artifact always identifies the cause, so correlate timestamps and IDs where possible.
What retries can and cannot do
A retry can classify an intermittent result and collect a trace on a later attempt. It can also keep a pipeline moving under a documented temporary policy. It does not remove the defect in the test, product, data, or environment. Track retry outcomes and assign ownership rather than treating a passed retry as equivalent to a stable first pass.
Prevention checklist
- Keep tests independent and safe to run in any order.
- Create unique, scoped test data and clean it predictably.
- Use stable locators and retrying assertions tied to expected state.
- Control third-party dependencies or test their contract at an appropriate boundary.
- Pin and record relevant browser, runtime, locale, and environment versions.
- Retain useful failure artifacts without exposing secrets.
- Review flaky-test rate and time lost, not only final pass rate.
Key takeaways
- Flakiness is an uncontrolled dependency, not a normal property of automation.
- Compare passing and failing evidence to find the first divergence.
- Fixed waits trade one timing assumption for another.
- Retries can reveal intermittency but do not fix its cause.
- Isolation, controlled data, observable waits, and good artifacts prevent recurring failures.
References and further reading
- Best practicesPlaywright documentation · Verified 2026-07-19
- RetriesPlaywright documentation · Verified 2026-07-19
- Trace viewerPlaywright documentation · Verified 2026-07-19
