Risk-based testing: check your scores against escaped defects
Take a QA lead whose team uses risk-based testing to plan each sprint. Every story gets a risk score, the team tests the high scorers first, and the low scorers…
You have read this

When a test that always passed turns red for no visible reason, the team often reaches for one word: flaky. That word usually means the test passes and fails for reasons nobody has found. Playwright's new --shuffle option can show that the cause is test order dependency, where a test only passes because an earlier test left something behind for it.
Playwright is a tool for testing web apps in a browser. Its 1.64.0 release notes came out on October 7, 2026. They say the new --shuffle option "schedules tests in a random order, which helps to find tests that accidentally depend on each other".
The notes show one command that shuffles the order and a second command that repeats it.
npx playwright test --shuffle
npx playwright test --shuffle 271828182
The first command runs the tests in a random order. The notes' sample run reports a shuffle seed, which is a number that recreates one specific order. The second command passes a seed back in. Above it, the notes put the comment "Pass the seed to reproduce the same order". The number 271828182 is only the notes' sample, because a real run reports its own seed.
Take a project-tracking app with two tests, where one creates a project named Demo and checks that Demo appears in the project list. The other test opens the list and archives Demo, which takes it off the list of active projects.
In the usual order, the creating test runs first. The archiving test then finds Demo in the list, archives it and passes. It passes on every run, yet it never created Demo itself. It relies on the first test having left Demo behind, and nothing in the test says so.
If the tests are shuffled and the archiving test runs first, the project list is empty, so there is nothing to archive and the test fails. Nothing changed in the app or in the test except the run order. Each plain --shuffle run picks a new order, so the failure can come and go and looks random.
Many suites run in a fixed order, which has a fair reason. The same order gives the same result, so one run is easy to compare with the last. While the creating test always runs before the archiving test, nothing shows that the archiving test depends on it. The suite stays green until something changes the order. A shuffled run changes the order on purpose.
When a shuffled run fails, copy the seed from its output and run again with that seed. The notes say a seed reproduces the same order, so a failure caused by the order should come back. A failure caused by something else, such as timing, may not. If the archiving test fails again in the same place, you have shown that the failure follows the order.
The failing test also tells you what it expects. Here the archiving test expects a project named Demo to exist, and nothing in the test creates one.
To fix the archiving test, give it its own setup. Setup means the steps that run before a test and create the data it needs. Here the setup creates a project, and the test archives that project. Then run the same seed again to confirm the test passes in the order that broke it.
Forcing the creating test to run first would also make the failure go away, but it would keep the dependency hidden. The archiving test would still fail when run alone, and anyone debugging it would need to know which other test to run first.
On your own suite, run with --shuffle more than once. When a test fails in one order, rerun with that seed. If it fails again, find the data it expects, move the step that creates that data into the test, and rerun the same seed to confirm the fix.
Nopmark's ISTQB CTFL training covers testing fundamentals for testers who want a structured course.
A rerun that passes tells you nothing about why the first run failed. A seed that reproduces the failure tells you that the failure follows the order. It also shows which data the test expected, and it lets you check the fix against the exact order that broke it.
What is test order dependency? A test has test order dependency when it only passes if another test has run before it. The first test leaves data or settings behind, and the second test uses them without creating them.
Is every flaky test an order problem? No. A flaky test can fail because of timing, shared accounts, a slow network or other causes. Order is one cause that a replayed seed can confirm or rule out. If the same seed does not bring the failure back, look at the other causes.
Does a passing shuffled run prove the suite has no order dependency? No. A random order tries only some of the possible orders. A pass means those orders did not expose a dependency, and running with --shuffle more than once tries more of them.
--shuffle option, the seed and both commands.Take a QA lead whose team uses risk-based testing to plan each sprint. Every story gets a risk score, the team tests the high scorers first, and the low scorers…
You have read this