Risk-based testing: check your scores against escaped defects
Take a QA lead whose team uses risk-based testing to plan each sprint. Every story gets a risk score, the team tests the high scorers first, and the low scorers…
You have read this

Flaky tests in CI start the same way every time. Your suite is green on your laptop. You push, and the pipeline lights up red. You run it again. Still red. You run it locally one more time. Green. So you shrug, label the test "flaky," and add a retry.
That label is usually wrong, and the retry is usually a cover-up. A test that passes locally and fails in CI isn't random. It's telling you that two environments are different, and the test depends on a difference you haven't named yet. Find the difference and the "flakiness" disappears.
Flaky tests in CI are one of the most common time sinks in test automation. A QA engineer recently described chasing a 25% failure rate on mobile login tests in CI for two weeks (auditing selectors, adding retries, treating it as a locator problem) before the real cause turned out to have nothing to do with locators at all. That two-week detour is avoidable if you diagnose in the right order.
When a test fails only sometimes, "flaky" feels like an explanation. It isn't. It's a description of the symptom that quietly closes the investigation.
A genuinely non-deterministic test is rare. What's common is a test whose result depends on something that varies between runs: how fast a response came back, whether an animation had finished, whether another test left data behind, whether the machine was under load. None of that is random. All of it is a difference you can find.
So reframe the question. Instead of "why is this test flaky," ask: "what is true on my laptop that isn't true in CI?" That question has answers. "Flaky" doesn't.
Five differences account for the large majority of cases. Work through them roughly in this order, since the earlier ones are both more common and cheaper to check.
This is the number one cause, by a wide margin. Your laptop is fast and lightly loaded. The CI runner is often slower, shared, and busy. A page that renders in 200ms locally might take 1.5 seconds in CI. If your test assumes the element is there "by now," it passes locally and fails in CI the moment the machine is slow.
Locally you probably watch the browser run. CI almost always runs headless. These are not the same environment. Headless browsers can differ in default viewport size, font rendering, how they handle focus, and how some events fire. A button that's in the viewport in your headed window might be below the fold, and not clickable, in the headless default size.
CI runners have less CPU and memory than your machine, and your pipeline probably runs tests in parallel to save time. Locally you may run them one at a time. Parallel tests on a constrained box compete for resources and, worse, can collide on shared state: the same user account, the same database row, the same fixture.
A test that depends on another test having run first will pass locally, where your runner happens to use one order, and fail in CI, where the order or sharding differs. The same goes for tests that create data and don't clean it up.
CI often sits behind different network rules, talks to a different test environment, or hits rate limits your single local run never triggers. And the data in the CI environment is rarely identical to your local seed.
Reaching for a retry is tempting because it makes the pipeline green. It also throws away the evidence. Here's a sequence that finds the cause instead of masking it.
Retries have one legitimate use: absorbing genuinely external, rare instability (a third-party sandbox that drops a connection once a week) while you have no control over the source. Even then, log every retry so you can see whether "rare" is quietly becoming "constant." A retry you can't see is a bug you're paying for and ignoring.
The deepest fix is structural: shrink the gap between the two environments so "passes locally" actually predicts "passes in CI."
When the environments converge, flakiness stops being a weekly mystery and becomes a normal, findable bug.
Diagnosing CI failures well is really an exercise in the testing fundamentals: understanding test independence, controlling preconditions, and reasoning about why a test gives the result it does. Those are the same principles the ISTQB Foundation syllabus builds, and they're what separate an engineer who adds a retry from one who finds the cause.
If you want to build that foundation deliberately rather than learning it two-week detour by two-week detour, Nopmark's ISTQB CTFL course is taught by accredited instructors and grounds you in the test-design and test-management thinking these problems demand.
Are flaky tests ever truly random? Rarely. Most "random" failures depend on timing, load, order, or data, all of which vary between runs but are findable. Treat true non-determinism as the last hypothesis, not the first.
Should I just add a retry to make the pipeline green? Only as a logged, deliberate last resort for genuinely external instability you can't control. As a default reflex, retries hide the signal and let real bugs ship. Diagnose first.
Why does my test pass alone but fail in the full suite? That's almost always leaked state or test-order dependence: another test left data behind, or your test relies on something a previous test set up. Make each test create and clean up its own preconditions.
What's the single fastest thing to check? Read the real error message and turn on failure screenshots. The exact error plus an image of the screen at the moment of failure resolves a large share of cases immediately.
How do I reproduce a CI failure locally? Run the failing test the way CI does: headless, at the CI viewport size, in the same container image, and with the same parallelism. A failure you can reproduce is a failure you can fix.
Ready to debug test failures by cause instead of by reflex? The ISTQB CTFL course from Nopmark grounds you in the test-design and independence principles behind problems like these, taught by accredited instructors, with hands-on practice. See the course.
Take a QA lead whose team uses risk-based testing to plan each sprint. Every story gets a risk score, the team tests the high scorers first, and the low scorers…
You have read this