Flaky on CI, Green Locally: The Real Causes (and How to Fix Them)

One screen green and one red with a puzzled developer, illustrating flaky tests in CI
In this post
  1. TL;DR
  2. Why "Flaky" Is the Wrong Word
  3. Why Flaky Tests in CI Pass Locally but Fail
  4. The Diagnosis Order That Beats Adding Retries
  5. Make CI Match Reality (or Reality Match CI)
  6. Where Nopmark Fits
  7. FAQ
  8. Ready to Stop Guessing at Test Failures?

Flaky tests in CI start the same way every time. Your suite is green on your laptop. You push, and the pipeline lights up red. You run it again. Still red. You run it locally one more time. Green. So you shrug, label the test "flaky," and add a retry.

That label is usually wrong, and the retry is usually a cover-up. A test that passes locally and fails in CI isn't random. It's telling you that two environments are different, and the test depends on a difference you haven't named yet. Find the difference and the "flakiness" disappears.

Flaky tests in CI are one of the most common time sinks in test automation. A QA engineer recently described chasing a 25% failure rate on mobile login tests in CI for two weeks (auditing selectors, adding retries, treating it as a locator problem) before the real cause turned out to have nothing to do with locators at all. That two-week detour is avoidable if you diagnose in the right order.

TL;DR

  • "Flaky" usually means "environment-dependent," not "random." Treat the gap between local and CI as the bug.
  • The usual culprits, in rough order of frequency: timing and async waits, headless vs. headed browsers, resource contention, leaked state and test order, and network or test data.
  • Adding retries hides the signal. Diagnose first; retry only as a deliberate, logged last resort.
  • The fastest fix is making CI reproducible locally (same container, same parallelism, same headless mode) so you can actually see the failure.

Why "Flaky" Is the Wrong Word

When a test fails only sometimes, "flaky" feels like an explanation. It isn't. It's a description of the symptom that quietly closes the investigation.

A genuinely non-deterministic test is rare. What's common is a test whose result depends on something that varies between runs: how fast a response came back, whether an animation had finished, whether another test left data behind, whether the machine was under load. None of that is random. All of it is a difference you can find.

So reframe the question. Instead of "why is this test flaky," ask: "what is true on my laptop that isn't true in CI?" That question has answers. "Flaky" doesn't.

Why Flaky Tests in CI Pass Locally but Fail

Five differences account for the large majority of cases. Work through them roughly in this order, since the earlier ones are both more common and cheaper to check.

Timing and async waits

This is the number one cause, by a wide margin. Your laptop is fast and lightly loaded. The CI runner is often slower, shared, and busy. A page that renders in 200ms locally might take 1.5 seconds in CI. If your test assumes the element is there "by now," it passes locally and fails in CI the moment the machine is slow.

  • The tell: failures are "element not found," "element not clickable," or timeouts, and they move around between runs.
  • The wrong fix: sleep(5) or a longer fixed wait. It slows every run and still breaks the day CI is slower than your guess.
  • The right fix: wait for the actual condition: the element to be visible and interactable, the network request to complete, the spinner to disappear. Assert on state, not on elapsed time.

Headless vs. headed browsers

Locally you probably watch the browser run. CI almost always runs headless. These are not the same environment. Headless browsers can differ in default viewport size, font rendering, how they handle focus, and how some events fire. A button that's in the viewport in your headed window might be below the fold, and not clickable, in the headless default size.

  • The tell: element-visibility and click failures that only happen in CI, especially on responsive layouts.
  • The right fix: run headless locally to reproduce, and set an explicit viewport size in your test config so both environments match.

Resource contention and parallelism

CI runners have less CPU and memory than your machine, and your pipeline probably runs tests in parallel to save time. Locally you may run them one at a time. Parallel tests on a constrained box compete for resources and, worse, can collide on shared state: the same user account, the same database row, the same fixture.

  • The tell: failures get worse as you add parallel workers, or a test passes alone and fails in the suite.
  • The right fix: give each parallel worker isolated data (unique users, unique records), and confirm whether the failure survives running single-threaded in CI. If it disappears, you have a contention problem, not a logic problem.

Leaked state and test order

A test that depends on another test having run first will pass locally, where your runner happens to use one order, and fail in CI, where the order or sharding differs. The same goes for tests that create data and don't clean it up.

  • The tell: a test passes in isolation but fails in the full suite, or failures shift when you change how tests are split across runners.
  • The right fix: make every test set up its own preconditions and tear down its own data. No test should rely on the leftovers of another.

Network and test data

CI often sits behind different network rules, talks to a different test environment, or hits rate limits your single local run never triggers. And the data in the CI environment is rarely identical to your local seed.

  • The tell: failures cluster on specific external calls, auth, or records that "should" exist.
  • The right fix: stub or mock third-party calls where you can, point both environments at known seeded data, and stop assuming a record exists just because it does locally.

The Diagnosis Order That Beats Adding Retries

Reaching for a retry is tempting because it makes the pipeline green. It also throws away the evidence. Here's a sequence that finds the cause instead of masking it.

  1. Read the actual error, not the test name. "Login test failed" tells you nothing. "Element not clickable at point (x, y)" points straight at viewport or timing.
  2. Reproduce CI locally. Run the failing test headless, at the CI viewport, in the same container image if you use one. A failure you can see is a failure you can fix.
  3. Isolate one variable at a time. Single-threaded vs. parallel. Headed vs. headless. Alone vs. in suite. Each flip tells you which category you're in.
  4. Capture artifacts on failure. Screenshots, video, DOM snapshots, and logs from the CI run. Most frameworks do this on demand, so turn it on. The screenshot at the moment of failure often ends the investigation in seconds.
  5. Fix the difference, not the symptom. Once you know it's timing, fix the wait. Once you know it's data collision, isolate the data. Then remove any retry you added while flailing.

Retries have one legitimate use: absorbing genuinely external, rare instability (a third-party sandbox that drops a connection once a week) while you have no control over the source. Even then, log every retry so you can see whether "rare" is quietly becoming "constant." A retry you can't see is a bug you're paying for and ignoring.

Make CI Match Reality (or Reality Match CI)

The deepest fix is structural: shrink the gap between the two environments so "passes locally" actually predicts "passes in CI."

  • Run the same container locally that CI runs. If your pipeline uses a Docker image, run your tests in that image on your machine. Most "works on my machine" failures evaporate when the machine is the same machine.
  • Default to headless and a fixed viewport in your test config, everywhere. Don't let your headed local window be the only place tests pass.
  • Match parallelism at least once before you trust a suite. Run it locally with the same number of workers CI uses.
  • Treat environment differences as first-class config, not accidents. Base URLs, viewport, worker count, and data seeds should be explicit settings, not whatever your laptop happened to default to.

When the environments converge, flakiness stops being a weekly mystery and becomes a normal, findable bug.

Where Nopmark Fits

Diagnosing CI failures well is really an exercise in the testing fundamentals: understanding test independence, controlling preconditions, and reasoning about why a test gives the result it does. Those are the same principles the ISTQB Foundation syllabus builds, and they're what separate an engineer who adds a retry from one who finds the cause.

If you want to build that foundation deliberately rather than learning it two-week detour by two-week detour, Nopmark's ISTQB CTFL course is taught by accredited instructors and grounds you in the test-design and test-management thinking these problems demand.

FAQ

Are flaky tests ever truly random? Rarely. Most "random" failures depend on timing, load, order, or data, all of which vary between runs but are findable. Treat true non-determinism as the last hypothesis, not the first.

Should I just add a retry to make the pipeline green? Only as a logged, deliberate last resort for genuinely external instability you can't control. As a default reflex, retries hide the signal and let real bugs ship. Diagnose first.

Why does my test pass alone but fail in the full suite? That's almost always leaked state or test-order dependence: another test left data behind, or your test relies on something a previous test set up. Make each test create and clean up its own preconditions.

What's the single fastest thing to check? Read the real error message and turn on failure screenshots. The exact error plus an image of the screen at the moment of failure resolves a large share of cases immediately.

How do I reproduce a CI failure locally? Run the failing test the way CI does: headless, at the CI viewport size, in the same container image, and with the same parallelism. A failure you can reproduce is a failure you can fix.

Ready to Stop Guessing at Test Failures?

Ready to debug test failures by cause instead of by reflex? The ISTQB CTFL course from Nopmark grounds you in the test-design and independence principles behind problems like these, taught by accredited instructors, with hands-on practice. See the course.

Keep reading