AI Tests Aren't Faster. The Data Says They're Slower.

A weary engineer waiting on a slow loading spinner late at night, illustrating slow AI test automation
In this post
  1. TL;DR
  2. What did vendors actually ship in May 2026?
  3. What does the practitioner data say?
  4. Where AI Test Automation Genuinely Wins
  5. Where AI in testing breaks down
  6. The honest 2026 setup
  7. What to do Monday morning
  8. Where Nopmark fits
  9. FAQ

The most upvoted post in r/softwaretesting this week is a 2,000-word rant about AI test automation being a nightmare. The tester is not wrong, and the rest of the industry is pretending otherwise.

The headline-grabbing finding from the Reddit thread is simple. Outside of API test generation, the 2026 testing toolkit looks almost identical to 2020. The new part is AI-driven UI tests that eat tokens, run slower than the deterministic tests they replaced, and fail in non-deterministic ways your CI logs cannot explain.

In the same week, four major vendors shipped or announced AI test automation features. The gap between the vendor narrative and the practitioner reality is the entire 2026 testing story.

This post is the honest version.

TL;DR

  • AI test generation is genuinely useful for API tests and selector self-healing. Most other claims do not survive contact with production.
  • AI test execution (where an LLM decides what to do at runtime) is slower, more expensive, and flakier than deterministic Playwright or Cypress in 2026.
  • The honest setup for 2026: AI-assisted authoring, deterministic runtime. Not the other way around.
  • Track CI minutes per test and flake rate before and after adopting any AI tool. If either metric moves the wrong way, the tool is making testing worse, not better.

What did vendors actually ship in May 2026?

Look at the announcements from one week of trade press.

  • Cypress moved cy.prompt() into beta. Write tests in plain English. (blog)
  • Playwright released v1.59 with page.screencast and v1.60 with HAR recording on tracing. Not AI features per se, but a clear bet on observability over agent runtime.
  • SmartBear added AI test generation to ReadyAPI, claiming up to 80 percent faster test creation.
  • Parasoft launched an MCP server for autonomous API mocking.
  • Applitools rebranded as "the Intelligent Testing Platform" and was named a Strong Performer in the Forrester Wave for Autonomous Testing Platforms.

Every one of those announcements positions AI as the future of testing. None of them publish independent CI-minute or flake-rate benchmarks against the deterministic tools they aim to replace.

What does the practitioner data say?

The same week as those announcements, the most-engaged posts in the testing community were:

  • "AI is a nightmare for QA": 150 upvotes, 81 comments
  • "What other career option for QA which is safe in this AI era": 52 upvotes, 53 comments
  • "Average QA: what does your day-to-day actually look like?": 58 upvotes, where the OP explicitly excludes "super-intelligent or MCP+AI users" from the question

Joe Colantonio's Automation Guild 2026 survey of 40,000-plus testers puts the contradiction in numbers. 72.8 percent picked "AI-powered testing & autonomous test generation" as their top priority. The same group's biggest concern was whether AI in testing actually helps.

That is not a community resisting change. That is a community waiting for the change to deliver.

Where AI Test Automation Genuinely Wins

This is not a "no AI" post. There are real wins. Three of them, specifically.

1. API test generation from contracts

If you have an OpenAPI spec, a Postman collection, or a working API call, AI generates the test boilerplate in seconds. The output is deterministic markup, not runtime intelligence. You read it, you commit it, it runs the same way every CI build. SmartBear's 80-percent-faster claim for ReadyAPI is in this category, and the claim is plausible.

2. Selector self-healing for known-stable flows

When a button id changes from submit-btn to checkout-submit, AI-driven self-healing locates the new element using surrounding context and updates the selector. For high-traffic flows that change shape often, this catches the test before a human does. Applitools and Mabl both do this well.

The win is bounded. Self-healing works when the change is cosmetic. When the underlying flow actually changed (a new modal, a different validation order), self-healing happily clicks the wrong thing and tells you everything passed.

3. Authoring-time assistance, not runtime decisions

cy.prompt("log in as a test user and add three items to the cart") is useful as a scaffolding tool. You read the generated test. You edit it. You commit it. Cypress runs it deterministically from then on.

This is the model that works in 2026. AI helps you write the test. The runtime stays deterministic.

Where AI in testing breaks down

The opposite model, AI as the runtime, is where the Reddit thread comes from.

Token cost per CI run

A deterministic Playwright test costs the CPU time of running Playwright. An AI-driven test that asks an LLM to interpret each step at runtime costs CPU plus API tokens plus network round-trip per decision. For a 100-test suite running on every PR, the bill compounds fast.

Teams I have spoken with this quarter report 3 to 8x cost increases when moving from deterministic to AI-runtime tests, with the high end on JavaScript-heavy SPAs where the AI has to re-orient itself across more DOM updates. None of the vendor press releases mention this number.

Probabilistic flake

A deterministic test fails when the app is broken. An AI-runtime test can fail when the app is broken, OR when the LLM made a different decision than last run, OR when the page took longer to settle and the AI tried to click before the DOM was ready. The flake budget for your suite doubles or triples.

Debugging the failures takes longer too. With a Playwright failure, you read the trace. With an LLM-runtime failure, you read the trace AND try to reconstruct what the LLM was thinking. Often you cannot.

The "self-healed" lie

When an AI test "self-heals" through a change, your test passes. Your CI is green. The metric on your dashboard says coverage is fine. But the test is now exercising a different code path than the one you originally meant to test, and nobody told you.

For a regression suite, this is worse than a flaky test. A flaky test annoys you. A self-healed test silently stops protecting the thing it was written to protect.

The honest 2026 setup

If you ignore the vendor noise and listen to the practitioners, the setup that works in 2026 is consistent across teams:

  • Runtime: deterministic. Playwright for web, native frameworks for mobile, deterministic API frameworks. The same test, run twice in a row, produces the same result.
  • Authoring: AI-assisted. Use cy.prompt, Playwright's MCP integration, or your IDE's inline AI to generate test scaffolding. A human reads and commits the result.
  • Maintenance: AI for selectors, humans for flow changes. Self-healing is fine for "the button moved." Any change in flow logic gets a human review.
  • Observability: trace-first. Playwright's page.screencast and HAR-on-tracing land in this category. Cypress's video recording does too. Pick whichever your team will actually look at when a test fails at 2am.
  • One metric to watch: CI minutes per test, before and after. If a new AI tool moves it the wrong way, the tool is costing you money, not saving it.

Most importantly: do not let a vendor convince you that the test runtime should be probabilistic. The whole point of a regression suite is that the same input produces the same output. An LLM-driven runtime breaks that contract.

What to do Monday morning

Concrete moves a QA lead can make this week, in order of effort.

  1. Run the math on one AI test. Pick a single test in your suite. Measure its run time and CI cost today. Reimplement it with whichever AI runtime your vendor is pitching. Run it 20 times. Average the cost and the flake rate. Bring the numbers to your next test strategy meeting.
  2. Add cy.prompt or Playwright MCP to one authoring workflow. Pick a new feature being shipped this sprint. Use AI-assisted authoring to generate the first draft of the tests. Have a human review and commit. Measure how much faster the human got to "tests merged."
  3. Audit your self-healing config. If you use Applitools, Mabl, or similar, find the dashboard that shows which tests have been "auto-fixed" in the last month. Manually review each fix. If any of them silently changed the code path, you have a problem larger than one test.
  4. Start a "AI in test" log. One file in your repo. Every time you adopt an AI feature, write the date, the tool, the use case, the cost change, and the flake change. In three months, you will have data nobody else in your industry has.

Where Nopmark fits

This is exactly the gap our CT-GenAI program is built around. It is the only ISTQB-aligned AI testing certification that teaches the "AI-assisted authoring, deterministic runtime" model rather than the vendor-pitched "AI as runtime" model. The lab work walks you through making the cost-and-flake measurements above on a real test suite.

The CT-GenAI program is taught by accredited instructors. The labs are vendor-agnostic. The exam aligns to the ISTQB Certified Tester AI Testing syllabus.

FAQ

Q: Are you saying AI in testing is a scam? No. AI test generation from API contracts is the real deal. AI-assisted authoring saves a measurable amount of time. AI selector self-healing prevents some real outages. Those are the wins. The marketing-vs-reality gap is in AI runtime, where an LLM decides what to test in the moment, and that gap is real.

Q: My team is already on cy.prompt / Mabl / testRigor. Should we rip it out? Run the math first. If CI minutes per test and flake rate look fine, keep going. If either is materially worse, you have a vendor whose marketing has outrun their engineering.

Q: What about agentic QA from Katalon or others? The agentic-QA pitch is the strongest version of "AI as runtime." Same caveats. Some teams will get value from it for genuinely exploratory test runs where the AI surfaces issues you would not have looked for. As a regression-suite replacement, the data is not there yet.

Q: Does this mean I should not learn AI testing skills? The opposite. Learn AI testing skills aggressively, then apply them with judgment. The senior end of QA is migrating toward AI-adjacent engineering. Angie Jones just retitled her LinkedIn role to "VP, Agentic AI Foundation." The skills are valuable. The vendor narrative around them is what to question.

Q: How do I get budget for the proper setup without sounding like I am anti-AI? Bring numbers, not opinions. Show the CI-minute cost of your current AI tool against a deterministic baseline. Decision-makers respond to dollars. The Monday-morning exercises above are how you build that case in two weeks.

Ready to learn AI testing the honest way? CT-GenAI from Nopmark covers exactly this gap. Real setup, not hype. Taught by accredited instructors with vendor-agnostic labs.

Keep reading