Risk-based testing: check your scores against escaped defects
Take a QA lead whose team uses risk-based testing to plan each sprint. Every story gets a risk score, the team tests the high scorers first, and the low scorers…
You have read this

The most upvoted post in r/softwaretesting this week is a 2,000-word rant about AI test automation being a nightmare. The tester is not wrong, and the rest of the industry is pretending otherwise.
The headline-grabbing finding from the Reddit thread is simple. Outside of API test generation, the 2026 testing toolkit looks almost identical to 2020. The new part is AI-driven UI tests that eat tokens, run slower than the deterministic tests they replaced, and fail in non-deterministic ways your CI logs cannot explain.
In the same week, four major vendors shipped or announced AI test automation features. The gap between the vendor narrative and the practitioner reality is the entire 2026 testing story.
This post is the honest version.
Look at the announcements from one week of trade press.
Every one of those announcements positions AI as the future of testing. None of them publish independent CI-minute or flake-rate benchmarks against the deterministic tools they aim to replace.
The same week as those announcements, the most-engaged posts in the testing community were:
Joe Colantonio's Automation Guild 2026 survey of 40,000-plus testers puts the contradiction in numbers. 72.8 percent picked "AI-powered testing & autonomous test generation" as their top priority. The same group's biggest concern was whether AI in testing actually helps.
That is not a community resisting change. That is a community waiting for the change to deliver.
This is not a "no AI" post. There are real wins. Three of them, specifically.
If you have an OpenAPI spec, a Postman collection, or a working API call, AI generates the test boilerplate in seconds. The output is deterministic markup, not runtime intelligence. You read it, you commit it, it runs the same way every CI build. SmartBear's 80-percent-faster claim for ReadyAPI is in this category, and the claim is plausible.
When a button id changes from submit-btn to checkout-submit, AI-driven self-healing locates the new element using surrounding context and updates the selector. For high-traffic flows that change shape often, this catches the test before a human does. Applitools and Mabl both do this well.
The win is bounded. Self-healing works when the change is cosmetic. When the underlying flow actually changed (a new modal, a different validation order), self-healing happily clicks the wrong thing and tells you everything passed.
cy.prompt("log in as a test user and add three items to the cart") is useful as a scaffolding tool. You read the generated test. You edit it. You commit it. Cypress runs it deterministically from then on.
This is the model that works in 2026. AI helps you write the test. The runtime stays deterministic.
The opposite model, AI as the runtime, is where the Reddit thread comes from.
A deterministic Playwright test costs the CPU time of running Playwright. An AI-driven test that asks an LLM to interpret each step at runtime costs CPU plus API tokens plus network round-trip per decision. For a 100-test suite running on every PR, the bill compounds fast.
Teams I have spoken with this quarter report 3 to 8x cost increases when moving from deterministic to AI-runtime tests, with the high end on JavaScript-heavy SPAs where the AI has to re-orient itself across more DOM updates. None of the vendor press releases mention this number.
A deterministic test fails when the app is broken. An AI-runtime test can fail when the app is broken, OR when the LLM made a different decision than last run, OR when the page took longer to settle and the AI tried to click before the DOM was ready. The flake budget for your suite doubles or triples.
Debugging the failures takes longer too. With a Playwright failure, you read the trace. With an LLM-runtime failure, you read the trace AND try to reconstruct what the LLM was thinking. Often you cannot.
When an AI test "self-heals" through a change, your test passes. Your CI is green. The metric on your dashboard says coverage is fine. But the test is now exercising a different code path than the one you originally meant to test, and nobody told you.
For a regression suite, this is worse than a flaky test. A flaky test annoys you. A self-healed test silently stops protecting the thing it was written to protect.
If you ignore the vendor noise and listen to the practitioners, the setup that works in 2026 is consistent across teams:
Most importantly: do not let a vendor convince you that the test runtime should be probabilistic. The whole point of a regression suite is that the same input produces the same output. An LLM-driven runtime breaks that contract.
Concrete moves a QA lead can make this week, in order of effort.
This is exactly the gap our CT-GenAI program is built around. It is the only ISTQB-aligned AI testing certification that teaches the "AI-assisted authoring, deterministic runtime" model rather than the vendor-pitched "AI as runtime" model. The lab work walks you through making the cost-and-flake measurements above on a real test suite.
The CT-GenAI program is taught by accredited instructors. The labs are vendor-agnostic. The exam aligns to the ISTQB Certified Tester AI Testing syllabus.
Q: Are you saying AI in testing is a scam? No. AI test generation from API contracts is the real deal. AI-assisted authoring saves a measurable amount of time. AI selector self-healing prevents some real outages. Those are the wins. The marketing-vs-reality gap is in AI runtime, where an LLM decides what to test in the moment, and that gap is real.
Q: My team is already on cy.prompt / Mabl / testRigor. Should we rip it out? Run the math first. If CI minutes per test and flake rate look fine, keep going. If either is materially worse, you have a vendor whose marketing has outrun their engineering.
Q: What about agentic QA from Katalon or others? The agentic-QA pitch is the strongest version of "AI as runtime." Same caveats. Some teams will get value from it for genuinely exploratory test runs where the AI surfaces issues you would not have looked for. As a regression-suite replacement, the data is not there yet.
Q: Does this mean I should not learn AI testing skills? The opposite. Learn AI testing skills aggressively, then apply them with judgment. The senior end of QA is migrating toward AI-adjacent engineering. Angie Jones just retitled her LinkedIn role to "VP, Agentic AI Foundation." The skills are valuable. The vendor narrative around them is what to question.
Q: How do I get budget for the proper setup without sounding like I am anti-AI? Bring numbers, not opinions. Show the CI-minute cost of your current AI tool against a deterministic baseline. Decision-makers respond to dollars. The Monday-morning exercises above are how you build that case in two weeks.
Ready to learn AI testing the honest way? CT-GenAI from Nopmark covers exactly this gap. Real setup, not hype. Taught by accredited instructors with vendor-agnostic labs.
Take a QA lead whose team uses risk-based testing to plan each sprint. Every story gets a risk score, the team tests the high scorers first, and the low scorers…
You have read this