Agentic QA, Decoded: What the Demos Don't Show You

Agentic QA concept: a QA engineer scrutinizing a monitor full of green passing checkmarks where a single red flag stands out, in a bright modern office
In this post
  1. What does agentic QA actually mean?
  2. What agentic QA does well today
  3. The oracle problem: who decides a test passed?
  4. The maintenance bill nobody demos
  5. Where to let an agent in, and where to keep a human
  6. Where Nopmark fits
  7. FAQ
  8. Learn to test AI without taking the demo's word for it

Every testing vendor is now selling agentic QA, and the demos look like magic. An agent reads your requirements, writes the tests, runs them, and reports back all green. Before you trust that green, ask one question: who checked the agent's work?

This year the pitch is everywhere. SmartBear shipped BearQ. Katalon published a full guide to agentic QA. Applitools released Autonomous 2.2. The framing is identical across all of them: software that plans, executes, and maintains your tests from a goal instead of a script.

Some of it is real and useful. Some of it is a demo trick that costs you later. This post separates the two, so you can sit in any vendor call and tell the difference in the first five minutes.

What does agentic QA actually mean?

Strip the marketing and agentic QA is a loop. An agent takes a goal, something like "verify checkout works," explores the app, decides what to test, generates the test code, runs it, reads the result, and sometimes repairs the test when it breaks. The word "agentic" just means it makes its own decisions at each step instead of following a fixed script you wrote.

Compare that to traditional automation. You decide what to test. You write the assertion. You maintain it. The machine only executes what you already designed. With an agent, those decisions move from you to the model.

That shift is the whole story. Everything good and everything risky about agentic QA comes from the same place: the agent is now making testing decisions that used to be yours. Keep that sentence in your head for the rest of this post. It is the lens for every claim a vendor will make.

What agentic QA does well today

Not all of this is hype. Three categories deliver real value right now, and they map to problems QA teams have complained about for a decade.

  • Visual validation: an agent compares what the screen actually renders against a known-good baseline and flags the difference. This catches layout and rendering regressions that a DOM assertion sails straight past. A button can be present in the markup and invisible to a user, and a pixel-level check is the only thing that notices.
  • Autonomous test generation: point a tool at an app, let it crawl, and get a first-draft suite without writing it line by line. This is genuinely useful for getting from zero to baseline coverage on a screen nobody has had time to test. It is a starting point, not a finished suite.
  • Self-healing execution: when a selector changes and the old locator breaks, the agent updates it and keeps the run going instead of failing. Selector churn is the single biggest reason teams abandon automation, so cutting it has real payback.

Joe Colantonio, who runs the largest automation community, said it plainly in his 2026 survey writeup: most "AI testing tools" are GPT wrappers, and the ones that earn their keep fall into those three buckets. In TestGuild's 2026 Automation Guild survey, 72.8% of testers ranked AI-powered and autonomous test generation as their top priority for the year, and these were not beginners. More than 60% had ten or more years of experience. The demand is real. Discernment is what's missing.

The oracle problem: who decides a test passed?

Here is the question the demo never answers. When an agent writes a test, runs it, and tells you it passed, what did "passed" actually mean?

Testing has a concept called the oracle. It is the source of truth that decides whether the behavior you observed is correct. When you write a test, you are the oracle. You decided a successful checkout shows an order number, so you asserted on the order number.

When an agent writes the assertion and also grades the result, it is its own oracle. Green stops meaning "the software did the right thing." It starts meaning "the agent agrees with itself." Those are not the same claim, and the space between them is exactly where bugs live.

I watched a tool generate 200 tests for a sample app and pass all 200. Impressive, until you read the assertions. Half of them checked that a page loaded, not that it did anything correct. The agent had written tests it was guaranteed to pass. No human chose those assertions, so no human noticed how weak they were.

This is why Applitools, in its own 2026 writeup, names the real risk of AI-driven testing as loss of trust. When results are not repeatable or explainable, teams slow down and quietly bring back manual checks. A test you cannot explain is not a test. It is a vibe with a pass status.

The maintenance bill nobody demos

The demo shows you day one. The cost shows up on day ninety.

  • Drift: an agent that regenerates tests on every run gives you a suite that keeps changing shape. You lose the stable baseline that makes a regression suite worth having, because last week's tests are not this week's tests.
  • False confidence: a growing pile of weak assertions reports green while real coverage stays flat. The dashboard looks healthier than the product. That gap is invisible until something ships broken and the suite swore everything was fine.
  • Triage moves, it does not vanish: when an agentic suite flags 30 failures, someone still has to decide which are real. Now you are debugging the agent's judgment on top of the application's behavior. That is two problems, not zero.

Surveys from Tricentis and Sembi this year found something the vendors gloss over. AI-generated code is increasing testing demand, not shrinking it. Tricentis surveyed more than 2,500 IT and quality leaders and found 60% of organizations are knowingly shipping untested code, with nearly a third saying the volume of AI-generated code is too much to test fully. Sembi surveyed nearly 4,000 QA, security, and engineering professionals, and 61% reported moderate to dramatic increases in testing load driven by AI-written code. More code, written faster, read by fewer humans. The work did not disappear. It changed shape.

Where to let an agent in, and where to keep a human

You do not have to be a purist. Agents earn a place. The skill is knowing which decisions you keep and which you hand over.

Keep the oracle. Let the agent draft and execute, but you own the assertions that define "correct" for anything that matters: money, data integrity, regulated behavior, anything a customer would file a bug about. If a wrong result there would be a real incident, a human writes that check.

Hand the agent the boring middle. Self-healing for cosmetic selector changes. Visual diffs flagged for your review. First-draft coverage on a new screen that you then sharpen. These save real time and the downside of a mistake is small.

Alan Page made the deeper point in March this year, and it lands hard here. Quality is not a result of testing. Quality is a result of the system you work in. Better testing on a broken system just gives you a more detailed picture of how broken it is. An agent that tests faster does not change that. If your process ships untested AI-generated code on a Friday afternoon, a smarter agent measures the damage. It does not prevent it.

So the question to bring to any agentic QA pitch is not "how much can it automate?" It is "what decisions does this take from my team, and do I want those decisions made by a model that grades its own homework?"

Three questions expose a weak tool fast:

  • Where is the independent oracle? If the agent writes and grades the test, what is the second source of truth?
  • What happens when the UI changes in a way that should fail the test? Does it heal right past a real bug?
  • Show me a test it refused to write. A tool that never says "I am not sure" is not reasoning about risk. It is generating output.

Where Nopmark fits

Testing AI systems, and testing well with AI agents, is a skill set most QA training has not caught up to. Knowing what an oracle is, why non-determinism breaks assertions, and how to design tests for probabilistic systems is exactly the gap these agentic tools expose. Nopmark's ISTQB Certified Tester AI Testing (CT-GenAI) course covers how to test AI-driven systems and how to use AI in your testing without handing over your judgment. It is built for the tester who wants to use these tools well, not be replaced by a demo.

FAQ

Is agentic QA going to replace manual testers? No, and the 2026 survey data points the other way. More AI-generated code means more testing demand, and the judgment calls (what to test, what "correct" means) still need a person. The role shifts toward defining and reviewing, not toward the exit.

What is the difference between agentic QA and test automation? Traditional automation executes decisions you already made. Agentic QA makes the decisions itself: what to test, how to test it, and whether it passed. The value and the risk both come from that handoff of judgment.

Are AI testing tools just GPT wrappers? Many are. The ones worth paying for cluster into three categories: visual validation, autonomous test generation, and self-healing execution. If a tool does not clearly sit in one of those, ask what it does that a prompt and an API key could not.

Why do agent-written tests pass so easily? Because the agent often writes assertions it knows it can satisfy. A test that checks "the page loaded" passes reliably and proves almost nothing. Read a sample of the assertions, not just the pass rate.

Can I trust a self-healing test? Sometimes. Self-healing is fine for cosmetic selector changes. It is dangerous when it heals past a change that should have failed the test. Healing should be logged and reviewable, never silent.

How do I start evaluating an agentic tool without getting burned? Run it on a real screen you already understand, then read every assertion it writes. Compare its verdict to yours. If you cannot explain why it passed or failed, that is your answer.

Learn to test AI without taking the demo's word for it

Ready to test AI systems with confidence instead of trusting a green dashboard? Nopmark's ISTQB CT-GenAI course teaches exactly this, from oracles and non-determinism to using AI agents without outsourcing your judgment, taught by accredited instructors with hands-on labs.

Keep reading