AI Test Generation Won't Tell You What to Test

Software QA engineer reviewing AI-generated test code and deciding what to test
In this post
  1. What AI test generation is genuinely good at
  2. The bottleneck was never writing the script
  3. Where AI test generation quietly fails
  4. What "agentic QA" actually means for you
  5. Where Nopmark fits
  6. FAQ

So AI test generation can write your automated scripts now, and half of LinkedIn has decided that means the tester is finished. The panic rests on a wrong assumption: that testing is mostly typing. It never was. The slow part of testing was always deciding what to interrogate, and that is exactly the part these tools cannot do for you.

Go read the testing subreddits this week and you will see the fear in plain language. One company is making developers quality-check their own work with an LLM and demoting the QA team to a consultancy. Another team, in the words of one poster, "started using AI and stopped using their own minds." A tester with 20 years of experience got laid off and is panicking. A different one says the developers on their team make them feel like a fraud on a regular basis.

The fear is real. The diagnosis behind it is wrong. Let me walk through what AI test generation actually changes, because the mechanism is the whole story.

What AI test generation is genuinely good at

Credit first. There is real value here, and pretending otherwise makes you look like the person who insisted Selenium was fine forever.

Frontier models are good at the mechanical middle of the job:

  • Boilerplate. Setup, teardown, page objects, fixtures, the repetitive scaffolding that every suite needs and nobody enjoys writing. A model produces this in seconds.
  • Bug reports. One of the manual testers posting this week said the single biggest win was bug reports. He used to stare at a blank report for far too long. Now he drafts the reproduction steps and expected behavior in a fraction of the time. That is a genuine gain, and notice he uses it for a narrow, well-defined task.
  • Self-healing locators. When a selector breaks because a class name changed, a model can often repair it without a human touching the test. This kills a specific category of flakiness that used to eat afternoons.

None of that is nothing. If you are still hand-writing every page object in 2026, you are leaving time on the table. The tester who uses AI for three things and refuses it for the rest has the right instinct.

But look closely at what all of those wins have in common. Each one is a task where you already know what you want: the setup you need, the bug you found, the element you are locating. In every case the model is filling in known structure faster. It is not deciding anything.

The bottleneck was never writing the script

Here is the part the "testers are cooked" crowd keeps missing. Writing the code was never the bottleneck in QA.

As one poster put it plainly this week, AI helps juniors write automated test scripts, but writing code was not the slow part of the job to begin with. The slow, hard, valuable part is figuring out what to test: which inputs matter, which workflows break under load, which edge case the requirements author never considered, and which sequence of user actions no test case in the suite covers.

That work is not typing. It is judgment. It is what a good tester does when they critically interrogate the requirements, the code, and the engineers' own assumptions, predicting where a mistake was made before anyone has run a single test.

I once worked with a tester who found bugs like it was a sixth sense. Ask him how he did it and he would say he was just doing what came naturally to him. He could not explain it because it was not a procedure but a mind trained to model human error, always asking where this system would betray the person using it. No model has that about your product. It was not in the training data, because your product is not in the training data.

Where AI test generation quietly fails

This is the trap. The vendor demo of AI test generation runs beautifully, and then the pilot on your own software produces slop.

The reason is domain knowledge. A vendor demo runs on a simple web form, the kind of trained, generic territory a model has seen a million times. Point the same tool at real domain software, a finance engine with its own business rules or a clinical workflow with its own constraints, and the output falls apart. The model does not know your business domain. It does not know your business processes. It does not know your customer workflow. It generates confident test cases that check the wrong things, and a junior who trusts them ships a suite that looks green and proves nothing.

Vendors know this, which is why the pitch usually pivots to selling you a retrieval setup to feed the model your context. Sometimes that helps. Often it burns money for a marginal gain over what you would get by pairing a frontier model with a tester who already holds the domain knowledge in their head.

There is a second failure that is easier to miss. Self-healing locators fix flakiness and hide regressions in the same motion. A locator that quietly repairs itself when the UI changed might be papering over a real defect. The test stays green. The bug ships. You traded a visible failure for an invisible one.

What "agentic QA" actually means for you

The buzzword this quarter is agentic QA. Read past the marketing and there are two very different things wearing the same name.

One is a product: an autonomous testing platform a vendor sells you, wrapped around the same domain-knowledge gap as everything else. The other is a capability you build yourself, using frontier models plus your own skills, your own agents, and your own domain context. The second one is real and it moves fast, faster than any single vendor can ship, because you are extending general-purpose models with the one thing they lack, which is knowledge of your system. The first one is mostly a rename.

The distinction matters for where you point your energy. Learning to drive frontier models against your own domain is a durable skill. Waiting for a vendor to sell you autonomy is a purchase order.

Where Nopmark fits

Nopmark's ISTQB Certified Tester AI Testing (CT-GenAI) course is built for exactly this moment: how to use AI in testing without handing it the judgment it cannot exercise. It covers where generative models help, where they introduce risk, and how to keep test design in human hands while the model handles the mechanical work. If your team is being told to "just use AI," this is the training that turns that mandate into a defensible practice rather than a pile of green tests that prove nothing.

FAQ

Does AI test generation replace manual testers? No. It replaces some of the typing manual testers do. The judgment about what to test, which is the actual job, stays with the human. Testers who move their effort up into test design become more valuable, not less.

Is AI test generation reliable on complex applications? It is reliable on well-defined, generic tasks and unreliable on domain-heavy software it was never trained on. The more your application depends on private business rules, the less you should trust generated tests without review.

Should juniors learn to write tests by hand if AI can do it? Yes. If you never learn to write and reason about tests, you cannot judge whether a generated test is any good. The skill you need is evaluation, and you only get it by doing the work first.

What about self-healing locators? Useful for genuine cosmetic changes, dangerous when they mask a real regression. Treat an auto-healed locator as a signal to look, not a problem solved.

How do I use AI in testing without lowering quality? Use it for known, bounded tasks (boilerplate, bug reports, locator repair) and keep the decisions about coverage, edge cases, and risk with a tester who understands the domain.

Ready to use AI in your testing without letting it make the calls it can't make? The ISTQB CT-GenAI course from Nopmark covers exactly this, taught by accredited instructors with hands-on labs, and you can see the full course at https://www.nopmark.com/pages/istqb-ct-genai

Keep reading