Risk-based testing: check your scores against escaped defects
Take a QA lead whose team uses risk-based testing to plan each sprint. Every story gets a risk score, the team tests the high scorers first, and the low scorers…
You have read this

Locator maintenance has been the burden on UI automation for about as long as UI automation has existed. Self-healing test automation is the current answer to it, and the pitch is a good one: when a locator breaks, the framework repairs it and the suite keeps running. However it is worth being precise about what that repair is allowed to do without asking you. A healing layer that picks the wrong element will report a pass, and nothing in your pipeline will tell you it happened.
Pranav Pandit, writing on Ministry of Testing on August 4, 2026, describes a single button rename that broke more than 40 tests at once. None of those 40 tests were testing the button. They were testing checkout, permissions, search, a dozen unrelated things, and they all went red because one string in one attribute changed underneath them.
A team hits that on a Monday, spends the morning on find-and-replace across a test repository, and gets no new coverage out of the day. Multiply it across a year of an interface that ships weekly and you can see why "the framework repairs its own locators" sells so well. Pandit calls it "the hidden tax of UI automation", and he is right about that.
Selenium teams carried this cost for 15 years. Playwright reduced it with better selectors and auto-waiting, but it did not remove it, because the underlying issue sits below the tool. A test refers to an element by a description of that element, and that description is owned by a dev that may have no idea your test exists.
So a recovery mechanism is a reasonable thing to want. The question is what that mechanism is allowed to do on its own.
When your locator fails, the healing layer looks at the page, finds the element it believes you meant, and substitutes it. If it is right, your test proceeds and passes, and that is the demo everybody has seen. If it is wrong, your test also proceeds. It clicks a different button. It fills a different field. It asserts against a different node. And depending on what your assertions actually check, it can still finish green.
A test suite exists to interrogate the software on your behalf while you sleep, and the only signal it produces is a binary one. Red means look at this. Green means carry on. A healing layer with no gate on it converts a category of red into green, silently, in the one direction you can never audit later.
So what does that suite still need to provide you at that point? What did it verify, and what did it only appear to verify? You cannot tell from the result, and you will not reconstruct it afterwards either, because the substitution may have happened at run time and left nothing behind. A failing test tells you something changed. A test that heals wrongly and passes reports that nothing changed, and your release process acts on that report.
Pandit's own conclusion after building one of these is that the context you feed the model governs the result, and the architecture he lands on reaches for AI only after the deterministic path has already failed. There are four rules that follow from that, and the ordering is also important.
Three of those four rules have nothing to do with artificial intelligence. They are about determinism, scope and reproducibility, which are the properties you would demand of any other piece of test infrastructure.
So what should a team have in hand at the end of a sprint running one of these? They should have a recovery log that records every attempt: which locator failed, what replaced it, what the validation gate saw, and whether the substitution held.
Each line on that log is one of two things. Either the interface changed and your test should be permanently updated to match, or your test was making an assumption about the page that was never safe and it got away with it for a while. Either way there is work to do, and a silent heal hides both cases equally well.
James Cullimore's idea from Software Testing Weekly #321 fits here, because he adds confidence ratings alongside coverage rather than reporting coverage as a flat number. A suite where 12% of last week's passes went through a repair path is in a different condition from one where none did, and a coverage report showing the same figure for both is not telling you the truth about your own tests.
A healed locator is a to-do item, and the weekly review is where it gets closed out. Promote the good repairs into the codebase where they become ordinary reviewed code, and delete the tests whose repairs you cannot justify.
Most teams will not build this. They will switch it on inside a commercial platform, which is fine, and it does mean the gate above is now someone else's implementation detail. So ask directly, and ask for answers you can verify rather than a marketing page:
A suite you can run twice, once with recovery on and once with it off, gives you the difference between what your tests verify and what your tooling is papering over. If you cannot get that number, you do not know what your regression suite is worth.
Setting the level of rigor here is a judgment call. You have to look at a mechanism, work out where it quietly changes the meaning of a passing test, and tailor the rigor to what a wrong answer would cost you. Nopmark's ISTQB CT-GenAI course covers testing systems that use AI and using AI inside your own testing, including where a model's output needs a deterministic check wrapped around it before anyone acts on it. If you are being asked to sign off on an AI-assisted testing capability this quarter, the course works through those decisions directly.
Is self-healing test automation worth enabling at all? On a large UI suite against a fast-moving interface, yes, provided the healing is gated and logged. On a suite of 40 tests, the maintenance you save will not pay for the review process you now owe.
Does Playwright have built-in self-healing? No. Playwright reduces locator fragility through user-facing selectors and auto-waiting, and its strict mode fails loudly when a locator is ambiguous. That strictness is a feature to preserve.
How do I know if a healed test is still testing the right thing? Only the recovery log will tell you. The result comes back green either way, so reading the substitution is the only way to close the question out.
What is the difference between self-healing and flaky-test retries? A retry runs the same test again and hopes for a different outcome. A heal changes what the test does and then runs it. Retries hide timing problems, heals hide interface changes, and both of them work by turning a signal into silence.
Can I audit heals after the fact if my platform does not export them? You cannot do it reliably. This is why the export question belongs in the evaluation and not in the first incident review after something reaches production.
Switch healing on without the gate and the log, and the failures you stop seeing are the ones that used to tell you the interface moved, so the first person to notice becomes a customer rather than your CI. Nopmark's ISTQB CT-GenAI certification training is taught by accredited instructors and works through where a model's output needs a hard deterministic check around it before anyone acts on it.
Take a QA lead whose team uses risk-based testing to plan each sprint. Every story gets a risk score, the team tests the high scorers first, and the low scorers…
You have read this