AI Governance Testing: The Evidence a Tester Can Own

AI governance testing: a test lead files a tab into an evidence binder beside a monitor of line charts
In this post
  1. What AI governance testing asks of a test team
  2. Map your tests to the NIST AI RMF Measure function
  3. A worked example: a returns fraud model
  4. Who owns the evidence on your team?
  5. Sources

From 2012 to 2020, Rite Aid used facial recognition in hundreds of its stores to flag people it suspected of shoplifting. In December 2023, the US Federal Trade Commission said in a complaint that the system had produced thousands of false matches. Some of what went wrong, as the FTC tells it, was missing tests and monitoring. That makes the case a good place to start on AI governance testing.

According to the complaint, employees acted on false alerts. They followed customers around stores, searched them, told them to leave, and sometimes called the police. The FTC said false matches were more likely at stores where Black or Asian residents were the largest group in the area. They were less likely where white residents were the largest group. To settle the charges, Rite Aid agreed to a five-year ban on using facial recognition for surveillance.

What AI governance testing asks of a test team

AI governance means the rules, roles and records a company uses to decide if an AI system is safe to use. Much of it is policy work. Several failures on the FTC's list concern missing test evidence. Others involve risk mitigation, image quality, training and oversight, which testing can support but not replace. The FTC said the company did not "test, assess, measure, document, or inquire about the accuracy" of the system before using it. It also said Rite Aid did not regularly monitor or test accuracy after deployment. That included having no procedure for tracking the rate of false matches.

These are FTC allegations that Rite Aid settled, not findings after a trial. This post does not guess at what happened inside the company. It uses the case to show the evidence a team could have kept.

A team may have a governance policy and a review board. Without test evidence, that board might still be approving a system nobody has interrogated. Testers already produce this kind of evidence in their test cases, expected results and test logs.

Map your tests to the NIST AI RMF Measure function

The NIST AI Risk Management Framework (AI RMF 1.0, NIST AI 100-1) came out on January 26, 2023. It is voluntary guidance, not a law.

The framework has four functions: Govern, Map, Measure and Manage. Measure is the one closest to test work. NIST describes it as using "rigorous software testing and performance assessment methodologies." It names TEVV, short for test, evaluation, verification and validation.

A test lead does not need to own the whole framework. Several Measure items in the AI RMF Core line up with work testers already do:

  • MEASURE 2.1: test sets, metrics and the tools used during TEVV are documented. In plain words, you write down what you tested with, how you scored it, and which tool ran it.
  • MEASURE 2.3: performance is shown under conditions like the real deployment. Blurry store camera images are part of the real deployment, so they belong in the test set.
  • MEASURE 2.5: the system is shown to be valid and reliable, and its limits are documented. This is where a tester states what the results do not cover.
  • MEASURE 2.11: fairness and bias are evaluated and the results are documented. For a tester, that means splitting error rates by group instead of reporting one average.
  • MEASURE 2.4: the system's behavior is monitored once it is in production. Testing does not stop at release.

The FTC's proposed order reads like the same list. If Rite Aid uses a covered biometric system again, it must test it before deployment and at least once every 12 months. The tests must copy real store conditions, measure how often outputs are wrong, and check whether that rate differs by race, sex, age or other traits.

A worked example: a returns fraud model

Take an online store that uses a model to flag product returns as possible fraud. A flagged return goes to a person for review, and the customer may wait days for a refund. The model might score thousands of returns a week. Here is the evidence a tester could own, item by item.

Build a test set from past decisions. Pull returns where an investigator checked the final outcome and confirmed fraud or an honest return. A refund paid with no complaint is not proof, so mark those cases as unknown and leave them out. That confirmed outcome is your test oracle, which means the source you trust to decide whether the model was right. Include messy cases, like blurry photos and missing order details, because the real queue has them.

Measure false positives by group. A false positive here is an honest customer flagged as fraud. Report that rate per region, per payment type and per customer age band, not as one overall score. If one group gets flagged twice as often, the review board needs to see it before release.

Write a test summary report the board can read. Name the test set version, the date, the model version, and the tool that ran the tests. Give the counts behind every rate. List the real-world conditions you tested, such as blurry photos and missing order details. This report adds evidence toward MEASURE 2.1 and 2.3 for that release.

State the limits. Say which groups had too few cases to measure, which return types were left out, and how old the data is. This adds evidence toward MEASURE 2.5. Showing that the model is valid and reliable takes more. The team also tracks how many confirmed fraud cases the model missed (the missed-fraud count, or recall), because a model that flags nothing would have a perfect false positive rate. The pass/fail thresholds are agreed with the business before the run.

Track false positives after release. When a reviewer clears a flagged return as honest, log it as a false positive. The false positive rate is those false positives divided by all honest returns that week, flagged or not. Nobody can check every unflagged return, so have an investigator check a random sample of them. Then scale up: if 1,000 returns went unflagged and 96 of a 100-return sample were honest, estimate 960 honest unflagged returns. Add the reviewer-cleared false positives to get the denominator, and count both from the same week's returns. Chart that estimated rate every week. The FTC complaint said Rite Aid later added a way for employees to report a "bad match," but did not make sure they used it. A tracked rate is only useful if someone checks that the reviewers are logging it.

Watch for concept drift. Fraud patterns change, and a model trained on last year's returns may slowly get worse. The ISTQB AI Testing syllabus (CT-AI) covers testing for concept drift. A rising false positive rate in the weekly chart can be a reason to check for drift. It can also come from new kinds of returns, labeling mistakes or a changed alert threshold.

Route complaints back into the test set. When a customer disputes a flag and wins, add that return to the test set with its correct outcome. Then the next model version is tested against a failure that already happened once.

Who owns the evidence on your team?

On many teams, policy writers, lawyers and managers may own most of the governance program. The test team can own the record that shows the AI system was checked. That means the test set, the oracle, the error rates by group, the stated limits and the production tracking. If your team wants to build these skills on its own product, talk to Nopmark about team training.

When that record exists, a review board can approve a release based on what was tested. When a complaint arrives, the team can show the error rate for that group, when it was last measured, and what changed. When it does not exist, the company may have nothing to show a regulator who asks how accuracy was tested.

Sources

Keep reading