In any testing situation, when you’re using a tool, you must understand its working principles. You must know what it can and cannot do. You must know how to configure it, and how to calibrate it, how to observe it in action, and how to adjust or repair it when it’s not working properly.
To do THAT effectively, you must be able to recognize when your tool is not working. You must be able to recognize when it might be fooling you into believing that is working when it isn’t. You must be prepared to test your testing tools (examples here and here). You must become alert to bugs in it.
Now: thanks to the gold rush, there is a big push from executives, managers, and from some testers themselves to use AI in testing.
If you want to use AI in testing, you must be able to test AI-based systems; otherwise, you run the serious risk of being fooled. If you want to be able test AI-based systems, you must know how to test.
Automated output checks have never represented particularly deep testing. They are completely powerless when we’re trying to test something whose input is free-form and whose output is non-deterministic and variable by design. Green bars based on consistent output simply aren’t available here.
The good news is we’ve known for a long time what it means to test, and how to do it. Testing is the enactment of critical thinking and risk investigation about a product. Testing is the process of evaluating a product by learning about it through experiencing, exploring and experimenting. That includes to some degree questioning, studying, modeling, observation, inference, etc. — with the intention of finding out the true state of the product or system.
A tester doesn’t see a pleasing demo and exclaim “Marvelous, marvelous!” A tester must be professionally skeptical when everyone around us is dazzled, distracted, and determined that everything is okay. A tester tests (lots of examples here).
Testing, the way we see it, starts with a premise: there is a risk of problems in the product that have escaped everyone else’s notice, because no one else on the project focuses on investigating the possibility of trouble as their primary role.
That’s why it’s important to have people on your project, and in your organization, who are trained to think like testers. When you don’t focus on looking for trouble, trouble will cheerfully come looking for you… or your clients.
(A prototype version of this post appeared on LinkedIn.)
I would add that you need to know what a testing tool is really good at and appropriate for the testing being done.
A shiny new tool or a testing tool you are familiar with might not be the best one for the job and you have to think about that. I learnt the hard way.