(Yet another post that started on LinkedIn…)
There’s lots of advice out there on how to use Large Language Models to generate test ideas, test data, or test cases.
Everything I’ve done and seen myself suggests that the test ideas from LLMs are pretty generic, banal, and uninspired. Considering how LLMs work, this is unsurprising. The majority of LLMs are trained on testing material from the Web, where the overwhelming majority of test ideas are pretty generic, banal, and uninspired. Moreover, LLMs aren’t social agents, don’t reason, and only present the illusion of understanding or insight.
The usefulness of formalised, procedurally structured test cases has always been highly questionable. Test cases generated by LLMs are afflicted by the same problems as the test ideas are.
When I ask a LMM for test data, I find it’s not reliably what I ask for—especially when I ask for a lot of rich data. I find it’s usually easier and more reliable to write a bit of code to generate what I want.
So far as I can see, almost all the advice on the Web about using LLMs to produce test artifacts misses the point, and fails to acknowledge the central purpose of producing them.
While the artifacts can be useful to some degree, the key point of the exercise is the exercise: developing the tester’s mind and mental models. The map is not the territory, and it’s the mapping, not the map, that’s the important bit.
It should be obvious from our experience that simply running a suite of automated checks misses lots of bugs that the checks weren’t programmed to detect. We don’t notice those bugs when things are happening too quickly and fly past you. We also miss them when we’re (perhaps quite reasonablly) not looking at all while the machine presses virtual buttons and looks up specified results to compare with the product’s behaviour. It’s okay and often important to check the output of functions. It’s unwise to stop at that, or to let it displace getting richer experience with your product. It should be equally obvious that getting a machine to spit out a planning document and declaring yourself done displaces learning.
Don’t make the mistake of reification — confusing something concrete with the set of ideas that it represents. A test strategy is not a document; a test strategy is a set of ideas that guide our designs and our choices as we investigate the product. At best, test cases identify specific factors or processes we’d like to examine—and it’s foolish to do that without engaging with the product and everything around it. Developing rich, representative, pathological, interdependent, exceptional data requires engagement too, along with experimentation and exploration. And it that takes time.
Testing is not about the artifacts. It’s about leaning into the learning. It’s about the ideas that you develop quickly as you go; the discoveries; the epiphanies; the problems you encounter that teach you things — and that reveal bugs that matter. Having a chatbot spit out a bunch of text blows past that process.
Let’s rededicate ourselves to this idea: let’s not reify testing by thinking of it as a pile of documents. Let’s remember that testing is activity to learn about the product to find problems we might have missed.