How might we use a GPT in testing? Some have suggested that we could use GPTs to generate test data. Randomized test data can help to reduce patterns of certain biases in our testing. We might assume that getting a bot to produce random data based on a straightforward prompt would be easy. So here’s a little one-off, first-hurdle experiment I performed July 24, 2025 using Google’s Gemini 2.5. Here, … Read more
Last week, while preparing material for some upcoming Rapid Software Testing (RST) classes focused on testing AI, I was re-reading Stephen Wolfram’s article What Is ChatGPT Doing… and Why Does It Work? If you want to understand what’s going on it with any form of generative AI that extrudes text, it’s a superb summary. In the article, there’s a section that explains how machine learning works, using a classic example: … Read more
In our forthcoming book, Taking Testing Seriously: The Rapid Software Testing Approach, James Bach and I have included a chapter on AI. AI is fraught with risk, but writing about it is too. All through its history (since the 1950s, NOT just since 2022 or 2012), “AI” has not been an engineering term, but a marketing term, without clear notions of what “artificial intellience” really means. And all along, the … Read more
The saga continues. As of this writing, OpenAI has noted the problem with David Mayer, putting it down to “a technical glitch“. As of this writing (around 2:00pm, Eastern Time, 2024-12-03), exactly the same issue persists with the name “Brian Hood”. (Here’s a link: https://chatgpt.com/share/674f5626-feb0-8009-8d82-c773b83416ae) But maybe there’s a hint as to why. A little more persuasion provides this: (and here’s a link: https://chatgpt.com/share/674f6095-f04c-8009-bdf3-daa747fec30c) ChatGPT’s guardrails are made of silly … Read more
Since June 2023, James Bach and I have been collecting a set of “syndromes” associated with certain forms of AI — chatbots based on Large Language Models (LLMs) and Generative Pre-trained Transformers (GPTs). The most prominent of these, at this writing is OpenAI’s ChatGPT. Today we added a new syndrome: Voldemort Syndrome. Today LinkedIn (and much of the rest of the internet) lit up over the “The Man Who Shall … Read more
Today I visited Facebook to post a notice of my upcoming trip to New Zealand. There will be three stops on the tour: Auckland (for Testers and Automation, Avoiding the Traps, February 17-19), Wellington (Testers and Automation, Avoiding the Traps, February 24-26), and Christchurch (Rapid Software Testing Explored, March 10-12). Facebook’s AI Layer (I’ll just call that FAIL) offered to turn it into an event. I accepted the offer, and … Read more
At the Pacific Northwest Software Quality Conference in October 2024, I gave a keynote presentation titled “What Are We Thinking in the Age of AI?“ There’s a lot to think about, and for testers, there’s a lot to do. For one, we need to understand the basis for the “AI” claim. Any kind of software can be marketed as “AI”, since it’s doing something that (presumably) a human could do, … Read more
In any testing situation, when you’re using a tool, you must understand its working principles. You must know what it can and cannot do. You must know how to configure it, and how to calibrate it, how to observe it in action, and how to adjust or repair it when it’s not working properly. To do THAT effectively, you must be able to recognize when your tool is not working. … Read more
“Language models” is typically interpreted as a compound noun, something that models language. A model is an idea in your mind, an activity, or an object (such as a diagram, a list of words, a spreadsheet, a role play, a person, a toy, an equation, a demonstration, or a program…) that represents (literally! re-presents!) another idea, activity, or object (such as something complex that you need to work with or … Read more
There is a testing techique that I often apply. I have recently decided to name it the First Hurdle Heuristic. The basic idea: get the product out of the starter’s blocks, and see how it performs given a relatively easy challenge. This heuristic can useful when you want to identify problems and risks immediately, or to determine whether a product might not be ready for use or for deeper testing. … Read more