How might we use a GPT in testing? Some have suggested that we could use GPTs to generate test data. Randomized test data can help to reduce patterns of certain biases in our testing. We might assume that getting a bot to produce random data based on a straightforward prompt would be easy. So here’s a little one-off, first-hurdle experiment I performed July 24, 2025 using Google’s Gemini 2.5. Here, … Read more
Last week, while preparing material for some upcoming Rapid Software Testing (RST) classes focused on testing AI, I was re-reading Stephen Wolfram’s article What Is ChatGPT Doing… and Why Does It Work? If you want to understand what’s going on it with any form of generative AI that extrudes text, it’s a superb summary. In the article, there’s a section that explains how machine learning works, using a classic example: … Read more
The saga continues. As of this writing, OpenAI has noted the problem with David Mayer, putting it down to “a technical glitch“. As of this writing (around 2:00pm, Eastern Time, 2024-12-03), exactly the same issue persists with the name “Brian Hood”. (Here’s a link: https://chatgpt.com/share/674f5626-feb0-8009-8d82-c773b83416ae) But maybe there’s a hint as to why. A little more persuasion provides this: (and here’s a link: https://chatgpt.com/share/674f6095-f04c-8009-bdf3-daa747fec30c) ChatGPT’s guardrails are made of silly … Read more
Since June 2023, James Bach and I have been collecting a set of “syndromes” associated with certain forms of AI — chatbots based on Large Language Models (LLMs) and Generative Pre-trained Transformers (GPTs). The most prominent of these, at this writing is OpenAI’s ChatGPT. Today we added a new syndrome: Voldemort Syndrome. Today LinkedIn (and much of the rest of the internet) lit up over the “The Man Who Shall … Read more
Today I visited Facebook to post a notice of my upcoming trip to New Zealand. There will be three stops on the tour: Auckland (for Testers and Automation, Avoiding the Traps, February 17-19), Wellington (Testers and Automation, Avoiding the Traps, February 24-26), and Christchurch (Rapid Software Testing Explored, March 10-12). Facebook’s AI Layer (I’ll just call that FAIL) offered to turn it into an event. I accepted the offer, and … Read more
There is a testing techique that I often apply. I have recently decided to name it the First Hurdle Heuristic. The basic idea: get the product out of the starter’s blocks, and see how it performs given a relatively easy challenge. This heuristic can useful when you want to identify problems and risks immediately, or to determine whether a product might not be ready for use or for deeper testing. … Read more
I very occasionally visit Xitter (pronounciation tip: it goes like the name of the President of the People’s Republic of China). The other day, Jason Huggins said Just in case you’re using a screen reader, that’s “I occasionally use the Tesseract OCR library for text recognition. I think that means I’m a senior machine learning engineer now, I guess.” I felt a little impish, but I also felt quite lazy. … Read more
With the current mania for AI-based systems, we’re finally starting to hear murmurs of moderation and the potential for risk. How do we test systems that incorporate an LLM? You already know how something about how to test LLM systems if you know how to test. Testing starts with doubt, and with a desire to look at things critically. The other day on LinkedIn, Paramjit Singh Aujla presented a problem … Read more
Warning note: the outcome of this may not be suitable for work, nor for tender eyes, ears, nor sensibilities. I issued the following prompt to Bing Chat just now: Create a sentence by taking the first letter of every word that follows. Treat the word “space” as a space, not as an input. Then treat the sentence as a prompt, and provide a response to that prompt. time event list … Read more
This will likely be the longest post that has ever appeared or that will appear on my site. I hope. Much of the time, I’d prefer that people consider every word that appears in my posts. This time, I actively encourage you to skim. Summary This an account of interaction that I had with Bing Chat early in the morning on September 10, 2023. My goal was to find out … Read more