Michael Bolton ChatGPT may produce inaccurate information about people, places, or facts. https://chat.openai.com/ Testing work comes with a problem: the more we test, the more we learn. The more we learn, the more we recognize other things to learn. When we investigate a problem, there’s a non-zero probability that we’ll encounter other problems — which in turn leads to the discovery of more problems. In the Rapid Software Testing namespace, we’ve come … Read more
Code is created by people. HTML elements are code, which is created by people. As part of developing those elements, people can tag them with attributes (including but not limited to “id”) that make them easy to find, via tools, for testing purposes. This can be done deliberately and consciously, or automatically as part of the creation of the element. When that doesn’t happen, testers and developers have difficulty locating … Read more
In preparation for our automation and management classes, James Bach and I are currently analysing ROI calculators (“Here’s what you can save over N years using our product!”) and we’re going deep on reverse-engineering one from a prominent tool vendor. Opaque formulas; undefined and unexplained terminology; unnamed and inexplicable constants; weird, inexplicable coefficients in the calculation; poor testability; hilarious bugs sitting right there on the surface; impossible conclusions due to … Read more
In the previous post, I described ChatGPT as being a generator of bullshit. Some might say that’s unfair to ChatGPT, because bullshit is “speech intended to persuade without regard for truth”. ChatGPT, being neither more nor less than code, has no intentions of its own; nor does it have a concept of truth, never mind regard for it, and therefore can’t be held respsonsible for the text that it produces. … Read more
A little while back, on LinkedIn, Jason Arbon posted a long article that included a lengthy conversation he had with ChatGPT. The teaser for the article is “A little humility and curiosity will keep you one step ahead of the competition — and the machines.” The title of the article is “Testing: Bolt-on AI” and in Jason’s post linking to it, I’m tagged, along with my Rapid Software Testing colleague … Read more
The other day on LinkedIn, Wayne Roseberry posted a puzzle that (he says) ChatGPT solved correctly. Here’s the puzzle. “Bob and Alice have a rectangular backyard that has an area of 2500 square feet.Every morning, Alice walks the 50 feet from her back door to the neighbor to pick up their laundry as well. What is the longest straight line that can bisect Bob and Alice’s back yard?” According to … Read more
This ChatGPT getting dumber? This paper raises the question; this blog post questions the conclusions; and this article has more to say. That’s not a very useful question, because “dumber” is not exactly a property of ChatGPT (or anything else). It’s a set of relationships between ChatGPT’s behaviour; people’s notion(s) of dumb and smart; and the context. Evaluating that requires a complex set of perspectives, values, and social judgements. For … Read more
Yet another post plucked and adapted from the walled garden of LinkedIn “What the large language models are good at is saying what an answer should sound like, which is different from what an answer should be.” —Rodney Brooks, https://spectrum.ieee.org/gpt-4-calm-down Note for testers and their clients: the problem that Rodney Brooks identifies with large language models applies to lots of test procedures and test results as well. People often have … Read more
“A test that is defined in terms of one expected result is undefined against the other types of results available from that test.” —Cem Kaner, 2004 (https://lnkd.in/gjFZYNGs) Almost 20 years on, that message is still lost on many testers, developers, and managers. Yet, as of today, we have another chance to acknowledge it and to spread the word! Excellent testing is not really about obtaining the expected result. Excellent testing … Read more
This post started as a LinkedIn post, which got started as a comment replying to this poll: It’s depressing to see ideas about testing and risk reduced to dopey formulas that are great for softball “certification” exam questions, but terribly limited for investigating and revealing product and business risk. Here’s how we describe “boundary” in Rapid Software Testing: A means by which something is classified or filtered This means that … Read more