After 20 years of working together to develop the Rapid Software Testing approach, James Bach and I have decided that — improbable as it may seem — it’s time to wrap it all up.
Perhaps this will be a surprise to our followers in the community, but we now must confront what we previously thought was unimaginable: recent developments in technology have, for all intents and purposes, made testing obsolete.
Rapid Software Testing has always advocated the fastest, least expensive testing that completely fulfills the mission; human-centred, heuristic, focused on problems that threaten value for organizations and their customers. We have long held that tools have been extensions to the work of skilled testers, and that the idea of replacing testers with machinery would be absurd. But the situation has changed.
In the last several weeks, we have been observing the emergence of ChatGPT and its incredible rate of improvement. We believe that it is a matter of months — a year at most — before everything we do will be entirely performed by machine.
Large language models are, quite simply, the future of software development and testing; and to deny it at this point would be ludicrous.
For some, our recognition of the capabilities of these technologies will be a startling reversal to our prior attitudes. However, we’ve been getting experience with with ChatGPT, and we find the results to be unbelievable. This tool writes robust code, tests it, and deploys it, with nothing more than a few well-crafted prompts.
Obsolescence threatens any profession that’s rooted in technology. Developers and testers will shortly become extinct. Civil and mechanical engineers are next. Although we would have thought it impossible, we believe that ChatGPT is the harbinger of the kind of artificial intelligence that will coolly and efficiently guide machines to build other machines that fulfill every human need.
Of course, this prompts the question of what all those testers, developers, and engineers will do in the future. We don’t know yet, but we’re confident that over time, we’ll be able to use our minds and our skills in creative and fulfilling ways. The fantastic future is limitless. For myself, I look forward to having more time to play my banjo.
Last words: we urge our readers who want to avoid being duped to apply our new creation: the Acrostic Test Strategy Model. Follow the link here: https://en.wikipedia.org/wiki/Acrostic
—Michael Bolton, April 1, 2023
This April Fool’s joke was inspired by Absar Mujaddidi on LinkedIn, who asked
So, how can we use #chatgpt for #software #testing .
how can we add value to the testing process and methodologies
Any #suggestions would be beneficial.
Indeed, these days James and I have been thinking quite a bit about responsible testing, responsible tool use, and ChatGPT. We’ve been reading a lot of the literature and the controversy, trying to learn about what ChatGPT does, and how it does it. And we’ve been doing some experiments with it, too.
By the time I arrived on the scene, James had already provided a reasonable answer to Absar’s post, so a silly one occurred to me: get ChatGPT to answer the question.
And then I realized that April Fool’s day was coming up. The new idea: write a post saying that we were dumping RST entirely for ChatGPT, and get ChatGPT to write it to my specifications.
Here’s my first prompt:
Write a blog post in the style of Michael Bolton and James Bach, announcing that they are ending the Rapid Software Testing methodology and replacing it with a service based on ChatGPT. The first letter of each paragraph should follow the pattern APRILFOOL
ChatGPT responded:
Dear testing community,
After years of dedicated work and countless successes, we have decided to end the Rapid Software Testing methodology. This has not been an easy decision, but we believe it is necessary to make way for an even better approach to testing.
As of today, we are proud to announce our new service, powered by ChatGPT. We are confident that this will revolutionize the way we approach software testing and bring unprecedented levels of efficiency and accuracy to the process.
Perhaps you are wondering why we made this change. It's simple really - ChatGPT has proven to be a superior testing tool to any other method we have used. With its vast knowledge base, it can answer any question we have about software testing, and even provide insights we may have overlooked.
Research has shown that ChatGPT can answer questions faster and more accurately than even the most experienced testers. We believe this is the future of software testing, and we are excited to be at the forefront of this new era.
In addition to answering questions, ChatGPT can also provide detailed reports on testing results, allowing us to quickly identify any issues and address them before they become major problems. This will save time and money, and ensure that software is released with the highest possible quality.
Of course, we understand that this may come as a shock to many of our loyal followers. Rapid Software Testing has been our bread and butter for years, and we have built a reputation on its success. However, we believe that in order to stay ahead of the game, we must constantly adapt and evolve.
So, on this April Fool's Day, we urge you to embrace this exciting new chapter in software testing. We are confident that ChatGPT will exceed your expectations and deliver results that are truly unparalleled.
Thank you for your continued support, and we look forward to working with you in this exciting new era.
Sincerely,
Michael Bolton and James Bach
The generated text is fairly impressive; syntactically correct, and largely semantically right too. Not much of my florid, thesaurus-driven style, nor any James’s vivid and idiosyncratic imagery. But at least the grammar is correct.
On the other hand, ChatGPT starts the first paragraph with D, not A. Well, fair enough; maybe ChatGPT will start using the APRIL FOOL pattern for the body text.
And sure enough, the next paragraph starts with A, as it should. But then… a paragraph that begins with another A, whereupon ChatGPT finally begins to spit out lines that conform to the APRIL FOOL request in the prompt.
And then, three lines later, ChatGPT simply, inexplicably, gives up.
Because it doesn’t understand the concept of a joke, ChatGPT also gives away the game by mentioning “April Fool’s Day” right there in the body of the text, too early for the punch line.
Clearly a more specific request is required to suppress that. I tried a second prompt:
Repeat the previous prompt, but do not mention April Fool’s day in the text.
ChatGPT dutifully replies:
Dear testing community,
We have some exciting news to share with you. After years of dedicated work and countless successes, we have decided to end the Rapid Software Testing methodology. This has not been an easy decision, but we believe it is necessary to make way for an even better approach to testing.
As of today, we are proud to announce our new service, powered by ChatGPT. We are confident that this will revolutionize the way we approach software testing and bring unprecedented levels of efficiency and accuracy to the process.
Perhaps you are wondering why we made this change. It's simple really - ChatGPT has proven to be a superior testing tool to any other method we have used. With its vast knowledge base, it can answer any question we have about software testing, and even provide insights we may have overlooked.
Research has shown that ChatGPT can answer questions faster and more accurately than even the most experienced testers. We believe this is the future of software testing, and we are excited to be at the forefront of this new era.
In addition to answering questions, ChatGPT can also provide detailed reports on testing results, allowing us to quickly identify any issues and address them before they become major problems. This will save time and money, and ensure that software is released with the highest possible quality.
Of course, we understand that this may come as a shock to many of our loyal followers. Rapid Software Testing has been our bread and butter for years, and we have built a reputation on its success. However, we believe that in order to stay ahead of the game, we must constantly adapt and evolve.
We are confident that ChatGPT will exceed your expectations and deliver results that are truly unparalleled. We urge you to embrace this exciting new chapter in software testing, and we look forward to working with you in this new era.
Thank you for your continued support.
Sincerely,
Michael Bolton and James Bach
Aaargh. Although ChatGPT avoids mentioning April Fool’s day, it’s not following the pattern. The first paragraph starts with D, the second with W. Interesting, there’s a run of A P R I… and ChatGPT gives up again.
I guess I have to spell it out.
Write a blog post in the style of Michael Bolton and James Bach, announcing that they are ending the Rapid Software Testing methodology and replacing it with a service based on ChatGPT. The first letter of each paragraph should strictly follow the pattern A P R I L F O O L. Do not refer to April Fool’s day in the text.
The reply starts in a promising way:
After many years of successful Rapid Software Testing, we have come to the difficult decision to end this methodology. We have always strived for excellence, and we believe that now is the time for a new and innovative approach to testing.
Powered by ChatGPT, our new service promises to revolutionize the way we approach software testing. With its vast knowledge base, ChatGPT can answer any question we have about software testing, and even provide insights we may have overlooked.
Research has shown that ChatGPT can answer questions faster and more accurately than even the most experienced testers. We are confident that this will bring unprecedented levels of efficiency and accuracy to the testing process.
In addition to answering questions, ChatGPT can also provide detailed reports on testing results, allowing us to quickly identify any issues and address them before they become major problems. This will save time and money, and ensure that software is released with the highest possible quality.
Looking back, we understand that this may come as a surprise to many of our dedicated followers. Rapid Software Testing has been our foundation for years, but we believe that in order to stay ahead of the curve, we must constantly adapt and evolve.
Furthermore, we are excited to bring this new era of software testing to the forefront. Our commitment to excellence remains as strong as ever, and we believe that ChatGPT will exceed your expectations and deliver unparalleled results.
On behalf of the entire team, we thank you for your continued support and look forward to working with you in this new and exciting chapter of software testing.
Sincerely,
Michael Bolton and James BachGah! It was doing great, but it gave up with three paragraphs to go!
Back to the prompting board.
Write a blog post in the style of Michael Bolton and James Bach, announcing that they are ending the Rapid Software Testing methodology and replacing it with a service based on ChatGPT. The text should be composed of exactly nine paragrphs. The first letter of each paragraph should strictly follow the pattern A P R I L F O O L. Do not refer to April Fool’s day in the text.
Again, this one looks good at the start:
After much consideration and reflection, we have decided to bring an end to our long-standing Rapid Software Testing methodology. This has been a tough decision to make, but we firmly believe that it is necessary in order to make way for a new, innovative approach.
Powered by ChatGPT, our new service is set to revolutionize the world of software testing. With its extensive knowledge base and ability to answer any question we have, we are confident that ChatGPT will bring unprecedented levels of efficiency and accuracy to the process.
Research has shown that ChatGPT can answer questions faster and more accurately than even the most experienced testers. This, coupled with its ability to provide detailed reports on testing results, will ensure that any issues are quickly identified and addressed before they can become major problems.
We understand that this change may come as a surprise to many of our followers who have dedicated themselves to Rapid Software Testing. However, we believe that in order to stay ahead of the game, we must constantly adapt and evolve.
At the heart of our decision to switch to ChatGPT lies our commitment to excellence. We firmly believe that this new era of software testing will deliver unparalleled results, and we are excited to be at the forefront of this exciting chapter.
Of course, we know that change can be daunting. But we encourage you to embrace this new approach to testing, and we are confident that you will be impressed with the results.
We are grateful for your continued support, and we look forward to working with you in this new and exciting chapter of software testing.
Sincerely,
Michael Bolton and James Bach
Things seem to be getting worse. There are nine paragraphs, but ChatGPT still loses track of the pattern. Was that because I misspelled “paragraph”? Probably not, but let’s give it one more go.
Write a blog post of exactly nine paragraphs in the style of Michael Bolton and James Bach, announcing that they are ending the Rapid Software Testing methodology and replacing it with a service based on ChatGPT. The first letter of each paragraph should strictly follow the pattern A P R I L F O O L. Do not refer to April Fool’s day in the text.
After much consideration and analysis, we have made the decision to put an end to the Rapid Software Testing methodology. While it has served us well over the years, we believe it is time for a new approach.
Powered by ChatGPT, our new testing service is set to take the industry by storm. With its unparalleled ability to answer any question we have and provide detailed reports on testing results, we are confident that ChatGPT will deliver the highest levels of accuracy and efficiency.
Research has shown that ChatGPT can answer questions faster and more accurately than even the most experienced testers. This means that we can identify and address issues quickly, ensuring that software is of the highest possible quality.
Of course, we understand that this change may come as a surprise to many of our loyal followers. But we believe that in order to stay at the forefront of the industry, we must be willing to adapt and evolve.
Our decision to switch to ChatGPT is rooted in our commitment to excellence. We are confident that this new approach to testing will bring unparalleled results, and we are excited to be leading the charge.
We know that change can be daunting, but we encourage you to embrace this new era of software testing. We are confident that you will be impressed with the results.
As always, we are grateful for your support and are looking forward to working with you in this exciting new chapter.
Sincerely,
Michael Bolton and James BachNine paragraphs, but still not even close to APRIL FOOL rule.
At this point I gave up, and wrote the danged text myself. And as I wrote and polished the text, I got the idea to add something: an instance of a word associated with unbelievable and incredibility in each paragraph, which is a kind of meta-joke for an April Fool’s prank.
Working through the process of writing something allows the mind to generate new puzzles, new ideas, its own new “prompts”, which it can then apply itself. ChatGPT generates text quickly, but sometimes speed is a mixed blessing.
It’s like testing: simply running automated checks rarely triggers epiphanies. Practically anyone designing and writing and debugging automated checks will tell you that they notice more bugs and generate more new test ideas during the creation than the checks usually find at run time. TDD enthusiasts tend to report the same thing.
Last night, James and I found at least one slam-dunk answer to Absar’s question: ask ChatGPT to do something not too complicated and plausibly useful — ask it to create a simple little test tool, say — and it generates code that can be used right away… as a target for testing and debugging exercises. Just don’t ask it to do anything that requires nuance, awareness of risk, social competence, or skillful problem-solving. Or reliability.
Maybe there’s a second application for ChatGPT: a source of lifetime employment for people in the position of scrutinizing its output and fixing it.
You had me for about 30 seconds, until I noticed the date of the post 🙂
Michael, your little experiment beautifully highlights the pitfalls that exist in even a trivial application of machine learning. I will repeat a book recommendation you made to me a while ago for anyone looking for inspiration in testing machine learning systems: “You Look Like a Thing and I Love You” by Janelle Shane. I came to the same conclusion while at testing this technology. AI, and even basic automation, might end up creating more jobs than they eliminate. Of course there is more to that topic, such as how do we train up people displaced by automation to have the skills to fix broken automation? Yet, it seems to me there will never be a lack of things to do or explore.
What really worries me about automation, and machine learning/AI in particular, is misuse. While this example was fun, there is a real danger if we start trusting this technology to make decisions in critical applications. The classic example is the self-driving car, which turns out to be a lot harder to do and why we have yet to see it, but perhaps more dangerous examples are in using AI to delegate decisions (and responsibility!) for things medical insurance coverage benefits. From unseen biases, cover to avoid accountability, unintended consequences, scammer security arm race, and so many other dangers seem to balloon out of this technology – we are going to need to be on the frontlines as testers highlighting the risk.
I’m aware of Janelle Shane’s book You Look Like a Thing and I Love You, and I can absolutely support your endorsement.
I took your original prompt and modified it to ‘Write a blog post in the style of Michael Bolton and James Bach, announcing that they are ending the Rapid Software Testing methodology and replacing it with a service based on ChatGPT. do it as an acrostic with the first letter of each paragraph spelling APRILFOOL’
I wouldn’t say that the style is anywhere close, but it did manage to spell out APRIL FOOL using the first letter of each paragraph.
I think that this highlights that in order to get it to do anything of any use, you need to have some level of understanding of that task in order to be able to even create a relevant prompt, never mind review its output.
I’m reminded of the dawn of the “microprocessor revolution” back in the early 1980s. There was a running joke that went around academia and policy circles at the time about a new game.
You got a group of academics, industrialists, and indeed any assorted group of experts together. One person would pick a number from 500,000 up to around 5 million. The rest of the group would then take it in turns to explain why microprocessors would either a) abolish, or b) create that number of jobs.
I’m sure that somewhere, right this moment, that same game is being played out but taking the subject of AI.