A One- or Two-Day Rapid Software Testing Class
Over the last few years, “AI”—artificial intelligence—has gone from a set of emerging technologies to an industry obsession. Developers are being asked to develop AI-based products and to use AI in their work — and to accept responsibility for the outcome. Testers are being pressed to test products with AI features, and to apply AI to testing tasks. Meanwhile, there is uncertainty and controversy about what “AI” even means, since AI is a marketing term, rather than an engineering term.
Whether designed for classification or prediction, or for generating text or code, all forms of AI have this in common: they are black boxes, imbued with magical behaviours and properties, whose behaviour is neither controlled, nor understood, nor otherwise known to be safe. When applying AI to anything, there’s question that cannot be avoided: is it doing a good job? A responsible approach to AI products and tools requires knowing enough about them to be aware of the risks—which means you must know how to test them, to look at them critically and avoid the traps. This class is designed to help you to learn how.
Traditionally, much testing has been framed as formalized, procedurally structured test cases that check output from deterministic functions against specific, prescribed results. That won’t work for GPT-based technologies whose output is non-deterministic by design. The Rapid Software Testing approach (RST) is designed to address that problem by helping you learn you to test for real, starting with this fundamental idea: testing is evaluating a product by learning about it through experiencing, exploring and experimenting.
In this class, offered over one or two days, we’ll help you—and through you, your organization—to test AI-based technologies and apply them responsibly in your work, looking through the lenses of the RST methodology. We’ll work through exercises, experiments, and discussions to develop expertise in revealing and observing problems and risks.
Maybe you’re a coder; maybe you’re not. This class is designed to help you either way. If you don’t write code, we don’t teach you how, but we will help you learn to work immediately and productively with people who do.
This class is not affiliated with any tool vendor. We’ll use tools in class, but we don’t teach you to operate any specific tool. We may demonstrate or mention particular tools during the class, but we have not accepted and will not accept any payment or benefit from commercial interests. (We may be biased, but we have not been bribed.) We can show you how to analyze vendor claims critically.
In short, this class is about the essence of testing and AI: risk focus, setting goals, and developing strategies that make sense for you and your organization.
Rapid Software Testing Focused: AI can be taught onsite and in person, or online.
About the Authors
This class has been co-developed by James Bach and Michael Bolton, the authors of the Rapid Software Testing methodology.
Michael Bolton started in technology work as a programmer in 1988. Since then, he has worked in testing, program management, customer support, and documentation, developing and using tools all the way along.
James Bach is a developer-turned-tester involved with automation in testing since 1987. James’ team was among the first to use spreadsheets to implement data-driven and keyword-driven automation. One of his most popular articles ever was Test Automation Snake Oil, written about the exaggerations and lies told by test tool companies in the 1990’s — the same silliness common among tool vendors today.
Who Should Take This Training
RST Focused: AI is for you if…
- you are a tester, developer, or manager responsible for testing an AI-based product. We will help you develop approaches to test capably and efficiently, with a focus on problems and business risk.
- you are a tester or developer under pressure to use AI in your work. We will help you to evaluate and apply AI-based tools in responsible ways.
- you are a developer or a tester who is comfortable with writing code. We will help you see many ways of applying your coding skills to testing.
- you are a tester who does not code. We will help you understand possibilities and challenges of creating automation, and help you to learn to ask for what you need.
- you are a quality coach or manager who is responsible for testing an AI-based product, or applying AI-based tools in testing work. We will help you create a strategy that avoids the common traps.
- you are a supporter of skilled, responsible testing, and you feel that you are under attack by technologists who think they can automate everything. We will help you defend your team and your work.
Goals of RSTF-AI
- The primary goal of this class is to teach you how to plan and administer an effective strategy for working with AI software testing.
- A secondary goal is to help you avoid common traps problems that undermind the quality of AI-based products and of testing work.
Main Topics Covered
This class is taught Socratically, with exercises, discussions and illustrations of automation within the RST methodology. Class discussions and debate address students’ questions and specific needs. We all learn from the unique perspective that each student brings to the class.
Here are some of the topics we can cover based on time and on the needs of the participants:
- Six factors that make AI a special testing problem
- Strategies for testing AI-based products, including machine learning models, LLMs, and generative AI
- How you might apply AI in testing work—and how AI can go wrong
- Approaches for interacting with AI safely and productively for testing work
- “LLM Syndromes”—patterns of undesirable behaviour in GenAI
- Exercises in evaluating output from GenAI
- Thinking critically about “ROI” — really cost, value, and risk — when considering AI in production or testing
- Demonstrations of creative ways to apply tools to testing
- Thinking critically about the claims made by commercial tool companies and other AI promoters
How This Class Compares To Our Other RST Classes
We talk a lot about test strategy and a little about automation in each of our classes. This class focuses on incorporating tools and automation into your test strategy. However, this class does not attempt to teach you the mechanics of how to code or how to test.