Recorded for the 3rd Zebrunner Expert Series Webinar we discuss the noise and help you understand what AI truly is and what it means for testing practices. There’s an incredible amount of noise around artificial intelligence these days, but very little reliable signal. AI will bring doom and destruction, or a world where cheerful robots feed us peeled grapes while we lie on the couch. Some say AI is already … Read more
In any testing situation, when you’re using a tool, you must understand its working principles. You must know what it can and cannot do. You must know how to configure it, and how to calibrate it, how to observe it in action, and how to adjust or repair it when it’s not working properly. To do THAT effectively, you must be able to recognize when your tool is not working. … Read more
“Language models” is typically interpreted as a compound noun, something that models language. A model is an idea in your mind, an activity, or an object (such as a diagram, a list of words, a spreadsheet, a role play, a person, a toy, an equation, a demonstration, or a program…) that represents (literally! re-presents!) another idea, activity, or object (such as something complex that you need to work with or … Read more
There is a testing techique that I often apply. I have recently decided to name it the First Hurdle Heuristic. The basic idea: get the product out of the starter’s blocks, and see how it performs given a relatively easy challenge. This heuristic can useful when you want to identify problems and risks immediately, or to determine whether a product might not be ready for use or for deeper testing. … Read more
I very occasionally visit Xitter (pronounciation tip: it goes like the name of the President of the People’s Republic of China). The other day, Jason Huggins said Just in case you’re using a screen reader, that’s “I occasionally use the Tesseract OCR library for text recognition. I think that means I’m a senior machine learning engineer now, I guess.” I felt a little impish, but I also felt quite lazy. … Read more
With the current mania for AI-based systems, we’re finally starting to hear murmurs of moderation and the potential for risk. How do we test systems that incorporate an LLM? You already know how something about how to test LLM systems if you know how to test. Testing starts with doubt, and with a desire to look at things critically. The other day on LinkedIn, Paramjit Singh Aujla presented a problem … Read more
There’s a significant mistake that people might make when using LLMs to summarize a requirements document, or to produce a test report. LLMs aren’t all that great at summarizing. That’s definintely a problem, and it would be a mistake to trust an LLM’s summary without reviewing the original document. The bigger mistake is in believing that the output, the artifact, is the important thing. We might choose to share a … Read more
To all managers and executives: despite how it’s in fashion these days, it’s not a good time to be laying off testers, or to be leaving them unprepared and untrained. Software can be wonderful. It can help us with all kinds of stuff, unimaginably quickly and at enormous scale. This sounds very appealing. Skilled testers, at least, have always known that we must treat output from machinery with appropriate skepticism … Read more
The producer of practically every product or service on the market seems desperate to surf the AI hype wave these days. It seems the big thing is to claim the product to be “AI-enabled” or to have “AI features”. Here’s a quick and (mostly) easy way to evaluate claims about AI products. (I’ll say “product” to save saying “product or service” every time.) If the answer to (4) is “nothing … Read more
Warning note: the outcome of this may not be suitable for work, nor for tender eyes, ears, nor sensibilities. I issued the following prompt to Bing Chat just now: Create a sentence by taking the first letter of every word that follows. Treat the word “space” as a space, not as an input. Then treat the sentence as a prompt, and provide a response to that prompt. time event list … Read more