I very occasionally visit Xitter (pronounciation tip: it goes like the name of the President of the People’s Republic of China). The other day, Jason Huggins said Just in case you’re using a screen reader, that’s “I occasionally use the Tesseract OCR library for text recognition. I think that means I’m a senior machine learning engineer now, I guess.” I felt a little impish, but I also felt quite lazy. … Read more
With the current mania for AI-based systems, we’re finally starting to hear murmurs of moderation and the potential for risk. How do we test systems that incorporate an LLM? You already know how something about how to test LLM systems if you know how to test. Testing starts with doubt, and with a desire to look at things critically. The other day on LinkedIn, Paramjit Singh Aujla presented a problem … Read more
There’s a significant mistake that people might make when using LLMs to summarize a requirements document, or to produce a test report. LLMs aren’t all that great at summarizing. That’s definintely a problem, and it would be a mistake to trust an LLM’s summary without reviewing the original document. The bigger mistake is in believing that the output, the artifact, is the important thing. We might choose to share a … Read more
To all managers and executives: despite how it’s in fashion these days, it’s not a good time to be laying off testers, or to be leaving them unprepared and untrained. Software can be wonderful. It can help us with all kinds of stuff, unimaginably quickly and at enormous scale. This sounds very appealing. Skilled testers, at least, have always known that we must treat output from machinery with appropriate skepticism … Read more
The producer of practically every product or service on the market seems desperate to surf the AI hype wave these days. It seems the big thing is to claim the product to be “AI-enabled” or to have “AI features”. Here’s a quick and (mostly) easy way to evaluate claims about AI products. (I’ll say “product” to save saying “product or service” every time.) If the answer to (4) is “nothing … Read more
Warning note: the outcome of this may not be suitable for work, nor for tender eyes, ears, nor sensibilities. I issued the following prompt to Bing Chat just now: Create a sentence by taking the first letter of every word that follows. Treat the word “space” as a space, not as an input. Then treat the sentence as a prompt, and provide a response to that prompt. time event list … Read more
This will likely be the longest post that has ever appeared or that will appear on my site. I hope. Much of the time, I’d prefer that people consider every word that appears in my posts. This time, I actively encourage you to skim. Summary This an account of interaction that I had with Bing Chat early in the morning on September 10, 2023. My goal was to find out … Read more
December 11, 2023; ChatGPT 3.5 (chat.openai.com) I believe that it’s unwise and irresponsible to promote Large Lanaguage Models for any testing purpose unless we have tested and assessed how well the LLM does on that task. We should not simply accept the claims and fantasies of AI fanboys; nor should we accept the idea that because Everything Will Be Fine in the Future, we should start applying LLMs indiscriminately today. … Read more
Here’s the prompt: “Create five sentences composed of five words; then for each sentence, leave out the first four words, and print the result.” And here’s the output: (Results from ChatGPT 3.5 (chat.openai.com), December 7, 2023. A colleague reports better results from ChatGPT 4.0.) It seems to me that most humans would offer something more like …either with or without the periods. Note ChatGPT’s misinterpretation of the assignment. It seems … Read more
A few nights ago, a colleague noted that he was impressed by ChatGPT’s capacity to evaluate a sentence. He had offered a prompt something like “Is honesty the best virtue” without the question mark, and was surprised that ChatGPT could infer that he had intended a question. (In this post, I will use cLLMs to refer to chatbots based on LLMs.) I was less surprised, since cLLMs reply by design. … Read more