It might okay for an organization to use LLMs in technical support — to parse a request for content, and then provide links to curated answers, call services that can provide correct answers, or summon a human agent.
Why do companies use GPTs — generative AI — for technical support? Remember Brooks on GPTs: a GPT is not designed to produce a correct answer; it’s designed to produce an answer that looks good.
Here Zoom’s virtual agent is telling me something:
The response is wrong. It’s not only a point version behind; it’s a whole version family behind.
Another trial:
The current version isn’t listed in the Zoom support content? It is, actually:
Another trial:
That response is correct in a sense; there’s no single build of Zoom. But it appears (from the reference above) that the version numbers of Zoom seem to be consistent across platforms.
Another:
The response here could provide a link to the release history, but it doesn’t. The bot is providing a link to the system requirements, which are, for this question, not exactly relevant.
When you’re testing a chatbot-based system, the accuracy, timeliness, relevance, and usefulness of the resonse presumably matter. Look for problems with those. Perform several trials using exactly the same prompt. Look for patterns of unhelpfulness. You can use our list of LLM syndromes to guide searches for those problems, or to help provide classifications for them.
Here’s one really good use for LLMs, though: use them to generate variations on the same request. For instance, I asked ChatGPT:
Here is some text that forms the basis of a customer request for a chatbot.
“What is the current version of Zoom?”
Provide 30 variations of that request.
ChatGPT helpfully provided this:
- What’s the latest version of Zoom?
- Which version of Zoom is currently available?
- What is the most recent Zoom release?
- What version of Zoom is available now?
- Can you tell me the current Zoom version?
- What is Zoom’s latest software version?
- Which Zoom version is the newest?
- What version number is Zoom currently on?
- What is the latest Zoom version number?
- Which version of the Zoom app is current?
- What is the newest version of the Zoom client?
- What is the current release of Zoom?
- Which Zoom release is currently supported?
- What version of Zoom should I be using?
- How can I find the latest Zoom version?
- Has a new version of Zoom been released?
- Is my version of Zoom up to date?
- What is the latest update available for Zoom?
- Which Zoom update is the most recent?
- What version does the latest Zoom update install?
- What is the current Zoom desktop client version?
- What is the latest version of Zoom Workplace?
- Which version of Zoom Workplace is currently available?
- What is the newest Zoom app release?
- What is Zoom’s current build number?
- Can you check the latest version of Zoom?
- Do you know the current Zoom release number?
- What is the most up-to-date Zoom version?
- Which Zoom version was released most recently?
- What version of Zoom can I download today?
Then use tools to feed each instance of the input to the bot several times. Again, look for patterns in the output — focusing on things that might mislead, confuse, or frustrate the customer looking for help.
But that’s going to take a lot of work! It will be expensive! Yes; maybe. Here’s something that’s a lot less expensive: a well-maintained, curated frequently-asked-questions list. The bot can programmed to refer the user to specific questions on this list, or quote them directly — and the bot can offer to connect the customer to an adaptable human agent.
At very least, a support chatbot should be system-prompted to give pointers to a correct answer, and NOT to generate one from training data — one that has a high probability of being unhelpful or false.