There is a testing techique that I often apply. I have recently decided to name it the First Hurdle Heuristic. The basic idea: get the product out of the starter’s blocks, and see how it performs given a relatively easy challenge.
This heuristic can useful when you want to identify problems and risks immediately, or to determine whether a product might not be ready for use or for deeper testing. Sometimes one or several First Hurdle tests add up to what people sometimes call “smoke testing” or “sanity testing”.
(Why would I apply a different name? Because, for me, it evokes a vivid image of an immediate stumble and faceplant. You can keep calling them smoke or sanity tests if you like, and if you insist, I’ll cheerfully adopt that language when we work together.)
Often the First Hurdle Heuristic can be applied to a claim. For instance: “ChatGPT can generate data for testing.” ChatGPT 3.5 tripped by hooking its foot over the first hurdle as it went over. ChatGPT4o didn’t even try to jump; it simply ran headlong into the first hurdle immediately.
You get to decide whether you want to use a product for any purpose you like. If it’s a Large Language Model, you can use all the tricks you might need to get the result you want: RAG, GAN, CoT, “Simon says do it right the first time.”
If it’s something important, it would be a really, really good idea to examine the output for errors. Then you might want to ask how much time you’d like to spend picking it up, dusting it off, and getting the cinders out of its shoes.
1 reply to “The First Hurdle Heuristic”