On August 5, 2005, James Bach posted in his blog a really interesting piece on intermittent problems. It’s, as usual, thoughtful and well-considered. You can read it at http://www.satisfice.com/blog/archives/34.
It’s a little agonizing to think that an intermittent problem might depend upon achieving a certain threshold value in some variable which might be difficult to reach. One form of intermittent problem might require hundreds of thousands of preceding transactions. This would be hard to reproduce if it first appeared after months of testing, and if you wanted to reproduce it on a new system. Another form of intermittent problem might happen only with the first use of the system. This would be hard to reproduce after the first test, of course.
James talks about input a lot in Possibility 4. However, the problem might be easier to spot or to model if we seek patterns in the output—not often, perhaps, but if you’re looking for a hard-to-find problem, a variety of models is generally preferable to one; we don’t know which model is the right one until we’ve finally found the bug.
I think it’s very important to consider that the heuristics for finding intermittent problems are, in the main, excellent heuristics for finding problems generally. For fun, I tried rereading the article removing the word “intermittent”. The article still made a lot of sense. “The ability and the confidence to investigate an intermittent bug is one of the things that marks an excellent tester”; “Many intermittent problems have not yet been observed at all, perhaps because they haven’t manifested, yet, or perhaps because they have manifested and not yet been noticed”; “Some General Suggestions for Investigating Intermittent Problems”; and so forth.
This underscores the point that all problems can be seen as intermittent; it’s just that, for “regular bugs”, the patterns are simpler on one level. We can see the nature of the patterns more easily in some circumstances than others. That’s why modelling, pattern-spotting, and systems thinking skills are so crucial for testers.