Flaky Testing

The expression “flaky tests” is evidence of flaky testing. No scientist refers to “flaky experimental results”. Scientists who observe inconsistency don’t dismiss it. They pay close attention to it, and probe it. They redesign their experiments or put better controls on them. When someone refers to an automated check (or a suite of them) as a “flaky test”, the suggestion is that it represents an unreliable experiment. That assumption is … Read more

Necessary Confusion and the Bootstrap Heuristic

I’m testing a test tool at the moment. I’m investigating it for a talk. The producers of the tool claim to have hundreds of thousands of users. A few positive remarks appear in a scrolling widget on the product’s web site from people purported to be users. Me, I can’t make head or tail of the product. It doesn’t seem to do what it’s supposed to do. It looks like … Read more