Test Tools

Selenium! Playwright! Cypress! Postman! There’s constant hubbub around these tools and others primarily focused on checking output—and now the din is getting louder about low-code or no-code tools for checking output. (Last year I reviewed a couple of those, here and here. Did these companies use their own tools to test their own tools? Either a Yes or No answer would be disturbing.)

Want some heuristics for choosing tools of that kind? You could look here.

On the other hand… why do people have such a limited view of test tools? Why not talk for a moment about the many ways in which tools can help us? Not just with output checking, but with the real, valuable, powerful, challenging, fun stuff of testing: learning about a product through experiencing, exploring and experimenting, finding problems that matter—and delivering the fruits of that learning to people who need to know about it.

Here’s a summary taken from A Context-Driven Approach to Automation in Testing, and refined in A Rote Less Travelled. This is how we think of testing tools in Rapid Software Testing.

In test design, we use tools to help us

• produce test data (tools like spreadsheets (why are some testers so eager to diss Excel? it’s like a Swiss Army knife for testing); state-model generators; Monte Carlo simulations; random number generators)
• obfuscate or cleanse production data for privacy reasons (data shufflers; name replacers)
• generate interesting combinations of parameters (all-pairs or combinatorial data generators)
• generate flows through the product that cover specific conditions (state-model or flow-model path generators)
• visualize elements of the product (mind-mapping tools, diagramming tools)

In product interaction, we use tools to help us

• set up and configure the product or test environments (like continuous deployment tools; version control tools; virtualization tools; or system cloning tools)
• submitting and timing transactions; perhaps for a long time; at high volume; under stress (profiling and benchmarking tools)
• encode procedures like operating the product and comparing its outputs to calculated results (this is automated checking)
• simulate software or hardware that has not been developed yet; or that we do not have immediately available to us (mocking or stubbing tools)
• probe the internal state of the system and analyze traffic within it as testing is being performed (instrumentation; log analysis; file or process monitors; debugging tools; “developer tools” on the browser)

In evaluation, we use tools to help us

• sort, filter, and parse output logs (text editors; spreadsheets; regular expressions)
• visualize output for comparative analysis (diffing, charting and graphing tools, conditional output formatting)
• develop, adapt and apply oracles that help us recognize potential problems (source file or output comparison tools; parallel or comparable algorithms; internal consistency checks within the application; statistical analysis tools)

In recording and reporting, we use tools to help us

• record our activities and document our procedures (note-taking tools; video-recording tools; built-in logging; word processing tools; user interaction recording tools)
• prepare reports for our clients (mind maps; word processors; spreadsheets; presentation software)

In managing the testing work, we use tools to help us

• map out our strategies (mind maps (yet again), outline processors, word processors)
• identify what has and has not been covered by testing (coverage tools; profilers; log file analysis)
• preserve information about our products, and to aid other people in future support and development (wikis; knowledge bases; file servers)

And we use tools (like text editors, programming languages, libraries, shell scripting, and integrated development environments) to build our own tools that fit into the categores above.

Tools can dramatically extend our power as testers. Maybe it would be a good idea if we told more stories about that.