Last week in the Netherlands, I presented a workshop on test framing, a skill that James Bach and I like to talk about. Test framing is the set of logical connections that structure, inform, and motivate a test.
Testing is a process of composing, editing, narrating, and justifying a story with three parallel threads. There is a product element—the part of the story that describes product, how it works, how it fails, and how it might fail in ways that matter to our clients. That story is backed by the activity element—the part of the story that describes what we did and how we did it. Here we describe what we did, how we designed our tests, and how we configured, operated, observed and evaluated the product. The narrative of our activities gives warrant to the product story, making it credible and accountable.
In turn, the testing story needs its own warrant, so there’s a justification element to supply that. The activity element is about what we did, how we did, when we did it, who was involved. The justification element is about why. In the justification part of the story, we talk about the quality of our work, and the obstacles and issues that might threaten it. We explain why we’ve done what we’ve done, why we haven’t done other things yet, or why we won’t do certain things at all unless things change. We relate the activities to the testing mission and the product story, and explain why the testing work we’ve done is good enough, and why it’s reasonable to have postponed or written off the testing work we haven’t done.
Framing is the network of logical statements and connectives that link the three story elements and their details. To frame a test means to create an unbroken line of logic from the mission—the information objective within some context—through the product story and our activities, down to the observed outcome of the test and our interpretation of it. (I’ve written about that here, too.) The logic consists of propositions—philospher-talk for “regular statements”—linked by logical connectives. In some kinds of formal logic, only specific operators are permitted, like “if”, “and”, “not”, “or”, “if and only if”. Test framing uses a somewhat less formal and more permissive form of logic, but the intention is to provide a means of reasoning through a test idea, its execution, and its interpretation. There are a couple of worked-out examples in this document, which was first presented at EuroSTAR in November, 2010, and which has been undergoing revision from time to time.
James and I have observed that many of the people in our classes and those who approach us for coaching really struggle with framing a test. Some of these testers are relatively inexperienced, of course. Yet we also observe that very experienced and skilled testers often struggle almost as hard. It seems that people aren’t used to the practice. It’s not that their testing is unmotivated, exactly; more that they’re not used to articulating and justifying their testing. That creates a feedback loop, which I’ll describe with a few gross generalizations:
a) Since testers don’t tell the story of their testing, clients don’t understand what testers are up to.
b) In order to evaluate the quality of testing, clients use a bunch of surrogate measures such as test case counts and defect escape ratios.
c) Testers get out of the habit of telling the testing story, and learn to emphasize the numbers (to please the client).
d) Go to (a).
So if you can work as a tester for years without framing your tests, and people ask you to frame tests rarely or not at all, is an inability to frame tests a problem? We think so.
- Test framing is an important element in bug advocacy—the skill of presenting a problem report to a testing client in a way that accurately conveys the meaning, significance, and risk associated with our actions, observations and evaluations. Without the capacity to trace the logical reasoning behind test activity, our bug advocacy will be considerably weaker.
- Decisions about quality are inherently subjective. As Jerry Weinberg points out, they’re political and emotional, yet the people who make them want to appear rational. Good test framing can help to provide some rationally based grounds for justifying quality decisions—and for providing reasonable alternatives.
- Test framing provides a way for more experienced testers to describe important parts of testing to novices or trainees.
- Usually people talk about traceability to requirements, when really they mean traceability to explicit requirements in requirements documents. Good test framing can provide traceability to implicit or as-yet-unstated requirements. That is, test framing can help to reveal and develop a richer and more representative requirements story.
- Test framing logically ties the elements of testing story together—the product element, the testing element, and the justification element. Test framing is like the sinew that joins the bones of the testing story.
- Practicing test framing can help us to frame other aspects of the project, including requirements, specifications, bugs, and management decisions. Is someone saying something that doesn’t make sense to you? Try framing it. Framing an observation, inference, or decision can help you to understand the elements that have gone into it, and the logic that links them—or the logical gaps that make it unreasonable, unjustified, or risky.
- Finally, and perhaps most importantly, framing is an important aspect of self-management and exploratory approaches. A tester that can connect the mission of testing to the test result and everything in between can also connect the logical and emotional aspects of testing. Moreover, a tester that can frame his or her work can show a rational basis for it, which is key to developing trust. A tester that can’t frame testing work into a logical structure is more likely to be directed and controlled by managers and clients. Whether true or not, managers will percieve that the testers don’t understand what they are doing. Even if the testers do understand, they won’t get credit for it.
After the aforementioned workshop, I posted a couple of documents on the Web and notified the people who attended. Shortly thereafter, I got this reply that I can quote with Francien’s kind permisison:
From: Francien Ramakers [mailto:xxxxxxxx@xxx.com]
Sent: 12 May 2011 05:36
To: Michael Bolton
Subject: RE: Test Framing at TestNet
Hi Michael,
The first link does not work :(.
When I click on it I get a page with “You know, I could have sworn that page was here a minute ago.” from your blog.
Framing of this test:
– If the purpose of the link was to provide the readers of your email with a document and if the idea was or this to be possible simply by clicking on the link within the email and if clicking on it does not result in display of a document, and if the other links do work fine (indicating that that the issue is unlikely to be caused by another problem, such as being offline) then something is wrong with the first link.
– If the purpose of the first link in your email was to test how many people download the material after you mail them and if all people that try to download any of it click on the first link and if all people that click on the link have got the same result as me and if they all mail you when they do then this is not a bug but a feature.
In both cases, I would still really like to receive the document it refers to. Could you please send it to me?
Hmmm… and now I am not sure whether I framed the test or the bug…
Kind regards,
Francien.
Note that Francien has supplied not only a logical basis for the theory that she’s witnessing a bug—but also that she’s providing another, equally logical basis for an alternative interpretation of her observations. That’s important, because a tester is someone who knows that things can be different. Locking on to a single interpretation is a risky business for a tester, since a single proposition somewhere in the frame can flip the interpretation of the test from bug to feature, and vice-versa.
So I replied.
From: Michael Bolton [mailto:michael@developsense.com]
Sent: donderdag 12 mei 2011 12:56
To: Francien Ramakers
Subject: RE: Test Framing at TestNet
Hi, Francien…
Well done!
I would like to suggest that you framed the test and the bug. I can frame the reason for the bug,
if you like.
IF I give a presentation AND
IF a part of my mission in delivering the presentation is to supply people potentially with useful notes and information
THEN I should prepare an email for them which has the correct links to that information in it
AND IF someone complains that one of those links doesn’t work
THEN I suspect that I’ve made an error in the links
THEREFORE I test the link
AND IF I can reproduce the problem
THEN I suspect the link is incorrect
UNLESS I didn’t upload the file in the first place (which I now realize I didn’t)
THEREFORE I really shouldn’t do all this stuff too late at night.
🙂
—Michael B.
Francien replied,
Hi Michael,
Thanks! 🙂
I think it’s cool that you framed the answer!
Mailing about it made me realize framing can be applied to other things then just tests.
Cheers,
Francien.
Francien, of course, is right. Framing can be about many other things: not only testing, but also about bugs, requirements, specifications, and so forth.
For extra points: in my framing, can you identify the proposition in my framing that is not logically connected to the others?
Hi Michael,
I think it’s the last line:
THEREFORE I really shouldn’t do all this stuff too late at night.
Michael replies: It IS the last line, although I’d contend it’s not for the reason that you cite.
‘All this stuff’ would logically refer to ‘giving the presentation’, ‘preparing an email’, ‘testing the link’, and ‘reproducing the problem’. But I doubt if you did ALL that too late at night. I know that the presentation was in the morning 😉
Indeed. But the real trouble with that line is that the conclusion is not linked or connected logically to any earlier statement. There is no statement preceding the conclusion that is connected by logic to it, so by logic there is no expressed warrant to that conclusion.
Now, it’s perfectly possible to make non-logical statements that are nonetheless true. But if we’re playing by the rules of logic, those statements are non-sequiturs; that is, they don’t follow from prior statements, and therefore they don’t have warrant under a system of logic.
It’s also possible to build logical arguments on false premises. Such arguments are dangerous, in that they look convincing because they follow logical structures. Logic doesn’t guarantee that truth will follow from a logical sequence of premises and propositions. The conclusion is guaranteed to be true only if the premises are true. Logic is like a computer program; it operates faithfully, but if you put garbage into an otherwise logically robust argument, you will get logically robust garbage out. So smart testers will not only question the logic that leads to a conclusion, but will also question the premises.
It is also a solution to the bug, probably one of many, but not the reason.
A contributing factor. And look! I’m doing it again!
Thanks for the comment!
Hi Michael,
Great post and ‘follow-up’. I think it highlights the difficulty in pure logic reasoning: we tend to ‘fill in the blanks’ and assume all kinds of things. And we do that automatically (most of the time I guess).
Being aware of that and trying to apply logic on our creative thinking is what makes it challenging and fun 😉
Ray
I don’t think explicit framing is needed very often in reality, often it is apparent that a test is needed.
But I think it is an essential skill, that when being trained, will bring along other testing skills as well.
And sometimes you definitely need it, e.g. when presenting a problem that might not be adressed otherwise.
However, framing would fit my testing world better with a few changes:
1) add “description” (or similar) so framing is about how you describe and present a test or a problem.
This also aligns better with how “framing” is used in other fields, e.g. you can frame something in different ways, so it suits the audience, and the most important aspects.
2) Use “a” instead of “the”. With current definition it is easy to think that there is one “true” frame, but in reality there can be many.
3) Remove “logical”, so framing can take advantage of people’s ability to connect things in other ways than logic, so it isn’t necessary to give all the details for all framings.
The logic part is one important bit, but there should be focus on other rationale/feelings as well.
This gives us something like:
Test framing is a description of connections that structure, inform, and motivate a test.
And here’s a made-up problem report example:
Main content of web site should be loaded first
When the web site is launched it starts to display sidebars and various frames before the main content of the page.
This is normally not a problem, but with slow connection, e.g. a mobile phone outside town, it gives a not so strong impression (see video available at …)
First impressions are important to our site, and my testing colleagues agree with me that a better display sequence would be:
company logo -> main content -> the rest
Here we have holes, subjective opinions, but still something that is easy to understand and make a decision from.
OK as framing or not?
Michael replies: These issues and questions are important enough to warrant a post of their own. Thank you for that.