A few months ago, James Bach introduced me to the idea of test framing. He identified it as a testing skill, and did some work in developing the concept by field-testing it with some of his online students. We’ve been refining it lately. I’ll be giving a brief talk on it at the Kitchener-Waterloo Software Quality Association on Thursday, September 30, 2010, and I’ll be leading a half-day workshop on it at EuroSTAR. Here’s our first public cut at a description.
The basic idea is this: in any given testing situation
- You have a testing mission (a search for information, and your mission may change over time).
- You have information about requirements (some of that information is explicit, some implicit; and it will likely change over time).
- You have risks that inform the mission (and awareness of those risks will change over time).
- You have ideas about what would provide value in the product, and what would threaten it (and you’ll refine those ideas as you go).
- You have a context in which you’re working (and that context will change over time).
- You have oracles that will allow you to recognize a problem (and you’ll discover other oracles as you go).
- You have models of the product that you intend to cover (and you’ll extend those models as you go).
- You have test techniques that you may apply (and choices about which ones you use, and how you apply them).
- You have lab procedures that you follow (that you may wish to follow more strictly, or relax).
- You configure, operate, and observe the product (using test techniques, as mentioned above), and you evaluate the product (by comparing it to the oracles mentioned above, in relation to the value of the product and threats to that value).
- You have skills and heuristics that you may apply.
- You have issues related to the cost versus the value of your activities that you must assess.
- You have time (which may be severely limited) in which to perform your tests.
- You have tests that you (may) perform (out of an infinite selection of possible tests that you could perform).
Test framing involves the capacity to follow and express a direct line of logic that connects the mission to the tests. Along the way, the line of logical reasoning will typically touch on elements between the top and the bottom of the list above. The goal of framing the test is to be able to answer questions like
The form of the framing is a line of propositions and logical connectives that relate the test to the mission. A proposition is a statement that expresses a concept that can be true or false. We could think of these as affirmative declarations or assumptions. Connectives are word or phrases (“and”, “not”, “if”, “therefore”, “and so”, “unless”, “because”, “since”, “on the other hand”, “but maybe”, and so forth) that link or relate propositions to each other, generating new propositions by inference. This is not a strictly formal system, but one that is heuristically and reasonably well structured. Here’s a fairly straightforward example:
GIVEN: (The Mission:) Find problems that might threaten the value of the product, such as program misbehaviour or data loss.
Proposition: There’s an input field here.
Proposition: Upon the user pressing Enter, the input field sends data to a buffer.
Proposition: Unconstrained input may overflow a buffer.
Proposition: Buffers that overflow clobber data or program code.
Proposition: Clobbered data can result in data loss.
Proposition: Clobbered program code can result in observable misbehaviour.
Connecting the propositions: IF this input field is unconstrained, AND IF it consequently overflows a buffer, THEREFORE there’s a risk of data loss OR program misbehaviour.
Proposition: The larger the data set that is sent to this input field, the greater the chance of clobbering program code or data.
Connection: THEREFORE, the larger the data set, the better chance of triggering an observable problem.
Connection: IF I put an extremely long string into this field, I’ll be more likely to observe the problem.
Conclusion: (Test:) THEREFORE I will try to paste an extremely long string in this input field AND look for signs of mischief such as garbage in records that I observed as intact before, or memory leaks, or crashes, or other odd behaviour.
Now, to some, this might sound quite straightforward and, well, logical. However, in our experience, some testers have surprising difficulty with tracing the path from mission down to the test, or from the test back up to mission—or with expressing the line of reasoning immediately and cogently.
Our approach, so far, is to give testers something to test and a mission. We might ask them to describe a test that they might choose to run; and to have them describe their reasoning. As an alternative, we might ask them why they chose to run a particular test, and to explain that choice in terms of tracing a logical path back to the mission.
If you have an unframed test, try framing it. You should be able to do that for most of your tests, but if you can’t frame a given test right away, it might be okay. Why? Because as we test, we not only apply information; we also reveal it. Therefore, we think it’s usually a good idea to alternate between focusing and defocusing approaches. After you’ve been testing very systematically using well-framed tests, mix in some tests that you can’t immediately or completely justify.
One of the possible justifications for an unframed test is that we’re always dealing with hidden frames. Revealing hidden or unknown frames is a motivation behind randomized high-volume automated tests, or stress tests, or galumphing, or any other test that might (but not certainly) reveal a startling result. The fact that you’re startled provides a reason, in retrospect, to have performed the test. So, you might justify unframed tests in terms of plausible outcomes or surprises, rather than known theories of error. You might encounter a “predictable” problem, or one more surprising to you. In that case, better that you should say “Who knew?!” than a customer.
To test is to tell three parallel stories: a story of the product, a story of our testing, and a story of the quality of our testing. James and I believe that test framing is a key skill that helps us to compose, edit, narrate, and justify the story of our testing in a logical, coherent, and rapid way. Expect to hear more about test framing, and please join us (or, if you like, argue with us) as we develop the idea.
See http://www.developsense.com/resources/TestFraming.pdf for current updates.
Idea of “Test Framing” is simply great!!
IMO, we testers more or less do Test Framing while do any testing though we were not cognizant with the term Test Framing, now we’re fortunate that one of our skill would recognize with a decent term 🙂
Moreover, according to my understanding somehow Test Framing can relate or extent from Mind Maps, isn’t it?
Michael replies: Yes, you could certainly illustrate a framing conversation with reference to a mind map. One of the longer-term items of business for us is to illustrate framing visually. While you’re waiting, the illustration of the Heuristic Test Strategy Model—tests at the centre with other elements arranged around it—would seem a useful point of departure.
Regards,
– Selim
I like this.
When you do your explicit framing, you will learn, and uncover information that will inspire you to even more and better test ideas.
When dealing with logic, we should also be careful not to try to fit all tests into one logical, coherent system. Multiple angles are needed, and they can/should be contradictive.
Michael replies: Yes, they certainly can be. Quality is, after all, subjective and multidimensional.
I would however prefer another example than “long string”, that is used too often?
Would you like an example using login? [grin /]
It was late. I’ll try to present some more creative examples.
Good idea. I really like it.
Initial thoughts:
I like the inclusion of formal deductive logic in the example (critical thinking). There’s room for expansion (maybe good & bad examples) of other aspects (on the critical thinking track) of using induction and also rhetorical ploys – there’s probably other aspects (must look up my Critical Reasoning references.)
I wondered if you could apply the same “test chaining” to some of the initial premises, e.g. “You have ideas about what would provide value in the product, and what would threaten it” – and using a similar proposition-conclusion statement. I think this is a way to capture the different perspectives of value in the product (not just mine as a tester but also my various stakeholders’ values). Then if someone asks “why did you test that” we can discuss the value aspects.
I then started thinking that there was too much logic and got onto the de-focussing train of thought – and there you captured it in the second-to-last paragraph 🙂
I’m going to ponder and cogitate on this some more…
Two typo type issues in there…
“Why did are you running” -first bullet under test framing.
Michael replies: Fixed. You can see how that problem sneaked in, I trust.
” We might ask them to ask them to describe a ” -Second sentence in paragraph beginning with “Our approach so far”
Thank you for asking me to asking me to fix this.
Not being a smartass, just thought you’d like to know.
You’re not being a smartass; you’re being more like… what’s it called? Oh, yeah: a tester. So, the appropriate response is: Thank you!
Great read, really eager to learn more of this kind of stuff re:Thought process exercises to improve testing skills. I’ll work on applying it in my work and see if I can’t come up with some thoughts of my own on the subject.
Please do, and keep in touch.
That sounds great indeed, I am however concerned about this test framing in a complex software. If you end up with 6 propositions for a simple text field then how can you limit the complexity of this frame?
Michael replies: It’s not that you have to do this explicitly for every test; it’s that you have to be able to do this for every test, and that you have to be able to do it at the speed of thought.
Also how would you take care of overlap between testers’ scope?
1) Manage it. Give testers guidance, in the form of charters, on what areas to test and where to direct their focus. If you find they’re covering the same space (and you don’t want that), then give them more explicit guidance.
2) Question whether overlap is a problem. Just as with customers, different testers will see many the same things from different perspectives. That’s a good thing; it allows us to find different aspects of value and threat to value in a product. If you decide that overlap is a problem, see (1).
Thanks!
Ahmet
Yet another documentation bug.
How are you testing (did you test, well you test) for this requirement?
should read
How are you testing (did you test, will you test) for this requirement?
Michael replies: It does (now). Thanks.
Logic chains are everything in being successful at testing. Perhaps a trained monkey can follow test scripts at tedious lengths and levels of detail, but in almost a decade and a half in the industry, I’ve never had the luxury to cover every realistic iteration, but have always had to rely on use cases, heuristics and understanding to determine where risk vectors are and make sure time gets allocated accordingly.
I agree. One finesse, though: The allocation of time is also a framing thing: if you want us to cover this AND that, it’s probably going to take more time or a different approach than the one required for this OR that. One thing I like about framing is that, done well, it allows to talk clearly not only about plans and not only about justifications, but also about choices for plans, and about justification for choices.
I like this article very much. Good job.
I like the idea of test framing, I’m going to have another read and think tomorrow. At the moment coming across as a melding of test axioms and BDD.
This is a great article Michael. Sometimes we are too caught up with expected deliverables of a “standard” test plan that we miss the whole point of it.
As a Tester I have always loathed Test Planning since I am forced to document in detail tests that i’m supposed to execute at a point where i knew least about the project.
Michael replies: Careful—that’s only one kind of test planning. You don’t like that kind of test planning (and I wouldn’t either, except in an extremely narrow selction of contexts). Test planning is the stuff that happens in your head; it’s the combination of strategy (the set of ideas that guide your test design) and logistics (the set of ideas that guide your application of resources). Such ideas can be quite lightly documented, or quite heavily documented. We choose the level of documentation based on how and when it might serve our purposes. The level of test planning you’re describing (documenting your tests in detail) sounds like scripting or programming to me. It’s tedious, but it’s especially brutal when you’re writing a program to be executed by a human machine. As such it’s not only a waste of time, but (at execution time) a real distraction for an observant, thinking person.
I have always resorted to creating test missions (so to speak) that gives me the ability to add/remove cases and techniques accordingly as the project progresses and as the requirements change. I just did not realize that i’m using some sort of quasi test framing.
The more you can frame your tests, I believe, in general the less documentation you need. I’ve often seen organizations that waste time and effort on producing documentation that specifies, in painful detail, what the tester should do, but those same documents leave out the motivation for doing it, or the risk that the test is intended to address. Yet if the tester—even a relatively inexperienced tester—understands the motivation and the risk, then there’s typically no need for the detailed instructions.
Now that I am managing my own team, I am less strict with documentation but i expect a certain precision and understanding from my team members as to what it is that they are testing. This would be a challenging habit to practice but once everyone gets into the groove of test framing, the quality of tests and released products will definitely go up.
I think so. I hope so. Let us know about your experience, and let us know if we can help.
And thank you for the reply!
I think this idea of test framing is a fantastic idea. I am amazed at how many people in the industry don’t understand the real purpose of testing, let alone how each individual test supports that purpose.
This has been the topic I have been focusing on lately, understanding why testers exist in the software industry. I really like the idea of being cognizant that every test we run should be able to be framed to support that goal.
I also think this principle could (and should) be applied outside of testing. There are plenty of self-help books out there that teach people to do everything they do for a reason. Framing is simply a way of connecting our actions to our purpose, be it in testing or any other aspect of life.
Michael replies: Nicely put. Thanks for the comment, Wade.
Here are two things you already have covered, but with other words:
James Bach at SWET1: “Test Framing – thread design”
Edward de Bono, Lateral Thinking p.41: “…a successful trial is still successful even if there was no good reason for trying it. It may also happen that once one has reached a particular point it becomes possible to construct a sound logical pathway back to the starting point.”
[…] will be doing a half-day tutorial on Test Framing (read his blogpost that coins test framing here). But that’s not all. There’s Lee Copeland too. And many, many more. L’embarras […]
[…] to a value (for customers) is not the same as testing that a program works. If they would use Test Framing approach, they’d probably discover the defect. Maybe a draft of a cheque book, displayed on […]
A few years ago I read an article by a professional framer who specialized in the restoration of old frames. My apologies for citing neither article nor author! How you frame any picture appears to change the picture itself, in profound ways. I suppose photographers know this as well. So do interior designers: the view from your dining room of one’s garden is so different if the window has draperies, and the choice of draperies is not an insignificant question.
The choice of frame may mean the difference between seeing a pointillist landscape as a collection of dots or as an unforgettably textured rendering of a field’s fleeting illumination by a setting sun.
When I think about test framing, I get more out of it if I begin with framing the testing of the entire SUT, an activity proper to forming a testing strategy, a very large part of which connects business objectives for the SUT to the quality objectives that are believed to validate those business objectives’ realization.
Ideally, individual tests would inherit their purpose from their containing context, and ultimately (iteratively) from the principal or master framework.
Michael replies: Ideally they would, yes. The framing idea got started when James, in his online coaching, realized that many testers seemed unable to frame a single test, much less an entire test strategy.
While thinking about your comment, I was reminded of the Barnes Museum, in which the artworks are displayed in an unusual way. “A notable aspect of the Barnes Foundation’s art collection is its display in ‘wall ensembles,’ which are intentional combinations of works from different time periods, geographic areas, and styles for the purpose of comparison and study.” If I recall correctly from the exhibit that came to Toronto, Mr. Barnes was adamant about setting the context of the artworks, and especially about establishing a frame for observing Western work in a more global context.
Of course, everything that we present (or is presented to us) appears in a frame. The trick, I think, is to be able to understand and explain the relationship of the figure to the ground. That’s a challenging but exciting task.
Sounds really interesting. There might be a small problem with the naming of the technique though.
In The Netherlands there is a book and way of testing that is Called Testframe (very short description: you define testcondition (things you want to test) and use a excel macro to generate testcase that should be easy to automate). More information (in very bad translation I should add): http://tinyurl.com/4wccuxq
Michael replies: That there’s another idea in the world called test framing doesn’t bother me. All definitions are heuristic—context-dependent and provisional. An integer might be any positive or negative whole-number value, or it might be a number in a range from 0 to 4294967295 (a 32-bit integer), or it might be a number in the range from -2147483648 to 2147483647 (a signed 32-bit integer)–and on 16-bit systems, ranges of smaller values. Given context and an explanation, smart people can cope with this kind of issue. Look at all the different ideas about testing itself. For example, there’s this thing that some people call testing and that my community calls checking…
This is someting quite different than your Testframing, but if you are going to give a talk in The Netherlands about Testframing, a lot of confusion might arise.
A lot of confusion? I think you underestimate your countrymen. For example, it seems to me that they’re capable of understanding that one doesn’t typically go to coffee shop to get coffee.
Just wanted to inform you 🙂
I believe the term is trademarked, would that hinder you using the term?
Perhaps not a lot of confusion, but if I would go to a Canadian coffee shop with my Dutch context setting, some weird scenario’s might arise. The same might true for a Canadian coming to a dutch coffee shop. 😉
So just wanting to help you :).
[…] session were met and others need to be improved in the future. Michael suggested one of his posts http://www.developsense.com/blog/2010/09/test-framing/ as a guide in a similar […]
Thank you Michael, I huge feeling of relief came over me as I read this. It is exactly what my mind was trying to create to process the various ‘levels’ of the testing environment.
I read about test framing for the first time in James RST I attended in June. I enjoy the buzz of discussing testing in the terms you guys use.
It also, very importantly, provides material to be able to pass along to those I mentor and train in testing. It is one of my favourite blog topics for 2011.
[…] testing landscape. James’ final point on learning testing is test framing. He referred to a blog entry from Michael Bolton on the topic. As with anything in testing, test framing had been around before James and Michael […]
[…] A Hint : Test Framing. […]
[…] Test Framing – tell the story of the test […]
[…] – and if not, hey, heuristics are fallible by definition, right? The same goes for ideas like test framing, considering the difference between checks and tests, or the practice of working hard to master […]
[…] test, how you are doing it, why it is valuable and what you learned from it. I believe that test framing is crucial in testing and to be able to tell a story of you testing. I think then we are getting […]
[…] with the mission of testing? It is extremely important that testers understand and adhere to their mission, as to replace it (either willfully or unintentionally) would be directly fogging the headlights on […]
[…] scientific thinking skills, using heuristics ; emphasizing the use of lightweight, flexible tools; framing testing ; applying oracles ; identifying coverage and the gaps in it; and reporting on all of those […]
[…] Framing: Test framing by Michael […]
[…] Test Framing […]