Framing Test Framing

Rikard Edgren is a testing philosopher in Sweden. He independently develops his own ideas on testing (an example here), collaborates with his colleagues on The Test Eye, and critiques and builds on the work of other people in the community. He offered a comment to my recent post on test framing, and the comment deserves a post on its own. He said,

I don’t think explicit framing is needed very often in reality, often it is apparent that a test is needed. But I think it is an essential skill, that when being trained, will bring along other testing skills as well. And sometimes you definitely need it, e.g. when presenting a problem that might not be adressed otherwise.

Yes, and that’s an important point. We will rarely be asked to frame a test, and we will rarely do it explicitly, or in writing. However, I argue that we should always and at any time be able to frame a test; that is, to tie the elements of the testing story together with logical connections.

However, framing would fit my testing world better with a few changes:

1) add “description” (or similar) so framing is about how you describe and present a test or a problem.
This also aligns better with how “framing” is used in other fields, e.g. you can frame something in different ways, so it suits the audience, and the most important aspects.

Well, maybe. To me, the description dominates the framing; the framing, the logical structure, is a subset of and a basis for the description. When you frame something in different ways—that is, when you reframe it—you’re presenting a new perspective, for sure. But to me the key element is that the new description is based on different premises, and a different set of propositions and logical connections.

2) Use “a” instead of “the”. With current definition it is easy to think that there is one “true” frame, but in reality there can be many.

Doh! You’ve caught me in violation of the “a vs. the” heuristic. The “the” in this case implies that you’ve got a particular frame in mind, and that it’s the one you’re going use to deliver your interpretation. Nonetheless, you’re right: there may be multiple ways of framing a given test. Lucky for both of us that the “a vs. the” heuristic is a heuristic, not a rule.

3) Remove “logical”, so framing can take advantage of people’s ability to connect things in other ways than logic, so it isn’t necessary to give all the details for all framings. The logic part is one important bit, but there should be focus on other rationale/feelings as well.

There is every opportunity for other rationales and feelings to appear in the framing of a test. Framing doesn’t deny emotions; on the contrary. I think you might be conflating logical oracles and logical argumentation. There’s a difference.

There are certain aspects of a program that are informed by logic, such that a logical principle or mechanism points to a problem. That is, the oracle is based on a logical decision rule. To use the trivial example of an adding program,

IF the program is intended to add numbers, AND
IF we are using the rules of everyday mathematics, AND
IF we add two numbers
AND the addition of those two numbers produces a result that is not equal to their sum,
THEN the result is mathematically false AND
IF we see a result that’s mathematically false
THEN we assume that there’s a problem with the program.

In this case, logical and mathematical truth or falsehood act as our oracle—while we’re also framing the test by establishing logical connections between proposition.

Yet we can use logical connections to frame tests based other oracles as well.

IF the program is intended to provide us with the service of adding numbers conveniently AND
IF we perform a test in which we tab from one control to the next AND
IF the controls are laid out such that tabbing from the first input field value to the second involves a tab stop on another control AND
IF that wastes the customer’s time or distracts her needlessly
THEN we can justify the claim that there’s a bug here.

Note that the basis for the bug is qualitative, but the line of reasoning used to frame the test is logical, based on propositions and logical connectors.

This gives us something like: Test framing is a description of connections that structure, inform, and motivate a test.

I’d see the “description” you have here as implicit in the way James and I define framing: the set of logical connections that stucture and inform a test.

And here’s a made-up problem report example:

Main content of web site should be loaded first
When the web site is launched it starts to display sidebars and various frames before the main content of the page.
This is normally not a problem, but with slow connection, e.g. a mobile phone outside town, it gives a not so strong impression (see video available at …)

First impressions are important to our site, and my testing colleagues agree with me that a better display sequence would be:
company logo -> main content -> the rest

Here we have holes, subjective opinions, but still something that is easy to understand and make a decision from.
OK as framing or not?

I’m glad you’ve given me an example, because it clarifies the misunderstanding. In answer to your question, I would suggest No, not okay as framing. There are propositions—statements—here, but they’re not linked by logical connectives. Let me propose a counter-example:

IF we colleagues are in agreement on principles that govern the order in which elements are displayed AND
IF we believe that our company’s logo should be the most prominent element in the Web site AND
IF we believe that the main content is the most important element of a web page to the users, AND
THEREFORE we further believe that the logo and the main content should be displayed before the other elements
UNLESS the connection is so fast that users cannot perceive a difference; WHEREAS
IF, when the web site is launched, it starts to display sidebars and various frames before the main content of the page, AND
IF there the order of loading is perceptible AND
IF the that order is not only perceptible but prominent on a slow connection AND
IF slow connections are common on mobile phones outside of town AND
IF we presume that a significant proportion of our users will be using such connections AND
IF we believe that those users will suffer annoyance AND
IF that annoyance is also the first impression that the user receives
THEREFORE we believe that this delay represents a threat to our image AND
IF we view threats to our image as a bug
THEREFORE we have reason to characterize our observation of the loading order as a bug or at least an issue.

Now, this is a pretty elaborate construction, but it can happen pretty much at the speed of thought. The point is that we can trace from premise to conclusion through logically connected statements and propositions. I think if I asked you to do that with your example above, you could do it easily, using informal yet rigorous logic. Yet, as my earlier post suggested, testers aren’t used to doing that. I think the craft could use practice in the art of logical argumentation, even when it’s about qualitative issues that aren’t themselves governed by logic.

2 replies to “Framing Test Framing”

  1. Awesome Michael!

    I can personally vouch for this part “…testers aren’t used to doing that.”

    Although now I intend to practice, so for that I thank you.

    Rikard’s example is definitely how I’m used to seeing it so at first your framing looked quite scary… Then after reviewing it thoroughly it makes perfect sense.

    Would love to see more on this subject as your work on it progresses.

    Michael replies: Yes, you can look forward to that.


    Thank you for the comment.

  2. OK, I’m with you.
    I rather describe “motivation”, your “framing” goes further, and the logic part is key in your concept.

    My proper framing of my example would be:
    IF you trust the observations and feelings of us testers
    AND our perceptions on start-up impressions isn’t optimal
    THEN it should be seen as a problem to take seriously
    AND THEREFORE you should consider changing the load sequence

    Notice that I removed the company logo, and the location of the cell phone, they are context-builders, not part of the logic.


Leave a Comment