“The key idea of the analytical domain is finding clever answers to interesting problems. We use models, mathematics, and other heuristics… Performing deep testing, which finds elusive bugs, generally requires a strong analytical approach.” – Domains of work; demands of testing (my previous blog post)

There is more to technology development than analysis. You can do a lot in a project that is not especially analytical. I would even say the majority of students who have come through my testing training classes find analysis intimidating, or at least unpleasant. This is why I split out five domains of work and list analytical work as just one of those domains. Still, it’s an important domain…and my personal favorite. That’s why I want to examine the analytical aspects of testing more deeply here.

The easiest way to see analysis in action in a testing situation is to ask and answer certain basic questions about the process:

• What exactly are you testing? Of all the things you could test, what are you not testing?
• What data are you testing with? What data should you be testing with?
• In the product you are testing right now, how do you recognize a bug when you see one? How might a bug be present and yet not visible to you? How might it be visible and yet not noticed by you?
• What kinds of people use your product? Who should be using it? How might that change over time?
• What changes could be made in the design of your product that would make it easier to test? What changes do we have to guard against because they would make it harder to test?
• If the product just changed, what is the best testing you can do in the next hour to catch problems that might have been introduced?
• What is the difference between an important problem and an unimportant one?

It’s true that you could answer these questions without performing any particular analysis. You could memorize answers you heard from someone else, or you could give only vague answers. But detailed, defensible, and responsible answers require analysis. And it’s not enough to perform that analysis in your own mind; you also need to be able to explain it to other people.

Tricentis offers tools that provide some assistance in answering these questions. I have been particularly impressed with the analytical features of Tricentis LiveCompare, a test impact analysis (TIA) tool that helps determine which subset of tests to execute for a given set of changes, fitting a certain risk profile. That is unfortunately not the norm. Most test tools focus more on administrative or technological aspects of the test process. It’s as if those tools are saying, “Wake me up when you decide what buttons to push. I can’t help you design tests, human!”

That will change, soon. As Technical Fellow at Tricentis, I am working to shape tooling that helps testers analyze better; tools that help testers design tests, not just push buttons. This is a key part of what we’re calling “the testing platform of the future.”

The joy of analysis

Here’s an example of one type of analytical work that I routinely encounter in testing. Imagine we are testing something that takes commands over a period of time. Lots of technology fits that description: vending machines, remote controls, calculators, a command-line interface, an operating system, etc. Let’s say there are four commands we can issue, and we can issue them in any order. Let’s also assume we are concerned that a particular sequence of commands might trigger a failure. We need to try every sequence, as cheaply as we can.

Question: What is the shortest single sequence of commands that will include all possible combinations of three-command sub-sequences?

Answer: We could use a de Bruijn sequence!

Just go to debruijnsequence.org and use the tools there to calculate this:

001330023011033101200311123332032102131323220221222312113030201000

This sequence packs all combinations of sequences into the shortest possible string. See for yourself. Is there a “121?” Yes. Is there a “330?” Yes.

If we’re testing a calculator, we could use each digit to represent one of the four commands:

0 = plus
1 = minus
2 = minus
3 = divide

Based on the above de Bruijn string, this translation table, and a simple sequence of digits, we could construct this expression:

1+2+3-4/5/6+7+8*9/8+7-6-5+4/3/2-1+2-3*4+5+6/7-8-9-8*7/6/5/4*3+2/1*2-3+4*5-6/7-8/9*8/7*6*5+4*3*2-1*2*3*4/5-6*7-8-9/8+7/6+5*4+3-2+1+2+3

When I paste that into the Microsoft Windows Calculator program, I get -19.1012. I get the same result when I run the calculation in a Perl program, and the same when I run it in Excel.

And that, my friends, is analytical testing.

But James…that isn’t analysis.

What?

No, that’s just applying some algorithm.

I see what you mean. Yes, like everyone else, I tend to fixate on the bright and shiny outcomes of analysis rather than the actual process that lurks and hovers behind the eyes of the analyst. What you see written above is from my first draft of this post. Indeed, the genuine acts of analysis are not represented. Let me go through it again and put a better spotlight on analysis.

The joy of analysis (second attempt)

Here’s an example of one type of analytical work that I routinely encounter in testing. Imagine we are testing something that takes commands over a period of time. Lots of technology fits that description: vending machines, remote controls, calculators, a command-line interface, an operating system, etc. Let’s say there are four commands we can issue, and we can issue them in any order…

Analysis: Order? That word immediately summons state models to my mind. Order matters. Technology is always sensitive to state, from the obvious case of trying to make a call on your phone without a SIM card, to the more subtle case of silently running too low on disk space and impacting virtual memory. States come and go, driven by events. Some systems are especially state-intensive, with many interacting and persistent variables. Others are affected by relatively few variables.

A command is an event, and an event modifies state. Therefore, the ordering of commands is also an exploration of how events affect the system in conjunction with the states produced or impacted by those events. In other words, when the user can do things in various orders, some of those sequences may trigger bugs, while others may not.

These thoughts flashed across my mind at the invocation of the word “order.” Have you also noticed that certain words or concepts light up your mind and trigger a cascade of actionable thoughts? That is one form of analysis. You improve this kind of analysis by giving yourself time and space to think, gaining experiences that create fodder for thinking, and studying General Systems Theory, mathematics, or just reading widely.

Let’s also assume that we are concerned that a particular sequence of commands might trigger a failure…

Analysis: All we have to do is try all the sequences. But wait, that would be impossible. “All the sequences” is an unlimited idea, whereas I test in a bounded world. How do I make infinity fit in me? This brings to mind sampling strategies. One way to make it tractable is to limit the length of the sequences that I try. Instead of no limit, I could limit the length to 100. That would require me to test 4100 sequences, which is still impossible. A length of 3 is quite tractable, though. There are only 64 of those.

To perform the required analysis, I had to be able to reason about combinations and permutations. I recommend that all testers either learn how to do that — or make friends who can. The basic things to learn about are combinations (different ways of grouping things) and permutations (different ways of ordering things). Assuming that any command can be issued at any time, and there are four commands, the full number of permutations for a sequence of three commands is given by the calculation 4 * 4 * 4, or 64.

I am also reasoning about risk. I know that any given long sequence is generally less likely to occur than any given shorter sequence. There are exceptions because user behavior is not at all random, but it’s a good general heuristic. Therefore, if I cover all the short sequences and I find a problem with one of them, it will probably be a more important bug than one that only occurs with ten specific commands performed in a specific order.

I chose the number 3 because it fits nicely in this blog post. In a natural test project, I would choose a length of 4 to 6.

Question: What is the shortest single sequence of commands that will include all possible combinations of three-command sub-sequences?
Answer: We could use a de Bruijn sequence!

Analysis: How did I decide that a de Bruijn sequence is the answer here? Step one is that I had to know about the existence and properties of de Bruijn sequences. This is a problem because I never learned about them in university (I never even finished high school, in fact). How, then?

Step zero was that my wife can’t afford to buy me what I want for Christmas, so she does the next best thing and gives me books about science and math. I have zillions of them. Ten years ago, she gave me Nets, Puzzles and Postmen: An Exploration of Mathematical Connections. That’s where I read about de Bruijn sequences, which are all about packing the most variation into the shortest set of steps. Of course, I immediately looked for ways to apply them to testing. (I relate everything to testing. My brother once challenged me to find a lesson about testing in the knitting section of a bookstore. It took less than a minute to discover something that I’ve used in my classes ever since.)

In this case, the analysis part is mostly in the learning process that preceded the moment of test design. I had prepared my mind to recognize certain patterns and connect them to certain methods. To do analysis well in the moment, you must become analytical over time. It’s an ongoing self-development process.

Are you feeling brainy?

The essence of analysis is the process of learning, which comes from the close examination of things. That leads to the apprehension of problems and the excitement of solving them. But you won’t do this unless you feel analytical; in other words, do you think of yourself as smart? A lot of people in our industry secretly feel stupid.

I’m lucky that I grew up around people who encouraged me to think. My family home was full of books. When my mother first subscribed me to Scientific American magazine, I was 10 years old. I struggled to read the articles, but I wanted to prove to her that she was right about me.

Not everyone gets that encouragement though. That’s why the most important work I do in my classes is to convince students that they can do analysis; that they can be good at it. It just requires patience and practice.

Testing needs analysis. It needs testers who have the ambition to think analytically. My ambition — and Tricentis’ ambition — is to create tools for thinking testers.

***

James Bach is a consulting software tester and Technical Fellow at Tricentis. He is also the founder and CEO of Satisfice, Inc., a software testing consultancy. James has been in the tech field as developer, tester, test manager, and consultant for 38 years. He is a founder of the Context-Driven school of testing, a charter member of the Association for Software Testing, the creator of Rapid Software Testing methodology and Session-based Test Management. He is also the author of two books: Lessons Learned in Software Testing and Secrets of a Buccaneer-Scholar: How Self-Education and the Pursuit of Passion Can Lead to a Lifetime of Success. For more about his work and online courses, see https://www.satisfice.com/.