Maximizing Your Testing Results

Testing computer software is more than just randomly executing portions of the software. Professional-level testing uses industry-accepted practices to write tests that are the most likely to find bugs. This article will examine how to use a combination of several common testing methods to maximize the results of the time you spend on testing.

Software test teams are often referred to as Quality Assurance teams, and rightfully so. Our job is to make sure we find as many defects (bugs) as possible, and see that they get fixed before the product gets into the hands of the customer. However, we cannot find all of the bugs, so we need to choose our test cases carefully, ensuring that they find the highest possible number of bugs. Cem Kaner in his book, Testing Computer Software, says, "A test that reveals a problem is a success. A test that did not reveal a problem was a waste of time." In Effective Testing Strategies in last month's Component Developer Magazine, Ellen Whitney covered some of the processes and techniques involved with testing computer software. This article will drill a little deeper into some of the testing techniques and areas that can help you make the most of your testing time.

You will not find all of the bugs

There are some folks who think that if they have enough testers and enough time, all of the bugs in a software product can be found. This is a silly assumption that is easily dismissed when one considers the complexity of computer software. Think of the last non-trivial program you wrote. How many possible inputs were there? How many possible outputs? Now, how many combinations of those inputs and outputs were there? Throw in the number of places the code can branch because of an IF or CASE statement, and you can be testing for a long time to get complete test coverage.

In 1975, Glenford J. Myers, author of The Art of Software Testing, wrote a "trivial" piece of code that would amount to about 20 lines in most languages. It had a few pieces of branching code, a loop, and that's about it. Myers calculated that it had 100 trillion different code paths. He estimated that a fast tester could test every possible code path in about a billion years.

Software can often fail after a number of iterations. Memory leaks add up and cause crashes, or sometimes something will fail after 1024 times, which just happens to be a whole hexadecimal number. But, you can't find all of those bugs, either. You can test something through a million iterations, but you would not have found the bug that fails on the millionth and first iteration.

Risk-Based Testing

Since we cannot test everything, we then need to prioritize what we can test. Using risk-based testing, we look for the biggest bang for our testing buck. This involves a two-pronged approach. First, we need to establish which code is most at risk in terms of potential bugs. What makes code risky? Here are a few items that I use to prioritize areas of code to test.

New code - Anything that has not been previously tested has no known level of quality.

Changed code - It has been estimated that the process of fixing a bug stands a 60% chance of breaking something else.

Complex code - Large, intricate sections of code are not only hard to write; they are hard to test.

Highly-dependant code ? If a function or other piece of code is called from many places in the program, we need to make sure it is stable.

Buggy code - If a lot of bugs have been found in an area of the program, there are probably still more bugs to be found.

Failing code - If there are regression bugs in some recently written code, there may be other bugs.

Next, we need to assess the impact potential bugs would have on users. Potentially risky areas are as follows:

Frequently used - Bugs that users see every day give the impression of instability in the product.

Critical areas - Users may close the general ledger in your accounting application only once a month, but it is crucial that the numbers balance.

Strategic features - If there is a feature in the product that makes the product stand above the competition, it had better work.

Once the risks have been established, they can be placed in a matrix from which test cases can be written (Figure 1).

As with test plans and test cases, it is important that this be documented. Inevitably, bugs will be found after the product is shipped, and management is going to want to know why the bugs were missed. Along with your list of test plans and test cases, the risk matrix can be used to demonstrate where your efforts were directed.

By: Mike Stewart

Mike is a software test lead for Microsoft's Visual FoxPro team. As a test lead, he directs the testing efforts of those on his team, as well as testing the Visual FoxPro product. He was an independent consultant for seven years before coming to Microsoft, specializing in custom database applications built with Visual FoxPro, C++, Delphi and Visual Basic.