Of automated test case management

We recently published an article on all-pairs testing (read it here) by Keith Vanden Eynden of Seapine Software, in which he was enjoined not to “push” Seapine’s tools.

Nevertheless, Seapine’s TestTrack TCM tool is still an interesting product, accessible to the smaller business, so we asked Keith about using it in practice – and, especially, about using it to implement All-Pairs Testing.

David Norfolk, for Reg Dev: OK, very basic, what does TCM stand for?

Vanden Eynden:TestTrack TCM is our Test Case Management solution.

Reg Dev: And what aspects of testing does it cover?

Vanden Eynden:TestTrack TCM handles all of the tasks associated with the software testing process including test case creation, test scheduling and execution, plus test results measurement and reporting.

Reg Dev: Surely, all test case management tools worthy of the name do all that – or am I being naive? What makes TestTrack TCM special?

Vanden Eynden: Other test case management tools don’t offer as comprehensive a set of functions and aren’t as easy to install, customize and use. The biggest differentiator [according to Vanden Eynden], however, is TestTrack TCM’s ability to push test failures immediately and directly into your defect management workflow. This saves time by eliminating manual data entry and the errors it inevitably introduces. In addition, with this connection between test case management and defect management, software engineers are provided with the steps needed to reproduce a bug. This also saves them time and increases the probability of a first time fix of the issue.

TestTrack TCM also supports automated testing tools, such as QA Wizard Pro, so it can automate everything from building the test runs, running the automated tests, to moving any resulting defects into the defect management workflow.

Reg Dev: OK, given that we’re using Test Track for now, how can we use it to demonstrate All-Pairs Testing?

Vanden Eynden:TestTrack TCM’s test variants allow you to define the variables that apply to the applications you are testing and also allow you to implement all-pairs testing easily. The sample project installed with TestTrack TCM contains the variants and values shown in Figure 1.

First you have to design and construct your all-pairs test runs. Then you begin by customizing the Database and Client Type variant values and adding a new Browser variant – see Figure 2.

With these variants in place, you can create one test case for the single sign-on feature and then generate test runs to cover different all-pairs combinations.

Reg Dev: OK then, how do you create the test case and run the tests?

Vanden Eynden:You can create a single test case and use different variant values to generate a range of test runs and provide the desired test coverage. For example, you create one test case for the single sign-on feature that contains all the information about how to set up and run the single sign-on test – see Figure 3.

Then you attach the all-pairs matrix to the test case to specify which variant value combinations to use when generating test runs – see Figure 4

After the test case is reviewed and approved, you can generate the 12 test runs identified earlier. You generate the test runs individually using the all-pairs matrix to complete the Generate Test Runs dialog box – see Figures 5 and 6.

After the first test run is generated, repeat the process to generate all the required combinations – see Figure 7.

Reg Dev: Just one point I’m not clear about. Where/when where does the actual test case code get written or generated? Does the QA team have to at least write the tests once? And then is it possible for test code to be stored in a `library’ and called as required?

Vanden Eynden: Of course someone has to author each test case, but, once written, they can be stored and used as templates for another project, or modified as needed to create new test cases.

Reg Dev: Well, very impressive, but what are the advantages of this approach?

Vanden Eynden: All-pairs testing enables you to achieve suitable test coverage without testing every possible combination. All-pairs testing can also reduce the number of test runs by 75 percent or more. With that time saving, you will be able to spend more time developing complex tests and less time performing mundane tests on the combination of variables.

Reg Dev: And, are there any disadvantages?

Vanden Eynden: Since you are not testing every possible combination, you introduce a little risk into the testing process, but it is a managed risk. One has to realize that all-pairs testing is a cost-benefit compromise, but a reasonable trade-off for most situations.

Reg Dev: Will you still need other approaches to testing too? What are they?

Vanden Eynden: No software testing technique can be expected to find all bugs. If all-pairs testing is an appropriate choice for your project, use it in combination with unit testing, code reviews, exploratory testing and other applicable tools that your organization regularly uses.

Reg Dev: What about managing testing as a whole?

Vanden Eynden: First, you have to invest in proper test planning and the creation of test cases. This is a big hurdle for many companies, but a necessary step to really getting control over quality. While it is hard to imagine developing software without defect tracking and source control tools in your arsenal, it wasn’t so long ago that these tools were seen as a luxury or non-essential. Tools for managing the testing process as a whole are still in that early stage. Products like TestTrack TCM make the management of test cases much easier and help companies see where they are light on test coverage. Once you have a tool like this, which saves you time and improves your ability to measure progress and quality, it is hard to go back to spreadsheets, Word documents, or paper.

Reg Dev: What about linking the things you’re are testing back to the Business Requirements which are, presumably, behind the application functions you are testing?

Vanden Eynden: This is desirable in many instances and required in some. TestTrack Pro can manage the business requirements and, as we discussed earlier, the integration between TestTrack TCM and our other tools provides this link between test outcomes and the managed requirements.

Reg Dev: Finally, how do you see Seapine’s tools developing in the face of evolving ALM (Application Lifecycle Management) offerings and the rise of Eclipse?

Vanden Eynden: Major releases of our flagship products are now entering the quality assurance phase and we have several new products on the drawing board. We see ourselves as part of the evolution of ALM so we think we are on the right track with our product set. Eclipse tools have their place and their supporters but there’s a need for commercial tools as well. Our goal is to offer a commercial ALM platform that is accessible from Eclipse wherever possible.

Reg Dev: Thanks. ®