Software Test Management

The effectiveness of Automated Testing wherever applicable in software test processes has been rarely disputed. Automated Testing reduces testing time, identifies bugs faster and facilitates continuous development (CD). However, Automated Testing environments tend to get extremely large (one with over 100 test scenarios would qualify as large and test scenarios can easily run into 1000’s) and tedious to maintain. It produces results that cannot be fully relied on. Therefore, it is critical to managing a large test environment effectively using well-defined processes and a decent set of tools.

If your test library is large, it is important that you find out if the individual test scenarios are properly organized, critical to the core application and relevant to the latest application version or to the usage pattern.

1. Test Organization: Your applications tests may involve testing various functional capabilities of an application. How do you determine which tests to run after you have made updates to just one section of your application (for example, your “billing” functionality has been updated)? You may not want to run all your test scenarios (unless doing regression tests) in order to test a small update. Your test framework should allow you to categorize your tests by functional areas so that you can quickly filter out everything else and run only those tests that belong to a specific function.

2. Test Criticality: Ideally you would create tests that cover both the critical and non-critical areas of your application. Your testing framework should provide an efficient way to manage bugs and failures. You should be able to categorize your test upfront by criticality (Critical, High, Medium or Low) so as to prioritize the testing as well as the resolution based on the criticality of the test.

3. Test Relevance: In most environments, software development and testing is a continuous process and over time you can have a large number of tests. Unfortunately, as software evolves (with new or updated features, changed processes, etc.) some of the tests become redundant or duplicate. It can be challenging managing test library to ensure that only quality test cases exist, especially if your testing framework does not allow for easy review of the individual test cases/scenarios. You should be able to log and keep a track of the entire test creation life cycle (including updates). For instance, you should be able to track the test creation date, the module/release version for which the test was created, the test case revision history along with the name of individuals who authored the tests. This would make a periodic review of test cases to identify redundant scenarios much easier and keep your test library lean and clean. It will ensure a test environment that provides high-quality test results that you can rely on to make decisions. You do not want to rely on the results from redundant tests by ignoring “passed” results or waste your time by acting on “failed” results that they produce.

4. Test Usage: If there are test scenarios exceeding 2 months sitting in your library, there is a possibility that they are just taking up space and may not be required anymore. It is vital that your testing framework should be able to identify such test scenarios so that you can make apt decisions, either to re-prioritize or re-categorize them for efficient usage or get rid of them.

By maintaining your test libraries through regular checks, you will certainly have a manageable test environment that produces reliable results on which you can make assertions and also ensure a faster and better software development process.

CloudQA and Effective Test Management