Test Strategy (OR Test Approaches)

A test strategy is an outline that describes the testing portion of the software development cycle. It is created to inform project managers, testers, and developers about some key issues of the testing process. This includes the testing objective, methods of testing new functions, total time and resources required for the project, and the testing environment.

In the test strategy is described how the product risks of the stakeholders are mitigated in the test levels, which test types are performed in the test levels, and which entry and exit criteria apply.

The test strategy is created based on development design documents. The system design document is the main one used and occasionally, the conceptual design document can be referred to. The design documents describe the functionalities of the software to be enabled in the upcoming release. For every set of development design, a corresponding test strategy should be created to test the new feature sets.

One way to classify test approaches or strategies is based on the point in time at which the bulk of the test design work is begun:
o Preventative approaches, where tests are designed as early as possible.
o Reactive approaches, where test design comes after the software or system has been
produced.

Typical approaches or strategies include:

* Analytical approaches, such as risk-based testing where testing is directed to areas of greatest
risk.
* Model-based approaches, such as stochastic testing using statistical information about failure
rates (such as reliability growth models) or usage (such as operational profiles).
* Methodical approaches, such as failure-based (including error guessing and fault-attacks),
experienced-based, check-list based, and quality characteristic based.
* Process- or standard-compliant approaches, such as those specified by industry-specific
standards or the various agile methodologies.
* Dynamic and heuristic approaches, such as exploratory testing where testing is more reactive
to events than pre-planned, and where execution and evaluation are concurrent tasks.
* Consultative approaches, such as those where test coverage is driven primarily by the advice
and guidance of technology and/or business domain experts outside the test team.
* Regression-averse approaches, such as those that include reuse of existing test material, extensive automation of functional regression tests, and standard test suites.

Different approaches may be combined, for example, a risk-based dynamic approach.

The selection of a test approach should consider the context, including:

* Risk of failure of the project, hazards to the product and risks of product failure to humans, the
environment and the company.
* Skills and experience of the people in the proposed techniques, tools and methods.
* The objective of the testing endeavour and the mission of the testing team.
* Regulatory aspects, such as external and internal regulations for the development process.
* The nature of the product and the business.

Test Levels

The test strategy describes the test level to be performed. There are primarily three levels of testing: unit testing, integration testing, and system testing. In most software development organizations, the developers are responsible for unit testing. Individual testers or test teams are responsible for integration and system testing.

Roles and Responsibilities

The roles and responsibilities of test leader, individual testers, project manager are to be clearly defined at a project level in this section. This may not have names associated: but the role has to be very clearly defined.

Testing strategies should be reviewed by the developers. They should also be reviewed by test leads for all levels of testing to make sure the coverage is complete yet not overlapping. Both the testing manager and the development managers should approve the test strategy before testing can begin.

Environment Requirements

Environment requirements are an important part of the test strategy. It describes what operating systems are used for testing. It also informs clearly the necessary OS patch levels and security updates required. For example, a certain test plan may require service pack 2 installed on the Windows XP OS as a prerequisite for testing.

Testing Tools

There are two methods used in executing test cases: manual and automation. Depending on the nature of the testing, it is usually the case that a combination of manual and automated testing is the most optimal testing method. Planner should find the appropriate automation tool to reduce total testing time.

Risks and Mitigation

Any risks that will affect the testing process must be listed along with the mitigation. By documenting the risks in this document, we can anticipate the occurrence of it well ahead of time and then we can proactively prevent it from occurring. Sample risks are dependency of completion of coding, which is done by sub-contractors, capability of testing tools etc.

Test Schedule

A test plan should make an estimation of how long it will take to complete the testing phase. There are many requirements to complete testing phases. First, testers have to execute all test cases at least once. Furthermore, if a defect was found, the developers will need to fix the problem. The testers should then re-test the failed test case until it is functioning correctly. Last but not the least, the tester need to conduct regression testing towards the end of the cycle to make sure the developers did not accidentally break parts of the software while fixing another part. This can occur on test cases that were previously functioning properly.

The test schedule should also document the number of tester available for testing. If possible, assign test cases to each tester.

It is often difficult to make an accurate approximation of the test schedule since the testing phase involves many uncertainties. Planners should take into account the extra time needed to accommodate contingent issues. One way to make this approximation is to look at the time needed by the previous releases of the software. If the software is new, multiplying the initial testing schedule approximation by two is a good way to start.

Regression Test Approach

When a particular problem is identified, the programs will be debugged and the fix will be done to the program. To make sure that the fix works, the program will be tested again for that criteria. Regression test will make sure that one fix does not create some other problems in that program or in any other interface. So, a set of related test cases may have to be repeated again, to make sure that nothing else is affected by a particular fix. How this is going to be carried out must be elaborated in this section. In some companies, whenever there is a fix in one unit, all unit test cases for that unit will be repeated, to achieve a higher level of quality.

Test Groups

From the list of requirements, we can identify related areas, whose functionality is similar. These areas are the test groups. For example, in a railway reservation system, anything related to ticket booking is a functional group; anything related with report generation is a functional group. Same way, we have to identify the test groups based on the functionality aspect.

Test Priorities

Among test cases, we need to establish priorities. While testing software projects, certain test cases will be treated as the most important ones and if they fail, the product cannot be released. Some other test cases may be treated like cosmetic and if they fail, we can release the product without much compromise on the functionality. This priority levels must be clearly stated. These may be mapped to the test groups also.

Test Status Collections and Reporting

When test cases are executed, the test leader and the project manager must know, where exactly we stand in terms of testing activities. To know where we stand, the inputs from the individual testers must come to the test leader. This will include, what test cases are executed, how long it took, how many test cases passed and how many failed etc. Also, how often we collect the status is to be clearly mentioned. Some companies will have a practice of collecting the status on a daily basis or weekly basis. This has to be mentioned clearly.

Test Records Maintenance

When the test cases are executed, we need to keep track of the execution details like when it is executed, who did it, how long it took, what is the result etc. This data must be available to the test leader and the project manager, along with all the team members, in a central location. This may be stored in a specific directory in a central server and the document must say clearly about the locations and the directories. The naming convention for the documents and files must also be mentioned.

Requirements Traceability Matrix

Ideally each software developed must satisfy the set of requirements completely. So, right from design, each requirement must be addressed in every single document in the software process. The documents include the HLD, LLD, source codes, unit test cases, integration test cases and the system test cases. Refer the following sample table which describes Requirements Traceability Matrix process. In this matrix, the rows will have the requirements. For every document {HLD, LLD etc}, there will be a separate column. So, in every cell, we need to state, what section in HLD addresses a particular requirement. Ideally, if every requirement is addressed in every single document, all the individual cells must have valid section ids or names filled in. Then we know that every requirement is addressed. In case of any missing of requirement, we need to go back to the document and correct it, so that it addressed the requirement.

Test Summary

The senior management may like to have test summary on a weekly or monthly basis. If the project is very critical, they may need it on a daily basis also. This section must address what kind of test summary reports will be produced for the senior management along with the frequency.
The test strategy must give a clear vision of what the testing team will do for the whole project for the entire duration. This document will/may be presented to the client also, if needed. The person, who prepares this document, must be functionally strong in the product domain, with a very good experience, as this is the document that is going to drive the entire team for the testing activities. Test strategy must be clearly explained to the testing team members right at the beginning of the project.