Heuristics of Software Testing

Testability
Software testability is how easily, completely and conveniently a computer program can be tested.
Software engineers design a computer product, system or program keeping in mind the product testability. Good programmers are willing to do things that will help the testing process and a checklist of possible design points, features and so on can be useful in negotiating with them.
Here are the two main heuristics of software testing.
1. Visibility
2. Control

1.Visibility
Visibility is our ability to observe the states and outputs of the software under test. Features to improve the visibility are
• Access to Code
Developers must provide full access (source code, infrastructure, etc) to testers. The Code, change records and design documents should be provided to the testing team. The testing team should read and understand the code.
• Event logging
The events to log include User events, System milestones, Error handling and completed transactions. The logs may be stored in files, ring buffers in memory, and/or serial ports. Things to be logged include description of event, timestamp, subsystem, resource usage and severity of event. Logging should be adjusted by subsystem and type. Log file report internal errors, help in isolating defects, and give useful information about context, tests, customer usage and test coverage.
The more readable the Log Reports are, the easier it becomes to identify the defect cause and work towards corrective measures.
• Error detection mechanisms
Data integrity checking and System level error detection (e.g. Microsoft Appviewer) are useful here. In addition, Assertions and probes with the following features are really helpful
 Code is added to detect internal errors.
 Assertions abort on error.
 Probes log errors.
 Design by Contract theory---This technique requires that assertions be defined for functions. Preconditions apply to input and violations implicate calling functions while post-conditions apply to outputs and violations implicate called functions. This effectively solves the oracle problem for testing.

• Resource Monitoring
Memory usage should be monitored to find memory leaks. States of running methods, threads or processes should be watched (Profiling interfaces may be used for this.). In addition, the configuration values should be dumped. Resource monitoring is of particular concern in applications where the load on the application in real time is estimated to be considerable.

2.Control
Control refers to our ability to provide inputs and reach states in the software under test.
The features to improve controllability are:
• Test Points
Allow data to be inspected, inserted or modified at points in the software. It is especially useful for dataflow applications. In addition, a pipe and filters architecture provides many opportunities for test points.
• Custom User Interface controls
Custom UI controls often raise serious testability problems with GUI test drivers. Ensuring testability usually requires:
 Adding methods to report necessary information
 Customizing test tools to make use of these methods
 Getting a tool expert to advise developers on testability and to build the required support.
 Asking third party control vendors regarding support by test tools.

• Test Interfaces
Interfaces may be provided specifically for testing e.g. Excel and Xconq etc.
Existing interfaces may be able to support significant testing e.g. InstallSheild, AutoCAD, Tivoli, etc.
• Fault injection
Error seeding---instrumenting low level I/O code to simulate errors---makes it much easier to test error handling. It can be handled at both system and application level, Tivoli, etc.
• Installation and setup
Testers should be notified when installation has completed successfully. They should be able to verify installation, programmatically create sample records and run multiple clients, daemons or servers on a single machine.


A BROADER VIEW
Below are given a broader set of characteristics (usually known as James Bach heuristics) that lead to testable software.


Categories of Heuristics of software testing

•Operability
The better it works, the more efficiently it can be tested.
The system should have few bugs, no bugs should block the execution of tests and the product should evolve in functional stages (simultaneous development and testing).
•Observability
What we see is what we test.
 Distinct output should be generated for each input
 Current and past system states and variables should be visible during testing
 All factors affecting the output should be visible.
 Incorrect output should be easily identified.
 Source code should be easily accessible.
 Internal errors should be automatically detected (through self-testing mechanisms) and reported.

•Controllability
The better we control the software, the more the testing process can be automated and optimized.
Check that
 All outputs can be generated and code can be executed through some combination of input.
 Software and hardware states can be controlled directly by the test engineer.
 Inputs and output formats are consistent and structured.
 Test can be conveniently, specified, automated and reproduced.
•Decomposability
By controlling the scope of testing, we can quickly isolate problems and perform effective and efficient testing.
The software system should be built from independent modules which can be tested independently.
•Simplicity
The less there is to test, the more quickly we can test it.
The points to consider in this regard are functional (e.g. minimum set of features), structural (e.g. architecture is modularized) and code (e.g. a coding standard is adopted) simplicity.
•Stability
The fewer the changes, the fewer are the disruptions to testing.
The changes to software should be infrequent, controlled and not invalidating existing tests. The software should be able to recover well from failures.
•Understandability
The more information we will have, the smarter we will test.
The testers should be able to understand well the design, changes to the design and the dependencies between internal, external and shared components.
Technical documentation should be instantly accessible, accurate, well organized, specific and detailed.
•Suitability
The more we know about the intended use of the software, the better we can organize our testing to find important bugs.

The above heuristics can be used by a software engineer to develop a software configuration (i.e. program, data and documentation) that is convenient to test and verify.