Friday, 19 May 2023
We all want quality software, it’s well-known the impact that problems or difficulties in using software can have on businesses today; in everyday life the software on our phones, in our cars, at work and during leisure time all play their part and any frustration can quickly drive customers into the arms of competitors. Testing plays an important part in ensuring quality is maximised, the more effective testing is, the more confidence software delivery teams and company executives can have in the software.
An old testing maxim says that exhaustive testing is not possible, which is still true today, so tests need to be targeted. This makes measuring the effectiveness of testing more important; there needs to be measurement to assess and improve. Organisations have set up a range of measures to gauge testing efficiency, we have included 8 of these below.
- Coverage
Ensuring the right level of test coverage means using either requirements or code based measurements. Which one is best will depend on the level of testing being done, as well as the complexity and criticality of the system. Code level coverage is often better suited to unit testing and can be measured via lines of code, decision coverage or statement coverage. These metrics are naturally available from testing tools, rather than via a manual process. Requirements based coverage relating test cases and conditions to requirements and while not as granular, can ensure that each requirement has some test coverage.
- Defect Density
Defect density is the number of bugs found in a particular software area. This KPI helps businesses assess the software’s quality and the performance of the development team in particular areas or system modules. For example, defect density can help determine when elements of a software application are ready to be released, or when it passes out of the alpha or beta phase (i.e., when the defect density goes below a certain percentage). Defect density is also a leading indicator of where regression tests should be focused, manual as well as automated.
- Tests Executed
Monitoring the total number of tests run is a critical indicator of the testing team's productivity. The number of tests executed is an example of a velocity KPI, which quantifies how quickly teams and businesses can carry out their tasks. The tests executed KPI fits together naturally with the next on the list, the time taken to test.
- Time to Test
Organisations can assess the quality and the effectiveness of their software testers by using time to test. This KPI can also identify more complex software features such as the longer the time it takes to test, the more intricate the feature’s design is. Time to test can be related to functionality, test level and number of cycles run. Time to test is a key consideration in manual testing when organisations are looking to assess the ROI of planned improvements, especially test automation.
- Defect Resolution Time
A team's level of software defect fix effort can be evaluated by tracking the time it takes to resolve a bug. The lower this KPI, the faster organizations can find and patch critical defects in their applications. To evaluate the effectiveness and productivity of various testers in the team, defect resolution times can be monitored both on a company-wide and on an individual basis. When working closely with development teams, this metric becomes a useful factor when estimating timescales for future effort / cycles.
- Quality Ratio
The quality ratio formula is simply the number of test cases that are executed successfully, divided by the total number of test cases, times 100. The testing manager and other stakeholders can quickly see the state of testing at a glance, using this ratio. For example, if the Quality Ratio declines between two different test cycles (without introducing any new test cases), this is a sign that new features or fixes are having a detrimental effect.
- Unresolved Vulnerabilities
Keeping a track of security vulnerabilities is an essential practice for any software development team. The unresolved vulnerabilities KPI helps gauge the quantity and severity of security issues, so that they can be triaged appropriately. This metric can be combined with other KPIs, such as time to resolution, to better understand the effectiveness of a team’s software security efforts.
- Automation Coverage
Evaluates the percentage of test cases automated vs those executed manually. While manual testing still has an important place (for example, acceptance and usability testing), a great deal of testing can and should now be automated. Considering automation is much faster and more efficient than manual testing, increasing the percentage of automated tests can help improve a team’s overall productivity, leaving more time for more meaningful manual tasks such as exploratory testing and error guessing.
Finally, it is worth mentioning that an overall Test Efficiency Index is a useful combination of the above metrics, each with a weighting assigned based on their relative importance to your organisation. This can be an easy way to quickly see a trend in how testing is (hopefully) being made more efficient.
Click here to connect with one of our experts for a no-obligation discussion and how we can help.
Vishal Jakkulwar - Principal Consultant
Prolifics Testing UK
