Why should you care about Test Metrics?
Software testing metrics are a powerful way to express data, project progression, ROI and anything else you want to present. Often, test coverage metrics is what we and other project stakeholders rely on when making decisions, so they need to be handled with care. We are all required to report our work efforts on a regular basis, as a means to communicate project progression status. Many times the test management tools we use produces reports automatically and we just use whatever default testing metrics and reports it spits out and then distribute them to all involved, but that is a big mistake!
Test metrics is, in fact, a powerful tool that can be used for good or for bad, much like a weapon. The quality metrics and reports you produce are trusted and used to make actionable decisions, which, can make or break any project. That is why it’s important to provide software testing metrics responsibly.
This ultimate guide to testing metrics will take you through the pros and cons of test metrics, and how to plan and maintain a useful software testing metrics program that can serve a real value.
The Pros of Software Testing Metrics
1. Effective means of communication – Not only for communicating status but also for conveying what is important and where effort is needed.
2. Usually clear – Good metrics are easy to understand by people both in and outside the team.
3. Benchmark and compare more objectively Between teams, processes, versions and more.
4. Track progress – See how things change over time, make educated predictions and corrections as you go.
5. Easy to distribute – Share or send simple spreadsheets, chart & reports by emails, or designated online dashboard displays.
6. People like them – They are compact and should be eye-pleasing ways of communicating information.
The Cons of Software Testing Metrics
1. People don’t like being measured – Do you like having your productivity measured by the number of hours you stay at the office? We didn’t think so…
2. Present only “one-side” of the story – test metrics can be contorted into showing anything you want, so you end up seeing one version of the truth and not the whole picture.
3. Software testing metrics are going to be used by managers as well, who might jump to conclusions without waiting to listen to all the information and will try to apply quick fixes to complex issues, while “hiding” those issues behind graphs and tables.
The Laws of (Conservation of) Metrics
- Anything that is measured will improve over time
- Whenever something improves, it will be at the expense of something else that will worsen accordingly
- You can only measure 5 (sometimes only 3) aspects of anything. Trying to measure more will lead to confusion and failure
- Metrics will become more ineffective after a number of iterations
Planning a Software Testing Metrics Program
1. Define the objectives of your metrics.
Who is your audience? Management? your team? your users? regulation or HR?
Each audience has different interests regarding project QA. Your managers might care more about time and budget spend on testing and what value that has brought, while HR might care more about team productivity, and your users would care more about when the latest feature update, for instance, will be released.
Tip: Customize your test metrics to your audience. Each of them needs to see different information to make different decisions
2. Make sure you have the right testing metrics. Ask yourself the 5 Why’s – To peel away the layers and reach your core metrics.
3. Make sure you build your test metrics correctly. Be S.M.A.R.T – all metrics need to be:
Simple -Clear and easy to follow
Measurable – Numbers and statistics over descriptions
Actionable – Propose solutions & not only point at Flaws
Repetitive – Show information over versions and products
Timely – Provide the information when it is still relevant
Tip: Some of your most important software testing metrics may be available ONLY after a project is done (external to the testing process) but should be included: Escaping Defects, Rejected Defects, Server Exception Levels, Support Calls Post-Release, etc.
Maintaining a Software Testing Metrics Program
1. Software testing metrics need to be constantly reviewed for both accuracy and relevancy. Therefore it is important to learn to adapt. As your project, technology and people change so should your metrics.
2. Don’t get too comfortable! You need to change your test metrics once in a while, bugs and people learn to work around them.
3. Stay Agile – even with your testing metrics: Be ready to change metrics, even halfway through your project
Remember! Testing is a service, and you are a service provider. Therefore, your software testing metrics plan should be of service as well. Understand who the client of your service is, and remember to serve his/her needs, not yours!
Tips and Tricks for working with Test Metrics
- The presentation is as important as the content.
- Have different metrics for different people.
- Capture more data than you need today, you may need it later.
- Provide analysis of your results with every graph (don’t trust the people to understand what they see).
- Make sure to provide multiple angles on the same data.
- Too many metrics can confuse – tell a story, but keep it interesting.
- Push metrics, don’t make people ask for them since it won’t be relevant by the time you provide them.
For more on software testing metric check out this webinar:
