The Ins and Outs of a Requirement Traceability Matrix (RTM)

In this article, we explore the concept of traceability in software testing and how to document bi-directional traceability in a requirements traceability matrix (RTM).

Introduction

One of the very first things a tester asks is, “What is this thing supposed to do?” Unfortunately, this is a question that often goes unanswered in any type of written specifications.

Ideally, the definition of functionality and attributes of what is built and tested would be found in a requirements specification document. The problem is, most organizations don’t have adequate requirements, if any at all.

Certainly, it is possible to test software without having any form of specification, but the problem of knowing what constitutes “correct” behavior remains.

The Basis of Testing

One of the fundamental things that is needed to test software in a measurable way, is to have something to use as a basis of testing. The test basis is anything that can be used from which to derive tests.

The basis of testing can also be used as a test oracle. A test oracle is simply the source of your knowledge about how something should or should not work. For example, if you are creating a new application to replace an older application, the older application could perhaps act as a test oracle to show how calculations are currently performed.

Other test oracles could be specifications such as requirements, user stories, use cases and so forth, or even people who are domain experts or users.

Without a basis of testing you are left guessing about not only what to test, but also if the results of that test are correct or not. Absence of a basis of testing typically means that you have to research the outcomes of each test to determine correctness. This can be a very time consuming and error prone task.

When everything is said and done, taking the time to document the basis of testing can actually save a lot of time and headaches.

Enter the Requirements Traceability Matrix (RTM)

The purpose of the requirements traceability matrix, also known as an RTM, is to document and trace the relationship between requirements and other testing assets. A testing asset in this context could be test cases, test scripts, test procedures and other similar items used in testing.

The goal is to have a single place where the level of coverage can be determined for both requirements and tests.

A Brief Background of the RTM

Requirements traceability matrices have been used since the very early days of software testing. If you examine some of the early works on software testing such as written by William E. Perry, Boris Beizer and others, you will see the use of RTM's as a way to plan, control, and measure software testing activities (Figure 1).

Figure 1 – A Low-Tech RTM

In the very early days of software testing, an RTM was implemented using a grid built in whatever means might be available. In some cases, it was simply a paper-based spreadsheet or grid. Then, as spreadsheets and word processing software became more prevalent, RTM's were implemented in those kinds of applications (Figure 2).

Figure 2 – A Spreadsheet-based RTM

One of the big issues with spreadsheets and other documents is that they can get scattered and hard to manage. It’s very easy to have many different versions of the same matrix when they are distributed by e-mail and other means. People may not know exactly which version of the matrix (or matrices) is the most current one.

Fortunately, the concept of traceability is a central feature of test management tools. Most test management tools, such as PractiTest, have the capabilities of tracing requirements to test cases and scripts. The great advantage of establishing traceability in a test management tool is that as tests are executed and results are captured, no extra effort is needed to update a separate matrix. Traceability is a common item to report from a test management tool (Figure 3).

Figure 3 – Requirements Traceability as Seen in PractiTest

Now that we have seen the basic idea of a Requirements Traceability Matrix, let’s look more at the rationale behind the RTM.

The Role of Requirements in Testing

As we start thinking about a requirements traceability matrix and how to implement such a matrix, we must also start thinking about the state of requirements in our organization.

For example, if we know that requirements are often missing or inadequate, then it will be very difficult to have a meaningful requirements traceability matrix. So, we will need a way to establish some basis of testing and traceability. Requirements are the ideal basis of testing, but as we will see, other items can be used as well for a test basis.

If we realize that we are not able to report if our test actually covered enough functionality, that alone could be a significant motivator to start addressing the need for requirements.

Requirements serve a very useful purpose in software development and testing. There are at least four major reasons for having some level of requirements.

If you can’t define something in a basic statement of need or requirements:

• How can you know you're building the right thing?
• How do you know what to test?
• How do you know if the thing you are testing is correct?
• How do you know when you're finished?

So, the ticket in the door to create an RTM is that you first must have some basis of testing such as requirements.

It is possible as a tester, to define a set of requirements after you have received a product to test. This is called “requirements reengineering” or “retrofitting” and may be helpful, but it might not actually be a productive effort in all cases.

If you expect to be maintaining and testing a particular application for the long-term, then requirements reengineering may have enough long-term value in traceability and other purposes to consider.

Requirements Can Take Other Forms

It is important to note that requirements may be expressed in a variety of ways.

While there is no substitute for a fully defined functional requirement, there are some other items that can enhance understanding of what is desired and may serve as a basis of testing. These include use cases and user stories.

Also, models such as process models, data models, state-transition models, and others can give additional understanding of requirements in a visual sense. However, models require interpretation which make them open to mis-interpretation.

A reasonable combination is a verbal written requirement, accompanied by models for visualization.

Why Traceability is Important in Software Testing

Actually, traceability is important in all of software development, not just testing.

Stakeholders naturally want to know how much of something has been built and tested. Traceability is how we can provide this information about test coverage

In software testing, one of the key areas of metrics is that of coverage. For example, how many of the requirements have been tested?

Warning: Test Coverage Can be Misleading

Test coverage is a tricky thing because you can test something completely in terms of percentage, but not necessarily well in terms of depth.

For example, it is possible to test 100% of all the defined requirements on a project with at least one test per requirement. But is one test case per requirement a good test? Probably not.

At best, you would be testing only the positive side of a condition.

Then, consider the need to test various requirements in conjunction with other requirements. That can make the goal of “complete” test coverage very elusive – perhaps even impossible.

Why Bi-Directional Traceability?

Traceability is one thing, but bi-directional traceability is another.

For example, if I can demonstrate that a particular test case relates back to a particular requirement, I have demonstrated traceability (Figure 4).

Figure 4 – Traceability to Requirements

In addition, if I can demonstrate that a particular requirement has an associated set of test cases to verify it, I have demonstrated traceability in a different way (Figure 5).

Figure 5 – Traceability to Tests

But if I can show traceability in both directions, to and from requirements and test cases, then I have shown bidirectional traceability (Figure 6).

Figure 6 – Bi-directional Traceability

Bi-directional traceability is important for multiple reasons:

1. When planning tests, it's good to know that I have considered all the available requirements in my tests. This requires that I can show trace ability from my test back to requirements.
2. When performing tests, I need to know if a result is correct or not, so traceability back to requirements gives me a way to know what constitutes a pass or fail result.
3. When reporting test results, I need to be able to convey to my stakeholders’ definite metrics of how many requirements or other bases of testing as a percentage have been covered. In addition, I need to be able to tell them how many of those requirements are working correctly and how many are not.
4. When a requirement changes, I need to know which test cases are impacted by the change so I can change the test cases appropriately.

How to Create a Requirement Traceability Matrix

The process to create an RTM is very straightforward:

1. Collect the requirements or other basis of testing (user stories, etc.).
2. Create a matrix in some form – spreadsheet, word processing document, etc. The leftmost column is an identifier for the requirement and the column to the right of that will contain a very brief, high-level name of the requirement. (Figure 7)
3. Fill in the left two columns with the IDs and requirement titles.
4. Collect the test cases or other test assets.
5. Working row by row, in each column to the right of the requirement title, place a test asset ID in each cell. (See Figure 2 for a basic example).

Requirements Traceability Matrix Template

Here is a sample blank RTM implemented in a spreadsheet. You can download it here.

Figure 7 – Blank RTM Template

Requirements Traceability Matrix Example

Here is an example RTM implemented in a spreadsheet.

Figure 8 – RTM Spreadsheet With Test Status

How Requirements Traceability is Seen in PractiTest

Requirements traceability in PractiTest is very easy. One reason for that is because you can also manage requirements in PractiTest. Once the requirements are captured (Figure 9), then you can easily associate any of your tests to them (Figure 10). Notice also that you can also link defect or issue reports to a particular requirement.

Figure 9 - A Sample Requirement Documented in PractiTest

Figure 10 – A Test Linked to a Requirement

In addition, you can also see the history of all changes to a requirement (Figure 11).

Figure 11 – Requirement History in PractiTest

Challenges With the RTM

While the Requirements Traceability Matrix has great value, it is not without its challenges. These challenges include:

1. Having good requirements at the start

   As stated earlier, the basis for any RTM is a set of specifications. If these are lacking, you will not be able to build a requirements traceability matrix. However, in the next topic I will give an idea for building an RTM without requirements, user stories or similar basis.

2. Staying current with requirements and tests

   Once you build the RTM, you must keep it current. This can be a very big challenge because both requirements and tests are constantly changing.

3. Having visibility to the RTM for everyone who needs it

   Whether you use a test management tool or a shared repository of documents, the RTM(s) need to be visible at all times to whoever needs to know the status of testing, and the extent of test coverage.

How to Build an RTM Without a Basis of Testing

Let’s take an example which I have seen in multiple projects.

The project is well underway and you are tasked with creating an RTM – with no requirements, user stories or anything of a similar nature. Plus, you don’t have the time to retrofit or reverse engineer requirements. Here is what I have successfully done.

Start with defining the test objectives in terms of system or application functionality. For example, in an e-commerce application:

“Validate order totaling is correct”
“Validate tax rates are applied correctly”

Test objectives are normally high-level. They are very important as they essentially define the scope of testing.

The great thing about this approach is that if you are writing a test plan, you need to define test objectives anyway.

From this point, you create the matrix as described earlier. Each row will be a test objective, with associated test cases in the columns to the right. (Figure 12).

Figure 12 – Test Objectives as a Substitute for Requirements

Conclusion

Of all the testing assets available, the Requirements Traceability Matrix is perhaps one of the most valuable items because it tells not only the extent of testing (coverage), but also the need for updating tests when requirements change. The RTM also can be a valuable tool to aid in evaluating test results if bi-directional traceability has been established. Tools such as PractiTest can greatly facilitate this effort.

Randall W. Rice, CTAL

Randall W. Rice is a leading author, speaker, consultant and practitioner in the field of software testing and software quality, with over 40 years of experience in building and testing software projects in a variety of domains, including defense, medical, financial and insurance.

You can read more at his website.