Pros and Cons of Using Jira as a Test Case Management Tool

This article explores both sides of the issue of trying to adapt the Jira incident tracking tool to also be used as a test case management tool. There are several key factors to consider, which can also be applied in a more general way to test tool usage in general. After reading this article, you should have the information needed to know which test management tool approach is best for you and whether or not it makes sense to try to use Jira as a

Table Of Contents

• Extending Software Testing Tools
• Jira Test Case Management
• How to Write Test Cases in Jira
• The Benefits of Applying Jira as a Test Management Tool
• The Issues With Using Jira as a Test Case Management Tool
• Tool Integration is Key
• What About Add-on Products for Test Management?
• Think Outside Of The Box When It Comes To Test Metrics
• Consider Performing a Proof-of-Concept and Pilot Project
• Conclusion

Extending Software Testing Tools

When using tools of any type, it is possible to apply them in ways other than their designed and intended use. For example, you could use a screwdriver for not only the insertion and removal of screws, but also to remove a cork from a bottle of wine. I think most people would agree that a corkscrew would be much better suited for opening the wine bottle than a screwdriver. That is, of course, unless you had no other option available.

The same thought can apply to software test tools. There is a great temptation sometimes to apply a tool in ways other than intended if it solves an immediate problem. After all, you don’t have to go through the time, effort and money to acquire another tool that now must be learned, supported and maintained.

One of the most popular talks I am known for is my conference tutorial on “Free and Cheap Tools.” I have been tracking these tools for over 20 years and I like people to be aware of the options. I have been a test manager with little or no budget for tools or anything else, for that matter. Any free or cheap tools I could find were helpful to some degree, even if it hit the “you get what you pay for” wall.

So, I am not inherently against the idea of trying to extend the value to tools by adapting them for certain other functions. However, I have also learned that tool adaptation can be a tricky thing that can sometimes lead to bad results.

The Principle of Using Tools as Designed, Not as Discovered

Before diving into the specifics of creating test cases in Jira, I would like to bring out a key principle in tool use that my friend Greg Paskal discussed in some of his articles and podcasts. Greg heard this at a Google Test Automation Conference, and I can confirm it from my own experiences as well.

The principle is that tools should be used as designed, not discovered. The idea is that when using a tool, you may learn you can do some other things with the tool that are not typically within the main features or objectives of the tool. For example, you may be able to use a capture/playback or other test automation tool as a rudimentary test data generation tool. But, test data generation tools are more robust and can do more things (and do them better) than a test automation tool.

The reason that ”use as discovered, not defined” is such an important principle is for at least two things:

1. Discovered functionality may be helpful at first, but may be short-lived. Since discovered functionality may be outside of a tool’s main feature set, it might go away at some point in the future. If the discovered feature disappears, then you are stuck and your entire investment may go away at any time. The vendor may not even know that people are using certain features of the tool for other purposes.
2. Discovered functionality is almost always less robust and harder to use than designed functionality. This has implications going forward for things like maintenance, knowledge transfer, etc.

Jira Test Case Management

Jira has been around for some time now, and is considered a key part of the Atlassian tool family to perform defect and issue tracking. Like many management-type tools, Jira is fundamentally a repository designed to facilitate documenting and tracing issue reports. And, it does that job very well.

The repository nature of Jira has prompted some people to explore the idea of using Jira as a test case management tool as well. In essence, one would be treating test cases the same as an issue/defect report.

So the question of “Can Jira be used as a test case management tool?” is “Yes, it can be.” But the process is not all that simple and straightforward.

Differences in Test Case Approaches

In my experience after reviewing thousands of test case approaches and documentation, I don’t think I have seen any two approaches exactly alike. This is both good and bad.

The good side is that you can be in control of how you define your tests. The bad side is that there are many nuances in test case design and it is very easy to commit to a test case approach and format that is not scalable or has similar issues.

That said, there is an architectural view of how test cases can be effectively structured to facilitate traceability, maintenance, reuse, scalability, execution and measurement.

Issue and defect reports also have attributes that can greatly facilitate tracking, traceability and resolution.

When comparing the way test cases and issue reports are formed and used, we can see some clear differences. If a tool is designed primarily for issue reporting, it will likely not be optimal for test case management.

In the following figure, we can see the differences between test case structures (Fig. 1) and issue report structures (Fig 2).

Figure 1 - Sample Test Case Structure

Figure 2 - Sample Issue Report Structure

How to Write Test Cases in Jira

To make Jira work as a test case management tool, a new issue type must be created to look like a test case, not a defect report. Each test case can have multiple test results associated with it just like an issue report can have sub-issues.

In fact, that is how you structure it in Jira. A test case will have test results as sub-issues. But, there is no way to go higher up, such as organizing sets of test cases into test suites.

Figure 3 - Add New Issue Types

To write test cases in Jira, you will need to also define screen layouts for the test case format and also for the test result format. Test case writing in Jira is not super difficult, but it can feel like you are creating a test case management tool, which in essence, you are.

Figure 4 - Add New Screen Scheme

So, now assume you have created a test case type of issue in Jira, along with test results as a sub-issues. You manually run those tests and record the results, which in itself can be tedious, but now it comes to test reporting.

The reporting features of Jira were designed for issue tracking and are quite robust. However, you may be constrained in how you might apply that reporting for test results. For example, you could see test results for a project or release, but anything more detailed or larger in scope would be difficult.

Figure 5 - Jira Dashboards

The Benefits of Applying Jira as a Test Management Tool

We have already discussed the larger context of extending tools, but let’s take a quick look at the advantages:

1. If you already own Jira, there are no additional licensing costs.
2. If you already know how to use Jira, the learning curve may be shorter.
3. You may be able to customize how you format test cases.

When budgets are tight, adapting and applying Jira for test management may be an attractive option, but keep in mind that in any tool equation initial cost savings can be far overshadowed by the cost of ongoing maintenance.

The Issues With Using Jira as a Test Case Management Tool

• Start-up - As with any tool, you must consider the up-front time it takes to get started. This can include learning the tool, configuration, and creating your first efforts. In the case of Jira as a test case management tool, the initial configuration and set-up will take some time. If you get the initial test case template design wrong (or at least not to your liking later on), it requires significant rework to restructure, depending on the volume of your test cases.

• Usability and Workflow – You will likely find that to enter test cases and associated results, you will have to access more screens than in a test management tool.

  This is because in Jira, the design assumption is that you are entering issue data, not test case data. Therefore, more time will be required for test case and test result entry. This also impacts the workflow as Jira test case management workflow typically requires more steps than a typical test management tool, which translates to more time and effort spent than optimally needed or available.

• Limitations on Reporting - As mentioned above, Jira is made to report and track issues, not test cases or test results. If you want to see metrics around test cases passes vs. test cases failed, you will probably have to have another way to do that, such as a spreadsheet. Obviously, this approach to test reporting is more cumbersome than it should be. A great benefit of test management tools are the reporting capabilities such as dashboards.

• Scalability – This is a big issue with any tool or technique. It is not uncommon for a tool or technique to work on a limited scale, but fail when applied on a larger scale.

  For example, ask yourself the questions, “How would Jira handle 10,000 test cases? Would I be satisfied with the level of effort needed to create and maintain 10,000 test cases in Jira? Would most of the people in our test organization be able to work through the process of using Jira as a test case management tool?

• Lack of Test Execution Ability– So far, we have only discussed the pros and cons of housing test cases and test results in Jira.

  When performing test execution In Jira, you don’t have the ability to initiate an automated test from within the tool. Therefore, you are limited to manual testing. This can be a problem if automation and CI testing are part of your testing needs.

Jira Test Plan

In Jira, there is no test plan feature other than to consider a project definition a test plan. However, the true nature of a test plan is to be a project plan for testing as opposed to the tool-centric view as simply a collection of test cases or test procedures.

Ideally, a test plan should define the scope of testing, the test objectives, the risks, resources (including environments), procedures (not for individual tests, but for the test as a whole), pass/fail criteria, schedules and more.

Tool Integration is Key

When considering the test tooling aspect of software testing, it is imperative to ensure that the tools being used can integrate with minimal external effort. Every external manual effort, such as manual exports to spreadsheets, is another burden to your testing workflow.

Ideally, you want to have integration between test tools that allows traceability to be achieved between requirements, user stories, test cases, test results and issue reports – at a minimum.

Figure 6 shows an example of how one tool like PractiTest can link together the key components of a lifecycle from requirements to issue reports.

Figure 6 – Types of Items Managed

5 Things you should ask you test management provider about they Jira connection

What About Add-on Products for Test Management?

Another option is to get an add-on plugin for test management. Examples of these add-ons include X-ray, Zephyr, Behave Pro, and qTest Scenario.

Each of these add-ons work differently, but they all share one thing in common – they extend the functionality of Jira but must also work within the framework and constraints of Jira. However, the add-ons do facilitate the initial set-up work discussed earlier. In addition, you can gain benefit from being able to define granular test steps that can be executed and documented without performing all the steps in a particular test.

In addition, agile teams can also benefit from add-ons that are more oriented to agile approaches such as Behavior-Driven Development (BDD) and Acceptance Test Driven Development (ATDD).

Add-ons can also enhance tool integration, such as with Continuous Integration tools.

Considering that each add-on has its own strengths and weaknesses, generally speaking, the downsides of add-ons are:

• Add-ons are developed and maintained by vendors who can take any action at any time. This is what I call “the vendor risk.”
• Versioning of test cases may be difficult due to a lack of change in history.
• Very limited ability to reuse test cycles. This can be significant since the Return on Investment of test planning and design is seen largely in test reuse.
• Sharing test cases between Jira projects isn’t possible due to the issue structure of Jira. Like reusing test cycles, the reuse of test cases is a key way to see ROI. In fact, lack of sharing actually adds to the workload of test case creation.

Importing of Test Cases into Jira or an Add-on Tool

It is quite possible you may already have an existing library or tool of some form, even if it is only a spreadsheet. If that is your case, the question that must be asked and answered is, “Can we import existing test cases into Jira or the add-on tool?”

This question is also valid for acquiring any new tool, test management or otherwise.

The answer may not be as straightforward as assumed. Even when a vendor promotes the importing and exporting of data, it may require a lot of gyrations to make it happen.

Just as we need to consider the need to import test cases, also consider the need to export test cases. Possible reasons might be the acquisition of a different test management tool in the future, sharing tests with another part of the organization that may be using a different test management tool, and so forth.

You always need to be thinking about how to preserve your investment in your test assets no matter which tool or which tooling approach you take.

Think Outside Of The Box When It Comes To Test Metrics

Whichever test management tool you use – Jira as a test case management tool, a Jira add-on, PractiTest, or any other tool, the gathering and reporting of test results and metrics should be robust enough so that people can make informed decisions.

In considering metrics, less is often more. But it can be possible to miss some great opportunities as well by staying on the common path of reporting and measuring things like the number of passed tests, failed tests, defects found, defects fixed, etc.

Think about also measuring how long it takes for a manual test to be performed, the time it takes for automated tests to run, etc. You can learn a lot from time-based metrics. Only a more robust test management tool like PractiTest can give insight into these kinds of metrics.

For example, if you have a set of automated tests and learn by tracking the metrics that one or more of the tests are running abnormally long or slow, this is something to investigate. It is possible for a test to pass functionally, all the while experiencing performance issues or inefficiency issues in the test itself. Slow tests impact your test cycle time and the ability to test and deliver projects on schedule.

Figure 7 - Test Report Metrics From PractiTest's Dashboards

Consider Performing a Proof-of-Concept and Pilot Project

Now that we have explored the pros and cons of applying Jira as a test case management tool, how do you know how best to proceed in your context? The only way to know if a particular tool or set of tools will work for you as desired in your context of projects, processes, technology and people, is to conduct a proof-of-concept and a pilot project. These go beyond a typical evaluation in terms of scope and complexity.

An evaluation is typically short and limited in terms of the extent of what can be learned about the tool. A demo is even more limited.

A proof-of-concept is like an extended demo where you can apply the tool in an in-depth way to your own projects and technology. Basically, you are doing a test of the tool in your environment.

In the case of Jira as a test case management tool, you would want to try to apply in that way and see how it works. Pay close attention to scalability. What works in the small proof-of-concept may not work well on a larger scale.

Pilot projects are an opportunity to try a tool or technique in actual practice before committing substantial time and money. In pilot projects, you want to manage risk by keeping negative exposure low. But, you will learn things in pilot projects you will learn in no other settings.

Conclusion

Each test tool has unique capabilities, strengths and weaknesses, and other factors that make tool selection a somewhat complex project. When selecting any type of test tool, it is good practice to start with a set of evaluation criteria based on your own needs and requirements.

In many cases, you will not get a perfect fit to your originally stated requirements, but a “best fit” may be more feasible. For those features that may not be present in the tool you eventually acquire, it may be necessary to consider a multi-tool approach with tool integration as a key attribute.

Remember the wisdom of using tools as designed, not as discovered. That wisdom was gained from experiences where the lessons were learned the hard way – trying to make an extension to a tool, then having to start over with the right tool.

While trying to extend tools beyond their designed purpose may be attractive in initial cost savings, consider the impact of investing a lot of time in creating test cases in a tool that may not meet enough of your needs. In many cases, the impact is that the work will need to be repeated, which of course, is a bad thing.

The better course of action is to do your research on the tools you need, define your needs, perform a proof-of-concept, and ideally, a pilot project to see how the candidate tool(s) perform in your environment, with your projects and with your people and processes.

After making this reasoned analysis, you will have both theoretical and practical knowledge of the test management tool solution that is best for you.