What Is Sanity Testing? A-Z Guide on Sanity Testing (Examples & Checklist)

What is Sanity Testing?

Sanity Testing is a term not found in many of the standard sources for software testing terminology, yet it is a term used not only by software testers, but also in other domains as well.

Sanity Testing Definition

Let’s start with this working definition of sanity testing, which also describes both the meaning of sanity testing and the purpose of sanity testing:

Sanity testing is a quick and basic test (or set of tests) to determine if a particular application or component is behaving reasonably correctly.

At the outset, we need to understand there is shared territory between the terms “sanity test”, “smoke test” and “regression test”. In many ways, you could see “sanity test” and “smoke test” as synonyms.

In the larger view, regression tests are typically more rigorous and larger in scope than sanity tests (See Figure 1).

Figure 1 – The Relationship Between Sanity Tests, Smoke Tests and Regression Tests

Sanity tests are often fewer in number and more focused than regression tests. However, it is not unusual for regression tests to start out as sanity tests.

What is a Sanity Test?

A sanity test is often a quick check on something to see if it makes sense at a basic, surface level. A sanity test might be one quick test, such as a calculation check. For example, if you are testing the checkout and payment function of a mobile app, you want to make sure taxes are computed correctly. Let’s say you know the tax rate is 9.95%.

You may run a quick test to buy something and then pay for it. You check the pre-tax total, then the after-tax total. In your mind, you expect the tax to be approximately 10% of the purchase price, an easy calculation to make.

Is this an exact calculation? No. But, the original concept of a sanity from other disciplines is that of an approximation just to see if the basic function generally makes sense.

What is a Sanity Test in Software Testing?

Software testing is a prime opportunity for sanity tests, but it is important to understand that sanity tests can also be performed by developers. In other words, you don’t have to wait for an independent tester to perform a software sanity test.

In software testing, sanity tests can be an early form of testing, either informally or formally.

What is a Sanity Test in QA?

Unfortunately, the terms “software testing” and “software QA” started to become synonymous over 20 years ago. The reason this is unfortunate is because software QA in the classic definition is much more than testing. True software QA involves process definition, metrics, defect prevention and other activities in addition to software testing - which is actually software Quality Control (QC). (Figure 2)

Figure 2 – The Relationship Between QA and QC

In the class definition of software QA, sanity checking could be applied not only in dynamic tests such as unit testing, system testing, etc., but also in static tests, such as software reviews. For example, in a review session, you might see a section of code that doesn’t look quite right. A sanity test to walkthrough a section of code could yield some very important early defects that might otherwise be missed.

In the more casual and merged definition of software QA, sanity testing would be the same as in software testing. That is, a quick and early way to check functional results.

Features of Sanity Testing

Some distinguishing features of sanity tests are:

Simple - Sanity tests are not intended to be complex but instead, easily designed and performed.

Defined or exploratory - Sanity tests may well be predefined in a format such as a checklist, or set of simple test cases. Yet, there is also a common practice of performing sanity tests based on intuition and experience.

Comprehensive enough - Sanity tests do not cover all functionality, but they should cover enough to give a basic assessment of behavior.

Quick - As compared to other forms of testing which may be conducted later, sanity tests are intended to be performed quickly for the purpose of quick feedback. The speed of feedback is important in rapid development and continuous integration to know if further development is possible.

How To Do Sanity Testing?

While sanity testing is sometimes performed in an exploratory approach, there is great value in having repeatable sanity tests. This implies taking a planned approach.

There are two levels of test planning to perform in sanity testing: High-level sanity test planning and detailed sanity test planning. We will explore both of these levels of test planning. (Figure 3)

Figure 3 – Levels of Sanity Test Planning

High Level Sanity Planning

At the high level of sanity test planning, the big picture is defined. This includes defining:

• Sanity test objectives
• The scope of sanity testing
• Risks
• Features to be covered in sanity testing
• Roles and responsibilities

Detailed Sanity Planning

At the detailed level, you are concerned with defining sanity test cases that achieve the objectives and cover the features outlined at the high level of sanity test planning.

The good news is that detailed test analysis and design are not required for sanity test cases. The whole concept is one of simplicity that provides a level of confidence that the application is performing correctly at a basic level.

What Should be Contained in a Sanity Test Case Definition?

Test case contents can vary widely from organization to organization. Much depends on your project context, application context (How much detail is needed? How complex is the domain?), the audience (how much knowledge of the software and problem domain do they have or need?), and the risk (higher risk tends to require more formally defined test cases).

With that understanding, at a very minimum a defined sanity test should include:

• The function to be tested (Ex: Order totaling)
• The test condition(s)
• The expected result(s)

In some situations, other items could also be in the test case definition, such as:

• Pre-requisite conditions
• Post-requisite conditions
• Steps to perform

Why Are Sanity Test Cases Important?

Sure, you could perform a few sanity tests manually in an ad-hoc manner. However, there is a hidden cost in this approach due to the extra time and effort needed to think of good tests, then evaluating those test results. Sometimes, the test evaluation requires some research to determine what is correct behavior.

In the ad-hoc sanity test approach, every test is in the moment and tends to fade away shortly after the test. Then, the next time you need to test the same application, you have to try to remember as many of the sanity tests as possible. That is an elusive goal.

Defining sanity tests is definitely an investment of time and effort. But, the payback is seen in the ease of repeating the tests reliably. This is especially true when sanity tests are automated.

So, there can be some healthy discussion as to whether or not to document sanity tests. The balance might be found in the need for repeatability.

Sanity Test Case Examples

Here are two examples of sanity tests that have been defined in PractiTest. The first sanity test example is a simple bank account balance check (Figure 4)

Figure 4 – Sanity Test in PractiTest

In the next example, we will go deeper in the test definition to see the steps involved. In Figure 5, we are seeing the reorder steps view to see the high-level steps in this sanity test. One of the great features in PractiTest is the ability to reorder steps, if desired.

Figure 5 – Reordering the Steps in a Sanity Test

Automated Sanity Tests

Sanity tests can certainly be automated, but a good question is, “Should sanity tests be automated?”

To answer this question, we need to distinguish between first-time, ad-hoc sanity tests and those that are repeated as a part of making changes on a regular basis. The latter would be a light touch regression test approach.

It would probably not make sense to automate first-time, ad-hoc sanity tests. The main reason is that it would take away from the quick and simple nature of the test. It typically takes much more time to create the automation for a test as compared to one manual test execution.

In this scenario, you would not have time to automate the sanity test for the job at hand. However, you could certainly document the sanity test, even if only on paper, for the purpose of documenting it as a test case. If the test has value you might consider adding it to a sanity test set for future test runs.

As a contrasting example, perhaps you are working on a project where you get a daily build and you have identified at least 20 functions that must work at a very basic level before running a more complete regression test. These functions are subject to changes in the way they behave, so you want to know about any problems as soon as possible. There are two compelling reasons for automating sanity tests in this situation.

First, your primary concern might be the number of times you must perform these sanity tests over the course of the project, and even after release. Let’s assume it takes twenty minutes to run and evaluate all twenty sanity tests. This time doesn’t include the time needed to research any anomalies or questions raised from the test.

Each release will require twenty minutes of manual sanity testing. After about day three, you will likely start feeling like a robot and wonder if there isn’t a better way. Yes, there is!

Let’s also assume the project will take one month, which works out to about twenty builds in twenty days. (There could actually be more builds than twenty builds.) This works out to at least 400 minutes or 6.67 hours of manual testing. This is also known as the Equivalent Manual Testing Effort (EMTE).

EMTE is an important metric because that is the time you are looking to reduce with test automation. In this example, if it takes longer than 6.67 hours to create the test automation, it would only make sense to automate the tests if you plan to repeat the tests many times in future efforts.

There is one other compelling reason for automating sanity tests, which is repeatability. In some situations, including sanity tests, it is imperative that the tests be performed exactly the same each time.

To know when a basic essential function starts to fail, you need high consistency in testing. It’s a very similar concept as in regression testing. You need a firm baseline of comparison to know when things break. Defined and automated sanity tests can give this level of repeatability not only for the project at hand, but for maintenance and other post-release testing.

What is Sanity Testing with Example

To get an idea of how sanity tests might look in a couple of contexts, consider the following examples of sanity tests in both new development and maintenance.

In the development context, the product is evolving through iterations of development. Typically, these iterations are very short. Sometimes, this results in several builds per day.

Each iteration undergoes a build verification test (BVT) and perhaps a smoke test to verify basic integration and functionality is correct as far as the build integrity is concerned. These tests are automated and functional, mainly for the purpose of making sure one or more changes do not “break” the build. (Figure 6)

Figure 6 – Build Verification Testing (BVT)

As a tester, your concern is that basic functionality works correctly after each build, but you know in a few hours you will be getting another build so you don’t want to run a regression test right now. That will come once you get a more fully fleshed out product.

In addition to testing the new functionality, you may learn there are existing functions that are impacted. Perhaps the impact is negative.

At this point sanity tests are defined, but manual. As you get the time, you can work to automate these tests.

In each build, you are adding new sanity tests. (Figure 7)

Figure 7 – Increasing Sanity Tests

In the maintenance context, the product is in full live use (also known as “production”). However, there are still many user stories in the backlog to enhance the application, as well as user stories to fix some defects that have been discovered during production.

Now, your concern as a tester is to perform sanity testing after each build based on the full set of sanity tests you have created throughout development, along with regression tests performed once the application has stabilized enough. Many of the tests have now been automated.

You are defining new sanity tests to test very basic functionality of the maintenance user stories. Like in the sanity testing of newly developed functions, you may discover failures in existing functionality. (Figure 8)

Figure 8 – Sanity Testing in Maintenance Releases

Sanity Testing Tools

The good news is that you don’t need special tools to perform sanity testing. In fact, as described earlier, sanity tests can be totally manual and unplanned.

However, tools can give you leverage in defining, organizing, performing and reporting sanity tests.

A test management tool such as PractiTest allow you to define tests and categorize them as sanity tests.

If you would like to try this yourself, it is easy to get a free trial account, and you can open one from here.

Sanity Testing Checklist

1. Do you have a clear understanding of newly implemented functionality?
2. Do you know what constitutes correct behavior of the function(s) you will be sanity testing?
3. Do you have the need to store and repeat sanity tests?
   • If so, do you have a tool that allows you to store, organize, perform and report your sanity tests?
   • If not, do you have a way to informally keep track of your tests, perhaps for future use?
4. If no tool is in place, do you have a plan to acquire a tool?
5. Do you know what should be contained in a sanity test definition in your context?
6. Do you have a way to report sanity test results quickly to developers?
7. Do you have criteria for informal vs. formal sanity tests?
8. Do you know what should be contained in your sanity test definitions?
9. Do you have a way to repeat your sanity tests, if needed?
10. Do developers also perform sanity tests?

Randall W. Rice, CTAL

Randall W. Rice is a leading author, speaker, consultant and practitioner in the field of software testing and software quality, with over 40 years of experience in building and testing software projects in a variety of domains, including defense, medical, financial and insurance.

You can read more at his website.