Introduction
Imagine you are assigned the task of testing a new software app or website. You have no prior knowledge or experience with the item(s) you are testing. Furthermore, there is no available documentation, such as requirements, user stories or use cases. What’s a tester to do?
This scenario is actually very common. Even when some basis of testing (requirements, user stories, etc.) exists it is many times incomplete or obsolete.
This lack of basic information about the items we are testing requires an approach that can be applied when there is no existing basis of knowledge. The good news is that much research and work has gone into defining a way to do this kind of testing. It has become known as exploratory testing and it is a recognized way to conduct testing.
In this article, we will look at exploratory testing in more detail and see ways to apply it in certain situations.
Table Of Contents
- What is Exploratory Testing
- When to Use Exploratory Testing
- Exploratory Testing in Agile
- A Little Background of Exploratory Testing
- Exploratory Testing Comparisons
- Differences Between Ad Hoc Testing and Exploratory Testing
- Advantages of Exploratory Testing
- How Effective is Exploratory Testing
- Difference Between Exploratory Testing and Agile Testing
- Exploratory Testing Tools – Passive vs. Active Tools
- Exploratory Testing in Jira
- How to Perform Exploratory Testing
- Exploratory Testing Examples
- Conclusion
What is Exploratory Testing?
Let’s start with some working definitions of exploratory testing. Over the years since the 1980’s, many definitions of exploratory testing have been proposed by many of its practitioners and evangelists.
One that seems to be a good overall fit is “Any testing to the extent that the tester actively controls the design of the tests as those tests are performed and uses information gained while testing to design new and better tests. (Cem Kaner and Andy Tinkham, “Exploring Exploratory Testing”)
While this definition is concise, there are other ways to understand exploratory testing by contrasting it to other approaches. Also, it helps to understand what exploratory testing is not, such as ad-hoc, chaotic, slow, or unplanned.
Another definition I find helpful is from James Bach: “Exploratory testing is simultaneous learning, test design, and test execution”.
And yet another way to understand exploratory testing is that of concurrent test case design and test execution of an application or system under test. The tests are not pre-designed, but instead developed during the test based on what has been seen so far in testing.
A major way to conduct exploratory testing is in short periods of time known as “time boxes” or “sessions”. This is called “Session-based Testing” and we will examine this in more detail a little later in this article.
Session-based testing helps to organize, measure and manage exploratory testing by defining an objective or charter for the session. The exact tests are not defined in advance, however.
Another helpful point of understanding exploratory testing is that it can be seen as part of a spectrum of testing, with highly defined or scripted testing at one end and fully exploratory testing on the other end (Figure 1). In actual practice, many organizations use a hybrid exploratory approach to deal with those situations when pre-defined tests are needed.
Figure 1 – The Spectrum of Exploratory and Scripted Testing
When to Use Exploratory Testing
There are some key factors which can be seen as great opportunities to use exploratory testing. These include:
- Lack of knowledge about the item(s) under test
- Lack of time in which to plan and perform testing
- Having people with the thinking skills needed for testing without pre-definition of tests
To contrast, factors that indicate exploratory testing is not a good fit include:
- The need for performing a very defined set of steps in the test, often with very defined test data. An example would be testing the configuration of a medical device
- The need to show test coverage of certain requirements or other basis of testing
- The need to show evidence of test results which can be reproduced repeatedly
Exploratory Testing in Agile
Exploratory testing can be done quickly, especially if the objectives are defined well and people are skilled in the practice. This speed of testing can be a good fit for agile methods with short iterations for testing. However, not all testing in agile is manual.
In agile testing, a lot of emphasis is placed on test automation to keep up with the needed pace of development. But the reality is that not every test can or should be automated. Plus, it can take significant time to develop automated tests. And…those automated tests must be maintained. This is a huge challenge in rapid change environments, such as seen in agile.
Therefore, exploratory testing offers an option to test quickly, even if the testers start to feel like they are robots, testing the same things over and over. However, when done correctly testers can always be branching out and testing new things which helps greatly to alleviate boredom.
A Little Background of Exploratory Testing
When recounting the history of nearly anything, accounts will vary. This is especially true in technology-related events. But this is the way I saw exploratory testing evolve from the perception of low-value ad-hoc testing to a recognized practice with substantial value.
In the 1980’s, software testing as a practice was starting to take a fairly defined form. Test plans, test cases, test scripts, and other items were becoming standard. The predominant message at testing conferences was one of highly defined tests and metrics.
Even during this time, there were some that recognized that regardless of the “standard” practice, there were also many people who “just tested” things as best they could.
In the early to mid-90’s, the rise of personal computing and personal computing software companies also gave momentum to the practice of “bug bashes” and other testing methods which were not pre-planned. It was around this time that the evangelists and pioneers in exploratory testing started to become more prevalent at testing conferences.
As time went on, exploratory testing became an accepted testing practice. Many of the proponents belonged to the “context-driven” school of testing. The context-driven view of testing is based on the idea that how you test something depends on the context.
For example, testing a medical device is different from testing a video game. Testing a medical device involves high risk and requires high levels of documentation and precision. Testing video games can be just as challenging, but requires a much more fluid approach to testing.
Exploratory Testing Comparisons
As mentioned above, sometimes the best way to understand something is by comparing and contrasting it with other practices.
Differences Between Ad Hoc Testing and Exploratory Testing
Ad hoc in Latin means “for this”. In English, the meaning is often “for this specific purpose”. Things that are ad hoc are used in situations that require a one-time effort.
So, an ad hoc test would be one performed for a specific product, release or other reason. There is typically little advance planning and almost no reuse of the tests. Everything done in an ad hoc manner is for the “here and now”.
Ad hoc testing may or may not cover all the features and conditions that should be tested. Therefore, many defects may be missed in testing and found by the users in live production use.
In ad hoc testing, the tests are derived as people think of them. This injects a degree of randomness to the test. Random tests can be helpful at times, but it has limitations. Learning is limited and you may or may not find any defects.
Exploratory testing on the other hand, has a much longer view of testing. The idea is that the things learned in one or more tests can be used as the basis for other tests. It’s much like the old game of 20 Questions where you try to guess what the object is by asking questions based on the questions of the previous players.
In fact, when I train people in how to do exploratory testing, we play “20 Software Questions”. I give people the object of testing just one feature in an application while I project the screen at the front of the room.
Each person gets to ask a question such as, “What if I do….”. I try that and we note the outcomes. Then, the next person asks a question built on the previous question(s). By the end of the game, we have tested an amazing number of things for just one detailed aspect of the item.
| Ad Hoc Testing | Exploratory Testing |
|---|---|
| For a single purpose | For the longer purpose of understanding |
| Often random | Purposeful |
| Little learning | Much learning |
| Almost impossible to repeat | Good notes or recording allow good repeatability |
| Few defects may be found | More defects may be found, depending on the tester |
| Quick and dirty | Quick and organized |
Difference Between Exploratory Testing and Agile Testing
As mentioned above, sometimes exploratory testing fits in the agile context. However, that is not always the case. Certainly, there is no automatic connection between agile testing and exploratory testing.
In agile testing, much emphasis may be placed upon test automation in order to keep up with the pace of short iterations and sprints. However, this depends on the ability of the team to be able to implement and maintain such automation.
Also, in agile testing it is common to see testers paired with developers to test as developers are implementing features. This level of test might be functional in nature, or structural based on the code under test.
In agile pairing, exploratory testing may not be used at all because the tester is seeing first-hand what is being developed so there is little need to explore and learn on their own.
However, there are some aspects of agile testing that can be achieved using exploratory testing. These come into play when the items being developed are released to testers who are not part of the development team. For example, in some agile organizations, a separate test team exists to test work from multiple agile teams in an integrated way. Exploratory testing can be used in those situations where testing is separate from development.
| Agile Testing | Exploratory Testing |
|---|---|
| Automated | Primarily manual |
| Paired with a developer | Often done by a separate team or person |
| Can use exploratory testing | Can be used in Agile testing |
| Under constant change | Can also respond to change |
| Requires fast test completion | Can be performed quickly |
Advantages of Exploratory Testing
Exploratory testing has some clear advantages in the right situations. As we will see, the “right” situation is more than just the product and extends to people, technology and processes.
Exploratory testing can be an effective way to:
- Test in the absence of documented knowledge
- Create new tests when existing tests grow stale
- Test quickly when time is short
- Learn about the item under test quickly
When Tests Grow Stale
Way back in the 1980’s, Boris Beizer, one of the early authors and practitioners of software testing, wrote about “The Pesticide Paradox”. To paraphrase, his assertion was that a pesticide is effective at first in killing bugs, over time the effectiveness diminishes until eventually another better pesticide is needed.
In a similar way, certain tests may be effective at first in discovering new defects but the more you run the tests (assuming the defects are fixed after the first discovery), those tests will pass and become more confirmatory or regression in nature. To find new and important defects, new tests are constantly needed.
A great way to find those new tests is by exploratory testing. One way is to think of exploratory tests as “tours”. James Whittaker has written a book specifically about this idea. (James Whittaker, “Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design”)
The Minefield Analogy
An analogy I heard James Bach relate once has to do with asking the question, “What is the best way to cross a minefield?” The answer is to follow exactly in the steps of someone who has done it successfully before. If we see a defect like a mine (when we hit it something blows up), then as testers, we want to find them. But if we stay on the beaten path, we will safely cross the “minefield” of our software. The red lines are paths that “hit” defects. See Figure 2
Figure 2 – The Minefield Analogy
Disadvantages of Exploratory Testing
However, there are some downsides to exploratory testing. To be fair, not all techniques are right in all situations. And, the application of any given technique depends on the people performing the technique. The downsides listed here are the common one I see.
- Difficult to repeat tests with precision
- Due to the lack of pre-defined expected results, it may take more time and effort to determine if a test has passed or failed
- Not every tester has the attributes to be a good exploratory tester
- Exploratory testing can be difficult to control unless combined with other techniques like Session-based testing
How Effective is Exploratory Testing?
There have been some initial but limited studies comparing exploratory testing to structured or scripted testing. At least one study also compared what they referred to as “hybrid testing” which combined the two techniques.
One study published in 2008 found that exploratory testing was more effective in finding defects than structured testing. (Juha Itkonen, “Do Test Cases Really Matter? An Experiment Comparing Test Case Based And Exploratory Testing”, Helsinki University Of Technology, 2008). However, it should be noted that this study was based on an experiment and not over the longer course of performing testing, such as in an organization with ongoing release schedules. Also, the study concludes that the difference in defects found in the two approaches is not statistically significant.
It is perhaps a reasonable conclusion that in the right context, exploratory testing can be highly effective, while in others not so effective. For example, if I as a tester must follow an exact set of steps to perform a test in the item under test, exploratory testing would not be a good fit.
Another example would be if a group of users were going to be performing user acceptance testing and needed a high degree of guidance in how to perform the test, test procedures and scripts would be of great benefit.
I was part of a “bug bash” where about twenty users were asked to test in an exploratory way for two hours. In one case, over one hundred potential defects were recorded after two hours of testing. After those reported defects were analyzed, only four of them were deemed as actual defects. The rest of the items looked odd, but turned out to be “works as designed”. Was that an effective test? Probably not.
But, those “testers” had not been trained in how to test in an exploratory way, which is a very important thing. Exploratory testing can be taught and learned, but there are some people who are better at it than others.
Automated Exploratory Testing
This is an interesting idea that gets into the Artificial Intelligence (AI) and Machine Learning (ML) aspects of testing in general, and test automation in specific.
There is a huge amount of AI and ML in testing, with much to be learned, applied and proven. Tool companies are working to make test automation more robust and less brittle with AI, but when it comes to exploratory testing there are some interesting questions, such as “Can a computer really take the place of a critically thinking human in testing?”
One possibility might be in the area of heuristics, which has been a long-standing part of exploratory testing. A heuristic is simply a way of thinking about something. For example, if I can do the positive of something, can I do the negative?
If a set of automated heuristics could be defined, then perhaps we might be closer to AI in testing, including exploratory testing. It remains to be seen if and when that might be truly a reality.
Exploratory Testing Tools – Passive vs. Active Tools
Over the years, tools have been developed to help exploratory testers. Some of the tools are passive in nature, while others may be more active.
Passive Exploratory Testing Tools
Passive exploratory testing tools are for documenting tests as they are performed. These tools can be a great help to testers as they can work at the same time in taking screenshots, taking notes, etc. These tools can help further by allowing testers to capture good tests learned through exploratory testing and document them as future defined tests.
PractiTest has the capability to indicate if a test is an exploratory test. (Figure 3) This has great value as the value from exploratory testing becomes very tangible as seen in fresh, repeatable tests that are clearly documented and managed.
Figure 3 – Defining an Exploratory Test Session in PractiTest
For example, you may capture your exploratory test session in PractiTest, then later convert the tests in that session to scripted tests. (Figure 4) You can try this feature by starting a PractiTest free trial.
Figure 4 – Creating a Scripted Test from an Exploratory Test in PractiTest
Active Exploratory Testing Tools
Active exploratory testing tools can drive the testing. This is where the need for AI and ML in exploratory test automation becomes apparent. However, active exploratory testing can be accomplished in other ways, such as using keyword-driven test frameworks.
In keyword-driven test automation, it is possible to combine and recombine various keywords or “action words” in ways to create larger test scenarios. Keyword test automation is not typically considered an exploratory testing technique, but with some adaptation, it is perhaps the most achievable way to automate exploratory testing in other than a purely scripted way.
The reason for this is that the testers need not be concerned with creating the test automation, but instead can focus on designing the input to the test automation. (Figure 5) However, some visual test automation tools do offer a level of AI for testing.
Figure 5 – An Example Generic Keyword-driven Test Automation Framework
Exploratory Testing in Jira
If you are using Jira as your test management tool, there is a plugin available for Jira Cloud to help document the results of your testing. And, there are test management plugins to Jira with their own exploratory testing features.
PractiTest offers the ability to have a robust test management solution that is totally integrated with Jira and automatically stays in sync with Jira. The integration between PractiTest and Jira allows users to report defects directly from a session-based test in PractiTest to their Jira project. This saves a lot of time!
How to Perform Exploratory Testing
Exploratory testing can be performed in many ways. In fact, a key advantage of exploratory testing is that it can be free-form in nature.
In this description, we will take a session-based approach to exploratory testing.
-
Define the target and scope of the test
The scope of the test should be achievable in a short time-frame, taking into account the status of the item(s) to be tested and the available testers. The question is, what is “short”?
One goal of exploratory testing is rapid testing, so you will want to consider that in your scope definition.
-
Define the objective of the test in a test charter
In reality, an exploratory testing effort may consist of many test sessions. However, it is important that each session has only one objective to maintain focus.
-
Define the tester(s)
An exploratory test can be a solo, paired, or even a small team effort. Pairing is often effective since one person may notice something that the other might miss. In addition, one person can take notes, while the other person tests.
-
Prepare for the test
This includes getting the test environment ready, making sure the item(s) under test are in place and any test data is obtained.
-
Perform the test
The test is conducted according to the session parameters defined in the charter. This includes limiting the test to the time box defined in the charter.
As anomalies, failures and other items are seen, they are documented. It is common for a significant amount of time and effort to be spent after the session in researching the results of the test.
The reason the follow-up research is done after the session is to keep the focus on the test during the session.
-
Perform a follow-up test, if needed
This test is intended to follow-up on the issues found in the initial test after more understanding has been gained. In the follow-up test, defects and failures can be confirmed and conclusions can be confirmed.
-
Document the test
In this step, you can tell the story of your test session, including what you learned, what you found (especially in terms of defects) and what your conclusions and recommendations might be.
-
Document helpful tests for future use
This is done after the test session and when you have the time to document the tests in a test management tool, such as PractiTest. These tests may get initially documented during the test but often require additional information.
Exploratory Testing Examples
Most exploratory tests will start with a test charter. The test charter will usually indicate the goal of the session. The charter can be chosen either by the executing tester or by the test lead/test manager. Test charter may include some guiding points that will further guide the tester while conducting the session.
The following example, presents an exploratory test case that will serve the tester as a plan for his/her exploratory session.
As you can see in this example, the session’s plan includes a test charter and guide points for the tester to follow when performing the session, as well as the environment to be used for the session (Operating system and browser in this case).
When the test charter is established (and guide points in some cases), and the time frame for the session is set, the tester can start conducting the session. During the session, the tester will take notes for further reference and future test case design, as well as report the issues and defects he/she discovered. Following is an example from an exploratory session with tester’s notes and reported issues:
Have questions regarding Exploratory and Session-Based testing?
Check out this recent LIVE Q&A webinar we hosted with James Bach, Consulting Software Tester and Buccaneer-Scholar.
In this unique Webinar, Both James Bach and Joel Montvelisky discuss questions regarding Exploratory and Session-Based Testing, from Theory to Methodology.
Conclusion
Exploratory testing is an effective software testing technique, especially in situations where little is known about the item to be tested and time for testing is limited.
Contrary to some beliefs, when done properly, exploratory testing is not random or ad hoc but instead is thoughtful and can be a great way to learn and test at the same time.
In addition, exploratory testing can be an effective way to create new, fresh and creative tests to pre-existing sets of tests that have grown stale and non-productive over time.
PractiTest offers solutions for exploratory testers by providing a way to capture and reuse exploratory tests on an ongoing basis.
By Randall W. Rice
Published on:
Randall W. Rice, CTAL
Randall W. Rice is a leading author, speaker, consultant and practitioner in the field of software testing and software quality, with over 40 years of experience in building and testing software projects in a variety of domains, including defense, medical, financial and insurance.
You can read more at his website.
