Exploratory testing

Exploratory testing is an approach in software testing that is concisely described as simultaneous learning, test design and test execution. Cem Kaner, who coined the term in 1983 [3] , now defines exploratory testing as "a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project." [4] .

While the software is being tested, the tester learns things that together with experience and creativity generates new good tests to run. Exploratory testing often thought of as a black box testing technique. Instead, those who have studied it consider it a test approach that can be applied to any test technique, at any stage in the development process. The key is not the test technique nor the item being tested or reviewed; the key is the cognitive engagement of the tester, and the tester's responsibility for managing his or her time. [5]

History

Exploratory testing has always been performed by skilled testers. In the early 1990s, ad hoc was too often synonymous with sloppy and careless work. As a result, a group of test methodologists (now calling themselves the Context-Driven School) began using the term "exploratory" seeking to emphasize the dominant thought process involved in unscripted testing, and to begin to develop the practice into a teachable discipline. This new terminology was first published by Cem Kaner in his book "Testing Computer Software" [3] and expanded upon in "Lessons Learned in Software Testing". Exploratory testing can be as disciplined as any other intellectual activity.

Description

Exploratory testing seeks to find out how the software actually works, and to ask questions about how it will handle difficult and easy cases.The testing is dependent on the tester's skill of inventing test cases and finding defects. The more the tester knows about the product and different test methods, the better the testing will be.

To further explain, comparison can be made of freestyle exploratory testing to the antithesis scripted testing, which basically means that test cases are designed in advance, including steps to reproduce and expected results. These tests are later performed by a tester who compares the actual result with the expected.

When performing exploratory testing, there are no exact expected results; it is the tester that decides what will be verified, critically investigating the correctness of the result.

In reality, testing almost always is a combination of exploratory and scripted testing, but with a tendency towards either one, depending on context.

According to Cem Kaner & James Bach, Exploratory testing is more a [mindset] or "...a way of thinking about testing" than a methodology. [1] They also say show that it crosses a continuum from slightly exploratory (slightly ambiguous or vaguely scripted testing) to highly exploratory (freestyle exploratory testing). [2]

The documentation of exploratory testing ranges from documenting all tests performed to just documenting the bugs. During pair testing, two persons create test cases together; one performs them, and the other documents.
Session-based testing is a method specifically designed to make exploratory testing auditable and measurable on a wider scale.

Benefits and drawbacks

The main advantage of exploratory testing is that less preparation is needed, important bugs are found fast, and is more intellectually stimulating than scripted testing.

Another major benefit is that testers can use deductive reasoning based on the results of previous results to guide their future testing on the fly. They do not have to complete a current series of scripted tests before focusing in on or moving on to exploring a more target rich environment. This also accelerates bug detection when used intelligently.

Another benefit is that, after initial testing, most bugs are discovered by some sort of exploratory testing. This can be demonstrated logically by stating, "Programs that pass certain tests tend to continue to pass the same tests and are more likely to fail other tests or scenarios that are yet to be explored."

Disadvantages are that the tests can't be reviewed in advance (and by that prevent errors in code and test cases), and that it can be difficult to show exactly which tests have been run.

When repeating freestyle exploratory tests, they will not be performed in the exact same manner, which can be an advantage if it is important to find new errors; or a disadvantage if it is more important to know that exact things are functional.

Usage

Exploratory testing is extra suitable if requirements and specifications are incomplete, or if there is lack of time. The approach can also be used to verify that previous testing has found the most important defects.It is common to perform a combination of exploratory and scripted testing where the choice is based on risk.

References

¹Exploratory & Risk Based Testing www.testingeducation.org 2004, P. 10

²Exploratory & Risk Based Testing www.testingeducation.org 2004, P. 14

³Kaner, Falk, and Nguyen, Testing Computer Software (Second Edition) Van Nostrand Reinhold, New York, 1993. p. 6, 7-11.

⁴Kaner, A Tutorial in Exploratory Testing, http://www.kaner.com/pdfs/QAIExploring.pdf, p. 36.

⁵Kaner, A Tutorial in Exploratory Testing, http://www.kaner.com/pdfs/QAIExploring.pdf, pp. 37-39, 40- .

External links

* [http://www.satisfice.com/articles/et-article.pdf Exploratory Testing Explained by James Bach]
* [http://www.context-driven-testing.com The Seven Basic Principles of the Context-Driven School]
* [http://www.testexplorer.com TestExplorer Software Testing Tools for Exploratory Testing]
* [http://www.manualtesting.com/why_manual_testing.htm Exploratory Testing Whitepapers, eSeminars and Solutions at ManualTesting.com]
* [http://www.checkpointech.com/sqa-software-solutions-4.php Exploratory Testing Theory, Tools and Training]
* [http://www.testingeducation.org/a/nature.pdf Exploratory & Risk Based Testing www.testingeducation.org 2004, by Cem Kaner & James Bach]
* [http://www.methodsandtools.com/archive/archive.php?id=65 Exploratory Testing: Finding the Music of Software Investigation by Jonathan Kohl]
* [http://www.testingcraft.com/ad_hoc_testing.pdf Ad Hoc Software Testing by Chris Agruss and Bob Johnson]

ee also

* Ad hoc testing
* Context-Driven School
* San Francisco depot
* Session-based testing
* Software testing
* Test case
* Test script