Oracle (software testing)

For other uses, see oracle (disambiguation).

An oracle is a mechanism used by software testers and software engineers for determining whether a test has passed or failed.^[1] It is used by comparing the output(s) of the system under test, for a given test case input, to the outputs that the oracle determines that product should have. The term was first used and defined in Howden's Introduction to the Theory of Testing. ^[2] . Additional work on different kinds of oracles was explored by Weyuker. ^[3]. Oracles are often separate from the system under test.^[4]. Method postconditions are commonly used as automated oracles in automated class testing.^[5]

Common oracles include:

• specifications and documentation,^[6]
• other products (for instance, an oracle for a software program might be a second program that uses a different algorithm to evaluate the same mathematical expression as the product under test)
• an heuristic oracle that provides approximate results or exact results for a set of a few test inputs,^[7]
• a statistical oracle that uses statistical characteristics,^[8]
• a consistency oracle that compares the results of one test execution to another for similarity,^[9]
• a model-based oracle that uses the same model to generate and verify system behavior,^{[citation needed]}
• or a human being's judgment (i.e. does the program "seem" to the user to do the correct thing?).^[4]

References

1. ^ A Course in Black Box Software Testing, Cem Kaner 2004
2. ^ Edward Miller and William E. Howden, "Software Testing and Validation Techniques", pp. 16-19, IEEE, 1978
3. ^ "The oracle assumption of program testing", 13th ICSS, pp. 44-49, Honolulu, 1980
4. ^ ^{a b} An Integrated Approach to Software Engineering, Pankaj Jalote 2005, ISBN 038720881X
5. ^ Bertrand Meyer, et al, "Programs that Test Themselves", IEEE Computer, 42-9, pp 46-55, Sept. 2009
6. ^ "Generating a test oracle from program documentation", Peters and Parnas, 1994 in Proceedings of the 1994 International Symposium on Software Testing and Analysis
7. ^ Heuristic Test Oracles, Douglas Hoffman, Software Testing & Quality Engineering Magazine, 1999
8. ^ Test Oracles Using Statistical Methods, Johannes Mayer and Ralph Guderlei
9. ^ Analysis of a Taxonomy for Test Oracles, Douglas Hoffman, Quality Week, 1998

Bibliography

• Binder, Robert V. (1999). "Chapter 18 - Oracles" in "Testing Object-Oriented Systems: Models, Patterns, and Tools". Addison-Wesley Professional, 7 November 1999. ISBN-13: 978-0201809381.