 Receiver operating characteristic
In
signal detection theory , a receiver operating characteristic (ROC), or simply ROC curve, is a graphical plot of the sensitivity vs. (1  specificity) for abinary classifier system as its discrimination threshold is varied. The ROC can also be represented equivalently by plotting the fraction oftrue positive s (TPR = true positive rate) vs. the fraction offalse positive s (FPR = false positive rate). Also known as a Relative Operating Characteristic curve, because it is a comparison of two operating characteristics (TPR & FPR) as the criterion changes. [Signal detection theory and ROC analysis in psychology and diagnostics : collected papers; Swets, 1996]ROC analysis provides tools to select possibly optimal models and to discard suboptimal ones independently from (and prior to specifying) the cost context or the class distribution. ROC analysis is related in a direct and natural way to cost/benefit analysis of diagnostic
decision making . Widely used inmedicine ,radiology ,psychology and other areas for many decades, it has been introduced relatively recently in other areas likemachine learning anddata mining .Basic concept
style="paddingleft:1em;"

style="paddingleft:1em;"  TPR = 0.63  style="paddingleft:2em;"  TPR = 0.77  style="paddingleft:2em;"  TPR = 0.24  style="paddingleft:2em;"  TPR = 0.88

style="paddingleft:1em;"  FPR = 0.28  style="paddingleft:2em;"  FPR = 0.77  style="paddingleft:2em;"  FPR = 0.88  style="paddingleft:2em;"  FPR = 0.24

style="paddingleft:1em;"  ACC = 0.68  style="paddingleft:2em;"  ACC = 0.50  style="paddingleft:2em;"  ACC = 0.18  style="paddingleft:2em;"  ACC = 0.82Plots of the four results above in the ROC space are given in the figure. The result A clearly shows the best among B and C. The result B lies on the random guess line (the diagonal line), and it can be seen in the table that the accuracy of B is 50%. However, when C is mirrored onto the diagonal line, as seen in C', the result is even better than A.
Since this mirrored C method or test simply reverses the predictions of whatever method or test produced the C contingency table, the C method has positive predictive power simply by reversing all of its decisions. When the C method predicts p or n, the C' method would predict n or p, respectively. In this manner, the C' test would perform the best. While the closer a result from a contingency table is to the upper left corner the better it predicts, the distance from the random guess line in either direction is the best indicator of how much predictive power a method has, albeit, if it is below the line, all of its predictions including its more often wrong predictions must be reversed in order to utilize the method's power.
Curves in ROC space
Discrete classifiers, such as
decision tree or rule set, yield numerical values or binary label. When a set is given to such classifiers, the result is a single point in the ROC space. For other classifiers, such as naive Bayesian and neural network, they produce probability values representing the degree to which class the instance belongs to. For these methods, setting a threshold value will determine a point in the ROC space. For instance, if probability values below or equal to a threshold value of 0.8 are sent to the positive class, and other values are assigned to the negative class, then a confusion matrix can be calculated. Plotting the ROC point for each possible threshold value results in a curve.Further interpretations
Sometimes, the ROC is used to generate a summary statistic. Three common versions are:
* the intercept of the ROC curve with the line at 90 degrees to the nodiscrimination line
* the area between the ROC curve and the nodiscrimination line
* the area under the ROC curve, or "AUC".
*d' (pronounced "dprime"), the distance between the mean of the distribution of activity in the system under noisealone conditions and its distribution under signal plus noise conditions, divided by theirstandard deviation , under the assumption that both these distributions are normal with the same standard deviation. Under these assumptions, it can be proved that the shape of the ROC depends only ond' .The AUC is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one. [Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27, 861874.] It can be shown that the area under the ROC curve is equivalent to the
MannWhitney U , which tests for the median difference between scores obtained in the two groups considered if the groups are of continuous data. It is also equivalent to the Wilcoxon test of ranks. The AUC has been found to be related to theGini coefficient (G) by the following formula [Hand, D.J., & Till, R.J. (2001). A simple generalization of the area under the ROC curve to multiple class classification problems. Machine Learning, 45, 171186.] $G\_1\; +\; 1\; =\; 2\; x\; AUC$, where::$G\_1\; =\; 1\; \; sum\_\{k=1\}^\{n\}\; (X\_\{k\}\; \; X\_\{k1\})\; (Y\_\{k\}\; +\; Y\_\{k1\})$
In this way, it is possible to calculate the AUC by using an average of a number of trapezoidal approximations.
However, any attempt to summarize the ROC curve into a single number loses information about the pattern of tradeoffs of the particular discriminator algorithm.
The machine learning community most often uses the ROC AUC statistic. This measure can be interpreted as the probability that when we randomly pick one positive and one negative example, the classifier will assign a higher score to the positive example than to the negative.In engineering, the area between the ROC curve and the nodiscrimination line is often preferred, because of its useful mathematical properties as a
nonparametric statistic . This area is often simply known as the discrimination. Inpsychophysics ,d' is the most commonly used measure.The illustration at the top right of the page shows the use of ROC graphs for the discrimination between the quality of different
epitope predicting algorithms. If you wish to discover at least 60% of the epitopes in avirus protein, you can read out of the graph that about 1/3 of the output would be falsely marked as an epitope. The information that is not visible in this graph is that the person that uses the algorithms knows what threshold settings give a certain point in the ROC graph.Sometimes it can be more useful to look at a specific region of the ROC Curve rather than at the whole curve. It is possible to compute partial AUC. [Cite journal doi = 10.1177/0272989X8900900307 volume = 9 issue = 3 pages = 190195 last = McClish first = Donna Katzman title = Analyzing a Portion of the ROC Curve journal = Med Decis Making accessdate = 20080929 date = 19890801 url = http://mdm.sagepub.com/cgi/content/abstract/9/3/190] For example, one could focus on the region of the curve with low false positive rate, which is often of prime interest for population screening tests. [Cite journal doi = 10.1111/15410420.00071 volume = 59 issue = 3 pages = 614623 last = Dodd first = Lori E. coauthors = Margaret S. Pepe title = Partial AUC Estimation and Regression journal = Biometrics accessdate = 20071218 date = 2003 url = http://www.blackwellsynergy.com/doi/abs/10.1111/15410420.00071]
History
The ROC curve was first used during the
World War II for the analysis of radar signals before it was employed insignal detection theory .cite bookauthor=D.M. Green and J.M. Swets title=Signal detection theory and psychophysics publisher=John Wiley and Sons Inc. date=1966 location=New York id=ISBN 0471324205] Following theattack on Pearl Harbor in 1941, the United States army began new research to increase the prediction of correctly detected Japanese aircraft from their radar signals.In the 1950s, ROC curves were employed in
psychophysics to assess human (and occasionally nonhuman animal) detection of weak signals. Inmedicine , ROC analysis has been extensively used in the evaluation ofdiagnostic test s. [cite journalauthor=M.H. Zweig and G. Campbell journal=Clinical chemistry pmid=8472349 title=Receiveroperating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine volume=39 issue=8 date=1993 pages=561–577] [cite bookauthor=M.S. Pepe title=The statistical evaluation of medical tests for classification and prediction location=New York publisher=Oxford date=2003] ROC curves are also used extensively inepidemiology andmedical research and are frequently mentioned in conjunction withevidencebased medicine . Inradiology , ROC analysis is a common technique to evaluate new radiology techniques. [cite journalauthor=N.A. Obuchowski title=Receiver operating characteristic curves and their use in radiology pmid=14519861 journal=Radiology volume=229 issue=1 date=2003 pages=3–8doi=10.1148/radiol.2291010898] . In the social sciences, ROC analysis is often called the ROC Accuracy Ratio, a common technique for judging the accuracy of default probability models.ROC curves also proved useful for the evaluation of
machine learning techniques. The first application of ROC in machine learning was by Spackman who demonstrated the value of ROC curves in comparing and evaluating different classificationalgorithm s. [cite conferenceauthor=Spackman, K. A. date=1989 title=Signal detection theory: Valuable tools for evaluating inductive learning booktitle=Proceedings of the Sixth International Workshop on Machine Learning location=San Mateo, CA pages=160–163 publisher=Morgan Kaufman]References
ee also
*
Constant false alarm rate
*Detection theory
*False alarm General references
*
Further reading
* Balakrishnan, N., "Handbook of the Logistic Distribution", Marcel Dekker, Inc., 1991, ISBN13: 9780824785871.
* Gonen M., "Analyzing Receiver Operating Characteristic Curves Using SAS", SAS Press, 2007, ISBN: 9781599942981.
* Green, William H., "Econometric Analysis", fifth edition, Prentice Hall, 2003, ISBN 0130661899.
* Hosmer, David W. and Stanley Lemeshow, "Applied Logistic Regression", 2nd ed., New York; Chichester, Wiley, 2000, ISBN 0471356328.
* Lasko, T. A., J.G. Bhagwat, K.H. Zou and L. OhnoMachado (Oct. 2005). The use of receiver operating characteristic curves in biomedical informatics. "Journal of Biomedical Informatics" 38(5):404415. PMID 16198999
* Mason, S. J. and N.E. Graham, Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation. "Q.J.R. Meteorol. Soc." (2002), 128, pp. 2145–2166.
* Pepe, M. S. (2003). "The statistical evaluation of medical tests for classification and prediction". Oxford. ISBN 0198565828
* Stephan, Carsten, Sebastian Wesseling, Tania Schink, and Klaus Jung. Comparison of Eight Computer Programs for ReceiverOperating Characteristic Analysis. Clin. Chem., Mar 2003; 49: 433  439. [http://www.clinchem.org/cgi/reprint/49/3/433]
* Swets, J.A. (1995). "Signal detection theory and ROC analysis in psychology and diagnostics: Collected papers." Lawrence Erlbaum Associates.
External links
* [http://gim.unmc.edu/dxtests/roc2.htm A simple example of a ROC curve]
* [http://www.csee.usf.edu/~candamo/site/papers/ROCintro.pdf An introduction to ROC analysis]
* [http://wwwpsych.stanford.edu/~lera/psych115s/notes/signal/ A more thorough treatment of ROC curves and signal detection theory]
* [http://www.medcalc.be/calc/diagnostic_test.php Diagnostic test evaluation  online calculator]
* [http://www.spl.harvard.edu/archive/splpre2007/pages/ppl/zou/roc.html Kelly H. Zou's Bibliography of ROC Literature and Articles]
* [http://home.comcast.net/~tom.fawcett/public_html/ROCCH/index.html Tom Fawcett's ROC Convex Hull: tutorial, program and papers]
* [http://www.cs.bris.ac.uk/~flach/ICML04tutorial/index.html Peter Flach's tutorial on ROC analysis in machine learning]
* [http://www.anaesthetist.com/mnm/stats/roc/ The magnificent ROC] — An explanation and interactive demonstration of the connection of ROCs to archetypal binormal test result plotsoftware
* [http://www.mskcc.org/mskcc/html/84563.cfm SAS and R code for ROC curves]
* [http://www.clinchem.org/cgi/content/full/49/3/433 Comparison of Eight Computer Programs for ReceiverOperating Characteristic Analysis, Clinical Chemistry. 2003;49:433439]
* [http://protein.bio.puc.cl/star.html StAR, a software for the statistical comparison of ROC curves]
* [http://rocr.bioinf.mpisb.mpg.de ROCR, a comprehensive R package for evaluating scoring classifiers] ( [http://bioinformatics.oxfordjournals.org/cgi/content/full/21/20/3940 Introductory article] )
* [http://epiweb.massey.ac.nz/ROC_analysis_software.htm List of ROC analysis software]
* [http://bioconductor.org/packages/1.9/bioc/html/ROC.html ROC package for R (part of the BioConductor suite)]
* [http://www.acm.org/sigs/sigkdd/kddcup/index.php?section=2004&method=soft Standalone PERF program used by the KDD Cup competition]
* [http://www.rad.jhmi.edu/jeng/javarad/roc/JROCFITi.html Webbased calculator of ROC curves from usersupplied data]
* [http://www.cs.bris.ac.uk/Research/MachineLearning/rocon/index.html ROC curve visualiser]
* [http://www.analyseit.com/method_evaluation/roc.aspx Analyseit ROC software]
* [http://www.gepsoft.com/LogisticRegression/Section03.htm GeneXproTools ROC Analysis]
* [http://www.pinel.qc.ca/GeneralList.aspx?nav_id=2984&lang_id=E ROCTools ROC software]
* [http://theoval.sys.uea.ac.uk/~gcc/matlab/#roc ROC Curve Tools, mfiles for MATLAB, written by Dr. Gavin C. Cawley]
* [http://www.medcalc.be/ MedCalc software]
* [http://pages.cs.wisc.edu/~richm/programs/AUC/ AUCCalculator, a Java program for finding AUCROC by Jesse Davis and Mark Goadrich]
Wikimedia Foundation. 2010.