 Bayesian probability

Bayesian statistics Theory Bayesian probability
Probability interpretations
Bayes' theorem
Bayes' rule · Bayes factor
Bayesian inference
Bayesian network
Prior · Posterior · Likelihood
Conjugate prior
Hyperparameter · Hyperprior
Principle of indifference
Principle of maximum entropy
Empirical Bayes method
Cromwell's rule
Bernstein–von Mises theorem
Bayesian information criterion
Credible intervalTechniques Bayesian linear regression
Bayesian estimator
Approximate Bayesian computationBayesian probability is one of the different interpretations of the concept of probability and belongs to the category of evidential probabilities. The Bayesian interpretation of probability can be seen as an extension of logic that enables reasoning with propositions, whose truth or falsity is uncertain. To evaluate the probability of a hypothesis, the Bayesian probabilist specifies some prior probability, which is then updated in the light of new, relevant data.^{[1]}
The Bayesian interpretation provides a standard set of procedures and formulae to perform this calculation. Bayesian probability interprets the concept of probability as "a degree of plausibility of a proposition (belief in a proposition) based on the given state of knowledge,"^{[2]} in contrast to interpreting it as a frequency or a "propensity" of some phenomenon.
The term "Bayesian" refers to the 18th century mathematician and theologian Thomas Bayes (1702–1761), who provided the first mathematical treatment of a nontrivial problem of Bayesian inference.^{[3]} Nevertheless, it was the French mathematician PierreSimon Laplace (1749–1827) who pioneered and popularised what is now called Bayesian probability.^{[4]}
Broadly speaking, there are two views on Bayesian probability that interpret the probability concept in different ways. According to the objectivist view, the rules of Bayesian statistics can be justified by requirements of rationality and consistency and interpreted as an extension of logic.^{[2]}^{[5]} According to the subjectivist view, probability measures a "personal belief".^{[6]} Many modern machine learning methods are based on objectivist Bayesian principles.^{[7]} In the Bayesian view, a probability is assigned to a hypothesis, whereas under the frequentist view, a hypothesis is typically tested without being assigned a probability.
Contents
Bayesian methodology
In general, Bayesian methods are characterized by the following concepts and procedures:
 The use of hierarchical models and marginalization over the values of nuisance parameters. In most cases, the computation is intractable, but good approximations can be obtained using Markov chain Monte Carlo methods.
 The sequential use of the Bayes' formula: when more data become available after calculating a posterior distribution, the posterior becomes the next prior.
 In frequentist statistics, a hypothesis is a proposition (which must be either true or false), so that the (frequentist) probability of a frequentist hypothesis is either one or zero. In Bayesian statistics, a probability can be assigned to a hypothesis.
Objective and subjective Bayesian probabilities
Broadly speaking, there are two views on Bayesian probability that interpret the 'probability' concept in different ways. For objectivists, probability objectively measures the plausibility of propositions, i.e. the probability of a proposition corresponds to a reasonable belief everyone (even a "robot") sharing the same knowledge should share in accordance with the rules of Bayesian statistics, which can be justified by requirements of rationality and consistency.^{[2]}^{[5]} Requirements of rationality and consistency are also important for subjectivists, for which the probability corresponds to a 'personal belief'.^{[6]} For subjectivists however, rationality and consistency constrain the probabilities a subject may have, but allow for substantial variation within those constraints. The objective and subjective variants of Bayesian probability differ mainly in their interpretation and construction of the prior probability.
History
Main article: History of statistics#Bayesian statisticsThe term Bayesian refers to Thomas Bayes (1702–1761), who proved a special case of what is now called Bayes' theorem in a paper titled "An Essay towards solving a Problem in the Doctrine of Chances".^{[8]} In that special case, the prior and posterior distributions were Beta distributions and the data came from Bernoulli trials. It was PierreSimon Laplace (1749–1827) who introduced a general version of the theorem and used it to approach problems in celestial mechanics, medical statistics, reliability, and jurisprudence.^{[9]} Early Bayesian inference, which used uniform priors following Laplace's principle of insufficient reason, was called "inverse probability" (because it infers backwards from observations to parameters, or from effects to causes).^{[10]} After the 1920s, "inverse probability" was largely supplanted by a collection of methods that came to be called frequentist statistics.^{[10]}
In the 20th century, the ideas of Laplace were further developed in two different directions, giving rise to objective and subjective currents in Bayesian practice. In the objectivist stream, the statistical analysis depends on only the model assumed and the data analysed.^{[11]} No subjective decisions need to be involved. In contrast, "subjectivist" statisticians deny the possibility of fully objective analysis for the general case.
In the 1980s, there was a dramatic growth in research and applications of Bayesian methods, mostly attributed to the discovery of Markov chain Monte Carlo methods, which removed many of the computational problems, and an increasing interest in nonstandard, complex applications.^{[12]} Despite the growth of Bayesian research, most undergraduate teaching is still based on frequentist statistics.^{[13]} Nonetheless, Bayesian methods are widely accepted and used, such as in the fields of machine learning^{[7]} and talent analytics.
Justification of Bayesian probabilities
The use of Bayesian probabilities as the basis of Bayesian inference has been supported by several arguments, such as the Cox axioms, the Dutch book argument, arguments based on decision theory and de Finetti's theorem.
Axiomatic approach
Richard T. Cox showed that^{[5]} Bayesian updating follows from several axioms, including two functional equations and the controversial hypothesis that probability should be treated as a continuous function. Here "continuity" is equivalent to countable additivity, as proved in measuretheoretic probability books. The countable additivity requirement is rejected (e.g. for being nonfalsifiable) by Bruno de Finetti, for example.
Dutch book approach
The Dutch book argument was proposed by de Finetti, and is based on betting. A Dutch book is made when a clever gambler places a set of bets that guarantee a profit, no matter what the outcome is of the bets. If a bookmaker follows the rules of the Bayesian calculus in the construction of his odds, a Dutch book cannot be made.
However, Ian Hacking noted that traditional Dutch book arguments did not specify Bayesian updating: they left open the possibility that nonBayesian updating rules could avoid Dutch books. For example, Hacking writes^{[14]} "And neither the Dutch book argument, nor any other in the personalist arsenal of proofs of the probability axioms, entails the dynamic assumption. Not one entails Bayesianism. So the personalist requires the dynamic assumption to be Bayesian. It is true that in consistency a personalist could abandon the Bayesian model of learning from experience. Salt could lose its savour."
In fact, there are nonBayesian updating rules that also avoid Dutch books (as discussed in the literature on "probability kinematics" following the publication of Richard C. Jeffrey's rule). The additional hypotheses sufficient to (uniquely) specify Bayesian updating are substantial, complicated, and unsatisfactory.^{[15]}
Decision theory approach
A decisiontheoretic justification of the use of Bayesian inference (and hence of Bayesian probabilities) was given by Abraham Wald, who proved that every admissible statistical procedure is either a Bayesian procedure or a limit of Bayesian procedures.^{[16]} Conversely, every Bayesian procedure is admissible.^{[17]}
Personal probabilities and objective methods for constructing priors
Following the work on expected utility theory of Ramsey and von Neumann, decisiontheorists have accounted for rational behavior using a probability distribution for the agent. Johann Pfanzagl completed the Theory of Games and Economic Behavior by providing an axiomatization of subjective probability and utility, a task left uncompleted by von Neumann and Oskar Morgenstern: their original theory supposed that all the agents had the same probability distribution, as a convenience.^{[18]} Pfanzagl's axiomatization was endorsed by Oskar Morgenstern: "Von Neumann and I have anticipated" the question whether probabilities "might, perhaps more typically, be subjective and have stated specifically that in the latter case axioms could be found from which could derive the desired numerical utility together with a number for the probabilities (cf. p. 19 of The Theory of Games and Economic Behavior). We did not carry this out; it was demonstrated by Pfanzagl ... with all the necessary rigor".^{[19]}
Ramsey and Savage noted that the individual agent's probability distribution could be objectively studied in experiments. The role of judgment and disagreement in science has been recognized since Aristotle and even more clearly with Francis Bacon. The objectivity of science lies not in the psychology of individual scientists, but in the process of science and especially in statistical methods, as noted by C. S. Peirce.^{[citation needed]} Recall that the objective methods for falsifying propositions about personal probabilities have been used for a half century, as noted previously. Procedures for testing hypotheses about probabilities (using finite samples) are due to Ramsey (1931) and de Finetti (1931, 1937, 1964, 1970). Both Bruno de Finetti and Frank P. Ramsey acknowledge^{[citation needed]} their debts to pragmatic philosophy, particularly (for Ramsey) to Charles S. Peirce.
The "Ramsey test" for evaluating probability distributions is implementable in theory, and has kept experimental psychologists occupied for a half century.^{[20]} This work demonstrates that Bayesianprobability propositions can be falsified, and so meet an empirical criterion of Charles S. Peirce, whose work inspired Ramsey. (This falsifiabilitycriterion was popularized by Karl Popper)^{[21]}^{[22]}
Modern work on the experimental evaluation of personal probabilities uses the randomization, blinding, and Booleandecision procedures of the PeirceJastrow experiment.^{[23]} Since individuals act according to different probability judgements, these agents' probabilities are "personal" (but amenable to objective study).
Personal probabilities are problematic for science and for some applications where decisionmakers lack the knowledge or time to specify an informed probabilitydistribution (on which they are prepared to act). To meet the needs of science and of human limitations, Bayesian statisticians have developed "objective" methods for specifying prior probabilities.
Indeed, some Bayesians have argued the prior state of knowledge defines the (unique) prior probabilitydistribution for "regular" statistical problems; cf. wellposed problems. Finding the right method for constructing such "objective" priors (for appropriate classes of regular problems) has been the quest of statistical theorists from Laplace to John Maynard Keynes, Harold Jeffreys, and Edwin Thompson Jaynes: These theorists and their successors have suggested several methods for constructing "objective" priors:
Each of these methods contributes useful priors for "regular" oneparameter problems, and each prior can handle some challenging statistical models (with "irregularity" or several parameters). Each of these methods has been useful in Bayesian practice. Indeed, methods for constructing "objective" (alternatively, "default" or "ignorance") priors have been developed by avowed subjective (or "personal") Bayesians like James Berger (Duke University) and JoséMiguel Bernardo (Universitat de València), simply because^{[citation needed]} such priors are needed for Bayesian practice, particularly in science. Each of these methods gives implausible priors for some problems, and so the quest for "the universal method for constructing priors" continues to attract statistical theorists.^{[citation needed]}
Thus, the Bayesian statistican needs either to use informed priors (using relevant expertise or previous data) or to choose among the competing methods for constructing "objective" priors.
See also
 Bertrand's paradox: a paradox in classical probability, solved by E.T. Jaynes in the context of Bayesian probability
 De Finetti's game – a procedure for evaluating someone's subjective probability
 Uncertainty
 An Essay towards solving a Problem in the Doctrine of Chances
Notes
 ^ Paulos, John Allen. "The Mathematics of Changing Your Mind," New York Times (US). August 5, 2011; retrieved 20110806
 ^ ^{a} ^{b} ^{c} ET. Jaynes. Probability Theory: The Logic of Science Cambridge University Press, (2003). ISBN 0521592712
 ^ Stephen M. Stigler (1986) The history of statistics. Harvard University press. pg 131.
 ^ Stephen M. Stigler (1986) The history of statistics. Harvard University press. pg 9798, pg 131.
 ^ ^{a} ^{b} ^{c} Richard T. Cox, Algebra of Probable Inference, The Johns Hopkins University Press, 2001
 ^ ^{a} ^{b} de Finetti, B. (1974) Theory of probability (2 vols.), J. Wiley & Sons, Inc., New York
 ^ ^{a} ^{b} Bishop, CM., Pattern Recognition and Machine Learning. Springer, 2007
 ^ McGrayne, Sharon Bertsch. (2011). The Theory That Would Not Die, p. 10. at Google Books
 ^ Stephen M. Stigler (1986) The history of statistics. Harvard University press. Chapter 3.
 ^ ^{a} ^{b} Stephen. E. Fienberg, (2006) "When did Bayesian Inference become "Bayesian"? Bayesian Analysis, 1 (1), 1–40. See page 5.
 ^ JM. Bernardo (2005), "Reference analysis", Handbook of statistics, 25, 17–90
 ^ Wolpert, RL. (2004) A conversation with James O. Berger, Statistical science, 9, 205–218
 ^ José M. Bernardo (2006) A Bayesian mathematical statistics prior. ICOTS7
 ^ Hacking (1967, Section 3, page 316), Hacking (1988, page 124)
 ^ van Frassen, B. (1989) Laws and Symmetry, Oxford University Press. ISBN 0198248601
 ^ Statistical Decision Functions. Abraham Wald. Wiley 1950.
 ^ Bayesian Theory. José Bernardo & Adrian F.M. Smith. John Wiley 1994. ISBN 0471924164.
 ^ Pfanzagl (1967, 1968)
 ^ Morgenstern (1976, page 65)
 ^ Davidson et al. (1957)
 ^ http://plato.stanford.edu/entries/popper/#ProDem
 ^ Popper, Karl. (2002) "The Logic of Scientific Discovery" 2nd Edition, Routledge ISBN 0415278430 (Reprint of 1959 translation of 1935 original) Page 57.
 ^ Pierce & Jastrow (1885)
References
 Berger, James O (1985). Statistical Decision Theory and Bayesian Analysis. Springer Series in Statistics (Second ed.). SpringerVerlag. ISBN 0387960988.
 Bernardo, José M.; Smith, Adrian F. M. (1994). Bayesian Theory. Wiley. ISBN 047149464x.
 Bickel, Peter J.; Doksum, Kjell A. (2001). Mathematical statistics, Volume 1: Basic and selected topics (Second (updated printing 2007) of the HoldenDay 1976 ed.). Pearson Prentice–Hall. ISBN 013850363X. MR443141.
 Davidson, Donald; Suppes, Patrick; Siegel, Sidney (1957). DecisionMaking: An Experimental Approach. Stanford University Press.
 de Finetti, Bruno. "Probabilism: A Critical Essay on the Theory of Probability and on the Value of Science," (translation of 1931 article) in Erkenntnis, volume 31, September 1989.
 de Finetti, Bruno (1937) "La Prévision: ses lois logiques, ses sources subjectives," Annales de l'Institut Henri Poincaré,
 de Finetti, Bruno. "Foresight: its Logical Laws, Its Subjective Sources," (translation of the 1937 article in French) in H. E. Kyburg and H. E. Smokler (eds), Studies in Subjective Probability, New York: Wiley, 1964.
 de Finetti, Bruno (1974–5). Theory of Probability. A Critical Introductory Treatment, (translation by A.Machi and AFM Smith of 1970 book) 2 volumes. Wiley ISBN 0471201413, ISBN 0471201421
 DeGroot, Morris (2004) Optimal Statistical Decisions. Wiley Classics Library. (Originally published 1970.) ISBN 047168029X.
 Hacking, Ian (December 1967). "Slightly More Realistic Personal Probability". Philosophy of Science 34 (4): 311–325. doi:10.1086/288169. JSTOR 186120. Partly reprinted in: Gärdenfors, Peter and Sahlin, NilsEric. (1988) Decision, Probability, and Utility: Selected Readings. 1988. Cambridge University Press. ISBN 0521336589
 Hajek, A. and Hartmann, S. (2010): "Bayesian Epistemology", in: Dancy, J., Sosa, E., Steup, M. (Eds.) (2001) A Companion to Epistemology, Wiley. ISBN 1405139005 Preprint
 Hald, Anders (1998). A History of Mathematical Statistics from 1750 to 1930. New York: Wiley. ISBN 0471179124.
 Hartmann, S. and Sprenger, J. (2011) "Bayesian Epistemology", in: Bernecker, S. and Pritchard, D. (Eds.) (2011) Routledge Companion to Epistemology. Routledge. ISBN 10415962196 (Preprint)
 Howson, C.; Urbach, P. (2005). Scientific Reasoning: the Bayesian Approach (3rd ed.). Open Court Publishing Company. ISBN 9780812695786.
 Jaynes E.T. (2003) Probability Theory: The Logic of Science, CUP. ISBN 9780521592710 (Link to Fragmentary Edition of March 1996).
 McGrayne, Sharon Bertsch. (2011). The Theory That Would Not Die: How Bayes' Rule Cracked The Enigma Code, Hunted Down Russian Submarines, & Emerged Triumphant from Two Centuries of Controversy. New Haven: Yale University Press. 13ISBN 9780300169690/10ISBN 0300169698; OCLC 670481486
 Morgenstern, Oskar (1978). "Some Reflections on Utility". In Andrew Schotter. Selected Economic Writings of Oskar Morgenstern. New York University Press. pp. 65–70. ISBN 9780814777718.
 Pierce, C.S. and Jastrow J. (1885). "On Small Differences in Sensation". Memoirs of the National Academy of Sciences 3: 73–83. http://psychclassics.yorku.ca/Peirce/smalldiffs.htm.
 Pfanzagl, J (1967). "Subjective Probability Derived from the Morgensternvon Neumann Utility Theory". In Martin Shubik. Essays in Mathematical Economics In Honor of Oskar Morgenstern. Princeton University Press. pp. 237–251.
 Pfanzagl, J. in cooperation with V. Baumann and H. Huber (1968). "Events, Utility and Subjective Probability". Theory of Measurement. Wiley. pp. 195–220.
 Ramsey, Frank Plumpton (1931) "Truth and Probability" (PDF), Chapter VII in The Foundations of Mathematics and other Logical Essays, Reprinted 2001, Routledge. ISBN 0415225469,
 Stigler, Stephen M. (1990). The History of Statistics: The Measurement of Uncertainty before 1900. Belknap Press/Harvard University Press. ISBN 067440341X.
 Stigler, Stephen M. (1999) Statistics on the Table: The History of Statistical Concepts and Methods. Harvard University Press. ISBN 0674836014
External links
 Online textbook: Information Theory, Inference, and Learning Algorithms, by David MacKay, has many chapters on Bayesian methods, including introductory examples; arguments in favour of Bayesian methods (in the style of Edwin Jaynes); stateoftheart Monte Carlo methods, messagepassing methods, and variational methods; and examples illustrating the intimate connections between Bayesian inference and data compression.
 An Intuitive Explanation of Bayesian Reasoning A very gentle introduction by Eliezer Yudkowsky
 An online introductory tutorial to Bayesian probability from Queen Mary University of London
 James Franklin The Science of Conjecture: Evidence and Probability Before Pascal, history from a Bayesian point of view.
Categories:
Wikimedia Foundation. 2010.