Maximum a posteriori

In statistics, the method of maximum a posteriori (MAP, or posterior mode) estimation can be used to obtain a point estimate of an unobserved quantity on the basis of empirical data. It is closely related to Fisher's method of maximum likelihood (ML), but employs an augmented optimization objective which incorporates a prior distribution over the quantity one wants to estimate. MAP estimation can therefore be seen as a regularization of ML estimation.

Introduction

Assume that we want to estimate an unobserved population parameter $heta$ on the basis of observations $x$. Let $f$ be the sampling distribution of $x$, so that $f(x|\; heta)$ is the probability of $x$ when the underlying population parameter is $heta$. Then the function

:$heta\; mapsto\; f(x\; |\; heta)\; !$

is known as the likelihood function and the estimate

:$hat\{\; heta\}\_\{mathrm\{ML(x)\; =\; argmax\_\{\; heta\}\; f(x\; |\; heta)\; !$

as the maximum likelihood estimate of $heta$.

Now assume that a prior distribution $g$ over $heta$ exists. This allows us to treat $heta$ as a random variable as in Bayesian statistics. Then the posterior distribution of $heta$ is as follows:

:$heta\; mapsto\; frac\{f(x\; |\; heta)\; ,\; g(\; heta)\}\{displaystyleint\_\{Theta\}\; f(x\; |\; heta\text{'})\; ,\; g(\; heta\text{'})\; ,\; d\; heta\text{'}\}\; !$

where $g$ is density function of $heta$, $Theta$ is the domain of $g$. This is a straightforward application of Bayes' theorem.

The method of maximum a posteriori estimation then estimates $heta$ as the mode of the posterior distribution of this random variable:

:$hat\{\; heta\}\_\{mathrm\{MAP(x)=\; argmax\_\{\; heta\}\; frac\{f(x\; |\; heta)\; ,\; g(\; heta)\}\; \{displaystyleint\_\{Theta\}\; f(x\; |\; heta\text{'})\; ,\; g(\; heta\text{'})\; ,\; d\; heta\text{'}\}=\; argmax\_\{\; heta\}\; f(x\; |\; heta)\; ,\; g(\; heta)!$

The denominator of the posterior distribution does not depend on $heta$ and therefore plays no role in the optimization. Observe that the MAP estimate of $heta$ coincides with the ML estimate when the prior $g$ is uniform (that is, a constant function). The MAP estimate is the Bayes estimator under the uniform loss function.

MAP estimates can be computed in several ways:
# Analytically, when the mode(s) of the posterior distribution can be given in closed form. This is the case when conjugate priors are used.
# Via numerical optimization such as the conjugate gradient method or Newton's method. This usually requires first or second derivatives, which have to be evaluated analytically or numerically.
# Via a modification of an expectation-maximization algorithm. This does not require derivatives of the posterior density.

While MAP estimation "is" a Bayes estimator (under the 0-1 loss function), it is not very representative of Bayesian methods in general. This is because MAP estimates are point estimates, whereas Bayesian methods are characterized by the use of distributions to summarize data and draw inferences: thus, Bayesian methods tend to report the posterior mean or median instead, together with posterior intervals. This is both because these estimators are optimal under squared-error and linear-error loss respectively - which are more representative of typical loss functions - and because the posterior distribution may not have a simple analytic form: in this case, the distribution can be simulated using Markov chain Monte Carlo techniques, while optimization to find its mode(s) may be difficult or impossible.

Example

Suppose that we are given a sequence $(x\_1,\; dots,\; x\_n)$ of IID $N(mu,sigma\_v^2\; )$ random variables and an a priori distribution of $mu$ is given by $N(0,sigma\_m^2\; )$. We wish to find the MAP estimate of $mu$.

The function to be maximized is then given by

:$pi(mu)\; L(mu)\; =\; frac\{1\}\{sqrt\{2\; pi\}\; sigma\_m\}\; expleft(-frac\{1\}\{2\}\; left(frac\{mu\}\{sigma\_m\}\; ight)^2\; ight)\; prod\_\{j=1\}^n\; frac\{1\}\{sqrt\{2\; pi\}\; sigma\_v\}\; expleft(-frac\{1\}\{2\}\; left(frac\{x\_j\; -\; mu\}\{sigma\_v\}\; ight)^2\; ight),$

which is equivalent to minimizing $mu$ in the following

:$sum\_\{j=1\}^n\; left(frac\{x\_j\; -\; mu\}\{sigma\_v\}\; ight)^2\; +\; left(frac\{mu\}\{sigma\_m\}\; ight)^2.$

Thus, we see that the MAP estimator for μ is given by

:$hat\{mu\}\_\{MAP\}\; =\; frac\{sigma\_m^2\}\{n\; sigma\_m^2\; +\; sigma\_v^2\; \}\; sum\_\{j=1\}^n\; x\_j.$

The case of $sigma\_m\; o\; infty$ is called a non-informative prior and leads to an ill-defined a priori probability distribution; in this case $hat\{mu\}\_\{MAP\}\; o\; hat\{mu\}\_\{ML\}.$

See also

* Maximum likelihood estimation, when no prior distribution is available.

References

* M. DeGroot, "Optimal Statistical Decisions", McGraw-Hill, (1970).
* Harold W. Sorenson, (1980) "Parameter Estimation: Principles and Problems", Marcel Dekker.