Normal-gamma distribution

Normal-gamma

parameters:	$\mu\,$ location (real) $\lambda > 0\,$ (real) $\alpha \ge 1\,$ (real) $\beta \ge 0\,$ (real)
support:	$x \in (-\infty, \infty)\,\!, \; \tau \in (0,\infty)$
pdf:	$f(x,\tau|\mu,\lambda,\alpha,\beta) = \frac{\beta^\alpha \sqrt{\lambda}}{\Gamma(\alpha)\sqrt{2\pi}} \, \tau^{\alpha-\frac{1}{2}}\,e^{-\beta\tau}\,e^{ -\frac{ \lambda \tau (x- \mu)^2}{2}}$
mean:	[1] $\operatorname{E}(X)=\mu\,\! ,\quad \operatorname{E}(\Tau)= \alpha \beta^{-1}$
variance:	[1] $\operatorname{var}(X)= \frac{\beta}{\lambda (\alpha-1)} ,\quad \operatorname{var}(\Tau)=\alpha \beta^{-2}$

In probability theory and statistics, the normal-gamma distribution is a bivariate four-parameter family of continuous probability distributions. It is the conjugate prior of a normal distribution with unknown mean and precision.^[2]

Definition

Suppose

$x|\tau, \mu, \lambda \sim N(\mu,1 /(\lambda \tau)) \,\!$

has a normal distribution with mean μ and variance 1 / (λτ), where

$\tau|\alpha, \beta \sim \mathrm{Gamma}(\alpha,\beta) \!$

has a gamma distribution. Then (x,τ) has a normal-gamma distribution, denoted as

$(x,\tau) \sim \mathrm{NormalGamma}(\mu,\lambda,\alpha,\beta) \! .$

Characterization

Probability density function

$f(x,\tau|\mu,\lambda,\alpha,\beta) = \frac{\beta^\alpha \sqrt{\lambda}}{\Gamma(\alpha)\sqrt{2\pi}} \, \tau^{\alpha-\frac{1}{2}}\,e^{-\beta\tau}\,e^{ -\frac{ \lambda \tau (x- \mu)^2}{2}}$

Properties

Scaling

For any t > 0, tX is distributed NormalGamma(tμ,λ,α,t²β)

Marginal distributions

By construction, the marginal distribution over τ is a gamma distribution, and the conditional distribution over x given τ is a Gaussian distribution. The marginal distribution over x is a three-parameter Student's t-distribution.

Posterior distribution of the parameters

Form of the posterior for a Normal random variable with a Normal-Gamma prior:

Presume the following hierarchy for a normal random variable X with unknown mean μ and precision λ.

$\begin{align} X & \sim \mathcal{N}(\mu, \lambda^{-1}) \\ \mu | \lambda &\sim \mathcal{N}(\mu_0, {(n_0 \lambda})^{-1}) \\ \lambda &\sim \mathcal{G}\left(\frac{\nu_0}{2},\frac{2}{S_0}\right) \end{align}$

Where:

μ₀ is the prior mean

S₀ is the prior sum of squared errors

n₀ is the prior sample size

ν₀ is the prior degrees of freedom

Note the joint distribution of the parameters is Normal-Gamma. The posterior distribution of the parameters can be analytically determined by Bayes' rule working with the likelihood $\mathbf{L(\lambda, \mu | X)}$, and the prior π(λ,μ).

$\begin{align} \mathbf{L(\lambda, \mu | X)} & \propto \prod_{i=1}^n \lambda^{1/2} \exp[\frac{-\lambda}{2}(x_i-\mu)^2] \\ & \propto \lambda^{n/2} \exp[\frac{-\lambda}{2}\sum_{i=1}^n(x_i-\mu)^2] \\ & \propto \lambda^{n/2} \exp[\frac{-\lambda}{2}\sum_{i=1}^n(x_i-\bar{x} +\bar{x} -\mu)^2] \\ & \propto \lambda^{n/2} \exp[\frac{-\lambda}{2}\sum_{i=1}^n\left((x_i-\bar{x})^2 + (\bar{x} -\mu)^2\right)] \\ & \propto \lambda^{n/2} \exp[\frac{-\lambda}{2}\left(S + n(\bar{x} -\mu)^2\right)] \end{align}$

where $S=\sum_{i=1}^n(x_i-\bar{x})^2$, the sum of squared errors.

Now consider the prior,

$\mathbf{\pi}(\mu,\lambda) \propto \lambda^{1/2}\exp[\frac{-\lambda n_0}{2}(\mu-\mu_0)^2] \lambda^{\frac{\nu_0}{2}-1}\exp[\frac{-\lambda S_0}{2}]$

The posterior distribution of the parameters is proportional to the prior times the likelihood.

$\begin{align} \mathbf{P(\lambda, \mu | X}) &\propto \lambda^{n/2} \exp[\frac{-\lambda}{2}\left(S + n(\bar{x} -\mu)^2\right)] \lambda^{1/2}\exp[\frac{-\lambda n_0}{2}(\mu-\mu_0)^2] \lambda^{\frac{\nu_0}{2}-1}\exp[\frac{-\lambda S_0}{2}] \\ &\propto \lambda^{\frac{\nu_0+n}{2}-1} \exp[\frac{-\lambda}{2}(S + S_0) ] \lambda^{1/2}\exp[\frac{-\lambda}{2}\left(n_0(\mu-\mu_0)^2 + n(\bar{x} -\mu)^2\right)] \\ \end{align}$

Notice the right half begins to look like the kernel of a normal pdf and the left like a gamma. After a bit of juggling and completing the square the result will appear.

$\begin{align} \mathbf{P(\lambda, \mu | X} )& \propto \lambda^{\frac{\nu_0+n}{2}-1} \exp[\frac{-\lambda}{2}(S + S_0) ] \lambda^{1/2}\exp[\frac{- \lambda}{2} \left(n_0 (\mu^2 - 2 \mu \mu_0 + \mu_0^2 ) + n(\bar{x}^2-2 \mu \bar{x} + \mu^2)\right)] \\ & \propto \lambda^{\frac{\nu_0+n}{2}-1} \exp[\frac{-\lambda}{2}(S + S_0 + n_0 \mu_0^2 + n \bar{x}^2) ] \lambda^{1/2}\exp[\frac{-\lambda}{2} (n+n_0) \left(\frac{n_0 \mu^2 + n \mu^2 }{n + n_0} - 2 \mu \frac{n\bar{x} +n_0\mu_0}{n+n_0} \right)] \\ & \propto \lambda^{\frac{\nu_0+n}{2}-1} \exp[\frac{-\lambda}{2}(S + S_0 + n_0 \mu_0^2 + n \bar{x}^2) ] \lambda^{1/2}\exp[\frac{-\lambda}{2} (n+n_0) \left(\mu^2 - 2 \mu \frac{n\bar{x} +n_0\mu_0}{n+n_0} + \left (\frac{n\bar{x} +n_0\mu_0}{n+n_0}\right )^2 - \left (\frac{n\bar{x} +n_0\mu_0}{n+n_0}\right )^2\right)] \\ & \propto \lambda^{\frac{\nu_0+n}{2}-1} \exp[\frac{-\lambda}{2}\left(S + S_0 + n_0 \mu_0^2 + n \bar{x}^2 - \frac{\left (n\bar{x} +n_0\mu_0 \right )^2}{n+n_0}\right) ] \lambda^{1/2}\exp[\frac{-\lambda}{2} (n+n_0) \left ( \mu - \frac{n\bar{x} +n_0\mu_0}{n+n_0}\right )^2] \\ & \propto \lambda^{\frac{\nu_0+n}{2}-1} \exp[\frac{-\lambda}{2}\left(S + S_0 + \frac{nn_0 (\bar{x}-\mu_0)^2}{n+n_0}\right) ] \lambda^{1/2}\exp[\frac{-\lambda}{2} (n+n_0) \left ( \mu - \frac{n\bar{x} +n_0\mu_0}{n+n_0}\right )^2] .\\ \end{align}$

This is a normal gamma pdf with parameters $\mathcal{NG} \left(\frac{n\bar{x} +n_0\mu_0}{n+n_0}, n+n_0, \frac{\nu_0+n}{2}, 2\left(S + S_0 + \frac{nn_0 (\bar{x}-\mu_0)^2}{n+n_0}\right)^{-1} \right) .$

The reference prior is^{[citation needed]} the limiting case as

$n_0, S_0, \mu_0 \rightharpoonup 0$

and $\nu_0 \rightharpoonup -1$

Generating normal-gamma random variates

Generation of random variates is straightforward:

1. Sample τ from a gamma distribution with parameters α and β
2. Sample x from a normal distribution with mean μ and variance 1 / (λτ)

Related distributions

• The normal-scaled inverse gamma distribution is essentially the same distribution parameterized by variance rather than precision
• The Normal-exponential-gamma distribution

Notes

1. ^ ^{a b} Bernardo & Smith (1993, p.434)
2. ^ Bernardo & Smith (1993, pages 136, 268, 434)

References

• Bernardo, J.M.; Smith, A.F.M. (1993) Bayesian Theory, Wiley. ISBN 0-471-49464-X
• Dearden et al. Bayesian Q-learning, Proceedings of the Fifteenth National Conference on Artificial Intelligence (AAAI-98), July 26–30, 1998, Madison, Wisconsin, USA.