Variational Bayesian methods

Variational Bayesian methods, also called ensemble learning, are a family of techniques for approximating intractable integrals arising in Bayesian statistics and machine learning. They can be used to lower bound the marginal likelihood (i.e. "evidence") of several models with a view to performing model selection, and often provide an analytical approximation to the parameter posterior which is useful for prediction. It is an alternative to Monte Carlo sampling methods for making use of a posterior distribution that is difficult to sample from directly.

Mathematical derivation

In variational inference, the posterior distribution over a set of latent variables $X\; =\; \{X\_1\; dots\; X\_n\}$ given some data $D$ is approximatedby a variational distribution

:$P(X|D)\; approx\; Q(X).$

The variational distribution $Q(X)$ is restricted to belong to a family of distributions of simplerform than $P(X|D)$. This family is selected with the intention that $Q$ can be made very similarto the true posterior. The difference between $Q$ and this true posterior is measured in terms ofa dissimilarity function $d(Q;\; P)$ and hence inference is performed by selecting the distribution$Q$ that minimises $d$. One choice of dissimilarity function where this minimisation is tractableis the Kullback-Leibler divergence (KL divergence), defined as

:$ext\{KL\}(Q\; ||\; P)\; =\; sum\_X\; Q(X)\; log\; frac\{Q(X)\}\{P(X|D)\}.$

We can write the log evidence as

:

As the log evidence is fixed with respect to $Q$, maximising the final term $mathcal\{L\}(Q)$ will minimise the KL divergence between $Q$ and $P$. By appropriate choice of $Q$, we can make $mathcal\{L\}(Q)$ tractable to compute and to maximise. Hence we have both a lower bound on the evidence $mathcal\{L\}(Q)$ and an analytical approximation to the posterior $Q$.

ee also

* Variational message passing: a modular algorithm for variational Bayesian inference.
* Expectation-maximization algorithm: a related approach which corresponds to a special case of variational Bayesian inference.

External links

* [http://www.variational-bayes.org Variational-Bayes.org] - a repository of papers, software, and links related to the use of variational Bayesian methods.
* [http://www.inference.phy.cam.ac.uk/mackay/itila/ The on-line textbook: Information Theory, Inference, and Learning Algorithms] , by David J.C. MacKay provides an introduction to variational methods (p. 422).
* [http://www.cse.buffalo.edu/faculty/mbeal/thesis/index.html Variational Algorithms for Approximate Bayesian Inference] , by M. J. Beal includes comparisons of EM to Variational Bayesian EM and derivations of several models including Variational Bayesian HMMs.