- Generalized linear model
In

statistics , the**generalized linear model**(**GLM**) is a flexible generalization of ordinary least squares regression. It relates the random distribution of the measured variable of the experiment (the "distribution function") to the systematic (non-random) portion of the experiment (the "linear predictor") through a function called the**link function**.Generalized linear models were formulated by

John Nelder and Robert Wedderburn as a way of unifying various other statistical models, includinglinear regression ,logistic regression andPoisson regression , under one framework. [*cite book | last = McCullagh | first = Peter | coauthors = Nelder, John | title = Generalized Linear Models | publisher = London: Chapman and Hall | date = 1989 | isbn = 0-412-31760-5 Chapter 1.*] This allowed them to develop a general algorithm formaximum likelihood estimation in all these models. It extends naturally to encompass many other models as well.**Overview**In a GLM, each outcome of the

dependent variable s,**Y**, is assumed to be generated from a particulardistribution function in theexponential family , a large range ofprobability distributions that includes the normal, binomial and poisson distributions, among others. The mean,**"μ**", of the distribution depends on the independent variables,**X**, through:: $operatorname\{E\}(mathbf\{Y\})\; =\; \backslash boldsymbol\{mu\}\; =\; g^\{-1\}(mathbf\{X\}\backslash boldsymbol\{eta\})$

where E(

**Y**) is theexpected value of**Y**;**X"β**" is the "linear predictor", a linear combination of unknown parameters,**"β**"; "g" is the link function.In this framework, the variance is typically a function,

**V**, of the mean::$operatorname\{Var\}(mathbf\{Y\})\; =\; operatorname\{V\}(\; \backslash boldsymbol\{mu\}\; )\; =\; operatorname\{V\}(g^\{-1\}(mathbf\{X\}\backslash boldsymbol\{eta\})).$

It is convenient if

**V**follows from the exponential family distribution, but it may simply be that the variance is a function of the predicted value.The unknown parameters,

**"β**", are typically estimated withmaximum likelihood , maximumquasi-likelihood , orBayesian techniques.**Model components**The GLM consists of three elements.: 1. A distribution function "f", from the exponential family. : 2. A linear predictor "η" =

**X"β**" .: 3. A link function "g" such that E(**Y**) =**μ**= "g"^{-1}("η").**Distribution function**The

exponential family of distributions are those probability distributions, parameterized by "θ" and "τ", whose density functions "f" (orprobability mass function , for the case of a discrete distribution) can be expressed in the form: $f\_Y(y;\; heta,\; au)\; =\; exp\{left(frac\{a(y)b(\; heta)\; -\; c(\; heta)\}\; \{h(\; au)\}\; +\; d(y,\; au)\; ight)\}.\; ,!$

"τ", called the "dispersion parameter", typically is known and is usually related to the variance of the distribution. The functions "a", "b", "c", "d", and "h" are known. Many, although not all, common distributions are in this family.

"θ" is related to the mean of the distribution. If "a" is the identity function, then the distribution is said to be in

canonical form . If, in addition, "b" is the identity and "τ" is known, then "θ" is called the "canonical parameter" and is related to the mean through:$mu\; =\; operatorname\{E\}(Y)\; =\; -c\text{'}(\; heta).\; ,!$

Under this scenario, the variance of the distribution can be shown to be [

*cite book | last = McCullagh | first = Peter | coauthors = Nelder, John | title = Generalized Linear Models | publisher = London: Chapman and Hall | date = 1989 | isbn = 0-412-31760-5 Chapter 2.*]:$operatorname\{Var\}(Y)\; =\; -c"(\; heta)\; h(\; au).\; ,!$

**Linear predictor**The linear predictor is the quantity which incorporates the information about the independent variables into the model. The symbol "η" (Greek "eta") is typically used to denote a linear predictor. It is related to the

expected value of the data (thus, "predictor") through the link function."η" is expressed as linear combinations (thus, "linear") of unknown parameters

**"β**". The coefficients of the linear combination are represented as the matrix of independent variables**X**. "η" can thus be expressed as:$eta\; =\; mathbf\{X\}\backslash boldsymbol\{eta\}.,$

The elements of

**X**are either measured by the experimenters or stipulated by them in the modeling design process.**Link function**The link function provides the relationship between the linear predictor and the mean of the distribution function. There are many commonly used link functions, and their choice can be somewhat arbitrary. It can be convenient to match the domain of the link function to the range of the distribution function's mean.

When using a distribution function with a canonical parameter "θ", a link function exists which allows for

**X**^{T}**Y**to be a sufficient statistic for**"β**". This occurs when the link function equates "θ" and the linear predictor. Following is a table of canonical link functions and their inverses (sometimes referred to as the mean function, as done here) used for several distributions in the exponential family.In the cases of the exponential and gamma distributions, the domain of the canonical link function is not the same as the permitted range of the mean. In particular, the linear predictor may be negative, which would give an impossible negative mean. When maximizing the likelihood, precautions must be taken to avoid this. An alternative is to use a noncanonical link function.

**Examples****General linear models**A possible point of confusion has to do with the distinction between generalized linear models and the

general linear model , two broad statistical models. The general linear model may be viewed as a case of the generalized linear model with identity link. As most exact results of interest are obtained only for the general linear model, the general linear model has undergone a somewhat longer historical development. Results for the generalized linear model with non-identity link areasymptotic (tending to work well with large samples).**Linear regression**A simple, very important example of a generalized linear model (also an example of a general linear model) is

linear regression . Here the distribution function is the normal distribution with constant variance and the link function is the identity, which is the canonical link if the variance is known. Unlike most other GLMs, there is a closed form solution for the maximum likelihood parameter estimates.**Binomial data**When the response data, "Y", are binary (taking on only values 0 and 1), the distribution function is generally chosen to be the

binomial distribution and the interpretation of "μ"_{i}is then the probability, "p", of "Y"_{i}taking on the value one.There are several popular link functions for binomial functions; the most typical is the canonical

logit link::$g(p)\; =\; ln\; left(\; \{\; p\; over\; 1-p\; \}\; ight).$

GLMs with this setup are

logistic regression models.In addition, the inverse of any continuous

cumulative distribution function (CDF) can be used for the link since the CDF's range is [0, 1] , the range of the binomial mean. The normal CDF $Phi$ is a popular choice and yields theprobit model . Its link is:$g(p)\; =\; Phi^\{-1\}(p).,!$

The identity link is also sometimes used for binomial data, but a drawback of doing this is that the predicted probabilities can be greater than one or less than zero. In implementation it is possible to fix the nonsensical probabilities outside of [0,1] but interpreting the coefficients can be difficult in this model. The model's primary merit is that near "p" = 0.5 it is approximately a linear transformation of the probit and logit―

econometrician s sometimes call this the Harvard model.The variance function for binomial data is given by:

:$operatorname\{Var\}(Y\_\{i\})=\; aumu\_\{i\}\; (1-mu\_\{i\}),!$

where the dispersion parameter "τ" is typically fixed at exactly one. When it is not, the resulting

quasi-likelihood model often described as binomial withoverdispersion or "quasibinomial".**Count data**Another example of generalized linear models includes

Poisson regression which modelscount data using thePoisson distribution . The link is typically the logarithm, the canonical link.The variance function is proportional to the mean

:$operatorname\{var\}(Y\_\{i\})\; =\; aumu\_\{i\},,$

where the dispersion parameter "τ" is typically fixed at exactly one. When it is not, the resulting

quasi-likelihood model is often described as poisson withoverdispersion or "quasipoisson".**Extensions****Correlated or clustered data**The standard GLM assumes that the observations are

uncorrelated . Extensions have been developed to allow forcorrelation between observations, as occurs for example inlongitudinal studies and clustered designs:

*(GEEs) allow for the correlation between observations without the use of an explicit probability model for the origin of the correlations, so there is no explicitGeneralized estimating equations likelihood . They are suitable when therandom effects and their variances are not of inherent interest, as they allow for the correlation without explaining its origin. The focus is on estimating the average response over the population ("population-averaged" effects) rather than the regression parameters that would enable prediction of the effect of changing one or more components of**X**on a given individual. GEEs are usually used in conjunction withHuber-White standard errors .

*(GLMMs) are an extension to GLMs that includesGeneralized linear mixed model srandom effects in the linear predictor, giving an explicit probability model that explains the origin of the correlations. The resulting "subject-specific" parameter estimates are suitable when the focus is on estimating the effect of changing one or more components of**X**on a given individual. GLMMs are a particular type ofmultilevel model (mixed model ). In general, fitting GLMMs is more computationally complex and intensive than fitting GEEs.

***Hierarchical generalized linear models**(HGLMs) are similar to GLMMs apart from two distinctions:

#The random effects can have any distribution in theexponential family , whereas current GLMMs nearly always have normal random effects;

#They are not as computationally intensive, as instead of integrating out the random effects they are based on a modified form of likelihood known as the "hierarchical likelihood" or "h"-likelihood.The theoretical basis and accuracy of the methods used in HGLMs have been the subject of some debate in the statistical literature. As of 2008, the method is only available in one statistical software package, namelyGenstat . [*cite book|author = Youngjo Lee|coauthors = John Nelder and Yudi Pawitan|title = Generalized Linear Models with Random Effects: Unified Analysis via H-likelihood|publisher = Chapman & Hall/CRC|year = 2006| url=http://www.crcpress.com/shopping_cart/products/product_detail.asp?sku=C6315*]**Generalized additive models**Generalized additive model s (GAMs) [*cite book|author = Hastie, T. J. and Tibshirani, R. J.|title = Generalized Additive Models|publisher = Chapman & Hall/CRC|year = 1990|isbn=9780412343902*] are another extension to GLMs in which the link function "η" is not restricted to be linear in the covariates**X**but is an additive function of the "x_{i}"s::$eta\; =\; eta\_0\; +\; f\_1(x\_1)\; +\; f\_2(x\_2)\; +\; ldots\; ,!$The smooth functions "f_{i}" are estimated from the data. In general this requires a large number of data points and is computationally intensive.**Multinomial regression**The binomial case may be easily extended to allow for a

multinomial distribution as the response. There are two ways in which this is usually done:**Ordered response**If the response variable is an ordinal measurement, then one may fit a model function of the form:

:$g(mu\_m)\; =\; eta\_m\; =\; eta\_0\; +\; X\_1\; eta\_1\; +\; ldots\; +\; X\_p\; eta\_p\; +\; gamma\_2\; +\; ldots\; +\; gamma\_m\; =\; eta\_1\; +\; gamma\_2\; +\; ldots\; +\; gamma\_m\; ,$ where $mu\_m\; =\; mathrm\{P\}(Y\; leq\; m)\; ,$.

for $m\; geq\; 2$. See:

*Ordered logit (also known as the proportional odds model).

*Ordered probit **Unordered response**If the response variable nominal measurement, or the data does not satisfy the assumptions of an ordered model, one may fit a model of the following form:

:$g(mu\_m)\; =\; eta\_m\; =\; eta\_\{m,0\}\; +\; X\_1\; eta\_\{m,1\}\; +\; ldots\; +\; X\_\{m,p\}\; eta\_p\; ,$ where $mu\_m\; =\; mathrm\{P\}(Y\; =\; m\; mid\; Y\; in\; \{1,m\}\; )\; ,$.

for $m\; geq\; 2$. This is much less efficient then the ordered response model, as it needs to estimate more parameters. See:

*Multinomial logit

*Multinomial probit **Etymology**The term "generalized linear model" and especially its abbreviation GLM can cause confusion with the

general linear model .John Nelder has expressed regret about this in a conversation with Stephen Senn:**Senn**: I must confess to having some confusionwhen I was a young statistician between general linearmodels and generalized linear models. Do you regretthe terminology?**Nelder**: I think probably I do. I suspect we shouldhave found some more fancy name for it that wouldhave stuck and not been confused with the generallinear model, although general and generalized are notquite the same. I can see why it might have been betterto have thought of something else. [*cite journal |last= Senn|first=Stephen |year=2003 |title=A conversation with John Nelder |journal=Statistical Science |volume=18 |issue=1 |pages=118–131 |doi=10.1214/ss/1056397489 |url=http://projecteuclid.org/euclid.ss/1056397489*]**External Sources*** [

*http://www.crcpress.com/shopping_cart/products/product_detail.asp?sku=C9500*]* [

*http://www.crcpress.com/shopping_cart/products/product_detail.asp?sku=SZ0149*]* [

*http://www.crcpress.com/shopping_cart/products/product_detail.asp?sku=C4390*]* [

*http://www.crcpress.com/shopping_cart/products/product_detail.asp?sku=C4746*]* [

*http://www.crcpress.com/shopping_cart/products/product_detail.asp?sku=C1760*]* [

*http://www.crcpress.com/shopping_cart/products/product_detail.asp?sku=C6315*]*

*

**References****ee also**For a more detailed discussion of the more common types of generalised linear models, see:

*Linear regression

*Logistic regression

*Poisson regression Also see

*Tweedie distributions **External links*** [

*http://www.eng.ox.ac.uk/samp Systems Analysis, Modelling and Prediction (SAMP), University of Oxford*] [*http://www.eng.ox.ac.uk/samp/glm_soft.html Open-source MATLAB code for GLM fitting.*]

* [*http://stats.ma.ic.ac.uk/j/jan01/public_html/ John Nelder FRS*]

* [*http://www.royalsoc.ac.uk/DServe/dserve.exe?dsqIni=Dserve.ini&dsqApp=Archive&dsqCmd=Show.tcl&dsqSearch=RefNo='EC/1981/28'&dsqDb=Catalog Royal Society citation for Nelder*]

*Wikimedia Foundation.
2010.*