 Maximum likelihood

In statistics, maximumlikelihood estimation (MLE) is a method of estimating the parameters of a statistical model. When applied to a data set and given a statistical model, maximumlikelihood estimation provides estimates for the model's parameters.
The method of maximum likelihood corresponds to many wellknown estimation methods in statistics. For example, one may be interested in the heights of adult female giraffes, but be unable due to cost or time constraints, to measure the height of every single giraffe in a population. Assuming that the heights are normally (Gaussian) distributed with some unknown mean and variance, the mean and variance can be estimated with MLE while only knowing the heights of some sample of the overall population. MLE would accomplish this by taking the mean and variance as parameters and finding particular parametric values that make the observed results the most probable (given the model).
In general, for a fixed set of data and underlying statistical model, the method of maximum likelihood selects values of the model parameters that produce a distribution that gives the observed data the greatest probability (i.e., parameters that maximize the likelihood function). Maximumlikelihood estimation gives a unified approach to estimation, which is welldefined in the case of the normal distribution and many other problems. However, in some complicated problems, difficulties do occur: in such problems, maximumlikelihood estimators are unsuitable or do not exist.
Contents
Principles
Suppose there is a sample x_{1}, x_{2}, …, x_{n} of n iid observations, coming from a distribution with an unknown pdf ƒ_{0}(·). It is however surmised that the function ƒ_{0} belongs to a certain family of distributions { ƒ(·θ), θ ∈ Θ }, called the parametric model, so that ƒ_{0} = ƒ(·θ_{0}). The value θ_{0} is unknown and is referred to as the "true value" of the parameter. It is desirable to find some estimator which would be as close to the true value θ_{0} as possible. Both the observed variables x_{i} and the parameter θ can be vectors.
To use the method of maximum likelihood, one first specifies the joint density function for all observations. For an iid sample this joint density function will be
Now we look at this function from a different perspective by considering the observed values x_{1}, x_{2}, ..., x_{n} to be fixed "parameters" of this function, whereas θ will be the function's variable and allowed to vary freely. From this point of view this distribution function will be called the likelihood:
In practice it is often more convenient to work with the logarithm of the likelihood function, called the loglikelihood:
or its scaled version, called the average loglikelihood^{[1]}^{[2]}:
The hat over indicates that it is akin to some estimator. Indeed, estimates the expected loglikelihood of a single observation in the model.
The method of maximum likelihood estimates θ_{0} by finding a value of θ that maximizes . This method of estimation is a maximum likelihood estimator (MLE) of θ_{0}:
A MLE estimate is the same regardless of whether we maximize the likelihood or the loglikelihood function, since log is a monotone transformation.
For many models, a maximum likelihood estimator can be found as an explicit function of the observed data x_{1}, …, x_{n}. For many other models, however, no closedform solution to the maximization problem is known or available, and a MLE has to be found numerically using optimization methods. For some problems, there may be multiple estimates that maximize the likelihood. For other problems, no maximum likelihood estimate exists (meaning that the loglikelihood function increases without attaining the supremum value).
In the exposition above, it is assumed that the data are independent and identically distributed. The method can be applied however to a broader setting, as long as it is possible to write the joint density function ƒ(x_{1},…,x_{n}  θ), and its parameter θ has a finite dimension which does not depend on the sample size n. In a simpler extension, an allowance can be made for data heterogeneity, so that the joint density is equal to ƒ_{1}(x_{1}θ) · ƒ_{2}(x_{2}θ) · … · ƒ_{n}(x_{n}θ). In the more complicated case of time series models, the independence assumption may have to be dropped as well.
A maximum likelihood estimator coincides with the most probable Bayesian estimator given a uniform prior distribution on the parameters.
Properties
Maximum likelihood is the extremum estimator obtained by maximizing, as a function of θ, the objective function
this being the sample analogue of the expected loglikelihood , where this expectation is taken with respect to the true density f(·θ_{0}).
The maximumlikelihood estimator has essentially no optimal properties for finite samples.^{[3]} However, the maximumlikelihood estimator possesses a number of attractive asymptotic properties, for many problems; these asymptotic properties include:
 Consistency: the estimator converges in probability to the value being estimated.
 Asymptotic normality: as the sample size increases, the distribution of the MLE tends to the Gaussian distribution with mean θ and covariance matrix equal to the inverse of the Fisher information matrix. (see e.g. Myung & Navarro 2004).
 Efficiency, i.e., it achieves the Cramér–Rao lower bound when the sample size tends to infinity. This means that no asymptotically unbiased estimator has lower asymptotic mean squared error than the MLE.
 Secondorder efficiency after correction for bias.
Consistency
Under the conditions outlined below, the maximum likelihood estimator is consistent. The consistency means that having a sufficiently large number of observations n, it is possible to find the value of θ_{0} with arbitrary precision. In mathematical terms this means that as n goes to infinity the estimator converges in probability to its true value:
Under slightly stronger conditions, the estimator converges almost surely (or strongly) to:
To establish consistency, the following conditions are sufficient:^{[4]}
 Identification of the model:
The identification condition is absolutely necessary for the ML estimator to be consistent. When this condition holds, the limiting likelihood function ℓ(θ·) has unique global maximum at θ_{0}.  Compactness: the parameter space Θ of the model is compact.
The identification condition establishes that the loglikelihood has a unique global maximum. Compactness implies that the likelihood cannot approach the maximum value arbitrarily close at some other point (as demonstrated for example in the picture on the right).
Compactness is only a sufficient condition and not a necessary condition. Compactness can be replaced by some other conditions, such as:
 both concavity of the loglikelihood function and compactness of some (nonempty) upper level sets of the loglikelihood function, or
 existence of a compact neighborhood N of θ_{0} such that outside of N the loglikelihood function is less than the maximum by at least some ε > 0.
 'Continuity: the function ln f(xθ) is continuous in θ for almost all xs:
 Dominance: there exists an integrable function D(x) such that
The dominance condition can be employed in the case of i.i.d. observations. In the noni.i.d. case the uniform convergence in probability can be checked by showing that the sequence is stochastically equicontinuous.
If one wants to demonstrate that the ML estimator converges to θ_{0} almost surely, then a stronger condition of uniform convergence almost surely has to be imposed:
Asymptotic normality
Maximumlikelihood estimators can lack asymptotic normality and can be inconsistent if there is a failure of one (or more) of the below regularity conditions:
Estimate on boundary. Sometimes the maximum likelihood estimate lies on the boundary of the set of possible parameters, or (if the boundary is not, strictly speaking, allowed) the likelihood gets larger and larger as the parameter approaches the boundary. Standard asymptotic theory needs the assumption that the true parameter value lies away from the boundary. If we have enough data, the maximum likelihood estimate will keep away from the boundary too. But with smaller samples, the estimate can lie on the boundary. In such cases, the asymptotic theory clearly does not give a practically useful approximation. Examples here would be variancecomponent models, where each component of variance, σ^{2}, must satisfy the constraint σ^{2} ≥0.
Data boundary parameterdependent. For the theory to apply in a simple way, the set of data values which has positive probability (or positive probability density) should not depend on the unknown parameter. A simple example where such parameterdependence does hold is the case of estimating θ from a set of independent identically distributed when the common distribution is uniform on the range (0,θ). For estimation purposes the relevant range of θ is such that θ cannot be less than the largest observation. Because the interval (0,θ) is not compact, there exists no maximum for the likelihood function: For any estimate of theta, there exists a greater estimate that also has greater likelihood. In contrast, the interval [0,θ] includes the endpoint θ and is compact, in which case the maximumlikelihood estimator exists. However, in this case, the maximumlikelihood estimator is biased. Asymptotically, this maximumlikelihood estimator is not normally distributed.^{[5]}
Nuisance parameters. For maximum likelihood estimations, a model may have a number of nuisance parameters. For the asymptotic behaviour outlined to hold, the number of nuisance parameters should not increase with the number of observations (the sample size). A wellknown example of this case is where observations occur as pairs, where the observations in each pair have a different (unknown) mean but otherwise the observations are independent and Normally distributed with a common variance. Here for 2N observations, there are N+1 parameters. It is wellknown that the maximum likelihood estimate for the variance does not converge to the true value of the variance.
Increasing information. For the asymptotics to hold in cases where the assumption of independent identically distributed observations does not hold, a basic requirement is that the amount of information in the data increases indefinitely as the sample size increases. Such a requirement may not be met if either there is too much dependence in the data (for example, if new observations are essentially identical to existing observations), or if new independent observations are subject to an increasing observation error.
Some regularity conditions which ensure this behavior are:
 The first and second derivatives of the loglikelihood function must be defined.
 The Fisher information matrix must not be zero, and must be continuous as a function of the parameter.
 The maximum likelihood estimator is consistent.
Suppose that conditions for consistency of maximum likelihood estimator are satisfied, and^{[6]}
 θ_{0} ∈ interior(Θ);
 f(xθ) > 0 and is twice continuously differentiable in θ in some neighborhood N of θ_{0};
 ∫ sup_{θ∈N}∇_{θ}f(xθ)dx < ∞, and ∫ sup_{θ∈N}∇_{θθ}f(xθ)dx < ∞;
 I = E[∇_{θ}lnf(xθ_{0}) ∇_{θ}lnf(xθ_{0})′] exists and is nonsingular;
 E[ sup_{θ∈N}∇_{θθ}lnf(xθ)] < ∞.
Then the maximum likelihood estimator has asymptotically normal distribution:
Proof, skipping the technicalities:
Since the loglikelihood function is differentiable, and θ_{0} lies in the interior of the parameter set, in the maximum the firstorder condition will be satisfied:
When the loglikelihood is twice differentiable, this expression can be expanded into a Taylor series around the point θ = θ_{0}:
where is some point intermediate between θ_{0} and . From this expression we can derive that
Here the expression in square brackets converges in probability to H = E[−∇_{θθ}ln f(xθ_{0})] by the law of large numbers. The continuous mapping theorem ensures that the inverse of this expression also converges in probability, to H^{−1}. The second sum, by the central limit theorem, converges in distribution to a multivariate normal with mean zero and variance matrix equal to the Fisher information I. Thus, applying the Slutsky's theorem to the whole expression, we obtain that
Finally, the information equality guarantees that when the model is correctly specified, matrix H will be equal to the Fisher information I, so that the variance expression simplifies to just I^{−1}.
Functional invariance
The maximum likelihood estimator selects the parameter value which gives the observed data the largest possible probability (or probability density, in the continuous case). If the parameter consists of a number of components, then we define their separate maximum likelihood estimators, as the corresponding component of the MLE of the complete parameter. Consistent with this, if is the MLE for θ, and if g(θ) is any transformation of θ, then the MLE for α = g(θ) is by definition
It maximizes the socalled profile likelihood:
The MLE is also invariant with respect to certain transformations of the data. If Y = g(X) where g is one to one and does not depend on the parameters to be estimated, then the density functions satisfy
and hence the likelihood functions for X and Y differ only by a factor that does not depend on the model parameters.
For example, the MLE parameters of the lognormal distribution are the same as those of the normal distribution fitted to the logarithm of the data.
Higherorder properties
The standard asymptotics tells that the maximumlikelihood estimator is √nconsistent and asymptotically efficient, meaning that it reaches the Cramér–Rao bound:
where I is the Fisher information matrix:
In particular, it means that the bias of the maximumlikelihood estimator is equal to zero up to the order n^{−1/2}. However when we consider the higherorder terms in the expansion of the distribution of this estimator, it turns out that θ_{mle} has bias of order n^{−1}. This bias is equal to (componentwise)^{[7]}
where Einstein's summation convention over the repeating indices has been adopted; I^{ jk} denotes the j,kth component of the inverse Fisher information matrix I^{−1}, and
Using these formulas it is possible to estimate the secondorder bias of the maximum likelihood estimator, and correct for that bias by subtracting it:
This estimator is unbiased up to the terms of order n^{−1}, and is called the biascorrected maximum likelihood estimator.
This biascorrected estimator is secondorder efficient (at least within the curved exponential family), meaning that it has minimal mean squared error among all secondorder biascorrected estimators, up to the terms of the order n^{−2}. It is possible to continue this process, that is to derive the thirdorder biascorrection term, and so on. However as was shown by Kano (1996), the maximumlikelihood estimator is not thirdorder efficient.
Least Squares as Maximum Likelihood Estimator
Suppose that we are given a data set of n points (x_{i},y_{i}) for i=1,...,n and we are to estimate m parameters a_{j} for i=1,...,m. The model gives y(x) as a function of :
One can do the LeastSquares fit to minimize over . This can be justified using Bayesian probability as follows^{[8]}:
Suppose each data point has an error uniformly and randomly (iid) distributed with Normal distribution around the "actual" model y(x) and suppose that σ_{i} is the standard deviation of the error at point x_{i}. Then the probability of the dataset is the product of probabilities at each point :
One can then invoke Bayes' theorem and get,
Where, is the prior probability distribution over all the models. This is often taken as constant (noninformative prior). One can then seek to maximize P(model  data) or minimize the negative logarithm of the same which is equivalent to minimizing the least squares sum.
Examples
Discrete uniform distribution
Main article: German tank problemConsider a case where n tickets numbered from 1 to n are placed in a box and one is selected at random (see uniform distribution); thus, the sample size is 1. If n is unknown, then the maximumlikelihood estimator of n is the number m on the drawn ticket. (The likelihood is 0 for n < m, 1/n for n ≥ m, and this is greatest when n = m. Note that the maximum likelihood estimate of n occurs at the lower extreme of possible values {m, m + 1, ...}, rather than somewhere in the "middle" of the range of possible values, which would result in less bias.) The expected value of the number m on the drawn ticket, and therefore the expected value of , is (n + 1)/2. As a result, the maximum likelihood estimator for n will systematically underestimate n by (n − 1)/2 with a sample size of 1.
Discrete distribution, finite parameter space
Suppose one wishes to determine just how biased an unfair coin is. Call the probability of tossing a HEAD p. The goal then becomes to determine p.
Suppose the coin is tossed 80 times: i.e., the sample might be something like x_{1} = H, x_{2} = T, ..., x_{80} = T, and the count of the number of HEADS "H" is observed.
The probability of tossing TAILS is 1 − p (so here p is θ above). Suppose the outcome is 49 HEADS and 31 TAILS, and suppose the coin was taken from a box containing three coins: one which gives HEADS with probability p = 1/3, one which gives HEADS with probability p = 1/2 and another which gives HEADS with probability p = 2/3. The coins have lost their labels, so which one it was is unknown. Using maximum likelihood estimation the coin that has the largest likelihood can be found, given the data that were observed. By using the probability mass function of the binomial distribution with sample size equal to 80, number successes equal to 49 but different values of p (the "probability of success"), the likelihood function (defined below) takes one of three values:
The likelihood is maximized when p = 2/3, and so this is the maximum likelihood estimate for p.
Discrete distribution, continuous parameter space
Now suppose that there was only one coin but its p could have been any value 0 ≤ p ≤ 1. The likelihood function to be maximised is
and the maximisation is over all possible values 0 ≤ p ≤ 1.
One way to maximize this function is by differentiating with respect to p and setting to zero:
which has solutions p = 0, p = 1, and p = 49/80. The solution which maximizes the likelihood is clearly p = 49/80 (since p = 0 and p = 1 result in a likelihood of zero). Thus the maximum likelihood estimator for p is 49/80.
This result is easily generalized by substituting a letter such as t in the place of 49 to represent the observed number of 'successes' of our Bernoulli trials, and a letter such as n in the place of 80 to represent the number of Bernoulli trials. Exactly the same calculation yields the maximum likelihood estimator t / n for any sequence of n Bernoulli trials resulting in t 'successes'.
Continuous distribution, continuous parameter space
For the normal distribution which has probability density function
the corresponding probability density function for a sample of n independent identically distributed normal random variables (the likelihood) is
or more conveniently:
where is the sample mean.
This family of distributions has two parameters: θ = (μ, σ), so we maximize the likelihood, , over both parameters simultaneously, or if possible, individually.
Since the logarithm is a continuous strictly increasing function over the range of the likelihood, the values which maximize the likelihood will also maximize its logarithm. Since maximizing the logarithm often requires simpler algebra, it is the logarithm which is maximized below. (Note: the loglikelihood is closely related to information entropy and Fisher information.)
which is solved by
This is indeed the maximum of the function since it is the only turning point in μ and the second derivative is strictly less than zero. Its expectation value is equal to the parameter μ of the given distribution,
which means that the maximumlikelihood estimator is unbiased.
Similarly we differentiate the log likelihood with respect to σ and equate to zero:
which is solved by
Inserting we obtain
To calculate its expected value, it is convenient to rewrite the expression in terms of zeromean random variables (statistical error) . Expressing the estimate in these variables yields
Simplifying the expression above, utilizing the facts that and , allows us to obtain
This means that the estimator is biased. However, is consistent.
Formally we say that the maximum likelihood estimator for θ = (μ,σ^{2}) is:
In this case the MLEs could be obtained individually. In general this may not be the case, and the MLEs would have to be obtained simultaneously.
Nonindependent variables
It may be the case that variables are correlated, that is, not independent. Two random variables X and Y are independent only if their joint probability density function is the product of the individual probability density functions, i.e.
Suppose one constructs an ordern Gaussian vector out of random variables , where each variable has means given by . Furthermore, let the covariance matrix be denoted by Σ,
The joint probability density function of these n random variables is then given by:
In the two variable case, the joint probability density function is given by:
In this and other cases where a joint density function exists, the likelihood function is defined as above, under Principles, using this density.
Applications
Maximum likelihood estimation is used for a wide range of statistical models, including:
 linear models and generalized linear models;
 exploratory and confirmatory factor analysis;
 structural equation modeling;
 many situations in the context of hypothesis testing and confidence interval formation.
 discrete choice models
These uses arise across applications in widespead set of fields, including:
 communication systems;
 psychometrics;
 econometrics;
 timedelay of arrival (TDOA) in acoustic or electromagnetic detection;
 data modeling in nuclear and particle physics;
 magnetic resonance imaging
 computational phylogenetics;
 origin/destination and pathchoice modeling in transport networks.
History
Maximumlikelihood estimation was recommended, analyzed (with flawed attempts at proofs) and vastly popularized by R. A. Fisher between 1912 and 1922^{[9]} (although it had been used earlier by Gauss, Laplace, Thiele, and F. Y. Edgeworth).^{[10]} Reviews of the development of maximum likelihood have been provided by a number of authors.^{[11]}
Much of the theory of maximumlikelihood estimation was first developed for Bayesian statistics, and then simplified by later authors.^{[12]}
See also
 Other estimation methods
 Restricted maximum likelihood, a variation using a likelihood function calculated from a transformed set of data.
 Quasimaximum likelihood estimator, a MLE estimator that is misspecified, but still consistent.
 Maximum a posteriori (MAP) estimator, for a contrast in the way to calculate estimators when prior knowledge is postulated.
 Method of support, a variation of the maximum likelihood technique.
 Mestimator, an approach used in robust statistics.
 Method of moments (statistics), another popular method for finding parameters of distributions.
 Generalized method of moments are methods related to the likelihood equation in maximum likelihood estimation.
 Minimum distance estimation
 Maximum spacing estimation, a related method that is more robust in many situations.
 Related concepts:
 Fisher information, information matrix, its relationship to covariance matrix of ML estimates
 Likelihood function, a description on what likelihood functions are.
 Mean squared error, a measure of how 'good' an estimator of a distributional parameter is (be it the maximum likelihood estimator or some other estimator).
 Extremum estimator, a more general class of estimators to which MLE belongs.
 The Rao–Blackwell theorem, a result which yields a process for finding the best possible unbiased estimator (in the sense of having minimal mean squared error). The MLE is often a good starting place for the process.
 Sufficient statistic, a function of the data through which the MLE (if it exists and is unique) will depend on the data.
 The BHHH algorithm is a nonlinear optimization algorithm that is popular for Maximum Likelihood estimations.
Notes
 ^ Ruppert, David (2010). Statistics and Data Analysis for Financial Engineering. Springer. p. 98. ISBN 9781441977861. http://books.google.com/books?id=i2bD50PbIikC&pg=PA98.
 ^ Wooldridge, Jeffrey (2009). Introductory econometrics: a modern approach. Cengage Learning. p. 761. ISBN 9780324660548. http://books.google.com/books?id=64vt5TDBNLwC&pg=PA761.
 ^ Pfanzagl (1994, p. 206)
 ^ Newey & McFadden (1994, Theorem 2.5.)
 ^ Lehmann and Casella.
 ^ Newey & McFadden (1994, Theorem 3.3.)
 ^ Cox & Snell (1968, formula (20))
 ^ Numerical Recipes: The Art of Scientific Computing, Third Edition in C++. 2007. pp. 773840.
 ^ Pfanzagl (1994)
 ^ Edgeworth (September 1908, December 1908)
 ^ Savage (1976), Pratt (1976), Stigler (1978, 1986, 1999), Hald (1998, 1999), Aldrich (1997).
 ^ Pfanzagl, Johann (1994). with the assistance of R. Hamböker. ed. Parametric Statistical Theory. Berlin: Walter de Gruyter. pp. 207–208. ISBN 3110138638.
References
 Aldrich, John (1997). "R. A. Fisher and the making of maximum likelihood 1912–1922". Statistical Science 12 (3): 162–176. doi:10.1214/ss/1030037906. MR1617519.
 Anderson, Erling B. 1970. "Asymptotic Properties of Conditional Maximum Likelihood Estimators". Journal of the Royal Statistical Society B 32, 283–301.
 Andersen, Erling B. 1980. Discrete Statistical Models with Social Science Applications. North Holland, 1980.
 Debabrata Basu. Statistical Information and Likelihood : A Collection of Critical Essays by Dr. D. Basu ; J.K. Ghosh, editor. Lecture Notes in Statistics Volume 45, SpringerVerlag, 1988.
 Cox, D. R.; Snell, E. J. (1968). "A general definition of residuals". Journal of the Royal Statistical Society. Series B (Methodological): 248–275. JSTOR 2984505.
 Edgeworth, F.Y. (Sep 1908). "On the probable errors of frequencyconstants". Journal of the Royal Statistical Society 71 (3): 499–512. doi:10.2307/2339293. JSTOR 2339293.
 Edgeworth, F.Y. (Dec 1908). "On the probable errors of frequencyconstants". Journal of the Royal Statistical Society 71 (4): 651–678. doi:10.2307/2339378. JSTOR 2339378.
 Ferguson, Thomas S. (1982). "An inconsistent maximum likelihood estimate". Journal of the American Statistical Association 77 (380): 831–834. JSTOR 2287314.
 Ferguson, Thomas S (1996). A course in large sample theory. Chapman & Hall. ISBN 0412043718.
 Hald, Anders (1998). A history of mathematical statistics from 1750 to 1930. New York: Wiley. ISBN 0471179124.
 Hald, Anders (1999). "On the history of maximum likelihood in relation to inverse probability and least squares". Statistical Science 14 (2): 214–222. JSTOR 2676741.
 Kano, Y. (1996). "Thirdorder efficiency implies fourthorder efficiency". Journal of the Japan Statistical Society 26: 101–117. http://www.journalarchive.jst.go.jp/english/jnlabstract_en.php?cdjournal=jjss1995&cdvol=26&noissue=1&startpage=101.
 Le Cam, Lucien (1990). "Maximum likelihood — an introduction". ISI Review 58 (2): 153–171.
 Le Cam, Lucien; Lo Yang, Grace (2000). Asymptotics in statistics: some basic concepts (Second ed.). Springer. ISBN 0387950362.
 Le Cam, Lucien (1986). Asymptotic methods in statistical decision theory. SpringerVerlag. ISBN 0387963073.
 Lehmann, E.L.; Casella, G. (1998). Theory of Point Estimation, 2nd ed. Springer. ISBN 0387985026.
 Myung, Jay I.; Navarro, Daniel J. (2004). "Information Matrix", Department of Psychology, Ohio State University, http://faculty.psy.ohiostate.edu/myung/personal/info_matrix.pdf
 Newey, Whitney K.; McFadden, Daniel (1994). "Chapter 35: Large sample estimation and hypothesis testing". In Engle, Robert; McFadden, Dan. Handbook of Econometrics, Vol.4. Elsevier Science. pp. 2111–2245. ISBN 0444887660.
 Pratt, John W. (1976). "F. Y. Edgeworth and R. A. Fisher on the efficiency of maximum likelihood estimation". The Annals of Statistics 4 (3): 501–514. doi:10.1214/aos/1176343457. JSTOR 2958222.
 Savage, Leonard J. (1976). "On rereading R. A. Fisher". The Annals of Statistics 4 (3): 441–500. doi:10.1214/aos/1176343456. JSTOR 2958221.
 Stigler, Stephen M. (1978). "Francis Ysidro Edgeworth, statistician". Journal of the Royal Statistical Society. Series A (General) 141 (3): 287–322. doi:10.2307/2344804. JSTOR 2344804.
 Stigler, Stephen M. (1986). The history of statistics: the measurement of uncertainty before 1900. Harvard University Press. ISBN 0674403401.
 Stigler, Stephen M. (1999). Statistics on the table: the history of statistical concepts and methods. Harvard University Press. ISBN 0674836014.
 van der Vaart, A.W. (1998). Asymptotic Statistics. ISBN 0521784506.
 Numerical Recipes: The Art of Scientific Computing, Third Edition in C++. 2007.
 Nikulin, M. (2006). "About one problem of D.Bernoulli and L.Euler from the theory of statistical estimation". SORT, v.30, #1, 91100. ISSN: 16962281.
External links
 Maximum Likelihood Estimation Primer (an excellent tutorial)
 Implementing MLE for your own likelihood function using R
 A selection of likelihood functions in R
 Tutorial on maximum likelihood estimation in Journal of Mathematical Psychology
Categories: Estimation theory
 Statistical theory
 Mestimators
Wikimedia Foundation. 2010.