M-estimator

M-estimator

In statistics, M-estimators are a broad class of statistics which are obtained as the solution to the problem of minimizing certain functions of the data. The process of obtaining an M-estimator is called M-estimation.

Some authors define M-estimators to be the root or roots of a system of equations consisting of certain functions of the data. This class is a subset of the class of minimization solutions. Typically these functions are the derivatives of the functions to be minimized in the broader definition.

Many classical statistics can be shown to be M-estimators. Their main utility, however, is as robust alternatives to classical statistical estimators.

Historical motivation

For a family of probability density functions "f" parameterized by "θ", the maximum likelihood estimate of "θ" (which could be vector valued) are computed by maximizing the likelihood function over "θ". The estimate is

:widehat{ heta} = operatorname{argmax}_{ heta}{ left( prod_{i=1}^n f(x_i, heta) ight) },!

or, equivalently,

:widehat{ heta} = operatorname{argmin}_{ heta}{ left( -sum_{i=1}^n log{( f(x_i, heta) ) } ight) }.,!

The performance of maximum likelihood estimators depends heavily on the assumed distribution family of the data being at least approximately true. In particular, maximum likelihood estimators can be inefficient and biased when the data are not from the assumed distribution. Of particular concern is the presence of outliers.

Definition

In 1964, Peter Huber proposed generalizing maximum likelihood estimation to the minimization of

:sum_{i=1}^n ho(x_i, heta),,!

where ρ is a function with certain properties (see below). The solutions

:hat{ heta} = operatorname{argmin}_{ heta}left(sum_{i=1}^n ho(x_i, heta) ight) ,!

are called M-estimators ("M" for "maximum likelihood-type" (Huber, 1981, page 43); other types of robust estimator include L-estimators, R-estimators and S-estimators). Maximum likelihood estimators are thus a special case of M-estimators.

The function ρ, or its derivative, ψ, can be chosen in such a way to provide the estimator desirable properties (in terms of bias and efficiency) when the data are truly from the assumed distribution, and 'not bad' behaviour when the data are generated from a model that is, in some sense, "close" to the assumed distribution.

Types of M-estimators

M-estimators are solutions, "θ", which minimize

:sum_{i=1}^n ho(x_i, heta).,!

This minimization can always be done directly. Often it is simpler to differentiate with respect to "θ" and solve for the root of the derivative. When this differentiation is possible, the M-estimator is said to be of ψ-type. Otherwise, the M-estimator is said to be of ρ-type.

In most practical cases, the M-estimators are of ψ-type.

ρ-type

For positive integer "r", let (mathcal{X},Sigma) and (Thetasubsetmathbb{R}^r,S) be measure spaces. hetainTheta is a vector of parameters. An M-estimator of ρ-type "T" is defined through a measurable function ho:mathcal{X} imesTheta ightarrowmathbb{R}. It maps a probability distribution "F" on mathcal{X} to the value T(F)inTheta (if it exists) that minimizesint_{mathcal{X ho(x, heta)dF(x):


T(F):=argmin_{ hetainTheta}int_{mathcal{X ho(x, heta)dF(x)

For example, for the maximum likelihood estimator, ho(x, heta)=-log(f(x, heta)), where f(x, heta)=frac{partial F(x, heta)}{partial x}.

ψ-type

If ho is differentiable, the computation of widehat{ heta} is usually much easier. An M-estimator of ψ-type "T" is defined through a measurable function psi:mathcal{X} imesTheta ightarrowmathbb{R}^r. It maps a probability distribution "F" on mathcal{X} to the value T(F)inTheta (if it exists) that solves the vector equation:int_{mathcal{Xpsi(x, heta)dF(x)=0
int_{mathcal{Xpsi(x,T(F))dF(x)=0

For example, for the maximum likelihood estimator, psi(x, heta)=left(frac{partiallog(f(x, heta))}{partial heta^1},cdots,frac{partiallog(f(x, heta))}{partial heta^p} ight)^t, where u^t denotes the transpose of vector "u" and f(x, heta)=frac{partial F(x, heta)}{partial x}.

Such an estimator is not necessarily an M-estimator of ρ-type, but if ρ has a continuous first derivative with respect to heta, then a necessary corresponding M-estimator of ψ-type to be an M-estimator of ρ-type is psi(x, heta)= abla_ heta ho(x, heta). The previous definitions can easily be extended to finite samples.

If the function ψ decreases to zero as x ightarrow pm infty, the estimator is called redescending. Such estimators have some additional desirable properties, such as complete rejection of gross outliers.

Computation

For many choices of ρ or ψ, no closed form solution exists and an iterative approach to computation is required. It is possible to use standard function optimization algorithms, such as Newton-Raphson. However, in most cases an iteratively re-weighted least squares fitting algorithm can be performed; this is typically the preferred method.

For some choices of ψ, specifically, "redescending" functions, the solution may not be unique. The issue is particularly relevant in multivariate and regression problems. Thus, some care is needed to ensure that good starting points are chosen. Robust starting points, such as the median as an estimate of location and the median absolute deviation as a univariate estimate of scale, are common.

Properties

Distribution

It can be shown that M-estimators are asymptotically normally distributed. As such, Wald-type approaches to constructing confidence intervals and hypothesis tests can be used. However, since the theory is asymptotic, it will frequently be sensible to check the distribution, perhaps by examining the permutation or bootstrap distribution.

Influence function

The influence function of an M-estimator of psi-type is proportional to its defining psi function.

Let "T" be an M-estimator of ψ-type, and "G" be a probability distribution for which T(G) is defined. Its influence function IF is :operatorname{IF}(x;T,G) = -frac{psi(x,T(G))} {intleft [frac{partialpsi(y, heta)} {partial heta} ight] mathrm{d}y }

A proof of this property of M-estimators can be found in Huber (1981, Section 3.2).

Applications

M-estimators can be constructed for location parameters and scale parameters in univariate and multivariate settings, as well as being used in robust regression.

Examples

Mean

Let ("X"1, ... , "X"n) be a set of independent, identically distributed random variables, with distribution "F".

If we define

: ho(x, heta)=frac{(x - heta)^2}{2},,!

we note that this is minimized when "θ" is the mean of the "X"s. Thus the mean is an M-estimator of ρ-type, with this ρ function.

As this ρ function is continuously differentiable in "θ", the mean is thus also an M-estimator of ψ-type for ψ("x", "θ") = "θ" - "x".

ee also

*Robust statistics
*Robust regression
*Redescending M-estimator

References

*

*

*

*


Wikimedia Foundation. 2010.

Игры ⚽ Нужно решить контрольную?

Look at other dictionaries:

  • estimator — es‧ti‧ma‧tor [ˈestmeɪtə ǁ ər] noun [countable] JOBS someone whose job is to calculate the probable cost, value, or price of something: • An estimator can call at the customer s home, take measurements and quote a price. * * * estimator UK US… …   Financial and business terms

  • estimator — estimatór s. m., pl. estimatóri Trimis de siveco, 10.08.2004. Sursa: Dicţionar ortografic  ESTIMATÓR estimatori m. Persoană care efectuează o estimare. /<it. estimatore Trimis de siveco, 22.08.2004. Sursa: NODEX  ESTIMATÓR, OÁRE s.m. şi f.… …   Dicționar Român

  • Estimator — Es ti*ma tor, n. [L. aestimator.] One who estimates or values; a valuer. Jer. Taylor …   The Collaborative International Dictionary of English

  • estimator — index juror Burton s Legal Thesaurus. William C. Burton. 2006 …   Law dictionary

  • estimator — 1660s, from L. aestimator, agent noun from aestimare (see ESTIMATE (Cf. estimate)) …   Etymology dictionary

  • Estimator — In statistics, an estimator is a function of the observable sample data that is used to estimate an unknown population parameter (which is called the estimand ); an estimate is the result from the actual application of the function to a… …   Wikipedia

  • estimator — įvertintuvas statusas T sritis automatika atitikmenys: angl. estimator vok. Schätzer, m; Schätzgerät, n rus. оцениватель, m pranc. estimateur, m …   Automatikos terminų žodynas

  • estimator error — įverčio paklaida statusas T sritis Standartizacija ir metrologija apibrėžtis Parametro įvertinimo paklaida, išreiškiama taip: (T – θ); čia T – įvertis, θ – tikroji parametro vertė. atitikmenys: angl. estimation error; estimator error vok.… …   Penkiakalbis aiškinamasis metrologijos terminų žodynas

  • estimator — įvertinys statusas T sritis Standartizacija ir metrologija apibrėžtis Statistiniai duomenys, vartojami tyrimų visumos parametrams įvertinti. atitikmenys: angl. estimator rus. оценочная функция, f pranc. estimateur, m …   Penkiakalbis aiškinamasis metrologijos terminų žodynas

  • estimator — įvertinys statusas T sritis fizika atitikmenys: angl. estimator vok. Schätzer, m; Schätzfunktion, f rus. оценочная функция, f pranc. estimateur, m …   Fizikos terminų žodynas

  • estimator — įverčio funkcija statusas T sritis fizika atitikmenys: angl. estimator vok. Schätzfunktion, f rus. оценочная функция, f pranc. estimateur, m …   Fizikos terminų žodynas

Share the article and excerpts

Direct link
Do a right-click on the link above
and select “Copy Link”