Scoring algorithm

In statistics, Fisher's Scoring algorithm is a form of Newton's method used to solve maximum likelihood equations numerically.

ketch of Derivation

Let $Y\_1,ldots,Y\_n$ be random variables, independent and identically distributed with twice differentiable p.d.f. $f(y;\; heta)$, and we wish to calculate the maximum likelihood estimator (M.L.E.) $heta^*$ of $heta$. First, suppose we have a starting point for our algorithm $heta\_0$, and consider a Taylor expansion of the score function, $V(\; heta)$, about $heta\_0$:

: $V(\; heta)\; approx\; V(\; heta\_0)\; -\; mathcal\{J\}(\; heta\_0)(\; heta\; -\; heta\_0)$,

where

: $mathcal\{J\}(\; heta\_0)\; =\; -\; sum\_\{i=1\}^n\; left.\; abla\; abla^\{\; op\}\; ight|\_\{\; heta=\; heta\_0\}\; log\; f(Y\_i\; ;\; heta)$

is the observed information matrix at $heta\_0$. Now, setting $heta\; =\; heta^*$, using that $V(\; heta^*)\; =\; 0$ and rearranging gives us:

: $heta^*\; =\; heta\_\{0\}\; +\; mathcal\{J\}^\{-1\}(\; heta\_\{0\})V(\; heta\_\{0\})$.

We therefore use the algorithm

: $heta\_\{m+1\}\; =\; heta\_\{m\}\; +\; mathcal\{J\}^\{-1\}(\; heta\_\{m\})V(\; heta\_\{m\})$,

and under certain regularity conditions, it can be shown that $heta\_m\; ightarrow\; heta^*$.

Fisher Scoring

In practice, $mathcal\{J\}(\; heta)$ is usually replaced by $mathcal\{I\}(\; heta)=\; mathrm\{E\}\; [J(\; heta)]$, the Fisher information, thus giving us the Fisher Scoring Algorithm:

: $heta\_\{m+1\}\; =\; heta\_\{m\}\; +\; mathcal\{I\}^\{-1\}(\; heta\_\{m\})V(\; heta\_\{m\})$.

References