Proof of Stein's example

Stein's example is an important result in decision theory which can be stated as

: "The ordinary decision rule for estimating the mean of a multivariate Gaussian distribution is inadmissible under mean squared error risk in dimesion at least 3".

The following is an outline of its proof. The reader is referred to the main article for more information.

Sketched proof

The risk function of the decision rule $d(mathbf\{x\})\; =\; mathbf\{x\}$ is

:$R(\; heta,d)\; =\; mathbb\{E\}\_\; heta\; [\; |mathbf\{\; heta\; -\; x\}|^2]$

::$=int\; (mathbf\{\; heta\; -\; x\})^T(mathbf\{\; heta\; -\; x\})\; left(\; frac\{1\}\{2pi\}\; ight)^\{n/2\}\; e^\{(-1/2)\; (mathbf\{\; heta\; -\; x\})^T\; (mathbf\{\; heta\; -\; x\})\; \}\; m(dx)$

::$=\; n.,$

Now consider the decision rule

:$d\text{'}(mathbf\{x\})\; =\; mathbf\{x\}\; -\; frac\{alpha\}\{|mathbf\{x\}|^2\}mathbf\{x\}$

where $alpha\; =\; n-2$. We will show that $d\text{'}$ is a better decision rule than $d$. The risk function is

:$R(\; heta,d\text{'})\; =\; mathbb\{E\}\_\; hetaleft\; [\; left|mathbf\{\; heta\; -\; x\}\; +\; frac\{alpha\}\{|mathbf\{x\}|^2\}mathbf\{x\}\; ight|^2\; ight]$

::$=\; mathbb\{E\}\_\; hetaleft\; [\; |mathbf\{\; heta\; -\; x\}|^2\; +\; 2(mathbf\{\; heta\; -\; x\})^Tfrac\{alpha\}\{|mathbf\{x\}|^2\}mathbf\{x\}\; +\; frac\{alpha^2\}\{|mathbf\{x\}|^4\}|mathbf\{x\}|^2\; ight]$

::$=\; mathbb\{E\}\_\; hetaleft\; [\; |mathbf\{\; heta\; -\; x\}|^2\; ight]\; +\; 2alphamathbb\{E\}\_\; hetaleft\; [frac\{mathbf\{(\; heta-x)^T\; x\{|mathbf\{x\}|^2\}\; ight]\; +\; alpha^2mathbb\{E\}\_\; hetaleft\; [frac\{1\}\{|mathbf\{x\}|^2\}\; ight]$

— a quadratic in $alpha$. We may simplify the middle term by considering a general sufficiently well behaved function $h:mathbf\{X\}\; mapsto\; h(mathbf\{X\})\; in\; mathbb\{R\}$ and using integration by parts. For any such $h$, for all $1leq\; i\; leq\; n$:

::$mathbb\{E\}\_\; heta\; [\; (\; heta\_i\; -\; mathbf\{X\}\_i)\; h(mathbf\{X\})\; ]\; =\; int\; (\; heta\_i\; -\; mathbf\{X\}\_i)\; h(mathbf\{X\})\; left(\; frac\{1\}\{2pi\}\; ight)^\{n/2\}\; e^\{\; -(1/2)(mathbf\{x-\; heta\})^T\; mathbf\{(x-\; heta)\}\; \}\; m(dx\_i)$

:$=\; left\; [\; h(mathbf\{X\})\; left(\; frac\{1\}\{2pi\}\; ight)^\{n/2\}\; e^\{-(1/2)\; (mathbf\{X\}-\; heta)^T\; (mathbf\{X\}-\; heta)\; \}\; ight]\; ^infty\_\{-infty\}$

:$-\; int\; frac\{partial\; h\}\{partial\; mathbf\{x\}\_i\}\; left(\; frac\{1\}\{2pi\}\; ight)^\{n/2\}\; e^\{-(1/2)mathbf\{(x-\; heta)\}^T\; mathbf\{(x-\; heta)\}\; \}\; m(dx\_i)$

:$=\; -\; mathbb\{E\}\_\; heta\; left\; [\; frac\{partial\; h\}\{partial\; mathbf\{x\}\_i\}\; ight]\; .$

(This result is known as Stein's lemma.)

Thus, if we set

:$h(mathbf\{X\})\; =\; frac\{mathbf\{X\}\_i\}\{|mathbf\{X\}|^2\}$

then assuming $h$ meets the "well behaved" condition (see end of proof), we have

:$frac\{partial\; h\}\{partial\; mathbf\{x\}\_i\}\; =\; frac\{1\}\{|mathbf\{X\}|^2\}\; -\; frac\{2mathbf\{X\}\_i^2\}\{|mathbf\{X\}|^4\}$

and so

::$mathbb\{E\}\_\; hetaleft\; [frac\{mathbf\{(\; heta-x)^T\; x\{|mathbf\{x\}|^2\}\; ight]\; =\; sum\_\{i=1\}^n\; mathbb\{E\}\_\; heta\; left\; [\; (\; heta\_i\; -\; mathbf\{X\}\_i)\; frac\{mathbf\{X\}\_i\}\{|mathbf\{X\}|^2\}\; ight]$

:$=\; -\; sum\_\{i=1\}^n\; mathbb\{E\}\_\; heta\; left\; [\; frac\{1\}\{|mathbf\{X\}|^2\}\; -\; frac\{2mathbf\{X\}\_i^2\}\{|mathbf\{X\}|^4\}\; ight]$

:$=\; -(n-2)mathbb\{E\}\_\; heta\; left\; [frac\{1\}\{|mathbf\{X\}|^2\}\; ight]\; .$

Then returning to the risk function of $d\text{'}$ :

:$R(\; heta,d\text{'})\; =\; n\; -\; 2alpha(n-2)mathbb\{E\}\_\; hetaleft\; [frac\{1\}\{|mathbf\{X\}|^2\}\; ight]\; +\; alpha^2mathbb\{E\}\_\; hetaleft\; [frac\{1\}\{|mathbf\{x\}|^2\}\; ight]\; .$

This quadratic in $alpha$ is minimized at

:$alpha\; =\; n-2,,$

giving

:$R(\; heta,d\text{'})\; =\; R(\; heta,d)\; -\; (n-2)^2mathbb\{E\}\_\; hetaleft\; [frac\{1\}\{|mathbf\{x\}|^2\}\; ight]$

which of course satisfies:

:

making $d$ an inadmissible decision rule.

It remains to justify the use of:$h(mathbf\{X\})=\; frac\{mathbf\{X\{|mathbf\{X\}|^2\}.$

This function is not in fact very "well behaved" since it is singular at $mathbf\{x\}=0$. However the function

:$h(mathbf\{X\})\; =\; frac\{mathbf\{X\{epsilon\; +\; |mathbf\{X\}|^2\}$

is "well behaved", and after following the algebra through and letting $epsilon\; o\; 0$ one obtains the same result.