Proof of the law of large numbers

Given "X"₁, "X"₂, ... an infinite sequence of i.i.d. random variables with finite expected value "E(X"₁")" = "E(X"₂")" = ... = µ < ∞, we are interested in the convergence of the sample average

:$overline\{X\}\_n=\; frac1n(X\_1+cdots+X\_n).$

__TOC__

The weak law

Theorem: $overline\{X\}\_n\; ,\; xrightarrow\{P\}\; ,\; mu\; qquad\; extrm\{for\}qquad\; n\; o\; infty.$

Proof using Chebyshev's inequality

This proof uses the assumption of finite variance $operatorname\{Var\}\; (X\_i)=sigma^2$ (for all $i$). The independence of the random variables implies no correlation between them, and we have that

:$operatorname\{Var\}(overline\{X\}\_n)\; =\; frac\{nsigma^2\}\{n^2\}\; =\; frac\{sigma^2\}\{n\}.$

The common mean μ of the sequence is the mean of the sample average:

:$E(overline\{X\}\_n)\; =\; mu.$

Using Chebyshev's inequality on $overline\{X\}\_n$ results in

:$operatorname\{P\}(\; left|\; overline\{X\}\_n-mu\; ight|\; geq\; varepsilon)\; leq\; frac\{sigma^2\}\{nvarepsilon^2\}.$

This may be used to obtain the following:

:

As "n" approaches infinity, the expression approaches 1. And by definition of convergence in probability (see Convergence of random variables), we have obtained

:$overline\{X\}\_n\; ,\; xrightarrow\{P\}\; ,\; mu\; qquad\; extrm\{for\}qquad\; n\; o\; infty.$

Proof using convergence of characteristic functions

By Taylor's theorem for complex functions, the characteristic function of any random variable, "X", with finite mean μ, can be written as

:$varphi\_X(t)\; =\; 1\; +\; itmu\; +\; o(t),\; quad\; t\; ightarrow\; 0.$

All "X"₁, "X"₂, ... have the same characteristic function, so we will simply denote this "φ"_"X".

Among the basic properties of characteristic functions there are

:$varphi\_\{frac\; 1\; n\; X\}(t)=\; varphi\_X(\; frac\; t\; n)\; quad\; extrm\{and\}\; quad\; varphi\_\{X+Y\}(t)=varphi\_X(t)\; varphi\_Y(t)\; quad\; extrm\{if,\}X,\; extrm\{and\},\; Y,\; extrm\{are,,independent\}.$

These rules can be used to calculate the characteristic function of $scriptstyleoverline\{X\}\_n$ in terms of "φ"_"X":

:$varphi\_\{overline\{X\}\_n\}(t)=\; left\; [varphi\_Xleft(\{t\; over\; n\}\; ight)\; ight]\; ^n\; =\; left\; [1\; +\; imu\{t\; over\; n\}\; +\; oleft(\{t\; over\; n\}\; ight)\; ight]\; ^n\; ,\; ightarrow\; ,\; e^\{itmu\},\; quad\; extrm\{as\}\; quad\; n\; ightarrow\; infty.$

The limit "e"^"it"μ is the characteristic function of the constant random variable μ, and hence by the Lévy continuity theorem, $scriptstyleoverline\{X\}\_n$ converges in distribution to μ:

:$overline\{X\}\_n\; ,\; xrightarrow\{mathcal\; D\}\; ,\; mu\; qquad\; extrm\{for\}qquad\; n\; o\; infty.$

μ is a constant, which implies that convergence in distribution to μ and convergence in probability to μ are equivalent. (See Convergence of random variables) This implies that

:$overline\{X\}\_n\; ,\; xrightarrow\{P\}\; ,\; mu\; qquad\; extrm\{for\}qquad\; n\; o\; infty.$

This proof states, in fact, that the sample mean converges in probability to the derivative of the characteristic function at the origin, as long as this exists.