Cochran's theorem

In statistics, Cochran's theorem, devised by William G. Cochran,^[1] is a theorem used in to justify results relating to the probability distributions of statistics that are used in the analysis of variance.^[2]

Statement

Suppose U₁, ..., U_n are independent standard normally distributed random variables, and an identity of the form

$\sum_{i=1}^n U_i^2=Q_1+\cdots + Q_k$

can be written, where each Q_i is a sum of squares of linear combinations of the Us. Further suppose that

$r_1+\cdots +r_k=n$

where r_i is the rank of Q_i. Cochran's theorem states that the Q_i are independent, and each Q_i has a chi-square distribution with r_i degrees of freedom.^{[citation needed]}

Here the rank of Q_i should be interpreted as meaning the rank of the matrix B⁽ⁱ⁾, with elements B_j,k⁽ⁱ⁾, in the representation of Q_i as a quadratic form:

$Q_i=\sum_{j=1}^n\sum_{k=1}^n U_j B_{j,k}^{(i)} U_k .$

Less formally, it is the number of linear combinations included in the sum of squares defining Q_i, provided that these linear combinations are linearly independent.

Examples

Sample mean and sample variance

If X₁, ..., X_n are independent normally distributed random variables with mean μ and standard deviation σ then

$U_i = \frac{X_i-\mu}{\sigma}$

is standard normal for each i. It is possible to write

$\sum U_i^2=\sum\left(\frac{X_i-\overline{X}}{\sigma}\right)^2 + n\left(\frac{\overline{X}-\mu}{\sigma}\right)^2$

(here, summation is from 1 to n, that is over the observations). To see this identity, multiply throughout by σ² and note that

$\sum(X_i-\mu)^2= \sum(X_i-\overline{X}+\overline{X}-\mu)^2$

and expand to give

$\sum(X_i-\mu)^2= \sum(X_i-\overline{X})^2+\sum(\overline{X}-\mu)^2+ 2\sum(X_i-\overline{X})(\overline{X}-\mu).$

The third term is zero because it is equal to a constant times

$\sum(\overline{X}-X_i)=0,$

and the second term has just n identical terms added together. Thus

$\sum(X_i-\mu)^2= \sum(X_i-\overline{X})^2+n(\overline{X}-\mu)^2 ,$

and hence

$\sum\left(\frac{X_i-\mu}{\sigma}\right)^2= \sum\left(\frac{X_i-\overline{X}}{\sigma}\right)^2 +n\left(\frac{\overline{X}-\mu}{\sigma}\right)^2 =Q_1+Q_2.$

Now the rank of Q₂ is just 1 (it is the square of just one linear combination of the standard normal variables). The rank of Q₁ can be shown to be n − 1, and thus the conditions for Cochran's theorem are met.

Cochran's theorem then states that Q₁ and Q₂ are independent, with chi-squared distributions with n − 1 and 1 degree of freedom respectively. This shows that the sample mean and sample variance are independent. This can also be shown by Basu's theorem, and in fact this property characterizes the normal distribution – for no other distribution are the sample mean and sample variance independent.^{[citation needed]}

Distributions

The result for the distributions is written symbolically as

$n(\overline{X}-\mu)^2\sim \sigma^2 \chi^2_1,$

$\sum\left(X_i-\overline{X}\right)^2 \sim \sigma^2 \chi^2_{n-1}.$

Both these random variables are proportional to the true but unknown variance σ². Thus their ratio is does not depend on σ² and, because they are statistically independent, the distribution of their ratio is given by

$\frac{n\left(\overline{X}-\mu\right)^2} {\frac{1}{n-1}\sum\left(X_i-\overline{X}\right)^2}\sim \frac{\chi^2_1}{\frac{1}{n-1}\chi^2_{n-1}} \sim F_{1,n-1}$

where F_1,n − 1 is the F-distribution with 1 and n − 1 degrees of freedom (see also Student's t-distribution). The final step here is effectively the defintion of a random variable having the F-distribution.

Estimation of variance

To estimate the variance σ², one estimator that is sometimes used is the maximum likelihood estimator of the variance of a normal distribution

$\widehat{\sigma}^2= \frac{1}{n}\sum\left( X_i-\overline{X}\right)^2.$

Cochran's theorem shows that

$\frac{n\widehat{\sigma}^2}{\sigma^2}\sim\chi^2_{n-1}$

and the properties of the chi-square distribution show that the expected value of $\widehat{\sigma}^2$ is σ²(n − 1)/n.

Alternative formulation

The following version is often seen when considering linear regression.^{[citation needed]} Suppose that Y∼N_n(0,σ²I_n) is a standard multivariate normal random vector (here I_n denotes the n-by-n identity matrix), and if $A_1,\ldots,A_k$ are all n-by-n symmetric matrices with $\sum_{i=1}^kA_i=I_n$. Then, on defining r_i = Rank(A_i), any one of the following conditions implies the other two:

• $\sum_{i=1}^kr_i=n ,$
• $Y^TA_iY\sim\sigma^2\chi^2_{r_i}$ (thus the A_i are positive semidefinite)
• Y^TA_iY is independent of Y^TA_jY for $i\neq j .$

See also

• Cramér's theorem, on decomposing normal distribution
• Infinite divisibility (probability)

References

1. ^ Cochran, W. G. (April 1934). "The distribution of quadratic forms in a normal system, with applications to the analysis of covariance". Mathematical Proceedings of the Cambridge Philosophical Society 30 (2): 178–191. doi:10.1017/S0305004100016595. 
2. ^ Bapat, R. B. (2000). Linear Algebra and Linear Models (Second ed.). Springer. ISBN 9780387988719.