Shannon index

The Shannon index, also known as the Shannon-Weaver Index and sometimes referred to as the Shannon-Wiener Index [Krebs, Charles (1989) "Ecological Methodology." HarperCollins, New York.] ), $H^\{prime\}$, is one of several diversity indices used to measure diversity in categorical data. It is simply the Information entropy of the distribution, treating species as symbols and their relative population sizes as the probability.

This article treats its use in the measurement of biodiversity. The advantage of this index is that it takes into account the number of species and the evenness of the species. The index is increased either by having additional unique species, or by having a greater species evenness.

The "Shannon-Weaver" name is a misnomer; apparently some biologists jumped to the conclusion that Warren Weaver, author of an influential preface to the book form [cite book|last=Weaver|first=W.|coauthors=C.E. Shannon|title=The Mathematical Theory of Communication|publisher=University of Illinois|location=Urbana, Illinois|date=1949] of Claude Shannon's 1948 paper [cite journal|last=Shannon|first=C.E.|title=A mathematical theory of communication|journal=Bell System Technical Journal|volume=27|month=July and October|pages=379–423 and 623–656|year=1948] founding information theory, was a cofounder of this theory. Weaver did play a crucial role in the rapid postwar development of information theory in a different way, however; as an influential early administrator of the Rockefeller Foundation, he ensured that the first information theorists received generous research grants. Norbert Wiener had no hand in the index either, although his influential popularisation of cybernetics was often conflated with information theory in the 1950s.

Definitions

*$n\_i$ The number of individuals in species i; the abundance of species i.
*$S$ The number of species. Also called species richness.
*$N$ The total number of all individuals
*$p\_i$ The relative abundance of each species, calculated as the proportion of individuals of a given species to the total number of individuals in the community: $n\_iover\; N$

Computing the index

:$H^prime\; =\; -sum\_\{i=1\}^S\; p\_i\; ln\; p\_i$

It can be shown that for any given number of species, there is a maximum possible $H^prime$, $H\_max=ln\; S$ which occurs when all species are present in equal numbers.

Proof that maximum evenness maximizes the index

The following will prove that any given population will have a maximum Shannon Index if and only if each species represented is composed of the same number of individuals.

Expanding the index:

:$H^prime\; =\; -sum\_\{i=1\}^S\; \{n\_iover\; N\}\; ln\; \{n\_iover\; N\}$

:$N\; H^prime\; =\; -sum\_\{i=1\}^S\; n\_i\; left\; (\; ln\; n\_i\; -\; ln\; N\; ight\; )=\; -sum\_\{i=1\}^S\; n\_i\; ln\; n\_i\; +\; ln\; N\; sum\_\{i=1\}^S\; n\_i$

:$N\; H^prime\; -\; N\; ln\; N\; =\; -sum\_\{i=1\}^S\; n\_i\; ln\; n\_i$

Now, let's define $H\_s\; =\; -sum\_\{i=1\}^S\; n\_i\; ln\; n\_i$ Clearly, since $N$ is a positive constant for a given population size, and $Nln\; N$ is also a constant, then maximizing $H\_s$ is equivalent to maximizing $H^prime$.

trategy

Let's split an arbitrarily sized population into two groups, with each group receiving an arbitrary number of individuals and an arbitrary number of species. Now, within each group, each species has the same number of individuals as any other species in that group, but the number of individuals per species in the first group may be different from the number of individuals per species in the second group.

Now, if it can be proven that $H\_s$ is maximized when the number of individuals per species in the first group matches the number of individuals per species in the second group, then it has been proved that the population has a maximum index only when each species in the population is evenly represented. $H\_s$ doesn't depend on the total population. So $H\_s$ may be built by simply adding the indices of two sub-populations. Since the population size is arbitrary, this proves that if you have two species (the smallest number that can be considered two groups), their index is maximized if they are present in equal numbers. So the rules of mathematical induction have been satisfied.

Proof

Now, divide the species into two groups. Within each group, the population is evenly distributed among the species present.
*$k$ The number of individuals in the second group.
*$p$ The number of species in the second group.
*$n\_\{i2\}\; =\; k/p$ Number of individuals in each species in the second group.
*$N-k$ The number of individuals in the first group.
*$S-p$ The species in the first group.
*$n\_\{i1\}\; =\; \{N-k\; over\; S-p\}$ The individuals in each species in the first group.

:$H\_s\; =\; -sum\_\{i=1\}^\{S-p\}\; \{N-k\; over\; S-p\}\; ln\; \{N-k\; over\; S-p\}\; -\; sum\_\{i=1\}^p\; \{kover\; p\}\; ln\; \{k\; over\; p\}\; =\; -left\; (\; N-k\; ight\; )\; ln\; \{N-k\; over\; S-p\}\; -\; k\; ln\; \{kover\; p\}.$

To find out which value of $k$ will maximize $H\_s$, we must find the value of $k$ which satisfies the equation:

:$\{dover\; dk\},\; H\_s=0.$

Differentiating,

:$ln\; \{\; N-k\; over\; S-p\}\; +\; (N-k)\{1\; over\; N-k\}\; -\; ln\; \{kover\; p\}\; -\; k\; \{1\; over\; k\}\; =\; 0,$

:$ln\; \{N-kover\; S-p\}\; =\; ln\; \{k\; over\; p\}$

Exponentiating:

:$\{N-kover\; S-p\}\; =\; \{k\; over\; p\}\; =\; \{pN\; over\; S\}.$

Now by applying the definitions of $N\_\{i1\}$ and $N\_\{i2\}$, we get

:$N\_\{i1\}\; =\; N\_\{i2\}\; =\; \{Nover\; S\}.$

Result

Now we have accomplished the proof that the Shannon index is maximized when each species is present in equal numbers (see #strategy). But what is the index in that case? Well, $n\_i\; =\; \{Nover\; S\}$, so $p\_i\; =\; \{1over\; S\}$ Therefore:

:$H\_max\; =\; -\; sum\_\{i=1\}^S\; \{1over\; S\}\; ln\; \{1over\; S\}\; =\; ln\; S.$

Notes

ee also

*biodiversity
*Species richness
*Information theory