Binary entropy function

In information theory, the binary entropy function, denoted $H(p)\; ,$ or $H\_\{mathrm\; b\}(p)\; ,$, is defined as the entropy of a Bernoulli trial with probability of success "p". Mathematically, the Bernoulli trial is modelled as a random variable "X" that can take on only two values: 0 and 1. The event $X\; =\; 1$ is considered a success and the event $X\; =\; 0$ is considered a failure. (These two events are mutually exclusive and exhaustive.)

If $mathrm\{Pr\}(X=1)\; =\; p,$ then $mathrm\{Pr\}(X=0)\; =\; 1-p$ and the entropy of "X" is given by

:$H(X)\; =\; H\_\{mathrm\; b\}(p)\; =\; -p\; log\; p\; -\; (1\; -\; p)\; log\; (1\; -\; p).\; ,$

where $0\; log\; 0$ is taken to be 0. The logarithms in this formula are usually taken (as shown in the graph) to the base 2. See "binary logarithm".

When $p=frac\; 1\; 2\; ,$ the binary entropy function attains its maximum value. This is the case of the unbiased bit, the most common unit of information entropy.

$H(p)$ is distinguished from the entropy function by its taking a single scalar constant parameter. For tutorial purposes, in which the reader may not distinguish the appropriate function by its argument, $H\_2(p)$ is often used; however, this could confuse this function with the analogous function related to Rényi entropy, so $H\_mbox\{b\}(p)$ (with "b" not in italics) should be used to dispel ambiguity.

Derivative

The derivative of the binary entropy function may be expressed as the negative of the logit function::$\{d\; over\; dp\}\; H\_\{mathrm\; b\}(p)\; =\; -\; operatorname\{logit\}(p)\; =\; -logleft(\; frac\{p\}\{1-p\}\; ight).\; ,$

ee also

* Information theory
* Information entropy

References

* David J. C. MacKay. " [http://www.inference.phy.cam.ac.uk/mackay/itila/book.html Information Theory, Inference, and Learning Algorithms] " Cambridge: Cambridge University Press, 2003. ISBN 0-521-64298-1

External links