Binary entropy function

Binary entropy function

In information theory, the binary entropy function, denoted H(p) , or H_{mathrm b}(p) ,, is defined as the entropy of a Bernoulli trial with probability of success "p". Mathematically, the Bernoulli trial is modelled as a random variable "X" that can take on only two values: 0 and 1. The event X = 1 is considered a success and the event X = 0 is considered a failure. (These two events are mutually exclusive and exhaustive.)

If mathrm{Pr}(X=1) = p, then mathrm{Pr}(X=0) = 1-p and the entropy of "X" is given by

:H(X) = H_{mathrm b}(p) = -p log p - (1 - p) log (1 - p). ,

where 0 log 0 is taken to be 0. The logarithms in this formula are usually taken (as shown in the graph) to the base 2. See "binary logarithm".

When p=frac 1 2 , the binary entropy function attains its maximum value. This is the case of the unbiased bit, the most common unit of information entropy.

H(p) is distinguished from the entropy function by its taking a single scalar constant parameter. For tutorial purposes, in which the reader may not distinguish the appropriate function by its argument, H_2(p) is often used; however, this could confuse this function with the analogous function related to Rényi entropy, so H_mbox{b}(p) (with "b" not in italics) should be used to dispel ambiguity.

Derivative

The derivative of the binary entropy function may be expressed as the negative of the logit function:: {d over dp} H_{mathrm b}(p) = - operatorname{logit}(p) = -logleft( frac{p}{1-p} ight). ,

ee also

* Information theory
* Information entropy

References

* David J. C. MacKay. " [http://www.inference.phy.cam.ac.uk/mackay/itila/book.html Information Theory, Inference, and Learning Algorithms] " Cambridge: Cambridge University Press, 2003. ISBN 0-521-64298-1

External links


Wikimedia Foundation. 2010.

Игры ⚽ Поможем решить контрольную работу

Look at other dictionaries:

  • Entropy (information theory) — In information theory, entropy is a measure of the uncertainty associated with a random variable. The term by itself in this context usually refers to the Shannon entropy, which quantifies, in the sense of an expected value, the information… …   Wikipedia

  • Binary symmetric channel — A binary symmetric channel (or BSC) is a common communications channel model used in coding theory and information theory. In this model, a transmitter wishes to send a bit (a zero or a one), and the receiver receives a bit. It is assumed that… …   Wikipedia

  • Binary logarithm — NOTOC In mathematics, the binary logarithm (log2 n ) is the logarithm for base 2. It is the inverse function of n mapsto 2^n. The binary logarithm is often used in computer science and information theory (where it is frequently written lg n , or… …   Wikipedia

  • Information theory — Not to be confused with Information science. Information theory is a branch of applied mathematics and electrical engineering involving the quantification of information. Information theory was developed by Claude E. Shannon to find fundamental… …   Wikipedia

  • Quantities of information — A simple information diagram illustrating the relationships among some of Shannon s basic quantities of information. The mathematical theory of information is based on probability theory and statistics, and measures information with several… …   Wikipedia

  • List of mathematics articles (B) — NOTOC B B spline B* algebra B* search algorithm B,C,K,W system BA model Ba space Babuška Lax Milgram theorem Baby Monster group Baby step giant step Babylonian mathematics Babylonian numerals Bach tensor Bach s algorithm Bachmann–Howard ordinal… …   Wikipedia

  • Logit — The logit function is an important part of logistic regression: for more information, please see that article. The logit function is the inverse of the sigmoid , or logistic function used in mathematics, especially in statistics. The logit of a… …   Wikipedia

  • Noisy-channel coding theorem — In information theory, the noisy channel coding theorem (sometimes Shannon s theorem), establishes that for any given degree of noise contamination of a communication channel, it is possible to communicate discrete data (digital information)… …   Wikipedia

  • Z-channel (information theory) — In information theory, a Z channel with crossover probability p is a binary input binary output channel that flips the input bit 0 with probability p , but maps input bit 1 to 1 with probability 1. The Z channel has a capacity of : log 2left(1+(1 …   Wikipedia

  • Temperature — This article is about the thermodynamic property. For other uses, see Temperature (disambiguation). A map of global long term monthly average surface air temperatures i …   Wikipedia

Share the article and excerpts

Direct link
Do a right-click on the link above
and select “Copy Link”