# Error exponent

Error exponent

In information theory, the error exponent of a channel code or source code over the block length of the code is the logarithm of the error probability. For example, if the probability of error of a decoder drops as "e"–"n"α, where "n" is the block length, the error exponent is α. Many of the information-theoretic theorems are of asymptotic nature, for example, the channel coding theorem states that for any rate less than the channel capacity, the probability of the error of the channel code can made to go to zero as the block length goes to infinity. In practical situations, there are limitations to the delay of the communication and the block length of the code cannot be taken to go to infinity. Therefore it is important to study how the probability of error drops as the block length go to infinity.

Error exponent in channel coding

For time-invariant DMC channels

The channel coding theorem states that for any ε > 0 and for any rate less than the channel capacity, there is an encoding and decoding scheme that can be used to ensure that the probability of block error is less than ε > 0 for sufficiently long message block "X". Also, for any rate greater than the channel capacity, the probability of block error at the receiver goes to one as the block length goes to infinity.

Assuming a channel coding setup as follows: the channel can transmit any of $M = 2^\left\{nR\right\} ;$ messages, by transmitting the corresponding codeword (which is of length "n"). Each component in the codebook is drawn i.i.d. according to some probability distribution with probability mass function "Q". At the decoding end, ML decoding is done.

Given that $y_1^n$ is received, "X"(1) or first message is transmitted, the probability that "X"(1) is incorrectly detected as "X"(2) is:

:$P_\left\{mathrm\left\{error\right\} 1 o 2\right\} = sum_\left\{x_1^n\left(2\right)\right\} Q\left(x_1^n\left(2\right)\right)1\left(p\left(y_1^n|x_1^n\left(2\right)\right)>p\left(y_1^n|x_1^n\left(1\right)\right)\right)$

The function $1\left(p\left(y_1^n|x_1^n\left(2\right)\right)>p\left(y_1^n|x_1^n\left(1\right)\right)$ has upper bound

:$left\left(frac\left\{p\left(y_1^n|x_1^n\left(2\right)\right)\right\}\left\{p\left(y_1^n|x_1^n\left(1\right)\right)\right\} ight\right)^s$

for $s > 0 ;$ Thus,

:$P_\left\{mathrm\left\{error\right\} 1 o 2\right\} le sum_\left\{x_1^n\left(2\right)\right\} Q\left(x_1^n\left(2\right)\right) left\left(frac\left\{p\left(y_1^n|x_2^n\left(2\right)\right)\right\}\left\{p\left(y_1^n|x_1^n\left(1\right)\right)\right\} ight\right)^s.$

Since there are a total of "M" messages, the Probability that "X"(1) is confused with any other message is "M" times the above expression. Since each entry in the codebook is i.i.d., the notation of "X"(2) can be replaced simply by "X". Using the Hokey union bound, the probability of confusing "X"(1) with any message is bounded by:

:$P_\left\{mathrm\left\{error\right\} 1 o mathrm\left\{any le M^ ho sum_\left\{x_1^n\right\} Q\left(x_1^n\right) left\left(frac\left\{p\left(y_1^n|x_2^n\right)\right\}\left\{p\left(y_1^n|x_1^n\left(1\right)\right)\right\} ight\right)^\left\{s ho\right\}.$

Averaging over all combinations of $X_1^n\left(1\right), y_1^n$:

:$P_\left\{mathrm\left\{error\right\} 1 o mathrm\left\{any le M^ ho sum_\left\{y_1^n\right\} left\left(sum_\left\{x_1^n\left(1\right)\right\} Q\left(x_1^n\left(1\right)\right) \left[p\left(y_1^n|x_1^n\left(1\right)\right)\right] ^\left\{1-s ho\right\} ight\right) left\left(sum_\left\{x_1^n\right\} Q\left(x_1^n\right) \left[p\left(y_1^n|x_1^n\right)\right] ^s ight\right)^\left\{ ho\right\}.$

Choosing $s = 1-s ho$ and combining the two sums over $x_1^n$ in the above formula:

:$P_\left\{mathrm\left\{error\right\} 1 omathrm\left\{any le M^ ho sum_\left\{y_1^n\right\} left\left(sum_\left\{x_1^n\right\} Q\left(x_1^n\right) \left[p\left(y_1^n|x_1^n\right)\right] ^\left\{frac\left\{1\right\}\left\{1+ ho ight\right)^\left\{1+ ho\right\}.$

Using the independence nature of the elements of the codeword, and the discrete memoryless nature of the channel:

:$P_\left\{mathrm\left\{error\right\} 1 omathrm\left\{any le M^ ho prod_\left\{i=1\right\}^n sum_\left\{y_i\right\} left\left(sum_\left\{x_i\right\} Q_i\left(x_i\right) \left[p_i\left(y_i|x_i\right)\right] ^\left\{frac\left\{1\right\}\left\{1+ ho ight\right)^\left\{1+ ho\right\}$

Using the fact that each element of codeword is identically distributed and thus stationary:

:$P_\left\{mathrm\left\{error\right\} 1 omathrm\left\{any le M^ ho left\left(sum_\left\{y\right\} left\left(sum_\left\{x\right\} Q\left(x\right) \left[p\left(y|x\right)\right] ^\left\{frac\left\{1\right\}\left\{1+ ho ight\right)^\left\{1+ ho\right\} ight\right)^n.$

Replacing "M" by 2"nR" and defining

:$E_o\left( ho,Q\right)= -lnleft\left(sum_\left\{y\right\} left\left(sum_\left\{x\right\} Q\left(x\right) \left[p\left(y|x\right)\right] ^\left\{frac\left\{1\right\}\left\{1+ ho ight\right)^\left\{1+ ho\right\} ight\right),$

probability of error becomes

:$P_mathrm\left\{error\right\} le exp\left(-n\left(E_o\left( ho,Q\right) - ho R\right)\right).$

"Q" and $ho$ should be chosen so that the bound is tighest. Thus, the error exponent can be defined as

:$E_\left\{r\right\}\left(R\right) = max_Q max_\left\{ ho varepsilon \left[0,1\right] \right\} E_\left\{o\right\}\left( ho,Q\right) - ho R. ;$

For time variant DMC channels

Error exponent in source coding

For time invariant discrete memoryless sources

The source coding theorem states that for any $varepsilon > 0$ and any discrete-time i.i.d. source such as $X$ and for any rate less than the entropy of the source, there is large enough $n$ and an encoder that takes $n$ i.i.d. repetition of the source, $X^\left\{1:n\right\}$, and maps it to $n.\left(H\left(X\right)+varepsilon\right)$ binary bits such that the source symbols $X^\left\{1:n\right\}$ are recoverable from the binary bits with probability at least $1 - varepsilon$.

Let $M=e^\left\{nR\right\},!$ be the total number of possible messages. Next map each of the possible source output sequences to one of the messages randomly using a uniform distribution and independently from everything else. When a source is generated the corresponding message $M=m ,$ is then transmitted to the destination. The message gets decoded to one of the possible source strings. In order to minimize the probability of error the decoder will decode to the source sequence $X_1^n$ that maximizes $P\left(X_1^n|A_m\right)$, where $A_m,$ denotes the event that message $m$ was transmitted. This rule is equivalent to finding the source sequence $X_1^n$ among the set of source sequences that map to message $m$ that maximizes $P\left(X_1^n\right)$. This reduction follows from the fact that the messages were assigned randomly and independently of everything else.

Thus, as an example of when an error occurs, supposed that the source sequence $X_1^n\left(1\right)$ was mapped to message $1$ as was the source sequence $X_1^n\left(2\right)$. If $X_1^n\left(1\right) ,$ was generated at the source, but $P\left(X_1^n\left(2\right)\right) > P\left(X_1^n\left(1\right)\right)$ then an error occurs.

Let $S_i ,$ denote the event that the source sequence $X_1^n\left(i\right)$ was generated at the source, so that $P\left(S_i\right) = P\left(X_1^n\left(i\right)\right) ,.$ Then the probability of error can be broken down as $P\left(E\right) = sum_i P\left(E|S_i\right) P\left(S_i\right), .$ Thus, attention can be focused on finding an upper bound to the $P\left(E|S_i\right),$.

Let $A_\left\{i\text{'}\right\} ,$ denote the event that the source sequence $X_1^n\left(i\text{'}\right)$ was mapped to the same message as the source sequence $X_1^n\left(i\right)$ and that $P\left(X_1^n\left(i\text{'}\right)\right) geq P\left(X_1^n\left(i\right)\right)$. Thus, letting $X_\left\{i,i\text{'}\right\},$ denote the event that the two source sequences $i ,$ and $i\text{'} ,$ map to the same message, we have that

:

and using the fact that $P\left(X_\left\{i,i\text{'}\right\}\right) = frac\left\{1\right\}\left\{M\right\},$ and is idependent of everything else have that

:$P\left(A_\left\{i\text{'}\right\}\right) = frac\left\{1\right\}\left\{M\right\} P\left(P\left(X_1^n\left(i\text{'}\right)\right) geq P\left(X_1^n\left(i\right)\right)\right), .$

A simple upper bound for the term on the left can be established as

:$left \left[ P\left(P\left(X_1^n\left(i\text{'}\right)\right) geq P\left(X_1^n\left(i\right)\right)\right) ight \right] leq left \left( frac\left\{P\left(X_1^n\left(i\text{'}\right)\right)\right\}\left\{P\left(X_1^n\left(i\right)\right)\right\} ight \right) ^s ,$

for some arbitrary real number $s>0 ,.$ This upper bound can be verified by noting that $P\left(P\left(X_1^n\left(i\text{'}\right)\right) > P\left(X_1^n\left(i\right)\right)\right) ,$ either equals $1 ,$ or $0 ,$ because the probabilities of a given input sequence are completely deterministic. Thus, if $P\left(X_1^n\left(i\text{'}\right)\right) geq P\left(X_1^n\left(i\right)\right) , ,$ then $frac\left\{P\left(X_1^n\left(i\text{'}\right)\right)\right\}\left\{P\left(X_1^n\left(i\right)\right)\right\} geq 1 ,$ so that the inequality holds in that case. The inequality holds in the other case as well because

:$left \left( frac\left\{P\left(X_1^n\left(i\text{'}\right)\right)\right\}\left\{P\left(X_1^n\left(i\right)\right)\right\} ight \right) ^s geq 0 ,$

for all possible source strings. Thus, combining everything and introducing some $ho in \left[0,1\right] ,$, have that

:

Where the inequalities follow from a variation on the Union Bound. Finally applying this upper bound to the summation for $P\left(E\right) ,$ have that:

:$P\left(E\right) = sum_i P\left(E|S_i\right) P\left(S_i\right) leq sum_i P\left(X_1^n\left(i\right)\right) left \left( frac\left\{1\right\}\left\{M\right\} sum_\left\{i\text{'}\right\} left \left( frac\left\{P\left(X_1^n\left(i\text{'}\right)\right)\right\}\left\{P\left(X_1^n\left(i\right)\right)\right\} ight \right) ^s ight \right) ^ ho , .$

Where the sum can now be taken over all $i\text{'} ,$ because that will only increase the bound. Ultimately yielding that

:$P\left(E\right) leq frac\left\{1\right\}\left\{M^ ho\right\} sum_i P\left(X_1^n\left(i\right)\right)^\left\{1-s ho\right\} left \left( sum_\left\{i\text{'}\right\} P\left(X_1^n\left(i\text{'}\right)\right)^s ight \right) ^ ho , .$

Now for simplicity let $1-s ho = s ,$ so that $s= frac\left\{1\right\}\left\{1+ ho\right\} , .$ Substituting this new value of $s ,$ into the above bound on the probability of error and using the fact that $i\text{'} ,$ is just a dummy variable in the sum gives the following as an upper bound on the probability of error:

:$P\left(E\right) leq frac\left\{1\right\}\left\{M^ ho\right\} left \left( sum_i P\left(X_1^n\left(i\right)\right)^\left\{frac\left\{1\right\}\left\{1+ ho ight \right) ^ \left\{1+ ho\right\} , .$

:$M=e^\left\{nR\right\},!$ and each of the components of $X_1^n\left(i\right) ,$ are independent. Thus, simplifying the above equation yields

:$P\left(E\right) leq exp left\left( -n left \left[ ho R - ln left\left( sum_\left\{x_i\right\} P\left(x_i\right)^\left\{frac\left\{1\right\}\left\{1+ ho ight\right)\left(1+ ho\right) ight\right] ight\right).$

The term in the exponent should be maximized over $ho ,$ in order to achieve the tightest upper bound on the probability of error.

Letting $E_0\left( ho\right) = ln left\left( sum_\left\{x_i\right\} P\left(x_i\right)^\left\{frac\left\{1\right\}\left\{1+ ho ight\right)\left(1+ ho\right) , ,$ see that the error exponent for the source coding case is:

:$E_r\left(R\right) = max_\left\{ ho in \left[0,1\right] \right\} left \left[ ho R - E_0\left( ho\right) ight\right] . ,$

For time-variant DMC sources

* Source coding
* Channel coding

Wikimedia Foundation. 2010.

### Look at other dictionaries:

• Exponent (Mathematik) — Das Potenzieren ist wie das Multiplizieren seinem Ursprung nach eine abkürzende Schreibweise für eine wiederholte mathematische Rechenoperation. Wie beim Multiplizieren ein Summand wiederholt addiert wird, so wird beim Potenzieren ein Faktor… …   Deutsch Wikipedia

• Index error — Index In dex, n.; pl. E. {Indexes}, L. {Indices}(?). [L.: cf. F. index. See {Indicate}, {Diction}.] [1913 Webster] 1. That which points out; that which shows, indicates, manifests, or discloses; as, the increasing unemployment rate is an index of …   The Collaborative International Dictionary of English

• List of exponential topics — This is a list of exponential topics, by Wikipedia page. See also list of logarithm topics. *Accelerating change *Artin Hasse exponential *Bacterial growth *Baker Campbell Hausdorff formula *Cell growth *Barometric formula *Basic infection number …   Wikipedia

• Shannon's source coding theorem — In information theory, Shannon s source coding theorem (or noiseless coding theorem) establishes the limits to possible data compression, and the operational meaning of the Shannon entropy.The source coding theorem shows that (in the limit, as… …   Wikipedia

• Channel capacity — In electrical engineering, computer science and information theory, channel capacity is the tightest upper bound on the amount of information that can be reliably transmitted over a communications channel. By the noisy channel coding theorem, the …   Wikipedia

• Exponentiation — Exponent redirects here. For other uses, see Exponent (disambiguation). Exponentiation is a mathematical operation, written as an, involving two numbers, the base a and the exponent (or power) n. When n is a positive integer, exponentiation… …   Wikipedia

• Floating point — In computing, floating point describes a method of representing real numbers in a way that can support a wide range of values. Numbers are, in general, represented approximately to a fixed number of significant digits and scaled using an exponent …   Wikipedia

• 2009–2011 Toyota vehicle recalls — Two of the vehicles under recall: the Toyota Camry (top) and the Toyota Corolla Three separate but related recalls of automobiles by Toy …   Wikipedia

• Rounding — This article is about numerical rounding. For lip rounding in phonetics, see Labialisation. For other uses, see Rounding (disambiguation). Rounding a numerical value means replacing it by another value that is approximately equal but has a… …   Wikipedia

• Large numbers — This article is about large numbers in the sense of numbers that are significantly larger than those ordinarily used in everyday life, for instance in simple counting or in monetary transactions. The term typically refers to large positive… …   Wikipedia