Limiting density of discrete points

The limiting density of discrete points is used to formulate an adjustment made by Edwin Thompson Jaynes to Claude Elwood Shannon's fundamental uniqueness theorem. The adjustment was created to allow Shannon's information measure formulation to be compatible with continuous distributions.

The limiting density of discrete points, as formulated by Jaynes, is proportional to the invariant measure $m(x)$.:

Whereas the information entropy, $H(X)$, of the discrete distribution $p(x)$ is as shown below in (1):: $H(X)=\; -\; sum\_\{i=1\}^np(x\_i)log\; p(x\_i)qquadqquad\; (1)$

the relative entropy of a continuous distribution $p(x)$ would be as shown in (2)

: $H(X)=-int\; p(x)logfrac\{p(x)\}\{m(x)\},dxqquadqquad\; (2)$

References

*Jaynes E.T.,Prior Probabilities IEEE, Transaction on System Science and Cybernetics SSSC-4 (1968), 227.
*Jaynes, E. T., 1983, `Papers On Probability, Statistics and Statistical Physics,' Edited by R. D. Rosenkrantz, D. Reidel publishing Co., Dordrecth, Holland