×

A comparison of the Shannon and Kullback information measures. (English) Zbl 1255.94042

Summary: Two widely used information measures are compared. It is shown that the Kullback measure, unlike the Shannon measure, provides the basis for a consistent theory of information which extends to continuous sample spaces and to nonconstant prior distributions. It is shown that the Kullback measure is a generalization of the Shannon measure, and that the Kullback measure has more reasonable additivity properties than does the Shannon measure. The results lend support to Jaynes’s entropy maximization procedure.

MSC:

94A17 Measures of information, entropy
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] C. E. Shannon,Bell System Tech. J. 27:379, 623 (1948); reprinted in C. E. Shannon and W. Weaver,The Mathematical Theory of Communication, Univ. of Illinois Press, Urbana, Ill. (1949). · Zbl 1154.94303 · doi:10.1002/j.1538-7305.1948.tb01338.x
[2] N. Wiener,Cybernetics, Wiley, New York (1948).
[3] F. M. Reza,An Introduction to Information Theory, McGraw-Hill, New York (1961), and references cited therein.
[4] R. C. Tolman,The Principles of Statistical Mechanics, Oxford Univ. Press, London (1938). · Zbl 0019.35902
[5] S. Kullback,Annals of Math. Statistics 22:79 (1951). · Zbl 0042.38403 · doi:10.1214/aoms/1177729694
[6] S. Kullback,Information Theory and Statistics, Wiley, New York (1951). · Zbl 0042.38403
[7] F. Schlögl,Z. Physik 249:1 (1971), and references cited therein. · doi:10.1007/BF01397920
[8] E. T. Jaynes,Phys. Rev. 106:620 (1957);108:171 (1957). · Zbl 0084.43701 · doi:10.1103/PhysRev.106.620
[9] E. T. Jaynes, inStatistical Physics (1962 Brandeis Lectures), ed. by K. W. Ford, Benjamin, New York (1963).
[10] A. Hobson,Concepts in Statistical Mechanics, Gordon and Breach, New York (1971).
[11] A. Katz,Principles of Statistical Mechanics, Freeman, San Francisco (1967). · Zbl 0188.29701
[12] R. Baierlein,Atoms and Information Theory, Freeman, San Francisco (1971).
[13] A. Feinstein,Foundations of Information Theory, McGraw-Hill, New York (1968); A. I. Khinchin,Mathematical Foundations of Information Theory, Dover, New York (1957). · Zbl 0082.34602
[14] A. Hobson,J. Stat. Phys. 1:383 (1969). · doi:10.1007/BF01106578
[15] Pierre Simon de Laplace,A Philosophical Essay on Probabilities, Dover, New York (1951). · Zbl 0045.06905
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.