Skip to main content
×
Home
    • Aa
    • Aa

Shannon entropy: a rigorous notion at the crossroads between probability, information theory, dynamical systems and statistical physics

  • ANNICK LESNE (a1)
Abstract

Statistical entropy was introduced by Shannon as a basic concept in information theory measuring the average missing information in a random source. Extended into an entropy rate, it gives bounds in coding and compression theorems. In this paper, I describe how statistical entropy and entropy rate relate to other notions of entropy that are relevant to probability theory (entropy of a discrete probability distribution measuring its unevenness), computer sciences (algorithmic complexity), the ergodic theory of dynamical systems (Kolmogorov–Sinai or metric entropy) and statistical physics (Boltzmann entropy). Their mathematical foundations and correlates (the entropy concentration, Sanov, Shannon–McMillan–Breiman, Lempel–Ziv and Pesin theorems) clarify their interpretation and offer a rigorous basis for maximum entropy principles. Although often ignored, these mathematical perspectives give a central position to entropy and relative entropy in statistical laws describing generic collective behaviours, and provide insights into the notions of randomness, typicality and disorder. The relevance of entropy beyond the realm of physics, in particular for living systems and ecosystems, is yet to be demonstrated.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Shannon entropy: a rigorous notion at the crossroads between probability, information theory, dynamical systems and statistical physics
      Available formats
      ×
      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about sending content to Dropbox.

      Shannon entropy: a rigorous notion at the crossroads between probability, information theory, dynamical systems and statistical physics
      Available formats
      ×
      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about sending content to Google Drive.

      Shannon entropy: a rigorous notion at the crossroads between probability, information theory, dynamical systems and statistical physics
      Available formats
      ×
Copyright
Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

P. H. Algoet and T. M. Cover (1988) A sandwich proof of the Shannon–McMillan–Breiman theorem. Annals of Probability 16 899909.

J. Avery (2003) Information theory and evolution, World Scientific.

R. Badii and A. Politi (1997) Complexity. Hierarchical structures and scaling in physics, Cambridge University Press.

R. Balian (2005) Information in statistical physics. Studies in History and Philosophy of Modern Physics 36 323353.

L. Breiman (1957) The individual ergodic theorem of information theory. Annals of Mathematical Statistics 28 809811. (Correction: (1957) 31 809–810.)

L. Brillouin (1951a) Maxwell's demon cannot operate: Information and entropy. Journal of Applied Physics 22 334337.

L. Brillouin (1951b) Physical entropy and information. Journal of Applied Physics 22 338343.

L. Brillouin (1953) Negentropy principle of information. Journal of Applied Physics 24 11521163.

M. Brin and A. Katok (1983) On local entropy. In: J. Palis (ed.) Geometric dynamics. Springer-Verlag Lecture Notes in Mathematics 1007 3038.

P. Castiglione , M. Falcioni , A. Lesne and A. Vulpiani (2008) Chaos and coarse-graining in statistical mechanics, Cambridge University Press.

C. Cercignani (1988) The Boltzmann equation and its applications, Springer-Verlag.

G. J. Chaitin (1966) On the length of programs for computing finite binary sequences. Journal of the ACM 13 547569.

E. G. D. Cohen and G. Gallavotti (1999) Note on two theorems of nonequilibrium statistical mechanics. Journal of Statistical Physics 96 13431349.

R. T. Cox (1946) Probability, frequency, and reasonable expectation. American Journal of Physics 14 113.

I. Csiszár (1975) I-divergence geometry of probability distributions and minimization problems. Annals of Probability 3 146158.

I. Csiszár (1998) The Method of types. IEEE Transactions on Information Theory 44 25052523.

B. Durand and A. Zvonkine (2007) Kolmogorov complexity. In: E. Charpentier , A. Lesne and N. Nikolski (eds.) Kolmogorov's Heritage in Mathematics, Springer-Verlag281300.

R. S. Ellis (1985) Entropy, large deviations and statistical mechanics, Springer-Verlag.

D. J. Evans and D. J. Searles (2002) The fluctuation theorem. Advances in Physics 51 15291585.

M. Falcioni , V. Loreto and A. Vulpiani (2003) Kolmogorov's legacy about entropy, chaos and complexity. In: A. Vulpiani and R. Livi (eds.) The Kolmogorov Legacy in Physics, Springer-Verlag85108.

D. P. Feldman and J. P. Crutchfield (1998) Measures of statistical complexity: Why? Physics Letters A 238 244252.

K. Ford (2007) From Kolmogorov's theorem on empirical distribution to number theory. In: E. Charpentier , A. Lesne and N. Nikolski (eds.) Kolmogorov's heritage in mathematics, Springer-Verlag97108.

S. A. Frank (2009) The common patterns of nature. Journal of Evolutionary Biology 22 15631585.

G. Gallavotti (1998) Chaotic dynamics, fluctuations, nonequilibrium ensembles. Chaos 8 384393.

G. Gallavotti (2006) Entropy, thermostats and the chaotic hypothesis. Chaos 16 043114.

P. Gaspard (2004) Time-reversed dynamical entropy and irreversibility in Markovian random processes. Journal of Statistical Physics 117 599615.

M. Gell-Mann and S. Lloyd (1996) Information measures, effective complexity, and total information. Complexity 2 4452.

E. Glasner (2003) Ergodic theory via joinings, American Mathematical Society.

A. N. Gorban (2007) Order-disorder separation: Geometric revision. Physica A 374 85102.

P. Grassberger (1986) Toward a quantitative theory of self-generated complexity. International Journal of Theoretical Physics 25 907938.

C. Gruber , S. Pache and A. Lesne (2004) On the second law of thermodynamics and the piston problem. Journal of Statistical Physics 117 739772.

B. Haegeman and R. S. Etienne (2010) Entropy maximization and the spatial distribution of species. American Naturalist 175 E74E90.

J. Honerkamp (1998) Statistical physics, Springer-Verlag.

S. Ihara (1993) Information theory for continuous systems, World Scientific.

E. T. Jaynes (1957a) Information theory and statistical mechanics Part I. Physical Review 106 620630.

E. T. Jaynes (1957b) Information theory and statistical mechanics Part II. Physical Review 108 171190.

E. T. Jaynes (1973) The well-posed problem. Foundations of Physics 3 477493.

E. T. Jaynes (1980) The minimum entropy production principle. Annual Review of Physical Chemistry 31 579601.

E. T. Jaynes (1982) On the rationale of maximum entropy methods. Proceedings of the IEEE 70 939952.

S. Karlin and H. M. Taylor (1975) A first course in stochastic processes, Academic Press.

W. Krieger (1970) On entropy and generators of measure-preserving transformations. Transactions of the American Mathematical Society 149 453464.

S. Kullback and R. Leibler (1951) On information and sufficiency. Annals of Mathematical Statistics 22 7986.

R. Landauer (1961) Irreversibility and heat generation in the computing process. IBM Journal of Research and Development 5 183191.

J. L. Lebowitz (1993a) Boltzmann's Entropy and Time's Arrow. Physics Today 46 3238.

J. L. Lebowitz (1993b) Macroscopic laws, microscopic dynamics, time's arrow and Boltzmann's entropy. Physica A 194 127.

F. Ledrappier and J. M. Strelcyn (1982) A proof of the estimation from below in Pesin's entropy formula. Ergodic Theory and Dynamical Systems 2 203219.

A. Lempel and J. Ziv (1976) On the complexity of finite sequences. IEEE Transactions on Information Theory 22 7581.

A. Lesne and A. Benecke (2008) Feature context-dependency and complexity reduction in probability landscapes for integrative genomics. Theoretical Biology and Medical Modelling 5 21.

A. Lesne , J. L. Blanc and L. Pezard (2009) Entropy estimation of very short symbolic sequences. Physical Review E 79 046208.

M. Li and P. Vitanyi (1997) An Introduction to Kolmogorov complexity and its applications, Springer.

H. Mahara and T. Yamaguchi (2010) Entropy balance in distributed reversible Gray-Scott model. Physica D 239 729734.

P. Martin-Löf (1966) The definition of random sequence. Information and Control 9 602619.

B. McMillan (1953) The basic theorems of information theory. Annals of Mathematical Statistics 24 196219.

G. Nicolis and P. Gaspard (1994) Toward a probabilistic approach to complex systems. Chaos, Solitons and Fractals 4 4157.

G. Parisi (2003) Complexity and intelligence. In: A. Vulpiani and R. Livi (eds.) The Kolmogorov Legacy in Physics, Springer-Verlag109122.

Y. Pesin (1997) Dimension theory in dynamical systems. Contemporary views and applications, University of Chicago Press.

S. J. Phillips and M. Dudík (2008) Modeling of species distributions with Maxent: new extensions and a comprehensive evaluation. Ecography 31 161175.

S. J. Phillips , R. P. Anderson and R. E. Schapire (2006) Maximum entropy modeling of species geographic distribution. Ecological Modelling 190 231259.

Z. Rached , F. Alajaji and L. Campbell (2001) Rényi's divergence and entropy rates for finite alphabet Markov sources. IEEE Transactions on Information Theory 47 15531562.

C. Robert (1990) An entropy concentration theorem: applications in artificial intelligence and descriptive statistics. Journal of Applied Probability 27 303313.

D. P. Ruelle (2003) Extending the definition of entropy to nonequilibrium steady states. Proceedings of the National Academy of Sciences of the United States of America 100 30543058.

I. Samengo (2002) Estimating probabilities from experimental frequencies. Physical Review E 65 046124.

T. Sagawa and M. Ueda (2009) Minimal energy cost for thermodynamic information processing: measurement and information erasure. Physical Review Letters 102 250602.

L. S. Schulman (2010) We know why coffee cools. Physica E 42 269272.

C. Shannon (1948) A mathematical theory of communication. Bell System Technical Journal 27 379423.

J. S. Shinner , M. Davison and J. T. Landsberg (1999) Simple measure for complexity. Physical Review E 59 14591464.

A. D. Sokal (1997) Monte Carlo methods in statistical mechanics: Foundations and new algorithms. In: C. C. De Witt-Morette and A. Folacci (eds.) Functional integration: basics and applications (1996 Cargèse summer school), Plenum Press.

A. D. Sokal and L. E. Thomas (1989). Exponential convergence to equilibrium for a class of random-walk models. Journal of Statistical Physics 54 797828.

R. Solomonoff (1978). Complexity-based induction systems: comparisons and convergence theorems. IEEE Transactions on Information Theory 24 422432.

L. Szilard (1929) Uber die Entropieverminderung in einem thermodynamischen System bei Eingriffen intelligenter Wesen. (On the lessening of entropy in a thermodynamic system by interference of an intelligent being). Zeitschrift für Physik 53 840856.

H. Touchette (2009) The large deviation approach to statistical mechanics. Physics Reports 478 169.

M. Tribus and E. C. McIrvine (1971) Energy and information. Scientific American 225 179188.

J. M. Van Campenhout and T. M. Cover (1981) Maximum entropy and conditional entropy. IEEE Transactions on Information Theory 27 483489.

V. Vovk and G. Shafer (2003) Kolmogorov's contributions to the foundations of probability. Problems of Information Transmission 39 2131.

H. White (1993) Algorithmic complexity of points in dynamical systems. Ergodic Theory and Dynamical Systems 13 807830.

A. D. Wyner and J. Ziv (1989) Some asymptotic properties of the entropy of a stationary ergodic data source with applications to data compression. IEEE Transactions on Information Theory 35 12501258.

J. Ziv and A. Lempel (1977) A universal algorithm for sequential data compression. IEEE Transactions on Information Theory 23 337343.

J. Ziv and A. Lempel (1978) Compression of individual sequences by variable rate coding. IEEE Transactions on Information Theory 24 530536.

O. Zuk , I. Kanter and E. Domany (2005) The entropy of a binary hidden Markov process. Journal of Statistical Physics 121 343360. (Conference version: Aymptotics of the entropy rate for a hidden Markov process. Proceedings DCC'05 173–182.)

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Mathematical Structures in Computer Science
  • ISSN: 0960-1295
  • EISSN: 1469-8072
  • URL: /core/journals/mathematical-structures-in-computer-science
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×

Metrics

Full text views

Total number of HTML views: 0
Total number of PDF views: 191 *
Loading metrics...

Abstract views

Total abstract views: 405 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 23rd September 2017. This data will be updated every 24 hours.