Thirty years ago, Claude Shannon published a paper with the title “A mathematical theory of communication”. In this paper, he defined a quantity, which he called entropy, that measures the uncertainty associated with random phenomena. The effects of this paper on communications in both theory and practice are still being felt, and his entropy function has been applied very successfully to several areas of mathematics. In particular, an extension of it to dynamic situations by A. N. Kolmogorov and Ja. G. Sinai led to a complete solution of a long-unsolved problem in ergodic theory, to a new invariant for differentiable dynamic systems, and to more precision in certain concepts in classical statistical mechanics.

Our intent in this book is to give a rather complete and self-contained development of the entropy function and its extension that is understandable to a reader with a knowledge of abstract measure theory as it is taught in most first-year graduate courses and to indicate how it has been applied to the subjects of information theory, ergodic theory, and topological dynamics. We have made no attempt to give a comprehensive treatment of these subjects; rather we have restricted ourselves to just those parts of the subject which have been influenced by Shannon's entropy and the Kolmogorov-Sinai extension of it. Thus, our purpose is twofold: first, to give a self-contained treatment of all the major properties of entropy and its extension, with rather detailed proofs, and second, to give an exposition of its uses in those areas of mathematics where it has been applied with some success.

Introduction

In this chapter we shall extend some of the theory developed so far by introducing the notion of the product integral of a measure. This extension is a quite natural one from several points of view. On the one hand, from a purely integration-theoretic viewpoint, one might envision the possibility of product integrating the most general objects of ordinary integration theory, especially in the light of the development so far. Of course, it would be expected that such a theory would be closely related to the theory of (linear) differential equations with measures as coefficients. Such equations have occasionally been considered ([VG], [DP]) at least in special cases, but usually not in any systematic manner. They have some importance, however; for example, one problem in (quantum mechanical) scattering theory concerns the description of self-adjoint operators H with a part unitarily equivalent to operating in L2(ℝn), and some of these operators are of the form H = H0 + μ, with μ a measure. The scattering theory associated with such operators may be developed by using the product integral to study the asymptotic form of solutions of the equation Hψ = k2ψ.

Entropy is a subject which has played a central role in a number of areas such as statistical mechanics and information theory. The connections between the various applications of entropy have become clearer in recent years by the introduction of probability theory into its foundations. It is now possible to see a number of what were previously isolated results in various disciplines as part of a more general mathematical theory of entropy.

This volume presents a self-contained exposition of the mathematical theory of entropy. Those parts of probability theory which are necessary for an understanding of the central topics concerning entropy have been included. In addition, carefully chosen examples are given in order that the reader may omit proofs of some of the theorems and yet by studying these examples and discussion obtain insight into the theorems.

The last four chapters give a description of those parts of information theory, ergodic theory, statistical mechanics, and topological dynamics which are most affected by entropy. These chapters may be read independently of each other. The examples show how ideas originating in one area have influenced other areas. Chapter III contains a brief description of how entropy as a measure of information flow has affected information theory and complements the first part of The Theory of Information and Coding by R. J. McEliece (volume 3 of this ENCYCLOPEDIA). Recent applications of entropy to statistical mechanics and topological dynamics are given in chapters V and VI.

In the present chapter, we shall discuss additional results and generalizations of the theory that we have presented so far. A large part of our discussion will focus on the work of other authors. Some of this work, e.g., product integration of nonlinear operator-valued functions, has not been developed in the previous chapters and will only be discussed briefly here. This is partly due to the fact that for these results, a really systematic and complete presentation is simply not available at the present time. On the other hand, what we have presented above has been somewhat determined by our own predispositions; a treatise of reasonable length covering all aspects of product integration in detail is hardly feasible, and we have therefore selected material which is of the most interest to us and which at the same time is not readily accessible in the modern literature. Nevertheless, the results we shall discuss in this chapter are of considerable significance and importance and certainly deserve some attention in any general discussion of product integration.

If we consider the theory of product integration of matrix-valued functions presented in Chapter 1 as a starting point, then there are several possible directions for generalization. For example, a first type of generalization might focus on reducing to a minimum the regularity assumptions on the function A(x) being product integrated with regard to its dependence on x, while not changing essentially the nature of the values of this function.

In this chapter we shall give a mathematical definition of the information in a random event and the entropy of experiments with a countable number of outcomes. We shall also indicate how the entropy is a measure of uncertainty and then give the main properties satisfied by both information and entropy. Then the definition of entropy will be extended to include experiments with an arbitrary number of outcomes, and the properties of entropy will be proven for this case. Finally we give the definitions of the rate of information generation and the entropy of a dynamical system and derive their most important properties. We conclude with several examples and a brief discussion of two useful extensions of these definitions.

Information and Uncertainty of Events

Let (Ω, F, P) be a Lebesgue space and E an event in F. Thinking of the Lebesgue space as being a mathematical model of some random experiment, suppose an outcome of this experiment results in the event E. We have gained some information because we know that E occurred. The purpose of this first section is to define a function I on the events in a Lebesgue space so that I(E) will give a quantitative measure of the information gained if the event E results from the outcome of the experiment.

Before the experiment is performed, the uncertainty of its outcome resulting in the event E should equal the information we have gained if the outcome does result in E.