In this preliminary chapter we shall give an exposition of certain topics in probability theory which are necessary to understand and interpret the definition and properties of entropy. We have tried to write the chapter in such a way that a reader with a knowledge of measure theory as given in Ash [15], Halmos [55], or any other basic measure theory text can follow the arguments and understand the examples. We introduce just those parts of probability theory which are necessary for the subsequent chapters and attempt to make them meaningful by use of very simple examples. We also restrict the discussion to “nice” probability spaces, so that conditional expectation and conditional probability are more intuitive and hopefully easier to understand. These “nice” spaces also make it possible to use partitions as models for random experiments, even those experiments which are limits of sequences of experiments.

Probability Spaces

Entropy is a quantitative measurement of uncertainty associated with random phenomena. In order to define this quantity precisely, it is necessary to have a mathematical model for random phenomena which is general enough to include many different physical situations and which has enough structure to allow us to use mathematical reasoning to answer questions about the phenomena.

Such a model is given by a mathematical structure called a probability space, which is nothing more than a measure space in which the measure of the universe set is 1.

A Model of an Information System

Information theory is concerned with constructing a mathematical model for systems which transmit information and then analyzing this model. The object of this analysis is to develop techniques to reproduce at one point (the destination) a message, or an adequate approximation to a message, which has been chosen at another point (the source) and transmitted over a channel.

Fundamental to this problem is a measure of the information transmitted by such a system. It is only in terms of quantity or rate of information processed by a system that one can judge the effectiveness of the system. Central to the notion of quantity of information is an interpretation of the idea of entropy. This interpretation equates the removal of uncertainty with the gain in information. Since entropy is a numerical measure of uncertainty, this interpretation of information gives entropy its central position. The interpretation and the resulting theories were initiated by C. E. Shannon [138] in his fundamental paper “A Mathematical Theory of Communication” published in 1948.

In this chapter we shall describe a standard model of an information system and show how the notion of entropy provides a quantitative basis for measuring the information processed by a system. We shall focus on the construction of the various models, and the definitions and theorems necessary to understand these models.

A large number of analytic functions are known to have continued fraction representations. Frequently a given function will be represented by several different continued fractions, each with its own convergence behavior. The purpose of this chapter is to give a wide selection of examples of continued-fraction representations of functions based on the methods of Chapter 5. Our selection is far from exhaustive. In Section 6.1 the examples consist of continued fractions of Gauss associated with various hypergeometric and confluent hypergeometric series. Section 6.2 deals with examples associated with minimal solutions of three-term recurrence relations. Further methods and examples will be discussed in Chapters 7 and 10. Additional examples can be found, among others, in the following books and articles as well as in the references contained therein: [Abramowitz and Stegun, 1964], [Andrews, 1968], [Askey and Ismail, to appear], [de Bruin, 1977], [Cody et al., 1970], [Erdelyi et al., 1953, Vols. 1, 2, 3], [Frank, 1958, 1960a, 1960b], [Gautschi, 1967, 1970, 1977], [Henrici, 1977, Vol. 2, Chapter 12], [Ince, 1919], [Khovanskii, 1963], [Luke, 1969], [Maurer, 1966], [McCabe, 1974], [Miller, 1975], [Murphy, 1971], [Murphy and O'Donohoe, 1976], [Perron, 1957a], [Phipps, 1971], [Schlömilch, 1871], [Sobhy, 1973], [Stegun and Zucker, 1970], [Tannery, 1882], [Thacher, 1967 and Tech. Rpt. 38–77], [Wall, 1948], [Widder, 1968], [Wynn, 1959].

Continued Fractions of Gauss

In this section we consider a number of examples of continued-fraction representations of functions associated with hypergeometric and confluent hypergeometric series. The method employed is described in Sections 5.2 and 5.4.

The concept of correspondence of continued fractions with formal Laurent series (fLs) was introduced in Chapter 5. Some general theory of correspondence was developed there, and two types of corresponding continued fractions were discussed, the C-fractions and the P-fractions. Some applications of correspondence to obtain continued-fraction representations of analytic functions were given in Chapters 5 and 6. The present chapter deals with special properties of the three most important types of corresponding continued fractions: regular C-fractions (Section 7.1), associated continued fractions (Section 7.2) and general T-fractions (Section 7.3). J-fractions are discussed in connection with associated continued fractions, since they are essentially of the same type. Also g-fractions are considered with regular C-fractions.

A great deal of this chapter is devoted to algorithms. For each type of continued fraction an algorithm is given to compute the coefficients of the continued fraction in terms of the corresponding fLs. The quotientdifference algorithm (Section 7.1.2) and the FG algorithms (Section 7.3.2) can also be used to compute zeros and poles of analytic functions. Among the power series to which the quotient-difference algorithm can be applied are all normal series. Subsumed among these are the Pólya frequency series discussed in Section 7.1.2. Some examples of analytic functions represented by general T-fractions are given in Section 7.3.3; these consist of ratios of confluent hypergeometric functions and include as special cases the error function and Fresnel integrals. The close connection between J-fractions and the general theory of orthogonal polynomials is described briefly in Section 7.2.2.

A large body of mathematics consists of facts that can be presented and described much like any other natural phenomenon. These facts, at times explicitly brought out as theorems, at other times concealed within a proof, make up most of the applications of mathematics, and are the most likely to survive changes of style and of interest.

This ENCYCLOPEDIA will attempt to present the factual body of all mathematics. Clarity of exposition, accessibility to the non-specialist, and a thorough bibliography are required of each author. Volumes will appear in no particular order, but will be organized into sections, each one comprising a recognizable branch of present-day mathematics. Numbers of volumes and sections will be reconsidered as times and needs change.

It is hoped that this enterprise will make mathematics more widely used where it is needed, and more accessible in fields in which it can be applied but where it has not yet penetrated because of insufficient information.

Introduction

This chapter stands at a higher level of abstraction than those that have preceded it. Some familiarity with the theory of Banach spaces is presupposed. In particular, the reader is assumed to be familiar with the various standard topologies on the set of bounded linear operators on a Banach space (norm topology, strong topology, and weak topology). For background material, we refer the reader to any standard text on functional analysis, for example, [KY3].

The situation to be considered in this chapter is as follows: X is a Banach space over the complex numbers and A is a function defined on the real interval [a, b] and taking values in the set B(X) of all bounded linear operators from X to itself. The problem will be to define the product integral of A and establish properties analogous to those found in Chapter 1 for product integrals of matrix-valued functions. As we shall see, for certain types of functions A the work of Chapter 1 generalizes directly. For other types the generalization is not so immediate and will require a considerable development.

In the last section of this chapter, we shall consider a problem involving unbounded operators.

Notation and Terminology

If φ is an element of the Banach space X, then ∥φ∥ denotes the norm of φ. X* denotes the dual of X (all bounded linear maps from X to ℂ).

Introduction

There exists such a wealth of convergence criteria for continued fractions that a selection has been unavoidable. We present the results that have been the most useful for applications, as well as the theorems that appear to us as having the most promise. That the potential of this latter group has not been fully realized we believe is due to the fact that it is relatively unknown. Many of these results are of recent vintage.

Among the best known and most often used convergence criteria are those of Worpitzky [1865, Corollary 4.36(B)], Pringsheim [1899, Theorem 4.35, Corollary 4.36] and Van Vleck [1901a, Theorem 4.29]. Not quite so well known, but surely of great importance, are the parabola theorems, of which we give a simple version in Theorem 4.40, and stronger versions in Theorems 4.42 and 4.43. Of the relatively unknown results we would like to emphasize Theorem 4.33, which is an extremely general criterion for continued fractions K(1/bn), and Theorem 4.49, which is a powerful twin-convergence-region theorem for continued fractions K(an/1). Not enough attention has been paid in the past to results which insure convergence of the even or odd part of a continued fraction. Such results have been considered as only a first step in proving convergence of the whole continued fraction. We feel that these criteria should be considered in their own right and give examples in Theorem 4.31 and Theorem 4.50. More results of this kind can be found in [Jones and Thron, 1970].

Introduction

Topological dynamics has its origin in the famous work of Poincaré [118] on the qualitative or geometric theory of ordinary differential equations. In Poincaré's work one finds many of the fundamental ideas and concepts of the modern theory of dynamical systems. The ideas were formalized and extended by Birkhoff [19], who undertook the systematic development of the theory of dynamical systems. The dynamical systems studied by Poincaré and Birkhoff, now called classical systems, are those that are defined from an ordinary differential equation defined on an open set or manifold contained in Euclidean n-space. In particular the second example mentioned in Section 2.8 and the example in Section 4.1 are of this type.

Specifically, a dynamical system is defined from an ordinary differential equation ẋ = H(x), where H is any smooth vector field on a smooth manifold M contained in Rn, in the following way. The fundamental existence and uniqueness theorem implies that for each p ∈ M there exists an open interval Ip of the reals R, which contains zero and a unique function Fp: Ip → M such that Fp(0) = p and for each Ip, F′p(t) = H(Fp(t)). If M is compact, then for each p ∈ M we may take Ip to be R and define a function ψ: M × R → M by ψ (p, t) = Fp(t). The function ψ turns out to be smooth because of theorems concerning the dependence of solutions on “initial conditions.”