In this chapter we revisit the coding theorem for a DMC. By definition, for any R > 0 below capacity, there exists a sequence of n-length block codes (fn, φn) with rates converging to R and maximum probability of error converging to zero as n → ∞. On the other hand, by Theorem 6.5, for codes of rate converging to a number above capacity, the maximum probability of error converges to unity. Now we look at the speed of these convergences. This problem is far more complex than its source coding analog and it has not been fully settled yet.

We saw in Chapter 6 that the capacity of a DMC can be achieved by codes, all codewords of which have approximately the same type. In this chapter we shall concentrate attention on constant composition codes, i.e., codes all codewords of which have the very same type. We shall investigate the asymptotics of the error probability for codes from this special class. The general problem reduces to this one in a simple manner.

Our present approach will differ from that in Chapter 6. In that chapter channel codes were constructed by defining the encoder and the decoder simultaneously, in a successive manner. Here, attention will be focused on finding suitable encoders; the decoder will be determined by the encoder in a way to be specified later.

When the first edition of this book went to print, information theory was only 30 years old. At that time we covered a large part of the topic indicated in the title, a goal that is no longer realistic. An additional 30 years have passed, the Internet revolution occurred, and information theory has grown in breadth, volume and impact. Nevertheless, we feel that, despite many new developments, our original book has not lost its relevance since the material therein is still central to the field.

The main novelty of this second edition is the addition of two new chapters. These cover zero-error problems and their connections to combinatorics (Chapter 11) and information-theoretic security (Chapter 17). Of several new research directions that emerged in the 30 years between the two editions, we chose to highlight these two because of personal research interests. As a matter of fact, these topics started to intrigue us when writing the first edition; back then, this led us to a last-minute addition of problems on secrecy.

Except for the new chapters, new results are presented only in the form of problems. These either directly complete the original material or, occasionally, illustrate a new research area. We made only minor changes, mainly corrections, to the text of the original chapters. (Hence the words recent and new refer to the time of the first edition, unless the context indicates otherwise.) We have updated the history part of each chapter and, in particular, we have included pointers to new developments.

A basic common characteristic of almost all channel coding problems treated in this book is that an asymptotically vanishing probability of error in transmission is tolerated. This permits us to exploit the global knowledge of the statistics of sources and channels in order to enhance transmission speed. We see again and again that in the case of a correct tuning of the parameters most codes perform in the same manner and thus, in particular, optimal codes, instead of being rare, abound. This ceases to be true if we are dealing with codes that are error-free.

The zero-error capacity of a DMC or compound DMC has been defined in Chapters 6 and 10 as the special case ε = 0 of ε-capacity. To keep this chapter self-contained, we give an independent (of course, equivalent) definition below.

A zero-error code of block length n for a DMC will be defined by a (codeword) set C ⊂ Xn, rather than by an encoder–decoder pair (f, φ), understanding that the message set coincides with the codeword set and the encoder is the identity mapping. This definition makes sense because if to a codeword set C there exists a decoder φ : Yn → C that yields probability of error equal to zero, this decoder is essentially unique.

Introduction

From different viewpoints and under different names, the so-called hidden Markov measures have received a lot of attention in the last fifty years [3]. One considers a (stationary) Markov chain (Xn)n∈ℕ with finite state space A and looks at its “instantaneous” image Yn ≔ π(Xn), where the map π is an amalgamation of the elements of A yielding a smaller state space, say B. It is well known that in general the resulting chain, (Yn)n∈ℕ, has infinite memory.

Introduction

Symbolic dynamics is the study of shift (and other) transformations on spaces of infinite sequences or arrays of symbols and maps between such systems. A symbolic dynamical system, with a shift-invariant measure, corresponds to a stationary stochastic process. In the setting of information theory, such a system amounts to a collection of messages. Markov measures and hidden Markov measures, also called sofic measures, on symbolic dynamical systems have the desirable property of being determined by a finite set of data. But not all of their properties, for example the entropy, can be determined by finite algorithms. This article surveys some of the known and unknown properties of hidden Markov measures that are of special interest from the viewpoint of symbolic dynamics. To keep the article self contained, necessary background and related concepts are reviewed briefly. More can be found in [47, 56, 55, 71].

We discuss methods and tools that have been useful in the study of symbolic systems, measures supported on them, and maps between them.

This volume is a collection of papers on hidden Markov processes (HMPs) involving connections with symbolic dynamics and statistical mechanics. The subject was the focus of a five-day workshop held at the Banff International Research Station (BIRS) in October 2007, which brought together thirty mathematicians, computer scientists, and electrical engineers from institutions throughout the world. Most of the papers in this volume are based either on work presented at the workshop or on problems posed at the workshop.

From one point of view, an HMP is a stochastic process obtained as the noisy observation process of a finite-state Markov chain; a simple example is a binary Markov chain observed in binary symmetric noise, i.e., each symbol (0 or 1) in a binary state sequence generated by a two-state Markov chain may be flipped with some small probability, independently from time instant to time instant. In another (essentially equivalent) viewpoint, an HMP is a process obtained from a finite-state Markov chain by partitioning its state set into groups and completely “hiding” the distinction among states within each group; more precisely, there is a deterministic function on the states of the Markov chain, and the HMP is the process obtained by observing the sequences of function values rather than sequences of states (and hence such a process is sometimes called a “function of a Markov chain”).

HMPs are encountered in an enormous variety of applications involving phenomena observed in the presence of noise.