Information is a fashionable concept with many facets, among which the quantitative one–our subject–is perhaps less striking than fundamental. At the intuitive level, for our purposes, it suffices to say that information is some knowledge of predetermined type contained in certain data or pattern and wanted at some destination. Actually, this concept will not explicitly enter the mathematical theory. However, throughout the book certain functionals of random variables will be conveniently interpreted as measures of the amount of information provided by the phenomena modeled by these variables. Such information measures are characteristic tools of the analysis of optimal performance of codes, and they have turned out to be useful in other branches of mathematics as well.

Intuitive background

The mathematical discipline of information theory, created by C. E. Shannon (1948) on an engineering background, still has a special relation to communication engineering, the latter being its major field of application and the source of its problems and motivation. We believe that some familiarity with the intuitive communication background is necessary for a more than formal understanding of the theory, let alone for doing further research. The heuristics, underlying most of the material in this book, can be best explained on Shannon's idealized model of a communication system (which can also be regarded as a model of an information storage system). The important question of how far the models treated are related to, and the results obtained are relevant for, real systems will not be addressed.

In Chapter 13 we formulated a fairly general model of noiseless communication networks. The absence of noise means that the coders located at the vertices of the network have direct access to the results of coding operations performed at immediately preceding vertices. By dropping this assumption, we now extend the model to cover communication in a noisy environment. We shall suppose that codewords produced at certain vertices are components of a vector input of a noisy channel, and it is the corresponding channel output that can be observed at some other vertex of the network.

The mathematical problems solved in this chapter will relate to the noisy version of the simplest multi-terminal network, the fork. In order to avoid clumsy notation, we give the formal definitions only for the case of two inputs.

Given finite sets X, Y, Z, consider channels with input set X × Y and output set Z. A multiple-access code (MA code) for such channels is a triple of mappings f : M1 → X, g : M2 → Y, φ : Z → M1 × M2, where M1 and M2 are arbitrary finite sets. The mappings f and g are called encoders, with message sets M1 resp. M2, while φ is the decoder. A MA code is also a code in the usual sense, with encoder (f, g) : M1 × M2 → X × Y and decoder φ.

In this chapter we revisit the coding theorem for a DMC. By definition, for any R > 0 below capacity, there exists a sequence of n-length block codes (fn, φn) with rates converging to R and maximum probability of error converging to zero as n → ∞. On the other hand, by Theorem 6.5, for codes of rate converging to a number above capacity, the maximum probability of error converges to unity. Now we look at the speed of these convergences. This problem is far more complex than its source coding analog and it has not been fully settled yet.

We saw in Chapter 6 that the capacity of a DMC can be achieved by codes, all codewords of which have approximately the same type. In this chapter we shall concentrate attention on constant composition codes, i.e., codes all codewords of which have the very same type. We shall investigate the asymptotics of the error probability for codes from this special class. The general problem reduces to this one in a simple manner.

Our present approach will differ from that in Chapter 6. In that chapter channel codes were constructed by defining the encoder and the decoder simultaneously, in a successive manner. Here, attention will be focused on finding suitable encoders; the decoder will be determined by the encoder in a way to be specified later.

When the first edition of this book went to print, information theory was only 30 years old. At that time we covered a large part of the topic indicated in the title, a goal that is no longer realistic. An additional 30 years have passed, the Internet revolution occurred, and information theory has grown in breadth, volume and impact. Nevertheless, we feel that, despite many new developments, our original book has not lost its relevance since the material therein is still central to the field.

The main novelty of this second edition is the addition of two new chapters. These cover zero-error problems and their connections to combinatorics (Chapter 11) and information-theoretic security (Chapter 17). Of several new research directions that emerged in the 30 years between the two editions, we chose to highlight these two because of personal research interests. As a matter of fact, these topics started to intrigue us when writing the first edition; back then, this led us to a last-minute addition of problems on secrecy.

Except for the new chapters, new results are presented only in the form of problems. These either directly complete the original material or, occasionally, illustrate a new research area. We made only minor changes, mainly corrections, to the text of the original chapters. (Hence the words recent and new refer to the time of the first edition, unless the context indicates otherwise.) We have updated the history part of each chapter and, in particular, we have included pointers to new developments.

A basic common characteristic of almost all channel coding problems treated in this book is that an asymptotically vanishing probability of error in transmission is tolerated. This permits us to exploit the global knowledge of the statistics of sources and channels in order to enhance transmission speed. We see again and again that in the case of a correct tuning of the parameters most codes perform in the same manner and thus, in particular, optimal codes, instead of being rare, abound. This ceases to be true if we are dealing with codes that are error-free.

The zero-error capacity of a DMC or compound DMC has been defined in Chapters 6 and 10 as the special case ε = 0 of ε-capacity. To keep this chapter self-contained, we give an independent (of course, equivalent) definition below.

A zero-error code of block length n for a DMC will be defined by a (codeword) set C ⊂ Xn, rather than by an encoder–decoder pair (f, φ), understanding that the message set coincides with the codeword set and the encoder is the identity mapping. This definition makes sense because if to a codeword set C there exists a decoder φ : Yn → C that yields probability of error equal to zero, this decoder is essentially unique.

Introduction

From different viewpoints and under different names, the so-called hidden Markov measures have received a lot of attention in the last fifty years [3]. One considers a (stationary) Markov chain (Xn)n∈ℕ with finite state space A and looks at its “instantaneous” image Yn ≔ π(Xn), where the map π is an amalgamation of the elements of A yielding a smaller state space, say B. It is well known that in general the resulting chain, (Yn)n∈ℕ, has infinite memory.