This book is meant to be a self-contained introduction to the basic results in the theory of information and coding. It was written during 1972–1976, when I taught this subject at Caltech. About half my students were electrical engineering graduate students; the others were majoring in all sorts of other fields (mathematics, physics, biology, even one English major!). As a result the course was aimed at nonspecialists as well as specialists, and so is this book.

The book is in three parts: Introduction, Part one (Information Theory), and Part two (Coding Theory). It is essential to read the introduction first, because it gives an overview of the whole subject. In Part one, Chapter 1 is fundamental, but it is probably a mistake to read it first, since it is really just a collection of technical results about entropy, mutual information, and so forth. It is better regarded as a reference section, and should be consulted as necessary to understand Chapters 2–5. Chapter 6 is a survey of advanced results, and can be read independently. In Part two, Chapter 7 is basic and must be read before Chapters 8 and 9; but Chapter 10 is almost, and Chapter 11 is completely, independent from Chapter 7. Chapter 12 is another survey chapter independent of every thing else.

The problems at the end of the chapters are very important. They contain verification of many omitted details, as well as many important results not mentioned in the text.

Transmission of information is at the heart of what we call communication. As an area of concern it is so vast as to touch upon the preoccupations of philosophers and to give rise to a thriving technology.

We owe to the genius of Claude Shannon the recongnition that a large class of problems related to encoding, transmitting, and decoding informatio can be approached in a systematic and disciplined way: his classic paper of 1948 marks the birth of a new chapter of Mathematics.

In the past thirty years there has grown a staggering literature in this fledgling field, and some of its terminology even has become part of our daily language.

The present monograph (actually two monographs in one) is an excellent introduction to the two aspects of communication: coding and transmission.

The first (which is the subject of Part two) is an elegant illustration of the power and beauty of Algebra; the second belongs to Probability Theory which the chapter begun by Shannon enriched in novel and unexpected ways.

Introduction

The theory of codes provides some jewels of combinatorics on words that we want to describe in this chapter.

A basic result is the defect theorem (Theorem 6.2.1), which states that if a set X of n words satisfies a nontrivial relation, then these words can be expressed simultaneously as products of at most n — 1 words. It is the starting point of the chapter. In Chapters 9 and 13, other defect properties are studied in different contexts.

A nontrivial relation is simply a finite word w which ambiguously factorizes over X. This means that X is not a code. The defect effect still holds if X is not an ω-code, i.e., if the nontrivial relation is an infinite, instead of a finite, word (Theorem 6.2.4).

The defect theorem implies several well-known properties on words that are recalled in this chapter. For instance, the fact that two words which commute are powers of the same word is a consequence. Another consequence is that a two-element code or more generally an elementary set is an co-code. The latter property appears to be a crucial step in one of the proofs of the DOL equivalence problem.

A remarkable phenomenon appears when, for a finite code X, neither the set X nor its reversal is an ω-code. In this case the defect property is stronger: the n elements of X can be expressed as products of at most n — 2 words (Theorem 6.3.4).

Introduction

In this chapter we shall be concerned with sesquipowers. Any nonempty word is a sesquipower of order 1. A word w is a sesquipower of order n if w = uvu, where u is a sesquipower of order n — 1. Sesquipowers have many interesting combinatorial properties which have applications in various domains. They can be defined by using bi-ideal sequences.

A finite or infinite sequence of words f1,…,fn,… is called a bi-ideal sequence if for all i > 0, fi is both a prefix and a suffix of fi+1 and, moreover, 2|fi| ≤ |fi+1|. A sesquipower of order n is then the nth term of a bi-ideal sequence. Bi-ideal sequences have been considered, with different names, by several authors in algebra and combinatorics (see Notes).

In Sections 4.2 and 4.3 we analyze some interesting combinatorial properties of bi-ideal sequences and the links existing between bi-ideal sequences, recurrence and n-divisions. From these results we will obtain in Section 4.4 an improvement (Theorem 4.4.5) of an important combinatorial theorem of Shirshov. We recall (see Lothaire 1983) that Shirshov's theorem states that for all positive integers p and n any sufficiently large word over a finite totally ordered alphabet will have a factor f which is a pth power or is n-divided, i.e., f can be factorized into nonempty blocks as f = x1 … xn with the property that all the words that one obtains by a nontrivial rearrangement of the blocks are lexicographically less than f.

Introduction

The notion of a dimension, when available, is a powerful mathematical tool in proving finiteness conditions in combinatorics. An example of this is Eilenberg's equality theorem, which provides an optimal criterion for the equality of two rational series over a (skew) field. In this example a problem on words, i.e., on free semigroups, is first transformed into a problem on vector spaces, and then it is solved using the dimension property of those algebraic structures. One can raise the natural question: do sets of words possess dimension properties of some kind?

We approach this problem through systems of equations in semigroups. As a starting point we recall the well-known defect theorem (see Chapter 6), which states that if a set of n words satisfies a nontrivial relation, then these words can be expressed simultaneously as products of at most n — 1 words. The defect effect can be seen as a weak dimension property of words. In order to analyze it further one can examine what happens when n words satisfy several independent relations, where independence is formalized as follows: a set E of relations on n words is independent, if E, viewed as a system of equations, does not contain a proper subset having the same solutions as E.

It is not difficult to see that a set of n words can satisfy two or more equations even in the case where the words cannot be expressed as products of fewer than n—1 words.

Introduction

This chapter deals with positional numeration systems. Numbers are seen as finite or infinite words over an alphabet of digits. A numeration system is defined by a pair composed of a base or a sequence of numbers, and of an alphabet of digits. In this chapter we study the representation of natural numbers, of real numbers and of complex numbers. We will present several generalizations of the usual notion of numeration system, which lead to interesting problems.

Properties of words representing numbers are well studied in number theory: the concepts of period, digit frequency, normality give rise to important results. Cantor sets can be defined by digital expansions.

In computer arithmetic, it is recognized that algorithmic possibilities depend on the representation of numbers. For instance, addition of two integers represented in the usual binary system, with digits 0 and 1, takes a time proportional to the size of the data. But if these numbers are represented with signed digits 0, 1, and —1, then addition can be realized in parallel in a time independent of the size of the data.

Since numbers are words, finite state automata are relevant tools to describe sets of number representations, and also to characterize the complexity of arithmetic operations. For instance, addition in the usual binary system is a function computable by a finite automaton, but multiplication is not.

Introduction

A seminal result of Makanin 1977 states that the existential theory of equations over free monoids is decidable. Makanin achieved this result by presenting an algorithm which solves the satisfiability problem for word equations with constants. The satisfiability problem is usually stated for a single equation, but this is no loss of generality.

This chapter provides a self-contained presentation of Makanin's result. The presentation has been inspired by Schulz 1992a. In particular, we show the result of Makanin in a more general setting, due to Schulz, by allowing that the problem instance is given by a word equation L = R together with a list of rational languages Lx ⊆ A*, where x ∈ Ω denotes an unknown and A is the alphabet of constants. We will see that it is decidable whether or not there exists a solution σ:Ω → A* which, in addition to σ(L) = σ(R), satisfies the rational constraints σ(x)∈ Lx for all x ∈ Ω. Using an algebraic viewpoint, rational constraints mean to work over some finite semigroup, but we do not need any deep result from the theory of finite semigroups. The presence of rational constraints does not make the proof of Makanin's result much harder; however, the more general form is attractive for various applications.

In the following we explain the outline of the chapter; for some background information and more comments on recent developments we refer to the Notes.