The main changes in this edition are in Part two. The old Chapter 8 (“BCH, Goppa, and Related Codes”) has been revised and expanded into two new chapters, numbered 8 and 9. The old chapters 9, 10, and 11 have then been renumbered 10, 11, and 12. The new Chapter 8 (“Cyclic codes”) presents a fairly complete treatment of the mathematical theory of cyclic codes, and their implementation with shift register circuits. It culminates with a discussion of the use of cyclic codes in burst error correction. The new Chapter 9 (“BCH, Reed-Solomon, and Related Codes”) is much like the old Chapter 8, except that increased emphasis has been placed on Reed-Solomon codes, reflecting their importance in practice. Both of the new chapters feature dozens of new problems.

This book is meant to be a self-contained introduction to the basic results in the theory of information and coding. It was written during 1972–1976, when I taught this subject at Caltech. About half my students were electrical engineering graduate students; the others were majoring in all sorts of other fields (mathematics, physics, biology, even one English major!). As a result the course was aimed at nonspecialists as well as specialists, and so is this book.

The book is in three parts: Introduction, Part one (Information Theory), and Part two (Coding Theory). It is essential to read the introduction first, because it gives an overview of the whole subject. In Part one, Chapter 1 is fundamental, but it is probably a mistake to read it first, since it is really just a collection of technical results about entropy, mutual information, and so forth. It is better regarded as a reference section, and should be consulted as necessary to understand Chapters 2–5. Chapter 6 is a survey of advanced results, and can be read independently. In Part two, Chapter 7 is basic and must be read before Chapters 8 and 9; but Chapter 10 is almost, and Chapter 11 is completely, independent from Chapter 7. Chapter 12 is another survey chapter independent of every thing else.

The problems at the end of the chapters are very important. They contain verification of many omitted details, as well as many important results not mentioned in the text.

Transmission of information is at the heart of what we call communication. As an area of concern it is so vast as to touch upon the preoccupations of philosophers and to give rise to a thriving technology.

We owe to the genius of Claude Shannon the recongnition that a large class of problems related to encoding, transmitting, and decoding informatio can be approached in a systematic and disciplined way: his classic paper of 1948 marks the birth of a new chapter of Mathematics.

In the past thirty years there has grown a staggering literature in this fledgling field, and some of its terminology even has become part of our daily language.

The present monograph (actually two monographs in one) is an excellent introduction to the two aspects of communication: coding and transmission.

The first (which is the subject of Part two) is an elegant illustration of the power and beauty of Algebra; the second belongs to Probability Theory which the chapter begun by Shannon enriched in novel and unexpected ways.

Introduction

In the previous chapter, we dealt with multirate systems in general, that is, systems in which more than one sampling rate coexist. Operations of decimation, interpolation, and sampling-rate changes were studied, as well as some filter design techniques using multirate concepts.

In a number of applications, it is necessary to split a digital signal into several frequency bands. After such decomposition, the signal is represented by more samples than in the original stage. However, we can attempt to decimate each band, ending up with a digital signal decomposed into several frequency bands without increasing the overall number of samples. The question is whether it is possible to exactly recover the original signal from the decimated bands. Systems which achieve this are generally called filter banks.

In this chapter, we first deal with filter banks, showing several ways in which a signal can be decomposed into critically decimated frequency bands, and recovered from them with minimum error. Later, wavelet transforms are considered. They are a relatively recent development of functional analysis that is generating great interest in the signal processing community, because of their ability to represent and analyze signals with varying time and frequency resolutions. Their digital implementation can be regarded as a special case of critically decimated filter banks. Finally, we provide a brief description of functions from the MATLAB Wavelet toolbox which are useful for wavelets and filter banks implementation.

Introduction

The theory of codes provides some jewels of combinatorics on words that we want to describe in this chapter.

A basic result is the defect theorem (Theorem 6.2.1), which states that if a set X of n words satisfies a nontrivial relation, then these words can be expressed simultaneously as products of at most n — 1 words. It is the starting point of the chapter. In Chapters 9 and 13, other defect properties are studied in different contexts.

A nontrivial relation is simply a finite word w which ambiguously factorizes over X. This means that X is not a code. The defect effect still holds if X is not an ω-code, i.e., if the nontrivial relation is an infinite, instead of a finite, word (Theorem 6.2.4).

The defect theorem implies several well-known properties on words that are recalled in this chapter. For instance, the fact that two words which commute are powers of the same word is a consequence. Another consequence is that a two-element code or more generally an elementary set is an co-code. The latter property appears to be a crucial step in one of the proofs of the DOL equivalence problem.

A remarkable phenomenon appears when, for a finite code X, neither the set X nor its reversal is an ω-code. In this case the defect property is stronger: the n elements of X can be expressed as products of at most n — 2 words (Theorem 6.3.4).

Introduction

In this chapter we shall be concerned with sesquipowers. Any nonempty word is a sesquipower of order 1. A word w is a sesquipower of order n if w = uvu, where u is a sesquipower of order n — 1. Sesquipowers have many interesting combinatorial properties which have applications in various domains. They can be defined by using bi-ideal sequences.

A finite or infinite sequence of words f1,…,fn,… is called a bi-ideal sequence if for all i > 0, fi is both a prefix and a suffix of fi+1 and, moreover, 2|fi| ≤ |fi+1|. A sesquipower of order n is then the nth term of a bi-ideal sequence. Bi-ideal sequences have been considered, with different names, by several authors in algebra and combinatorics (see Notes).

In Sections 4.2 and 4.3 we analyze some interesting combinatorial properties of bi-ideal sequences and the links existing between bi-ideal sequences, recurrence and n-divisions. From these results we will obtain in Section 4.4 an improvement (Theorem 4.4.5) of an important combinatorial theorem of Shirshov. We recall (see Lothaire 1983) that Shirshov's theorem states that for all positive integers p and n any sufficiently large word over a finite totally ordered alphabet will have a factor f which is a pth power or is n-divided, i.e., f can be factorized into nonempty blocks as f = x1 … xn with the property that all the words that one obtains by a nontrivial rearrangement of the blocks are lexicographically less than f.

This book originated from a training course for engineers at the research and development center of TELEBRAS, the former Brazilian telecommunications holding. That course was taught by the first author back in 1987, and its main goal was to present efficient digital filter design methods suitable for solving some of their engineering problems. Later on, this original text was used by the first author as the basic reference for the digital filters and digital signal processing courses of the Electrical Engineering Program at COPPE/Federal University of Rio de Janeiro.

For many years, former students asked why the original text was not transformed into a book, as it presented a very distinct view that they considered worth publishing. Among the numerous reasons not to attempt such task, we could mention that there were already a good number of well-written texts on the subject; also, after many years of teaching and researching on this topic, it seemed more interesting to follow other paths than the painful one of writing a book; finally, the original text was written in Portuguese and a mere translation of it into English would be a very tedious task.

In recent years, the second and third authors, who had attended the signal processing courses using the original material, were continuously giving new ideas on how to proceed. That was when we decided to go through the task of completing and updating the original text, turning it into a modern textbook.

Introduction

The notion of a dimension, when available, is a powerful mathematical tool in proving finiteness conditions in combinatorics. An example of this is Eilenberg's equality theorem, which provides an optimal criterion for the equality of two rational series over a (skew) field. In this example a problem on words, i.e., on free semigroups, is first transformed into a problem on vector spaces, and then it is solved using the dimension property of those algebraic structures. One can raise the natural question: do sets of words possess dimension properties of some kind?

We approach this problem through systems of equations in semigroups. As a starting point we recall the well-known defect theorem (see Chapter 6), which states that if a set of n words satisfies a nontrivial relation, then these words can be expressed simultaneously as products of at most n — 1 words. The defect effect can be seen as a weak dimension property of words. In order to analyze it further one can examine what happens when n words satisfy several independent relations, where independence is formalized as follows: a set E of relations on n words is independent, if E, viewed as a system of equations, does not contain a proper subset having the same solutions as E.

It is not difficult to see that a set of n words can satisfy two or more equations even in the case where the words cannot be expressed as products of fewer than n—1 words.

Introduction

This chapter deals with positional numeration systems. Numbers are seen as finite or infinite words over an alphabet of digits. A numeration system is defined by a pair composed of a base or a sequence of numbers, and of an alphabet of digits. In this chapter we study the representation of natural numbers, of real numbers and of complex numbers. We will present several generalizations of the usual notion of numeration system, which lead to interesting problems.

Properties of words representing numbers are well studied in number theory: the concepts of period, digit frequency, normality give rise to important results. Cantor sets can be defined by digital expansions.

In computer arithmetic, it is recognized that algorithmic possibilities depend on the representation of numbers. For instance, addition of two integers represented in the usual binary system, with digits 0 and 1, takes a time proportional to the size of the data. But if these numbers are represented with signed digits 0, 1, and —1, then addition can be realized in parallel in a time independent of the size of the data.

Since numbers are words, finite state automata are relevant tools to describe sets of number representations, and also to characterize the complexity of arithmetic operations. For instance, addition in the usual binary system is a function computable by a finite automaton, but multiplication is not.

Introduction

In practice, a digital signal processing system is implemented by software on a digital computer, either using a general-purpose digital signal processor, or using dedicated hardware for the given application. In either case, quantization errors are inherent due to the finite-precision arithmetic. These errors are of the following types:

• Errors due to the quantization of the input signals into a set of discrete levels, such as the ones introduced by the analog-to-digital converter.

• Errors in the frequency response of filters, or in transform coefficients, due to the finite-wordlength representation of multiplier constants.

• Errors made when internal data, like outputs of multipliers, are quantized before or after subsequent additions.

All these error forms depend on the type of arithmetic utilized in the implementation. If a digital signal processing routine is implemented on a general-purpose computer, since floating-point arithmetic is in general available, this type of arithmetic becomes the most natural choice. On the other hand, if the building block is implemented on special-purpose hardware, or a fixed-point digital signal processor, fixed-point arithmetic may be the best choice, because it is less costly in terms of hardware and simpler to design. A fixed-point implementation usually implies a lot of savings in terms of chip area as well.

For a given application, the quantization effects are key factors to be considered when assessing the performance of a digital signal processing algorithm.

Introduction

A seminal result of Makanin 1977 states that the existential theory of equations over free monoids is decidable. Makanin achieved this result by presenting an algorithm which solves the satisfiability problem for word equations with constants. The satisfiability problem is usually stated for a single equation, but this is no loss of generality.

This chapter provides a self-contained presentation of Makanin's result. The presentation has been inspired by Schulz 1992a. In particular, we show the result of Makanin in a more general setting, due to Schulz, by allowing that the problem instance is given by a word equation L = R together with a list of rational languages Lx ⊆ A*, where x ∈ Ω denotes an unknown and A is the alphabet of constants. We will see that it is decidable whether or not there exists a solution σ:Ω → A* which, in addition to σ(L) = σ(R), satisfies the rational constraints σ(x)∈ Lx for all x ∈ Ω. Using an algebraic viewpoint, rational constraints mean to work over some finite semigroup, but we do not need any deep result from the theory of finite semigroups. The presence of rational constraints does not make the proof of Makanin's result much harder; however, the more general form is attractive for various applications.

In the following we explain the outline of the chapter; for some background information and more comments on recent developments we refer to the Notes.