In this chapter, we consider block codes with a certain structure, which are defined over alphabets that are fields. Specifically, these codes, which we call linear codes, form linear spaces over their alphabets. We associate two objects with these codes: a generator matrix and a parity-check matrix. The first matrix is used as a compact representation of the code and also as a means for efficient encoding. The parity-check matrix will be used as a tool for analyzing the code (e.g., for computing its minimum distance) and will also be part of the general framework that we develop for the decoding of linear codes.

As examples of linear codes, we will mention the repetition code, the parity code, and the Hamming code with its extensions. Owing to their structure, linear codes are by far the predominant block codes in practical usage, and virtually all codes that will be considered in subsequent chapters are linear.

Definition

Denote by GF(q) a finite (Galois) field of size q. For example, if q is a prime, the field GF(q) coincides with the ring of integer residues modulo q, also denoted by ℤq. We will see more constructions of finite fields in Chapter 3.

An (n, M, d) code C over a field F = GF(q) is called linear if C is a linear subspace of Fn over F; namely, for every two codewords c1, c2 ∈ C and two scalars a1, a2 ∈ F we have a1c1 + a2c2 ∈ C.

For most of this chapter, we deviate from our study of codes to become acquainted with the algebraic concept of finite fields. These objects will serve as our primary tool for constructing codes in upcoming chapters. As a motivating example, we present at the end of this chapter a construction of a double-error-correcting binary code, whose description and analysis make use of finite fields. This construction will turn out to be a special case of a more general family of codes, to be discussed in Section 5.5.

Among the properties of finite fields that we cover in this chapter, we show that the multiplicative group of a finite field is cyclic; this property, in turn, suggests a method for implementing the arithmetic operations in finite fields of moderate sizes through look-up tables, akin to logarithm tables. We also prove that the size of any finite field must be a power of a prime and that this necessary condition is also sufficient, that is, every power of a prime is a size of some finite field. The practical significance of the latter property is manifested particularly through the special case of the prime 2, since in most coding applications, the data is sub-divided into symbols—e.g., bytes—that belong to alphabets whose sizes are powers of 2.

Prime fields

For a prime p, we let GF(p) (Galois field of size p) denote the ring of integer residues modulo p (this ring is also denoted by ℤp).

In Chapter 1, we introduced the concept of a block code with a certain application in mind: the codewords in the code serve as the set of images of the channel encoder. The encoder maps a message into a codeword which, in turn, is transmitted through the channel, and the receiver then decodes that message (possibly incorrectly) from the word that is read at the output of the channel. In this model, the encoding of a message is independent of any previous or future transmissions—and so is the decoding.

In this chapter, we consider a more general coding model, where the encoding and the decoding are context-dependent. The encoder may now be in one of finitely many states, which contain information about the history of the transmission. Such a finite-state encoder still maps messages to codewords, yet the mapping depends on the state which the encoder is currently in, and that state is updated during each message transmission. Finite-state encoders will be specified through directed graphs, where the vertices stand for the states and the edges define the allowed transitions between states. The mapping from messages to codewords will be determined by the edge names and by labels that we assign to the edges.

The chapter is organized as follows. We first review several concepts from the theory of directed graphs. We then introduce the notion of trellis codes, which can be viewed as the state-dependent counterpart of block codes: the elements of a trellis code form the set of images of a finite-state encoder.

In this chapter, we introduce the model of a communication system, as originally proposed by Claude E. Shannon in 1948. We will then focus on the channel portion of the system and define the concept of a probabilistic channel, along with models of an encoder and a decoder for the channel. As our primary example of a probabilistic channel—here, as well as in subsequent chapters—we will introduce the memoryless q-ary symmetric channel, with the binary case as the prevailing instance used in many practical applications. For q = 2 (the binary case), we quote two key results in information theory. The first result is a coding theorem, which states that information through the channel can be transmitted with an arbitrarily small probability of decoding error, as long as the transmission rate is below a quantity referred to as the capacity of the channel. The second result is a converse coding theorem, which states that operating at rates above the capacity necessarily implies unreliable transmission.

In the remaining part of the chapter, we shift to a combinatorial setting and characterize error events that can occur in channels such as the q-ary symmetric channel, and can always be corrected by suitably selected encoders and decoders. We exhibit the trade-off between error correction and error detection: while an error-detecting decoder provides less information to the receiver, it allows us to handle twice as many errors.

Generalized Reed–Solomon (in short, GRS) codes and their derivative codes are probably the most extensively-used codes in practice. This may be attributed to several advantages that these codes have. First, GRS codes are maximum distance separable, namely, they attain the Singleton bound. Secondly, being linear codes, they can be encoded efficiently; furthermore, as we see in this chapter, encoders for the sub-class of conventional Reed–Solomon (in short, RS) codes can be implemented by particularly simple hardware circuits. Thirdly, we will show in Chapters 6 and 9 that GRS codes can also be decoded efficiently.

As their names suggest, RS codes pre-dated their GRS counterparts. Nevertheless, we find it more convenient herein to define GRS codes first and prove several properties thereof; we then present RS codes as a special class of GRS codes.

One seeming limitation of GRS codes is the fact that their length is bounded from above by the size of the field over which they are defined. This could imply that these codes might be useful only when the application calls for a field size that is relatively large, e.g., when the field is GF(28) and the symbols are bytes. Still, we show that GRS codes can serve as building blocks to derive new codes over small alphabets as well. We present two methods for doing so. The first technique is called concatenation and is based on two stages of encoding, the first of which is a GRS encoder.

In Section 4.1, we defined MDS codes as codes that attain the Singleton bound. This chapter further explores their properties. The main topic to be covered here is the problem of determining for a given positive integer k and a finite field F = GF(q), the largest length of any linear MDS code of dimension k over F. This problem is still one of the most notable unresolved questions in coding theory, as well as in other disciplines, such as combinatorics and projective geometry over finite fields. The problem has been settled so far only for a limited range of dimensions k. Based on the partial proved evidence, it is believed that within the range 2 ≤ k ≤ q−1 (and with two exceptions for even values of q), linear [n, k] MDS codes exist over F if and only if n ≤ q+1. One method for proving this conjecture for certain values of k is based on identifying a range of parameters for which MDS codes are necessarily extended GRS codes. To this end, we will devote a part of this chapter to reviewing some of the properties of GRS codes and their extensions.

Definition revisited

We start by recalling the Singleton bound from Section 4.1. We will prove it again here, using a certain characterization of the minimum distance of a code, as provided by the following lemma.

The study of error-correcting codes concentrates primarily on codes in the Hamming metric. Such codes are designed to correct a prescribed number of errors, where by an error we mean a change of an entry in the transmitted codeword, irrespective of the (nonzero) error value. The assignment of the same weight to each nonzero error value is reflected also in the model of the q-ary symmetric channel, where all nonzero error values occur with the same probability.

In this chapter, we consider codes in the Lee metric. This metric is defined over the ring of integer residues modulo q and it corresponds to an error model where a change of an entry in a codeword by ±1 is counted as one error. This type of errors is found in noisy channels that use phase-shift keying (PSK) modulation, or in channels that are susceptible to synchronization errors.

Our focus herein will be on GRS codes and alternant codes: we first study their distance properties in the Lee metric, and then present an efficient decoding algorithm for these codes, which corrects any error pattern whose Lee weight is less than half the designed minimum Lee distance of the code.

We also describe another family of codes in the Lee metric, due to Berlekamp. For certain parameters, these codes are shown to be perfect in that metric; namely, they attain the Lee-metric analog of the sphere-packing bound.

In this chapter, we continue the discussion on concatenated codes, which was initiated in Section 5.4. The main message to be conveyed in this chapter is that by using concatenation, one can obtain codes with favorable asymptotic performance—in a sense to be quantified more precisely—while the complexity of constructing these codes and decoding them grows polynomially with the code length.

We first present a decoding algorithm for concatenated codes, due to Forney. This algorithm, referred to as a generalized minimum distance (in short, GMD) decoder, corrects any error pattern whose Hamming weight is less than half the product of the minimum distances of the inner and outer codes (we recall that this product is a lower bound on the minimum distance of the respective concatenated code). A GMD decoder consists of a nearest-codeword decoder for the inner code, and a combined error–erasure decoder for the outer code. It then enumerates over a threshold value, marking the output of the inner decoder as erasure if that decoder returns an inner codeword whose Hamming distance from the respective received sub-word equals or exceeds that threshold. We show that under our assumption on the overall Hamming weight of the error word, there is at least one threshold for which the outer decoder recovers the correct codeword. If the outer code is taken as a GRS code, then a GMD decoder has an implementation with time complexity that is at most quadratic in the length of the concatenated code.

In Chapter 6, we introduced an efficient decoder for GRS codes, yet we assumed that the number of errors does not exceed ⌊(d−1)/2⌋, where d is the minimum distance of the code. In this chapter, we present a decoding algorithm for GRS codes, due to Guruswami and Sudan, where this upper limit is relaxed.

When a decoder attempts to correct more than ⌊(d−1)/2⌋ errors, the decoding may sometimes not be unique; therefore, we consider here a more general model of decoding, allowing the decoder to return a list of codewords, rather than just one codeword. In this more general setting, a decoding is considered successful if the computed list of codewords contains the transmitted codeword. The (maximum) number of errors that a list decoder can successfully handle is called the decoding radius of the decoder.

The approach that leads to the Guruswami–Sudan list decoder is quite different from the GRS decoder which was introduced in Chapter 6. Specifically, the first decoding step now computes from the received word a certain bivariate polynomial Q(x, z) over the ground field, F, of the code. Regarding Q(x, z) as a univariate polynomial in the indeterminate z over the ring F[x], a second decoding step computes the roots of Q(x, z) in F[x]; these roots are then mapped to codewords which, in turn, form the returned list.

In this chapter, we establish conditions on the parameters of codes. In the first part of the chapter, we present bounds that relate between the length n, size M, minimum distance d, and the alphabet size q of a code. Two of these bounds—the Singleton bound and the sphere-packing bound—imply necessary conditions on the values of n, M, d, and q, so that a code with the respective parameters indeed exists. We also exhibit families of codes that attain each of these bounds. The third bound which we present—the Gilbert–Varshamov bound—is an existence result: it states that there exists a linear [n, k, d] code over GF(q) whenever n, k, d, and q satisfy a certain inequality. Additional bounds are included in the problems at the end of this chapter. We end this part of the chapter by introducing another example of necessary conditions on codes—now in the form of MacWilliams' identities, which relate the distribution of the Hamming weights of the codewords in a linear code with the respective distribution in the dual code.

The second part of this chapter deals with asymptotic bounds, which relate the rate of a code to its relative minimum distance δ = d/n and its alphabet size, as the code length n tends to infinity.

In the third part of the chapter, we shift from the combinatorial setting of (n, M, d) codes to the probabilistic framework of the memoryless q-ary symmetric channel.

Concatenated codes are examples of compound constructions, as they are obtained by combining two codes—an inner code and an outer code—with a certain relationship between their parameters. This chapter presents another compound construction, now combining an (inner) code C over some alphabet F with an undirected graph G = (V, E). In the resulting construction, which we refer to as a graph code and denote by (G, C), the degrees of all the vertices in G need to be equal to the length of C, and the code (G, C) consists of all the words of length ∣E∣ over F in which certain sub-words, whose locations are defined by G, belong to C. The main result to be obtained in this chapter is that there exist explicit constructions of graph codes that can be decoded in linear-time complexity, such that the code rate is bounded away from zero, and so is the fraction of symbols that are allowed to be in error.

We start this chapter by reviewing several concepts from graph theory. We then focus on regular graphs, i.e., graphs in which all vertices have the same degree. We will be interested in the expansion properties of such graphs; namely, how the number of outgoing edges from a given set of vertices depends on the size of this set.

This book has evolved from lecture notes that I have been using for an introductory course on coding theory in the Computer Science Department at Technion. The course deals with the basics of the theory of error-correcting codes, and is intended for students in the graduate and upper-undergraduate levels from Computer Science, Electrical Engineering, and Mathematics. The material of this course is covered by the first eight chapters of this book, excluding Sections 4.4–4.7 and 6.7. Prior knowledge in probability, linear algebra, modern algebra, and discrete mathematics is assumed. On the other hand, all the required material on finite fields is an integral part of the course. The remaining parts of this book can form the basis of a second, advanced-level course.

There are many textbooks on the subject of error-correcting codes, some of which are listed next: Berlekamp [36], Blahut [46], Blake and Mullin [49], Lin and Costello [230], MacWilliams and Sloane [249], McEliece [259], Peterson and Weldon [278], and Pless [280]. These are excellent sources, which served as very useful references when compiling this book. The two volumes of the Handbook of Coding Theory [281] form an extensive encyclopedic collection of what is known in the area of coding theory.

One feature that probably distinguishes this book from most other classical textbooks on coding theory is that generalized Reed—Solomon (GRS) codes are treated before BCH codes—and even before cyclic codes.