This chapter treats source coding, with a focus on the basics of lossless coding of image and graphic data. Data compression and image coding are widely used today when transmitting and storing data. Examples are transmission of images on the Internet and storage of images, audio files, and video on CDs or DVDs.

In Chapter 1 source coding of discrete sources generating independent symbols was introduced, specifically in the form of Huffman codes and arithmetic coding. In Chapter 2 some source models were considered. In this chapter we treat source coding of statistically dependent data. Furthermore, for real-world data the model is unknown, so this chapter also deals with issues concerning efficiently estimating model parameters and possibly the model order for source coding. Actually, the standard notion of a model may be abandoned altogether and the goal may be to code a single individual sequence, i.e. independently of other sequences and their statistics.

First we consider how to code the source data in relation to the models previously defined. In general terms, the model is represented by a context defining a dependency on the past data.

We shall refer to the approach as context-adaptive coding. The context may readily be defined in two or more dimensions. Doing so naturally leads to source coding of images. This forms the basis of the binary image-coding standards originally developed for fax coding and today used in a number of applications including the widely used PDF (Portable Document Format) format.

Introduction

Reed–Solomon codes are error-correcting codes defined over large alphabets. They were among the early constructions of good codes (1959), and are now one of the most important classes of error-correcting codes for many applications. At the same time these codes constitute a natural starting point for studying algebraic coding theory, i.e. methods of correcting errors by solving systems of equations.

Finite fields

To describe the codes and the decoding methods, the symbol alphabet is given a structure that allows computations similar to those used for rational numbers. The structure is that of a field. In a field there are two compositions, addition and multiplication, satisfying the usual associative and distributive rules. The compositions have neutral elements 0 and 1, every element has an additive inverse (a negative), and nonzero elements have a multiplicative inverse.

Well-known examples of fields include the rational, the real, and the complex numbers. The integers are not a field because only ±1 have multiplicative inverses. However, there are also fields with a finite number of elements, and we actually used the binary field in the previous chapter to construct binary codes. A finite field with q elements is referred to as F(q). Having the alphabet given this structure allows us to use concepts of matrices, vector spaces, and polynomials, which concepts are essential to the construction of codes and decoding algorithms.

The simplest examples of finite fields are sets of integers, [0, 1, 2, …, p − 1], with addition and multiplication modulo p.

Shannon's paper from 1948, which presented information theory in a way that already included most of the fundamental concepts, helped bring about a fundamental change in electronic communication. Today digital formats have almost entirely replaced earlier forms of signaling, and coding has become a universal feature of communication and data storage.

Information theory has developed into a sophisticated mathematical discipline, and at the same time it has almost disappeared from textbooks on digital communication and coding methods. This book is a result of the authors' desire to teach information theory to students of electrical engineering and computer science, and to demonstrate its continued relevance. We have also chosen to mix source coding and error-correcting codes, since both are components of the systems we focus on.

Early attempts to apply information-theory concepts to a broad range of subjects met with limited success. The development of the subject has mostly been fuelled by the advances in design of transmitters and receivers for digital transmission such as modern design and other related applications. However, more recently the extensive use of digitized graphics has made possible a vast range of applications, and we have chosen to draw most of our examples from this area.

The first five chapters of the book can be used for a one-semester course at the advanced-undergraduate or beginning-graduate level. Chapter 6 serves as a transition from the basic subjects to the more complex environments covered by current standards.

This book has been written primarily as an introductory text for graduate students interested in algebraic graph theory and related areas. It is also intended to be of use to mathematicians working in graph theory and combinatorics, to chemists who are interested in quantum chemistry, and in part to physicists, computer scientists and electrical engineers using the theory of graph spectra in their work. The book is almost entirely self-contained; only a little familiarity with graph theory and linear algebra is assumed.

In addition to more recent developments, the book includes an up-to-date treatment of most of the topics covered in Spectra of Graphs by D. Cvetković, M. Doob and H. Sachs [CvDSa], where spectral graph theory was characterized as follows:

In this chapter we discuss several instances of the following problem:

Given the spectrum, or some spectral characteristics of a graph, determine all graphs from a given class of graphs having the given spectrum, or the given spectral characteristics.

In some cases, the solution of such a problem can provide a characterization of a graph up to isomorphism (see Section 4.1). In other cases we can deduce structural details (see also Chapter 3). Non-isomorphic graphs with the same spectrum can arise as sporadic exceptions to characterization theorems or from general constructions. Accordingly, Section 4.2 is devoted to cospectral graphs; we include comments on their relation to the graph isomorphism problem, together with various examples and statistics. We also discuss the use of other graph invariants to strengthen distinguishing properties. In particular, in Section 4.3, we consider characterizations of graphs by eigenvalues and angles.

Spectral characterizations of certain classes of graphs

In this section we investigate graphs that are determined by their spectra. The three subsections are devoted to (i) elementary characterizations, (ii) characterizations of graphs with least eigenvalue -2, and (iii) characterizations of special types. In the case of (i), a graph is uniquely reconstructed from its spectrum, while in cases (ii) and (iii) various exceptions occur due to the existence of cospectral graphs.

Introduction

In the previous chapters we have presented the basic concepts of information theory, source coding, and channel coding. In Chapters 1−3 we have followed traditional information-theory terminology in distinguishing between sources, which produce information, and channels, which are used for transmission (or storage) of information. In many current forms of communication information passes through multiple steps of processing and assembly into composite structures. Since in such cases it can be difficult to make a distinction between sources and channels, we use the neutral term information medium to refer to structures, whether physical or conceptual, that are used for storing and delivering information. In short form the terms medium and media are used. The diverse forms of electronic media may serve as examples of the composite objects we have in mind and the range of meanings of the term. As a specific case one can think of a two-dimensional (2-D) barcode printed on an advertising display so that it can be read by a cell-phone camera and used as a way of accessing the website for the business.

In the case of highly structured composite objects we shall make no attempt to directly apply concepts like entropy or capacity. Instead we limit our applications of information theory tools to more well-defined components of such objects in digital form. The present chapter discusses how 2-D media can be described in the light of these concepts, and how the various tools can be used in such applications.

Introduction

This chapter describes fields of discrete symbols over finite alphabets A (with ∣A∣ symbols). Such fields can serve as models of graphics media or of other forms of two-dimensional (2-D) storage medium. By addressing the storage medium as a surface on which an image array may be written, the density may be increased. Besides optical storage, new storage techniques based on holography and nano-technology have been demonstrated. Two-dimensional barcodes have also been designed for coding small messages, but increasing the capacity compared with conventional barcodes, which carry information in one dimension only. The Datamatrix is an example of such a code. We focus on 2-D constrained coding for storage applications, but the models presented are more general and they may be used for other applications as well.

Two-dimensional fields with a finite constraint

The symbols are placed in a regular grid (commonly referred to as a lattice) and indexed a(i, j), where i, j are integers. The first index indicates rows and the symbols are spaced at equal intervals within the row. We usually consider rectangular grids on which the symbols are aligned in columns indicated by the second index. However, the symbols in row i + 1 can be shifted a fraction of a symbol interval relative to row i. In particular, a shift of half a symbol interval gives a hexagonal grid. The mutual dependency of symbols is often described in terms of neighbors.

This chapter is devoted to results which did not fit readily into earlier chapters. Section 8.1 is concerned with the behaviour of certain eigenvalues when a graph is modified, and with further bounds on the index of a graph. Section 8.2 deals with relations between the structure of a graph and the sign pattern of certain eigenvectors. Results from these first two sections enable us to give a general description of the connected graphs having maximal index or minimal least eigenvalue among those with a given number of vertices and edges. In Section 8.3 we discuss the reconstruction of the characteristic polynomial of a graph from the characteristic polynomials of its vertex-deleted subgraphs. In Section 8.4 we review what is known about graphs whose eigenvalues are integers.

More on graph eigenvalues

In this section we revisit two topics which have featured in previous chapters. The first topic concerns the relation between the spectrum of a graph G and the spectrum of some modification G′ of G. When the modification arises as a small structural alteration (such as the deletion or addition of an edge or vertex), the eigenvalues of G′ are generally small perturbations of those of G, and we say that G′ is a perturbation of G. In Subsection 8.1.1, we use algebraic arguments to establish some general rules which determine whether certain eigenvalues increase or decrease under particular graph perturbations.

Introduction

In Chapter 3 we introduced linear error-correcting codes, and discussed how long codes can be obtained from short codes by the product construction. Reed–Solomon (RS) codes over larger alphabets were presented in Chapter 4, where we discussed the Datamatrix format as an example of how RS codes can be used to protect binary data. In this chapter we continue the analysis of these themes insofar as they relate to error correction in video and other applications.

We describe constructions of long error-correcting codes that are suitable for encoding of two-dimensional (2-D) media. The size of the pages makes it desirable to have relatively long codes, and 2-D constructions are often used to obtain long codes. However, there is not necessarily a link between the 2-D structure of the media and the code. In the last section we suggest that very long codes could be given a structure that would allow the 2-D structures to be connected, and that such a code could be partially decoded in cases in which only a subset of the data has to be retrieved.

Binary images of Reed–Solomon codes

The RS codes that are used in applications are always based on the fields F(2m). Here m = 8 is the traditional choice, but future applications are likely to use larger fields. As discussed in Section 4.6, the field is often constructed from a so-called primitive polynomial, p(z).

The class of convolutional codes was invented by Elias in 1955 and has been widely in use for wireless, space, and broadcast communications since about 1970. Their popularity stems from the relative ease with which the maximum-likelihood sequence decoder may be implemented and from their effectiveness when concatenated with a Reed–Solomon code. Since the early 1990s, they have enjoyed new respect and popularity due to the efficacy of concatenations of multiple convolutional codes in turbo (-like) codes.

In this chapter, we introduce the class of convolutional codes. We first present algebraic descriptions of convolutional codes, which were pioneered by Forney and extended by Johannesson and Zigangirov. We also discuss various encoder realizations and matrix representations of convolutional codes. We then discuss a graphical (trellis) representation, which aids two optimal decoding algorithms, the Viterbi algorithm and the BCJR algorithm. Finally, we present techniques for bounding the performance of convolutional codes based on the technique of Viterbi. Our discussion in this chapter will include only binary convolutional codes. Extensions of the various details to the nonbinary case are straightforward. See for additional information on convolutional codes, including tables of optimal codes.

The Convolutional Code Archetype

Convolutional codes are linear codes with a very distinct algebraic structure. While they can be utilized in block-oriented (packet-based) situations, their encoders are frequently described as stream-oriented. That is, in contrast with a block code, whose encoder assigns an n-bit codeword to each block of k data bits, a convolutional encoder assigns code bits to an incoming information bit stream continuously, in a stream-oriented fashion.

Digital Data Communication and Storage

Digital communication systems are ubiquitous in our daily lives. The most obvious examples include cell phones, digital television via satellite or cable, digital radio, wireless internet connections via Wi-Fi and WiMax, and wired internet connection via cable modem. Additional examples include digital data-storage devices, including magnetic (“hard”) disk drives, magnetic tape drives, optical disk drives (e.g., CD, DVD, blu-ray), and flash drives. In the case of data-storage, information is communicated from one point in time to another rather than one point in space to another. Each of these examples, while widely different in implementation details, generally fits into a common digital communication framework first established by C. Shannon in his 1948 seminal paper, A Mathematical Theory of Communication. This framework is depicted in Figure 1.1, whose various components are described as follows.