Introduction

When we study computability, we study problems in an abstract sense. For example,

1. a. Addition is the problem of returning a third number that is the sum of two given numbers.

2. b. Travelling Salesman problem (TSP) is one in which a list of distances between some number of cities are given and the person is asked to find the shortest route so that he visits each city once and returns to the start.

3. c. Halting Problem (HP) is one in which a program is given some appropriate input and it needs to be decided whether the program, when run on that input, loops forever or halts.

Problems (b) and (c) are the two problems of computer interest.

In both these problems, the statement of the problem does not give the actual values needed to provide the result but just tells what kind of objects they are. A set of actual values for a problem is called an instance of the problem (by this terminology, all homework problems done by school students are considered as instances of problems).

In all these problems an instance is required that is the input and the relationship between the input and the output is to be understood. In order to solve these problems, there are certain things one should know.

1. Can it be solved algorithmically – is there any definite procedure that solves any instance of the problem in a finite amount of time? In other words, is it computable? Not all problems are computable.

Introduction

In any language such as English, Hindi or Sanskrit, words can be combined in several ways. Naturally, some combinations form valid sentences, while others do not. The validity of a sentence is determined by the grammar of a language, which comprises a set of rules. For instance, ‘The boy prepares tea quickly’, although meaningless, is a perfectly legal sentence. In other words, the sentences in a language may be nonsensical, but they must obey the rules of grammar. The discussion in this chapter deals with only the syntax of sentence (the way the words are combined) and not with the semantics of sentences (meaning).

In the previous chapters, two different (though equivalent) methods: finite automata and regular expressions were introduced for describing languages. These methods have their own limitations in the sense that some simple languages, such as {0n1n|n ≥ 0}, cannot be described by these methods. Formal languages and grammars are widely used in connection with programming languages. During programming, we proceed with an intuitive knowledge of the languages, which leads to errors. Therefore, a precise description of the language is needed, at almost every step, which helps to understand the syntax diagrams found in programming texts. Among the ways in which programming languages can be defined precisely, grammars or context-free grammars are most widely used. This method happens to be a very powerful method and such grammars can describe certain features which have a recursive structure. Context-free-grammars (CFG) were first used in the study of human languages.

Introduction

A language is a system of signs used to communicate information to others. However, the language of computation is a combination of both english and mathematics. Fundamentally, a computer is a symbol manipulator, in the sense, that it takes sequences of symbols as inputs and manipulates them as per the program specifications. These symbols are precise and unambiguous, unlike the language of humans. The first step in communicating a problem to a machine is the design of a proper language of computation and this is the fundamental object of computability.

While discussing about languages, it is important to note two cases:

1. a. During the evaluation of an input expression (e.g.: calculator), the language of arithmetic expression handles both the input and the output communication.

2. b. In the case of web form, the language simply describes all the legitimate inputs to the field and the output is simply a binary value-Yes or No.

Thus, the problems belonging to case (a), have both input and an output language, while, those belonging to case (b), have only an input language. It is of interest to note that, from the perspective of theory of computation, any type of problem can be expressed in terms of a language recognition.

Since a language is a medium of communication, it should be given some meaning (i.e. its semantics). But much of the manipulation of symbols, strings and language could be done effectively without understanding their semantics and this is the subject of this section. In other words, the mathematical study of the theory of computation begins with the understanding of the mathematics of symbols and strings.

Introduction

Let us now return to our original problem from Chapter 2: receiver design. Our ultimate goal is to recover (in an optimal manner) the transmitted information bits b from the received waveform r(t). In this chapter, we will formulate an inference problem that will enable us to achieve this task. In the first stage the received waveform is converted into a suitable observation y.We then create a factor graph of the distribution p(B, Y = y|M). This factor graph will contain three important nodes, expressing the relationship between information bits and coded bits (the decoding node), between coded bits and coded symbols (the demapping node), and between coded symbols and the observation (the equalization node). Correspondingly, the inference problem breaks down into three sub-problems: decoding, demapping, and equalization.

Decoding will be covered in Chapter 8, where we will describe some state-of-the-art iterative decoding take over schemes, including turbo codes, RA codes, and LDPC codes. The demapping problem will be considered in Chapter 9 for two common modulation techniques: bit-interleaved coded modulation and trellis-coded modulation. Equalization highly depends on the specific digital communication scheme. In Chapter 10, we will derive several general-purpose equalization strategies. In the three subsequent chapters, we will then show how these general-purpose strategies can be applied to the digital communication schemes from Chapter 2. In Chapter 11 the focus is on single-user, single-antenna communication.

Motivation

Claude E. Shannon was one of the great minds of the twentieth century. In the 1940s, he almost single-handedly created the field of information theory and gave the world a new way to look at information and communication. The channel-coding theorem, where he proved the existence of good error-correcting codes to transmit information at any rate below capacity with an arbitrarily small probability of error, was one of his fundamental contributions. Unfortunately, Shannon never described how to construct these codes. Ever since his 1948 landmark paper “A mathematical theory of communication” [1], the channel-coding theorem has tantalized researchers worldwide in their quest for the ultimate error-correcting code. After more than forty years, state-of-the-art errorcorrecting codes were still disappointingly far away from Shannon's theoretical capacity bound. No drastic improvement seemed to be forthcoming, and researchers were considering a more practical capacity benchmark, the cut-off rate [2], which could be achieved by practical codes.

In 1993, two until then little-known French researchers from the ENST in Bretagne, Claude Berrou and Alain Glavieux, claimed to have discovered a new type of code, which operated very close to Shannon capacity with reasonable decoding complexity. The decoding process consisted of two decoders passing information back and forth, giving rise to the name “turbo code.” They first presented their results at the IEEE International Conference on Communications in Geneva, Switzerland [3]. Quite understandably, they were met with a certain amount of skepticism by the traditional coding community. Only when their findings were reproduced by other labs did the turbo idea really take off.

Introduction

Before we can even consider designing iterative receivers, we have a lot of ground to cover. The ultimate goal of the receiver is to recover optimally the sequence of information bits that was sent by the transmitter. Since these bits were random in the first place, and they are affected by random noise at the receiver, we need to understand and quantify this randomness, and incorporate it into the design of our receiver.We need to specify what it means to recover optimally the information bits from the received signal. Optimal in what sense? To answer this problem, we call on estimation theory, which will allow us to formulate a suitable optimization problem, the solution of which will give our desired optimal data recovery. Unfortunately, as is common to most optimization problems of any practical relevance, the issue of finding a closed-form solution is intractable. There exists a set of tools that can solve these problems approximately by means of sampling. They go under the name of Monte Carlo (MC) techniques and can help in describing difficult distributions, and in obtaining their characteristics, by means of a list of samples. These MC methods will turn out to be very useful in the factor-graph framework.

This chapter is organized as follows.

• We will start with the basics of Bayesian estimation theory in Section 3.2, covering some important estimators both for discrete and for continuous parameters.

• In Section 3.3, we will provide a brief introduction to MC techniques, including particle representations of distributions, importance sampling, and Markov-chain Monte Carlo (MCMC) methods.

Introduction

As with any good story, it is best to start at the very beginning. Digital communication deals with the transmission of binary information (obtained as the output of a source encoder) from a transmitter to a receiver. The transmitter converts the binary information to an analog waveform and sends this waveform over a physical medium, such as a wire or open space, which we will call the channel. As we shall see, the channel modifies the waveform in several ways. At the receiver, this modified waveform is further corrupted due to thermal noise. Not only does the receiver have to recover the original binary information, but also it must deal with channel effects, thermal noise, and synchronization issues. All in all, the receiver has the bad end of the deal in digital communications. For this reason, this book deals mainly with receiver design, and only to a very small extent with the transmitter.

In this chapter we will describe several digital transmission schemes, detailing how binary information is converted into a waveform at the transmitter side, and how a corrupted version of this waveform arrives at the receiver side. Right now, our focus is not on how the corresponding receivers should be designed. Since there is a myriad of digital transmission schemes, we are obliged to limit ourselves to some of the most important ones.

In early 2002, I was absent-mindedly surfing the Internet, vaguely looking for a tutorial on turbo codes. My PhD advisor at Ghent University, Marc Moeneclaey, thought it wise for his new students to become familiar with these powerful error-correcting codes. Although I finally settled on W. E. Ryan's “A turbo code tutorial,” my search led me (serendipitously?) to the PhD thesis of NiclasWiberg. This thesis shows how to describe codes by means of a (factor) graph, and how to decode them by passing messages on this graph. Although interesting, the idea seemed a bit far-fetched, and I didn't fully appreciate or understand its significance. Nevertheless, Wiberg's thesis stayed in the back of my mind (or at least, I'd like to think so now).

During 2002 and 2003, I worked mainly on synchronization and estimation algorithms for turbo and LDPC codes. A colleague of mine, Justin Dauwels, who was at that time a PhD student of Andy Loeliger at the ETH in Zürich, had developed a remarkable synchronization algorithm for LDPC codes, based on Wiberg's factor graphs. Justin was interested in comparing his synchronization algorithm with ours, and became a visiting researcher in our lab for the first two months of 2004. I fondly remember many hours spent in the department lunchroom, with Justin (painstakingly) explaining the intricacies of factor graphs to me. His discussions motivated me to re-write my source code for decoding turbo and LDPC codes using the factor-graph framework.

Introduction

Error-correcting codes are a way to protect a binary information sequence against adverse channel effects by adding a certain amount of redundancy. This is known as encoding. The receiver can then try to recover the original binary information sequence, using a decoder. The field of coding theory deals with developing and analyzing codes and decoding algorithms. Although coding theory is fairly abstract and generally involves a great deal of math, our knowledge of factor graphs will allow us to derive decoding algorithms without delving too deep. As we will see, using factor graphs, decoding becomes a fairly straightforward matter. In contrast to conventional decoding algorithms, our notation will be the same for all types of codes, which makes it easier to understand and interpret the algorithms.

In this chapter, we will deal with four types of error-correcting codes: repeat–accumulate (RA) codes, low-density parity-check (LDPC) codes, convolutional codes, and turbo codes. Repeat–accumulate codes were introduced in 1998 as a type of toy code. Later they turned out to have a great deal of practical importance [81]. We then move on to the LDPC codes, which were invented by Gallager in 1963 [50], and reintroduced in the early 1990s by MacKay [82]. Both types of codes can easily be cast into factor graphs; these factor graphs turn out to have cycles, leading to iterative decoding algorithms. Convolutional codes, on the other hand, are based on state-space models and thus lead to cycle-free factor graphs [83].