The underlying purpose of any communication system is to transmit information. But what exactly is information? How is it measured? Are there limits to the amount of data that can be sent over a channel, even when all the parts of the system are operating at their best? This chapter addresses these fundamental questions using the ideas of Claude Shannon (1916–2001), who defined a measure of information in terms of bits. The number of bits per second that can be transmitted over the channel (taking into account its bandwidth, the power of the signal, and the noise) is called the bit rate, and can be used to define the capacity of the channel.

Unfortunately, Shannon's results do not give a recipe for how to construct a system that achieves the optimal bit rate. Earlier chapters have highlighted several problems that can arise in communication systems (including synchronization errors such as intersymbol interference). This chapter assumes that all of these are perfectly mitigated. Thus, in Figure 14.1, the inner parts of the communication system are assumed to be ideal, except for the presence of channel noise. Even so, most systems still fall far short of the optimal performance promised by Shannon.

There are two problems. First, most messages that people want to send are redundant, and the redundancy squanders the capacity of the channel. A solution is to preprocess the message so as to remove the redundancies.