The aim in this first chapter is to represent a message in as efficient or economical a way as possible, subject to the requirements of the devices that are to deal with it. For instance, computer memory stores information in binary form, essentially as strings of 0s and 1s. Everyone knows that English text contains far fewer letters q or j than e or t. So it is common sense to represent e and t in binary by shorter strings than are used for q and j. It is that common-sense idea that we shall elaborate in this chapter.

We do not consider at this stage any devices that corrupt messages or data. There is no error creation, so no need for error detection or correction. We are thus doing noiseless coding, and decoding. In later chapters we meet ‘noisy’ channels, that introduce occasional errors into messages, and will consider how to protect our messages against them. This will not make what we do in this chapter unnecessary, for we can employ coding and decoding for error correction as well as the noiseless coding and decoding to be met with here.

The first mathematical idea we shall consider about noiseless coding — beyond just setting up notation, though that carries ideas along with it — is that codes should be decipherable. We shall, naturally, insist on that! The mathematical expression of the idea, the Kraft inequality, limits how little code you can get away with to encode your messages. Under this limitation you still have much choice of code, and need therefore a criterion of what makes a code optimal. Now the problem is not to encode a single message, but to set up the method of encoding an indefinitely long stream, stretching into the future, of messages with similar characteristics. The likely characteristics of those prospective messages have to be specified probabilistically. That is, there is a message ‘source’ whose future output from the point of view of having to code it, is random, following a particular probability distribution or distributions which can be ascertained from the physical set-up or estimated statistically.