Published online by Cambridge University Press: 05 June 2012
Introduction
In this chapter we present the applicability of probability theory and random variables to the formulation of information theory pioneered by Claude Shannon in the late 1940s. Information theory introduces the general idea of source coding and channel coding. The purpose of source coding is to minimize the bit rate required to represent the source (represented mathematically by a discrete random variable) with a specified efficiency at the output of the source coder. On the other hand, the goal of channel coding is to maximize the bit rate at the input of the channel encoder so that code words can be transmitted through the channel with a specified reliability. Both source coding and channel coding can be achieved with the knowledge of the statistics of both the source and channel.
Entropy of a discrete source
Information comes from observing the outcome of an event. Common events occur frequently (high probability) and therefore carry little information. On the other hand, rare events occur infrequently (low probability) and hence carry much more information. In 1928 R. V. L. Hartley proposed a logarithmic measure of information that illustrates this observation.
To save this book to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.
Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.
Find out more about the Kindle Personal Document Service.
To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.
To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.