This chapter explores how state evolution impacts uncertainty in classical systems using majorization and related principles.

Chapter 2 is a study of divergence (also known as information divergence, Kullback–Leibler (KL) divergence, relative entropy), which is the first example of a dissimilarity (information) measure between a pair of distributions P and Q. Defining KL divergence and its conditional version in full generality requires some measure-theoretic acrobatics (Radon–Nikodym derivatives and Markov kernels) that we spend some time on. (We stress again that all this abstraction can be ignored if one is willing to work only with finite or countably infinite alphabets.) Besides definitions we prove the “main inequality” showing that KL divergence is non-negative. Coupled with the chain rule for divergence, this inequality implies the data-processing inequality, which is arguably the central pillar of information theory and this book. We conclude the chapter by studying the local behavior of divergence when P and Q are close. In the special case when P and Q belong to a parametric family, we will see that divergence is locally quadratic, with Hessian being the Fisher information, explaining the fundamental role of the latter in classical statistics.