Could a European swallow fly a coconut from the African continent to the British Isles? You can think of bagging as convening a committee of general experts to answer some questions, perhaps questions involving aerodynamics (the study of flying), carpology (the study of seeds and fruits like coconuts), and ornithology (the study of birds like swallows).

In Chapter 19 we apply methods developed in the previous chapters (namely the weak converse and the random/maximal coding achievability) to compute the channel capacity. This latter notion quantifies the maximal amount of (data) bits that can be reliably communicated per single channel use in the limit of using the channel many times. Formalizing the latter statement will require introducing the concept of a communication channel. Then for special kinds of channels (the memoryless and the information-stable ones) we will show that computing the channel capacity reduces to maximizing the (sequence of the) mutual information. This result, known as Shannon’s noisy channel coding theorem, is very special as it relates the value of a (discrete, combinatorial) optimization problem over codebooks to that of a (convex) optimization problem over information measures. It builds a bridge between the abstraction of information measures (Part I) and practical engineering problems.

Eudaimonism was the dominant theory in ancient Greek ethics. The name derives from the Greek word ‘eudaimonia’, which is often translated as ‘happiness’ but is sometimes translated as ‘flourishing.’ Many scholars in fact prefer the latter translation because they believe it better captures the concern of the ancient Greeks with the idea of living well. This preference suggests that a useful way of distinguishing between eudaimonism and egoism is to observe, when formulating their fundamental principles, the distinction between well-being and happiness that we drew in Chapter 2. Accordingly, the fundamental principle of eudaimonism is that the highest good for each person is his or her well-being; the fundamental principle of egoism remains, as before, that the highest good for a person is his or her happiness. Admittedly, this way of distinguishing between the two theories would be theoretically pointless if the determinants of how happy a person was were the same as the determinants of how high a level of well-being the person had achieved. Thus, in particular, when hedonism is the favored theory of well-being, this way of distinguishing between eudaimonism and egoism comes to nothing. It fails in this case to capture any real difference between them. For when hedonism is the favored theory of well-being, determinations of how happy a person is exactly match the determinations of how high a level of well-being a person has achieved.

Both egoism and eudaimonism share an outlook of self-concern. They both identify the perspective from which a person judges what ought to be done as that of someone concerned with how best to promote his own good. On either theory, then, the highest good for a person is that person’s own good, whether this be his own happiness or his own well-being. Hence, on either theory, ethical considerations are understood to have the backing of reason insofar as they help to advance this good.

Chapter 1 introduces the first information measure – Shannon entropy. After studying its standard properties (chain rule, conditioning), we will briefly describe how one could arrive at its definition. We discuss axiomatic characterization, the historical development in statistical mechanics, as well as the underlying combinatorial foundation (“method of types”). We close the chapter with Han’s and Shearer’s inequalities, which both exploit the submodularity of entropy.

Chapter 2 is a study of divergence (also known as information divergence, Kullback–Leibler (KL) divergence, relative entropy), which is the first example of a dissimilarity (information) measure between a pair of distributions P and Q. Defining KL divergence and its conditional version in full generality requires some measure-theoretic acrobatics (Radon–Nikodym derivatives and Markov kernels) that we spend some time on. (We stress again that all this abstraction can be ignored if one is willing to work only with finite or countably infinite alphabets.) Besides definitions we prove the “main inequality” showing that KL divergence is non-negative. Coupled with the chain rule for divergence, this inequality implies the data-processing inequality, which is arguably the central pillar of information theory and this book. We conclude the chapter by studying the local behavior of divergence when P and Q are close. In the special case when P and Q belong to a parametric family, we will see that divergence is locally quadratic, with Hessian being the Fisher information, explaining the fundamental role of the latter in classical statistics.

Chiral perturbation theory is the systematic low-energy effective theory of QCD, interms of low-energy parameters and pseudo-Nambu–Goldstone boson fields representing pions,kaons, and the η. We discuss their masses in leading order, and the correspondingelectromagnetic corrections, where we arrive at Dashen’s theorem. We show how thislow-energy scheme even encompasses nucleons, and how QCD provides corrections to the weakgauge boson masses. In that context, we comment on a technicolor extension and on thehypothesis of minimal flavor violation, which is described by spurions.

This enthusiastic introduction to the fundamentals of information theory builds from classical Shannon theory through to modern applications in statistical learning, equipping students with a uniquely well-rounded and rigorous foundation for further study. The book introduces core topics such as data compression, channel coding, and rate-distortion theory using a unique finite blocklength approach. With over 210 end-of-part exercises and numerous examples, students are introduced to contemporary applications in statistics, machine learning, and modern communication theory. This textbook presents information-theoretic methods with applications in statistical learning and computer science, such as f-divergences, PAC-Bayes and variational principle, Kolmogorov’s metric entropy, strong data-processing inequalities, and entropic upper bounds for statistical estimation. Accompanied by additional stand-alone chapters on more specialized topics in information theory, this is the ideal introductory textbook for senior undergraduate and graduate students in electrical engineering, statistics, and computer science.

We outline the main concepts of the Standard Model, illustratively describing itscentral features and some open questions, as a preparation for the following chapters.

This chapter introduces the first fermion generation. We begin with the electron and theleft-handed neutrino, their CP invariance as well as anomalies in triangle diagrams andWitten’s global SU(2) anomaly. They are both canceled by adding up and down quarks. Wediscuss the constraints that anomaly cancelation imposes on the electric charges of thefermions. Finally we also add a right-handed neutrino, extend the anomaly discussion tothe lepton and baryon numbers, and further extend the model by proceeding totechnicolor.

In Chapter 13 we will discuss how to produce compression schemes that do not require a priori knowledge of the generative distribution. It turns out that designing a compression algorithm able to adapt to an unknown distribution is essentially equivalent to the problem of estimating an unknown distribution, which is a major topic of statistical learning. The plan for this chapter is as follows: (1) We will start by discussing the earliest example of a universal compression algorithm (of Fitingof). It does not talk about probability distributions at all. However, it turns out to be asymptotically optimal simultaneously for all iid distributions and with small modifications for all finite-order Markov chains. (2) The next class of universal compressors is based on assuming that the true distribution belongs to a given class. These methods proceed by choosing a good model distribution serving as the minimax approximation to each distribution in the class. The compression algorithm for a single distribution is then designed as in previous chapters. (3) Finally, an entirely different idea are algorithms of Lempel–Ziv type. These automatically adapt to the distribution of the source, without any prior assumptions required.

In this chapter we introduce the problem of analyzing low-probability events, known as large deviation theory. It is usually solved by computing moment-generating functions and Fenchel-Legendre conjugation. It turns out, however, that these steps can be interpreted information-theoretically in terms of information projection. We show how to solve information projection in a special case of linear constraints, connecting the solution to exponential families.

Dirac, Weyl, and Majorana fermions are now formulated in terms of functional integralsof Grassmann fields in Euclidean space. We discuss continuous and discrete symmetries, thespin-statistics theorem as well as the transfer matrix on the lattice. Regarding thetransformations C, P, and T, we highlight a little known subtlety of the parity behaviorof Majorana fermions.

In Chapter 20 we study data transmission with constraints on the channel input. For example, how many bits per channel use can we transmit under constraints on the codewords? To answer this question in general, we need to extend the setup and coding theorems to channels with input constraints. After doing that we will apply these results to compute the capacities of various Gaussian channels (memoryless, with intersymbol interference and subject to fading).