Chapter 2 is a study of divergence (also known as information divergence, Kullback–Leibler (KL) divergence, relative entropy), which is the first example of a dissimilarity (information) measure between a pair of distributions P and Q. Defining KL divergence and its conditional version in full generality requires some measure-theoretic acrobatics (Radon–Nikodym derivatives and Markov kernels) that we spend some time on. (We stress again that all this abstraction can be ignored if one is willing to work only with finite or countably infinite alphabets.) Besides definitions we prove the “main inequality” showing that KL divergence is non-negative. Coupled with the chain rule for divergence, this inequality implies the data-processing inequality, which is arguably the central pillar of information theory and this book. We conclude the chapter by studying the local behavior of divergence when P and Q are close. In the special case when P and Q belong to a parametric family, we will see that divergence is locally quadratic, with Hessian being the Fisher information, explaining the fundamental role of the latter in classical statistics.

This enthusiastic introduction to the fundamentals of information theory builds from classical Shannon theory through to modern applications in statistical learning, equipping students with a uniquely well-rounded and rigorous foundation for further study. The book introduces core topics such as data compression, channel coding, and rate-distortion theory using a unique finite blocklength approach. With over 210 end-of-part exercises and numerous examples, students are introduced to contemporary applications in statistics, machine learning, and modern communication theory. This textbook presents information-theoretic methods with applications in statistical learning and computer science, such as f-divergences, PAC-Bayes and variational principle, Kolmogorov’s metric entropy, strong data-processing inequalities, and entropic upper bounds for statistical estimation. Accompanied by additional stand-alone chapters on more specialized topics in information theory, this is the ideal introductory textbook for senior undergraduate and graduate students in electrical engineering, statistics, and computer science.

In Chapter 13 we will discuss how to produce compression schemes that do not require a priori knowledge of the generative distribution. It turns out that designing a compression algorithm able to adapt to an unknown distribution is essentially equivalent to the problem of estimating an unknown distribution, which is a major topic of statistical learning. The plan for this chapter is as follows: (1) We will start by discussing the earliest example of a universal compression algorithm (of Fitingof). It does not talk about probability distributions at all. However, it turns out to be asymptotically optimal simultaneously for all iid distributions and with small modifications for all finite-order Markov chains. (2) The next class of universal compressors is based on assuming that the true distribution belongs to a given class. These methods proceed by choosing a good model distribution serving as the minimax approximation to each distribution in the class. The compression algorithm for a single distribution is then designed as in previous chapters. (3) Finally, an entirely different idea are algorithms of Lempel–Ziv type. These automatically adapt to the distribution of the source, without any prior assumptions required.

In this chapter we introduce the problem of analyzing low-probability events, known as large deviation theory. It is usually solved by computing moment-generating functions and Fenchel-Legendre conjugation. It turns out, however, that these steps can be interpreted information-theoretically in terms of information projection. We show how to solve information projection in a special case of linear constraints, connecting the solution to exponential families.

In Chapter 20 we study data transmission with constraints on the channel input. For example, how many bits per channel use can we transmit under constraints on the codewords? To answer this question in general, we need to extend the setup and coding theorems to channels with input constraints. After doing that we will apply these results to compute the capacities of various Gaussian channels (memoryless, with intersymbol interference and subject to fading).

This chapter focuses on different research designs in experimental sociology. Most definitions of what constitutes an experiment converge on the idea that the experimenter "control" the phenomenon under investigation, thereby setting the conditions under which the phenomenon is observed and analyzed. Typically, the researcher exerts experimental control by creating two situations that are virtually identical, except for one element that the researcher introduces or manipulates in only one of the situations. The purpose of this exercise is to observe the effects of such manipulation by comparing it with the outcomes of the situation in which the manipulation is absent. One way to look at how the implementation of this rather straightforward exercise produces a variety of designs is by focusing on the relationship that experimental design bears with the theory that inspires it. Therefore, we begin this chapter with a discussion of the relationship between theory and experimental design before turning to a description of the most important features of various types of designs. The chapter closes with a short overview of experiments in different settings such as laboratory, field, and multifactorial survey experiments.

Experimental practices developed in different scientific disciplines following different historical trajectories. Thus, standard experimental procedures differ starkly between disciplines. One of the most controversial issues is the use of deception as a methodological device. Psychologists do not conduct a study involving deception unless they have determined that the use of deceptive techniques is justified by the study’s significant prospective scientific, educational, or applied value and that effective nondeceptive alternative procedures are not feasible. In experimental economics it is strictly forbidden and a ban on experiments involving deception is enforced by all major economic journals. In the sociological scientific community, there is no clear consensus on the matter. Importantly, the disagreement is sometimes based on ethical considerations, but more often it is based on pragmatic grounds: the anti-deception camp argues that deceiving participants leads to invalid results, while the other side argues that deception has little negative impact and, under certain conditions, can even enhance validity. In this chapter, we first discuss the historical reasons leading to the emergence of such different norms in different fields and then analyze and separate ethical and pragmatic concerns. Finally, we propose some guidelines to regulate the use of deception in sociological experiments.

Sociology is a science concerning itself with the interpretive understanding of social action and thereby with a causal explanation of its course and consequences. Empirically, a key goal is to find relations between variables. This is often done using naturally occurring data, survey data, or in-depth interviews. With such data, the challenge is to establish whether a relation between variables is causal or merely a correlation. One approach is to address the causality issue by applying proper statistical or econometric techniques, which is possible under certain conditions for some research questions. Alternatively, one can generate new data with experimental control in a laboratory or the field. It is precisely through this control via randomization and the manipulation of the causal factors of interest that the experimental method ensures – with a high degree of confidence – tests of causal explanations. In this chapter, the canonical approach to causality in randomized experiments (the Neyman–Rubin causal model) is first introduced. This model formalizes the idea of causality using the "potential outcomes" or "counterfactual" approach. The chapter then discusses the limits of the counterfactual approach and the key role of theory in establishing causal explanations in experimental sociology.

Field experiments have a long tradition in some areas of the social and behavioral sciences and have become increasingly popular in sociology. Field experiments are staged in "natural" research settings where individuals usually interact in everyday life and regularly complete the task under investigation. The implementation in the field is the core feature distinguishing the approach from laboratory experiments. It is also one of the major reasons why researchers use field experiments; they allow incorporating social context, investigating subjects under "natural" conditions, and collecting unobtrusive measures of behavior. However, these advantages of field experiments come at the price of reduced control. In contrast to the controlled setting of the laboratory, many factors can influence the outcome but are not under the experimenter’s control and are often hard to measure in the field. Using field experiments on the broken windows theory, the strengths and potential pitfalls of experimenting in the field are illustrated. The chapter also covers the nascent area of digital field experiments, which share key features with other types of experiments but offer exciting new ways to study social behavior by enabling the collection large-scale data with fine-grained and unobtrusive behavioral measures at relatively low variable costs.

This chapter focuses in more detail on the role of incentives in experimental sociology. Providing the right incentives in an experiment is an important precondition for drawing valid inferences. This is a predominant view in experimental economics based on the induced-value theory assuming that monetary incentives override any other human motivation in laboratory economic experiments. A slightly less demanding assumption is that subjects can be incentivized by monetary payoffs but are also motivated by other-regarding preferences or reciprocity. On the other hand, psychologists focus on motivations that subjects bring into the laboratory as a predisposition to behavior and on the framing of the situation. Sociological research takes elements from both perspectives and emphasizes institutional, cultural, and social determinants of human behavior. An important theoretical framework for experimental work is sociological work on framing. According to sociological framing theories, subjects interpret the situation in terms of the given cues and select an action that is appropriate to the situation. The chapter discusses the implications of these three views on the design of experiments in sociology.