An empirical average will converge, in some sense, to the corresponding expectation. This famous result, called the Law of Large Numbers, can be anticipated based on the concentration inequalities introduced in the previous chapter, but some appropriate notions of convergence for random variables need to be defined in order to make a rigorous statement. Beyond mere convergence, the fluctuations of an empirical average around the associated expectation can be characterized by the Central Limit Theorem, and are known to be Gaussian in some asymptotic sense. The chapter also discusses the limit of extremes such as the maximum of a sample.

Stochastic processes model experiments whose outcomes are collections of variables organized in some fashion. We focus here on Markov processes, which include random walks (think of the fortune of a person gambling on black/red at the roulette over time) and branching processes (think of the behavior of a population of an asexual species where each individual gives birth to a number of otherwise identical offsprings according to a given probability distribution) .

In this chapter we consider distributions on the real line that have a discrete support. It is indeed common to count certain occurrences in an experiment, and the corresponding counts are invariably integer-valued. In fact, all the major distributions of this type are supported on the (non-negative) integers. We introduce the main ones here.

We consider an experiment resulting in two paired numerical variables. The general goal addressed in this chapter is that of quantifying the strength of association between these two variables. By association we mean dependence. Contrary to the previous chapter, here the two variables can be measurements of completely different kinds (e.g., height and weight). Several measures of association are introduced, and used to test for independence.

In some areas of mathematics, physics, and elsewhere, continuous objects and structures are often motivated, or even defined, as limits of discrete objects. For example, in mathematics, the real numbers are defined as the limit of sequences of rational numbers, and in physics, the laws of thermodynamics arise as the number of particles in a system tends to infinity (the so-called thermodynamic or macroscopic limit). Taking certain discrete distributions (discussed in the previous chapter) to their continuous limits, which is done by letting their support size increase to infinity in a controlled manner, gives rise to continuous distributions on the real line. We introduce and discuss such distributions in this chapter, including the normal (aka Gaussian) family of distributions, and in the process cover probability densities.

We consider in this chapter experiments where the variables of interest are paired. Importantly, we assume that these variables are directly comparable (in contrast with the following two chapters). Crossover trials are important examples of such experiments. The main question of interest here is that of exchangeability, which reduces to testing for symmetry when there are only two variables.

When a die (with 3 or more faces) is rolled, the result of each trial can take one of as many possible values. The same is true in the context of an urn experiment, when the balls in the urn are of multiple different colors. Such models are broadly applicable. Indeed, even `yes/no’ polls almost always include at least one other option like `not sure’ or `no opinion’. Another situation where discrete variables arise is when two or more coins are compared in terms of their chances of landing heads, or more generally, when two or more (otherwise identical) dice are compared in terms of their chances of landing on a particular face. In terms of urn experiments, the analog is a situation where balls are drawn from multiple urns. This sort of experiments can be used to model clinical trials where several treatments are compared and the outcome is dichotomous. When the coins are tossed together, or when the dice are rolled together, we might want to test for independence. We thus introduce some classical tests for comparing multiple discrete distributions and for testing for the independence of two or more discrete variables that are observed together.

A prototypical (although somewhat idealized) workflow in any scientific investigation starts with the design of the experiment to probe a question or hypothesis of interest. The experiment is modeled using several plausible mechanisms. The experiment is conducted and the data are collected. These data are finally analyzed to identify the most adequate mechanism, meaning the one among those considered that best explains the data. Although an experiment is supposed to be repeatable, this is not always possible, particularly if the system under study is chaotic or random in nature. When this is the case, the mechanisms above are expressed as probability distributions. We then talk about probabilistic modeling --- albeit with not one but several probability distributions. It is as if we contemplate several probability experiments, and the goal of statistical inference is to decide on the most plausible one in view of the collected data. We introduce core concepts such as estimators, confidence intervals, and tests.

The chapter focuses on discrete probability spaces, where probability calculations are combinatorial in nature. Urn models are presented as the quintessential discrete experiments.

Statistics is the science of data collection and data analysis. We provide, in this chapter, a brief introduction to principles and techniques for data collection, traditionally divided into survey sampling and experimental design --- each the subject of a rich literature. While most of this book is on mathematical theory, covering aspects of Probability Theory and Statistics, the collection of data is, by nature, much more practical, and often requires domain-specific knowledge. And careful data collection is of paramount importance. Indeed, data that were improperly collected can be completely useless and unsalvageable by any technique of analysis. And it is worth keeping in mind that the collection phase is typically much more expensive that the analysis phase that ensues (e.g., clinical trials, car crash tests, etc). Thus the collection of data should be carefully planned according to well-established protocols or with expert advice. We discuss the basics of data collection in this chapter.

This chapter introduces Kolmogorov’s probability axioms and related terminology and concepts such as outcomes and events, sigma-algebras, probability distributions and their properties.

In this chapter we introduce and briefly discuss some properties of estimators and tests that make it possible to compare multiple methods addressing the same statistical problem. We discuss the notions of sufficiency and consistency, and various notions of optimality (including minimax optimality), both for estimators and for tests.