Chapter 3 discusses the field of machine learning from a theoretical perspective. The review will advance the discussion of advanced metrics in Chapter 5 and error estimation methods in Chapter 6. The specific concepts surveyed in this chapter include loss functions, empirical risk, generalization error, empirical and structural risk minimization, regularization, and learning bias. The unsupervised learning paradigm is also reviewed and the chapter concludes with a discussion of the bias/variance tradeoff.

This chapter addresses the often-misunderstood concept of validity. Much of the methodological discussion around sociological experiments is framed in terms of internal and external validity. The standard view is that the more we ensure that the experimental treatment is isolated from potential confounds (internal validity), the more unlikely it is that the experimental results can be representative of phenomena of the outside world (external validity). However, other accounts describe internal validity as a prerequisite of external validity: Unless we ensure internal validity of an experiment, little can be said of the outside world. We contend in this chapter that problems of either external or internal validity do not necessarily depend on the artificiality of experimental settings or on the laboratory–field distinction between experimental designs. We discuss the internal–external distinction and propose instead a list of potential threats to the validity of experiments that includes "usual suspects" like selection, history, attrition, and experimenter demand effects and elaborate on how these threats can be productively handled in experimental work. Moreover, in light of the different types of experiments, we also discuss the strengths and weaknesses of each regarding threats to internal and external validity.

The first sociological experiments have been conducted in the second and third decades of the twentieth century, accompanied by a fierce debate about the possibilities and limits of the approach, which anticipated many of the critiques currently raised against the method. The chapter traces the development of experimental research in sociology from these beginning to modern perspectives. One of the reasons for the marginal position of experimentation in sociology has been the reluctance to give up full control of potentially intervening variables (called the ex post facto method) in favor of randomization. Inspirations from social psychology and, later, economics, have finally resulted in the experimental designs that are currently used in sociology.

Chapter 9 is devoted to evaluation methods for an important category of classical learning paradigms left out of Chapter 8 so as to receive fuller coverage: unsupervised learning. In this chapter, a number of different unsupervised learning schemes are considered and their evaluation discussed. The particular tasks considered are clustering and hierarchical clustering, dimensionality reduction, latent variable modeling, and generative models including probabilistic PCA, variational autoencoders, and GANs. Evaluation methodology is discussed discussed for each of these tasks.

Chapter 11 completes the discussion of Chapter 10 by raising the question of how to practice machine learning in a responsible manner. It describes the dangers of data bias, and surveys data bias detection and mitigation methods; it lists the benefits of explainability and discusses techniques, such as LIME and SHAP, that have been proposed to explain the decisions made by opaque models; it underlines the risks of discrimination and discusses how to enhance fairness and prevent discrimination in machine learning algorithms. The issues of privacy and security are then presented, and the need to practice human-centered machine learning emphasized. The chapter concludes with the important issues of repeatability, reproducibility, and replicability in machine learning.

Chapter 1 discusses the motivation for the book and the rationale for its organization into four parts: preliminary considerations, evaluation for classification, evaluation in other settings, and evaluation from a practical perspective. In more detail, the first part provides the statistical tools necessary for evaluation and reviews the main machine learning principles as well as frequently used evaluation practices. The second part discusses the most common setting in which machine learning evaluation has been applied: classification. The third part extends the discussion to other paradigms such as multi-label classification, regression analysis, data stream mining, and unsupervised learning. The fourth part broadens the conversation by moving it from the laboratory setting to the practical setting, specifically discussing issues of robustness and responsible deployment.

This chapter provides the tools necessary to implement virtually any type of peril in the hazard module of a catastrophe (CAT) model. These tools comprise, for a given peril, the creation of the following: a set of simulated events, a catalogue of hazard intensity footprints, and the main metrics employed in probabilistic hazard assessment (hazard curves and hazard maps). Despite the general purpose of the standard CAT modelling framework, peril-specific CAT models are commonly developed in silos by dedicated experts. In view of the dozens of perils quantified in this textbook, a more generalist approach is employed. An ontology is proposed that harmonizes the description of different perils, going from (1) event source, to (2) event size distribution, to, finally, (3) event intensity footprint. To illustrate how all the previous steps can be wrapped up in one continuous modelling pipeline, an application to probabilistic seismic hazard assessment is also provided.

Chapter 8 introduces evaluation procedures for paradigms other than classification. In particular, it discusses evaluation for classical problems such as regression analysis, time-series analysis, outlier detection, and reinforcement learning, along with evaluation approaches for newer tasks such as positive-unlabelled classification, ordinal classification, multi-labeled classification, image segmentation, text generation, data stream mining, and lifelong learning.

In Chapter 7, the history of statistical analysis is reviewed and its legacy discussed. Four situations of interest to machine learning evaluation are subsequently discussed within different statistical paradigms: the comparison of two classifiers on a single domain; the comparison of multiple classifiers on a single domain; the comparison of two classifiers on multiple domains; and the comparison of multiple classifiers on multiple domains. The three statistical paradigms considered for each of these situations are the null hypothesis statistical testing (NHST) setting; an enhanced Fisher-flavored methodology that adds the notions of confidence intervals, effect size, and power analysis to NHST; and a newer approach based on Bayesian reasoning.

Chapter 10 is an introduction to the connections between probability theory and partial differential equations. At the beginning of §10.1, I show that martingales provide a link between probability theory and partial differential equations. More precisely, I show how to represent in terms of Wiener integrals solutions to parabolic and elliptic partial differential equations in which the Laplacian is the principal part. In the second part of §10.1, I derive the Feynman–Kac formula and use it to calculate various Wiener integrals. In §10.2 I introduce the Markov property of Wiener measure and show how it not only allows one to evaluate other Wiener integrals in terms of solutions to elliptic partial differential equations but also enables one to prove interesting facts about solutions to such equations as a consequence of their representation in terms of Wiener integrals. Continuing in the same spirit, I show in §10.2 how to represent solutions to the Dirichlet problem in terms of Wiener integrals, and in §10.3 I use Wiener measure to construct and discuss heat kernels related to the Laplacian and discuss ground states (a.k.a. stationary measures) for them.