This chapter explores three kinds of unsupervised task: clustering, density estimation and dimensionality reduction. Cluster analysis aims to group similar observations together. The K-means algorithm does this by repeatedly reassigning each point to the nearest cluster centre, reducing or maintaining the clustering inertia at each step. Density estimation involves learning a probabilistic model of a data-generating process. Gaussian mixture models represent the distribution as a weighted sum of multivariate normal components. The EM algorithm fits these models by alternating between assigning each component a responsibility for each point and updating component locations using responsibility-weighted averages. Cross-entropy measures how well an estimated density approximates the true one and is minimised when the two match. Dimensionality reduction compresses data into a lower-dimensional latent space via an encoder, with a decoder reconstructing the original data. Principal component analysis uses linear encoder–decoder pairs to minimise reconstruction error, offering a simple yet powerful form of dimensionality reduction.

This chapter explores Bayesian optimization and statistics, which provides a flexible framework for balancing prior knowledge about a system with new data. Many optimization techniques are either rooted in Bayesian statistics, or have a Bayesian interpretation. For example, the Tikhonov and LASSO regularized least-squares come from specific prior distributions on the unknown parameters. Bayesian methods are particularly useful when we have small data sets or complex models. They are used extensively for hyperparameter tuning of machine learning models, for experimental design, and for applications in engineering that involve complex partial differential equations. This chapter provides a high-level overview of Bayesian methods in optimization, starting with statistical estimation. Density estimation, regularization, and expectation maximization will be explored here. Estimating a statistical distribution from data is fundamentally an optimization problem, and is the basis of machine learning. The simplest approach is to assume a known distribution and estimate the parameters using optimization. If we include Bayesian prior information, this becomes maximum a posteriori estimation.

This chapter treats non-neural approaches to automatic speech recognition. We cover classical techniques like dynamic time warping and the structure of HMM- and GMM-HMM-based recognition systems. We spend a fair amount of time on the Kaldi system, as this is still very widely used.

This chapter discusses how to build probabilistic models that include both discrete and continuous variables. Mathematically, this is achieved by defining them as random variables within the same probability space. In practice, the variables are manipulated using their marginal and conditional distributions. We define the conditional pmf of a discrete random variable given a continuous variable, and the conditional probability density of a continuous random variable given a discrete variable. We use these objects to build mixture models and apply them to model height in a population. Next, we describe Gaussian discriminant analysis, a classification method based on mixture models with Gaussian conditional distributions, and apply it to diagnose Alzheimer's disease. Then, we explain how to perform clustering using Gaussian mixture models and leverage the approach to cluster NBA players. Finally, we introduce the framework of Bayesian statistics which enables us to explicitly encode our uncertainty about model parameters, and use it to analyze poll data from the 2020 United States presidential election.