This chapter explores three kinds of unsupervised task: clustering, density estimation and dimensionality reduction. Cluster analysis aims to group similar observations together. The K-means algorithm does this by repeatedly reassigning each point to the nearest cluster centre, reducing or maintaining the clustering inertia at each step. Density estimation involves learning a probabilistic model of a data-generating process. Gaussian mixture models represent the distribution as a weighted sum of multivariate normal components. The EM algorithm fits these models by alternating between assigning each component a responsibility for each point and updating component locations using responsibility-weighted averages. Cross-entropy measures how well an estimated density approximates the true one and is minimised when the two match. Dimensionality reduction compresses data into a lower-dimensional latent space via an encoder, with a decoder reconstructing the original data. Principal component analysis uses linear encoder–decoder pairs to minimise reconstruction error, offering a simple yet powerful form of dimensionality reduction.

This chapter introduces neural networks as flexible function approximators built by composing layers of simple processing units. A network with no hidden layers performs linear regression if its output layer is linear and logistic regression if its output layer uses softmax. Hidden layers increase expressivity: a network with one hidden layer and ReLU activations can approximate any continuous function on a closed and bounded input domain, though complex functions may require many units. Deep networks, with multiple hidden layers, are more efficient and scalable than shallow ones, especially for learning hierarchical structure. Neural networks are trained using gradient-based optimisation, with gradients computed via backpropagation. Training adjusts weights to minimise a loss function, using small batches of data. Techniques like early stopping and small batches act as implicit regularisers, while weight decay provides explicit regularisation. Convolutional neural networks use convolution and pooling layers to exploit spatial structure in image data. More broadly, architectural choices often reflect domain-specific assumptions.

This chapter introduces directed acyclic graphs (DAGs) as a way to represent multivariate probability distributions. DAGs help clarify the structure of probabilistic models and the dependencies among their variables and serve as a central tool in later chapters. Every DAG corresponds to a specific factorisation of a joint mass or density function into a product of conditional distributions. While a DAG encodes how the distribution breaks down into conditionals, it does not fully determine the distribution itself. Instead, it implies certain dependency constraints among variables. These constraints can be examined using the concept of d-separation, which allows us to infer conditional independence relationships directly from the graph.

This chapter introduces key concepts and methods in Bayesian statistical modelling. The posterior predictive distribution captures both epistemic uncertainty in model parameters and aleatory uncertainty in future outcomes. A Bayesian p-value gives the probability that a statistic computed from data output by a given model will be more extreme than the value of the same statistic computed from observed data. Bayesian p-values close to 0 or 1 suggest the model may be inadequate. Markov chain Monte Carlo is a general-purpose tool for sampling from complex, unnormalised distributions. It produces dependent samples, so the effective sample size is usually smaller than the number of iterations. Informative priors are useful when data leave large uncertainties in parameter values. Empirical Bayes combines information across related datasets by estimating a distribution over parameters using frequentist methods. Hierarchical modelling provides a unified Bayesian framework for handling multiple related datasets, capturing group structure via a hierarchical graph.

In this chapter, we explore an unsupervised learning problem: estimating a distribution function from two-dimensional data. Although there is no response variable, the workflow mirrors that of supervised learning. We select the best-fitting function within a family by maximising the sum of the log of the distribution's values at the observed data points. As in supervised learning, excessive flexibility leads to overfitting, while insufficient flexibility leads to underfitting. We use cross-validation to identify a function family that achieves a happy medium.

This chapter introduces simple and multiple linear regression models – core tools in predictive modelling due to their simplicity and interpretability. These models assume the response variable is a linear function of the predictor(s), plus a noise term. The regression function gives the expected response given the predictors. The coefficient of determination, R2, measures how much of the variance in the response is explained by the model. In simple linear regression, R2 equals the square of the Pearson correlation between response and predictor; in multiple regression, it equals the square of the correlation between response and predicted values. Each coefficient in multiple regression reflects the expected change in the response for a one-unit increase in that predictor, holding others fixed. Standardising predictors lets us compare coefficient sizes. Strong collinearity between predictors increases uncertainty in the fitted coefficients. Models using only a subset of predictors may generalise better than those using all and overfitting. The squared error risk of a modelling procedure – its expected test error – can be broken down into bias, variance and irreducible noise.