This chapter explores three kinds of unsupervised task: clustering, density estimation and dimensionality reduction. Cluster analysis aims to group similar observations together. The K-means algorithm does this by repeatedly reassigning each point to the nearest cluster centre, reducing or maintaining the clustering inertia at each step. Density estimation involves learning a probabilistic model of a data-generating process. Gaussian mixture models represent the distribution as a weighted sum of multivariate normal components. The EM algorithm fits these models by alternating between assigning each component a responsibility for each point and updating component locations using responsibility-weighted averages. Cross-entropy measures how well an estimated density approximates the true one and is minimised when the two match. Dimensionality reduction compresses data into a lower-dimensional latent space via an encoder, with a decoder reconstructing the original data. Principal component analysis uses linear encoder–decoder pairs to minimise reconstruction error, offering a simple yet powerful form of dimensionality reduction.

A1.1 Basic matrix operations

A1.1.1 Definition (matrix)

A matrix is an ordered rectangular array of numbers (or functions). The numbers (or functions) are called the elements or the entries of the matrix.

A1.1.2 Dimension/size of a matrix

A matrix having m rows and a columns is a matrix of dimension m x n.

This chapter introduces neural networks as flexible function approximators built by composing layers of simple processing units. A network with no hidden layers performs linear regression if its output layer is linear and logistic regression if its output layer uses softmax. Hidden layers increase expressivity: a network with one hidden layer and ReLU activations can approximate any continuous function on a closed and bounded input domain, though complex functions may require many units. Deep networks, with multiple hidden layers, are more efficient and scalable than shallow ones, especially for learning hierarchical structure. Neural networks are trained using gradient-based optimisation, with gradients computed via backpropagation. Training adjusts weights to minimise a loss function, using small batches of data. Techniques like early stopping and small batches act as implicit regularisers, while weight decay provides explicit regularisation. Convolutional neural networks use convolution and pooling layers to exploit spatial structure in image data. More broadly, architectural choices often reflect domain-specific assumptions.

CHANCE PERMEATES OUR physical and mental universe. While the role of chance in human lives has had a longer history, starting with the more authoritative influence of the nobility, the more rationally sound theory of probability and statistics has come into practice in diverse areas of science and engineering starting from the early to mid-twentieth century. Practical applications of statistical theories proliferated to such an extent in the previous century that the American government-sponsored RAND corporation published a 600-page book that wholly consisted of a random number table and a table of standard normal deviates. One of the primary objectives of this book was to enable a computer-simulated approximate solution of an exact but unsolvable problem by a procedure known as the Monte Carlo method devised by Fermi, von Neumann, and Ulam in the 1930s–40s.

Statistical methods are the mainstay of conducting modern scientific experiments. One such experimental paradigm is known as a randomized control trial, which is widely used in a variety of fields such as psychology, drug verification, testing the efficacy of vaccines, agricultural sciences, and demography. These statistical experiments require sophisticated sampling techniques in order to nullify experimental biases. With the explosion of information in the modern era, the need to develop advanced and accurate predictive capabilities has grown manifold. This has led to the emergence of modern artificial intelligence (AI) technologies. Further, climate change has become a reality of modern civilization. Accurate prediction of weather and climatic patterns relies on sophisticated AI and statistical techniques. It is impossible to think of a modern economy and social life without the influence and role of chance, and hence without the influence of technological interventions based on statistical principles. We must begin this journey by learning the foundational tenets of probability and statistics.

EMPIRICAL TECHNIQUES rely on abstracting meaning from observable phenomena by constructing relationships between different observations. This process of abstraction is facilitated by appropriate measurements (experiments), suitable organization of data generated by measurements, and, finally, rigorous analysis of the data. The latter is a functional exercise that synthesizes information (data) and theory (model) and enables prediction of hitherto unobserved phenomena.1 It is important to underscore that a good theory (model) that explains a certain phenomenon well by appealing to a set of laws and conditions is expected to be a good candidate for predicting the same using reliable data. For example, a good model for the weight of a normal human being is w = m * h, where w and h refer to weight and height of the person, and m can be set to unity if appropriate units are chosen. A rational explanation of such a formula for weight based on anatomical considerations is perhaps very reasonable. From an empirical standpoint, if we collect height and weight data of normal humans, we will notice that a linear model of the form w = m * h represents the data reasonably well and may be used to predict the weight of the person based on the height of the person. This fact ascertains a functional symmetry between explanation and prediction. Therefore, a good predictive model must automatically be able to explain the data (and related events) well.

This chapter introduces directed acyclic graphs (DAGs) as a way to represent multivariate probability distributions. DAGs help clarify the structure of probabilistic models and the dependencies among their variables and serve as a central tool in later chapters. Every DAG corresponds to a specific factorisation of a joint mass or density function into a product of conditional distributions. While a DAG encodes how the distribution breaks down into conditionals, it does not fully determine the distribution itself. Instead, it implies certain dependency constraints among variables. These constraints can be examined using the concept of d-separation, which allows us to infer conditional independence relationships directly from the graph.