In this chapter, we explore an unsupervised learning problem: estimating a distribution function from two-dimensional data. Although there is no response variable, the workflow mirrors that of supervised learning. We select the best-fitting function within a family by maximising the sum of the log of the distribution's values at the observed data points. As in supervised learning, excessive flexibility leads to overfitting, while insufficient flexibility leads to underfitting. We use cross-validation to identify a function family that achieves a happy medium.

In this chapter, we examine our first supervised learning problem, focusing on how to construct prediction functions and assess their performance. Given data consisting of predictor–response pairs, we can learn the parameters of a prediction function by minimising a loss, such as the residual sum of squares, which measures the discrepancy between actual and predicted responses. Using more flexible families of prediction functions typically reduces loss on the training data, but excessive flexibility can lead to overfitting: fitting to noise rather than the systematic component of the relationship. Overfitting results in poor prediction performance on new, unseen data. To estimate how a prediction method will perform on unseen data, we use cross-validation. However, when we compare many prediction methods using cross-validation, the best-performing method often appears better than it truly is; its apparent performance is an unreliable guide to its future accuracy. Prior knowledge is crucial for selecting plausible prediction methods to compare. Finally, we can use bootstrapping to quantify uncertainty in prediction functions and their predictions.

This chapter considers some basic concepts of essentail importance in supervised learning, of which the fundamental task is to model the given dataset (training set) so that the model prediction matches the given data optimally in certain sense. As typically the form of the model is predetermined, the task of supervised learning is essentially to find the optimal parameters of the model in either of two ways: (a) the least squares estimation (LSE) method that minimizes the squared error between the model prediction and observed data, or (b) the maximum A posteriori (MAP) method that maximizes the posterior probability of the model parameters given the data is maximized. The chapter further considers some important issues including overfitting, underfitting, and bias-variance tradeoff, faced by all supervised learning methods based on noisy data, and then some specific methods to address such issues, including cross-validation, regularization, and ensemble learning.