This chapter discusses a two-layer competitive learning network for unsupervised clustering based on a competitive learning rule. The weights of each node of the output layer are treated as a vector, in the same space for the input vectors. Given an input, all nodes compete to become the sole winner (winner-take-all) with the highest output value, so that its weights can be modified in such a way that it will be more likely to win when the same input is presented to the input layer next time. By the end of this iterative learning process, each weight vector is gradually moved toward the center of one of the clusters in the space, thereby representing the cluster. The chapter further considers the self organizing map (SOM), a network based on the same competitive learning rule, but modified in such a way that nodes in the neighborhood of the winner learn along with the winner by modifying their weights as well but to a lesser extent. Once fully trained, the SOM achieves the effect that neighboring nodes learn to respond to similar inputs, mimicking a typical behavior of certain visual and auditory cortex in the brain.

This chapter first introduces a simple two-layer perceptron network based on some straight forward learning rule. This perceptron network can be used as a linear classifier capable of multiclass classification if the classes are linearly separable, which can be further generalized for nonlinear classification when the kernel method is introduced into the algorithm. The main algorithm discussed in this chapter is the multi-layer (3 or more) back propagation network which is a supervised method most widely used for classification, and also serves as one of the building blocks of the much more powerful deep learning method and other artificial intelligence methods. Based on the labeled sample in the training set, the weights of the back propagation network are sequentially modified in the training process in such a way that the error, the difference between the actual out and the desired outputs, the ground truth labeling of its input, is reduced by the gradient descent method. Based on the same training process, this network can be modified to serve as an autoencoder for dimensionality reduction, similar to what the PCA can do.

This chapter discusses the basic methods for solving unconstrained optimization problems, which plays an important role in ML, as many learning problems are solved by either maximizing or minimizing some objective function. The solution of an optimization problem, the point at which the given function is minimized, can be typically found by either gradient descent or Newton’s method. The gradient descent method approaches the solution iteratively from an initial guess by moving in the opposite direction of the gradient, while Newton’s method finds the solution based on the second order derivative as well as the first order, the gradient. It is therefore a more effective method than the gradient method due to the extra piece of information, with a higher computational cost for calculating the second order derivatives. In fact, Newton’s method for minimizing a function is essentially solving an equation resulting from setting the derivative of the function to zero, i.e., it is essentially the same method used for solving equations considered previously. The chapter also considers some variants of Newton’s method, including the quasi-Newton methods and the conjugate gradient method requiring fewer iteration steps.

This chapter considers some basic concepts of essentail importance in supervised learning, of which the fundamental task is to model the given dataset (training set) so that the model prediction matches the given data optimally in certain sense. As typically the form of the model is predetermined, the task of supervised learning is essentially to find the optimal parameters of the model in either of two ways: (a) the least squares estimation (LSE) method that minimizes the squared error between the model prediction and observed data, or (b) the maximum A posteriori (MAP) method that maximizes the posterior probability of the model parameters given the data is maximized. The chapter further considers some important issues including overfitting, underfitting, and bias-variance tradeoff, faced by all supervised learning methods based on noisy data, and then some specific methods to address such issues, including cross-validation, regularization, and ensemble learning.

This chapter discusses the most basic frequentist method, the linear least squares (LLS) regression, for obtaining the optimal weights for the linear regression model that minimizes the squared error between the model prediction and observation. The chapter also considers how the goodness of its results can be evaluated quantitatively by the coefficient of determination (R-squared). The chapter then further discusses some variations of LLS, including ridge regression with an extra regularization term to make proper tradeoff between overfitting and underfitting, and linear method based on basis functions for nonlinear regression problems. Finally the last section of the chapter briefly discusses the Bayes methods that maximizes the likelihood and the posterior probability of the parameters of the linear regression model. This section serves to prepare the reader for discussion of various Bayesian learning algorithms in future chapters.

This chapter discusses the method of principal component analysis (PCA) for dimensionality reduction, by which the original high-dimensional feature space can be mapped into a much lower dimensional space still containing most of the separability information, based on either the total scatter matrix of the given dataset, or the within and/or between-class scatter matricies. This transformation from high to low dimensional space can be considered as a pre-processing stage before the main process for classification which can be carried out more efficiently and effectively in the low-dimensional space after the transformation.

This chapter is dedicated to the method of independent component analysis (ICA), which can be considered to be in parallel with PCA, as both methods are for the purpose of extracting some essential information, either the principal or independent components, from the given dataset, to be further processed. However, different from PCA, the ICA assumes the signals in the given data are linear combinations of a set of independent signal components (therefore also the name blind source separation or BSS), which can be recovered based on the fact that a linear combination of multiple random variables is more Gaussian than each of them individually. The ICA is therefore carried out based on the ICA is therefore carried out based on the principle of maximizing non-Gaussianity, often using measures such as kurtosis or negentropy to identify statistically independent components.

This chapter introduces a set of distances and scatter matrices of various kinds used to measure the difference or similarity between two sample points, one sample and one class/cluster, and two classes/clusters, and the within, between, and total scatteredness of classes/clusters, for the purpose of further measuring the separability of the classes/clusters in a subspace composed of features that are either selected or extracted from the original high dimensional feature space .

This chapter intruduces an important idea of kernel mapping, which can map the feature space to a much higher dimensional space where the class separability could be improved significantly for better classification results. Based on the assumption that all data samples only appear in the form of inner product in the algorithm, kernel mapping is actually carried out implicitly, in the sense that the mapping function never needs to be explicitly specified. The chapter then introduces the method of kernel PCA, as a variant of PCA, together with another variant probabilistic PCA. The chapter further considers the method of factor analysis based on two important concepts of latent variables and expectation maximization (EM), both playing some important roles in other learning algorithms to be discussed in future chapters. Finally the chapter moves on to discuss two additional methods, multidimensional scaling (MDS) and t-distributed stochastic neighbor embedding (t-SNE), for the same general purpose of dimensionality reduction.

This chapter discusses both supervised and unsupervised algorithms all to be carried out in a tree-like hierarchy, in which a classification or clustering problem is solved in a divide-and-conquer manner while traversing a binary tree. For supervised classification, the tree classifier is first constructed in the training phase, and then in the test phase, a set of classes are subdivided into two subsets at each node of the tree based on a subset of features specifically selected to best separate the two subsets. This operation is carried out along a path in the tree from the root node down to one of the leaf nodes representing one of the classes. For unsupervised clustering, the tree structure is constructed in either a top-down or ottom-up fashion. In the former case, the given dataset represented by root node is recursively split into two subsets represented by the two child nodes; while in the latter case, all samples each represented by one of the leaf nodes are merged sequentially until they form a single group at the root node. In either case, the splitting or merging is carried out based on certain distance previously considered. Such splitting or merging process can be truncated somewhere between the root and leaf nodes to obtain a set of clusters.

This chapter discusses nonlinear regression method based on gradient descent and its variations for obtaining the optimal parameters of any given nonlinear regression function.

This chapter is dedicated to the sole topic of support vector machine (SVM), a typical discriminative algorithm mostly for binary classification. The goal of the algorithm is to find a optimal hyperplane that separate the two classes (assumed to be linearly separable) in the feature space in such a way that the two classes are best separated, in the sense that the distances (called margin) between the plane and the samples closest to it (called support vectors) on either side of the plane are maximized. This is a constrained optimization problem which could be solved directly, but it is actually first converted to its dual problem and then solved by quadratis programming. The reason for solving the dual problem is due to the fact that all data points appear in the form of inner product, so that kernel method can be used to carry out the classification in a higher dimensional space in which the two classes become linearly separable even if they are not so in the original space. The chapter further considers some variants of SVM, such as sequential minimal optimization and generalized multiclass SVM.

Thia chapter considers methods for both regression and classification based on Gaussian process, a stochastic process with Gaussian distribution, of which the mean vector and covariance matrix can be obtained based on the labeled samples in the training set. The resulting Gaussian process serves as a nonlinear regression function that fits the given dataset. This function can be treated as the probability for data samples' the class identity and used for classificationas as shown before. This Gaussian process approach also has some two advantages: first, the certainty (or confidence) of the regression or classification result can be quantitatively measured; second proper tradeoff between overfitting and underfitting can be made by adjusting a parameter for the covariance of the Gaussian process model.