The data explosion

We live in a world that is rich in data, ever increasing in scale. This data comes from many different sources in science (bioinformatics, astronomy, physics, environmental monitoring) and commerce (customer databases, financial transactions, engine monitoring, speech recognition, surveillance, search). Possessing the knowledge as to how to process and extract value from such data is therefore a key and increasingly important skill. Our society also expects ultimately to be able to engage with computers in a natural manner so that computers can ‘talk’ to humans, ‘understand’ what they say and ‘comprehend’ the visual world around them. These are difficult large-scale information processing tasks and represent grand challenges for computer science and related fields. Similarly, there is a desire to control increasingly complex systems, possibly containing many interacting parts, such as in robotics and autonomous navigation. Successfully mastering such systems requires an understanding of the processes underlying their behaviour. Processing and making sense of such large amounts of data from complex systems is therefore a pressing modern-day concern and will likely remain so for the foreseeable future.

Machine learning

Machine learning is the study of data-driven methods capable of mimicking, understanding and aiding human and biological information processing tasks. In this pursuit, many related issues arise such as how to compress data, interpret and process it. Often these methods are not necessarily directed to mimicking directly human processing but rather to enhancing it, such as in predicting the stock market or retrieving information rapidly. In this probability theory is key since inevitably our limited data and understanding of the problem forces us to address uncertainty.

In this chapter we discuss some classical distributions and their manipulations. In previous chapters we've assumed that we know the distributions and have concentrated on the inference problem. In machine learning we will typically not fully know the distributions and need to learn them from available data. This means we need familiarity with standard distributions, for which the data will later be used to set the parameters.

Representing data

The numeric encoding of data can have a significant effect on performance and an understanding of the options for representing data is therefore of considerable importance. We briefly outline three central encodings below.

Categorical

For categorical (or nominal) data, the observed value belongs to one of a number of classes, with no intrinsic ordering, and can be represented simply by an integer. An example of a categorical variable would be the description of the type of job that someone does, e.g. healthcare, education, financial services, transport, homeworker, unemployed, engineering etc. which could be represented by the values 1, 2, …, 7. Another way to transform such data into numerical values would be to use 1-of-m encoding. For example, if there are four kinds of jobs: soldier, sailor, tinker, spy, we could represent a soldier as (1,0,0,0), a sailor as (0,1,0,0), a tinker as (0,0,1,0) and a spy as (0,0,0,1). In this encoding the distance between the vectors representing two different professions is constant.

Machine learning is the body of research related to automated large-scale data analysis. Historically, the field was centred around biologically inspired models and the long-term goals of much of the community are oriented to producing models and algorithms that can process information as well as biological systems. The field also encompasses many of the traditional areas of statistics with, however, a strong focus on mathematical models and also prediction. Machine learning is now central to many areas of interest in computer science and related large-scale information processing domains.

Styles of learning

Broadly speaking the main two subfields of machine learning are supervised learning and unsupervised learning. In supervised learning the focus is on accurate prediction, whereas in unsupervised learning the aim is to find compact descriptions of the data. In both cases, one is interested in methods that generalise well to previously unseen data. In this sense, one distinguishes between data that is used to train a model and data that is used to test the performance of the trained model, see Fig. 13.1. We discuss first the basic characteristics of some learning frameworks before discussing supervised learning in more detail.

Supervised learning

Consider a database of face images, each represented by a vector x. Along with each image x is an output class y ∈ {male, female} that states if the image is of a male or female.

Machine learning is fundamentally about extracting value from large datasets. Often themotivation is ultimately to produce an algorithm that can either mimic or enhance human/biological performance.

In Part III we begin by discussing some of the basic concepts in machine learning, namely supervised and unsupervised learning. We then move on to discuss some standard models in machine learning and related areas.

High-dimensional data is prevalent in machine learning and related areas. Indeed, there often arises the situation in which there are more data dimensions than there are data examples. In such cases we seek a lower-dimensional representation of the data. In this chapter we discuss some standard methods which can also improve the prediction performance by removing ‘noise’ from the representation.

High-dimensional spaces – low-dimensional manifolds

In machine learning problems data is often high dimensional – images, bag-of-word descriptions, gene-expressions etc. In such cases we cannot expect the training data to densely populate the space, meaning that there will be large parts in which little is known about the data. For the hand written digits from Chapter 14, the data is 784 dimensional and for binary valued pixels the number of possible images is 2784 ≈ 10236. Nevertheless, we would expect that only a handful of examples of a digit should be sufficient (for a human) to understand how to recognise a 7. Digit-like images must therefore occupy a highly constrained volume in the 784 dimensions and we expect only a small number of degrees of freedom to be required to describe the data to a reasonable accuracy. Whilst the data vectors may be very high dimensional, they will therefore typically lie close to a much lower-dimensional ‘manifold’ (informally, a two-dimensional manifold corresponds to a warped sheet of paper embedded in a high-dimensional space), meaning that the distribution of the data is heavily constrained.