The advances in performance measure characterization discussed in Chapters 3 and 4 have armed researchers with more precise estimates of classifier performance. However, these are not by themselves sufficient to fully evaluate the difference in performances between classifiers on one or more test domains. More precisely, even though the performance of different classifiers may be shown to be different on specified sets of data, it needs to be confirmed whether the observed differences are statistically significant and not merely coincidental. Chapter 5 started to look at this issue, but focused primarily on the objectivity and stability of the results. This can be construed as the first step to assessing the significance of a difference. Only in the case of the comparison of two classifiers on a single domain did the discussion actually move on to significance issues. Statistical significance testing, which is the subject of this chapter, enables researchers to move on to more precise assessments of significance of the results obtained (within certain constraints). The importance of statistical significance testing hence cannot be overstated. Nonetheless, the use of available statistical tools for such testing in the fields of machine learning and data mining has been limited at best. Researchers have concentrated on using the paired t test, many times inappropriately, to confirm the difference in classifiers' performance. Moreover, this has sometimes been done at the cost of excluding other, more appropriate, tests.

We conclude the discussion on various aspects of performance evaluation of learning algorithms by unifying these seemingly disparate parts and putting them in perspective. The raison d'être of the following discussion is to appreciate the breadth and depth of the overall evaluation process, emphasizing the fact that such evaluation experiments should not be put together in an ad hoc manner, as they are currently done in many cases, by merely selecting a random subset of some or all of the components discussed in various chapters so far. Indeed, a careful consideration is required of both the underlying evaluation requirements and, in this context, of the correlation between the different choices for each component of the evaluation framework.

This chapter attempts to give a brief snapshot of the various components of the evaluation framework and highlights some of their major dependencies. Moreover, for each component we also give a template of the various steps necessary to make appropriate choices along with some of the main concerns and interrelations to take into account with respect to both, other steps in a given component and other evaluation components themselves. Unfortunately, because of the intricate dependencies between various steps as well as components, it might seem necessary to make simultaneous choices and check their compatibility.

The general model evaluation framework should serve as a representative template and not as a definitive guide.

We reviewed the major components of the evaluation framework in the last chapters and described in detail the various techniques pertaining to each, together with their applications. The focus of this chapter is to complement this review by outlining various advancements that have been made relatively recently, but that have not yet become mainstream. We also look into some approaches aimed at addressing problems arising from the developments on the machine learning front in various application settings, such as ensemble classifiers.

Just as with the traditional developments in performance evaluation, recent attempts at improving as well as designing new performance metrics have led the way. These have resulted in both improvements to existing performance metrics, thereby claiming to ameliorate the issues with the current versions, and proposals for novel metrics aimed at addressing the areas of algorithm evaluation not satisfactorily addressed by current metrics. We discuss in brief some of these advancements in Section 8.1. In Section 8.2, we focus on the attempts at unifying these performance metrics as well as studying their interrelation in the form of both theoretical and experimental frameworks. A natural extension to such studies is the design of more general or broader measures of performance, either as a result of insights obtained from the theoretical framework or by combining existing metrics based on observations from the experimental frameworks. Such metric combinations for obtaining general measures are the focus of Section 8.3.

We have discussed different aspects and components pertaining to the evaluation of learning algorithms. Given one or more fixed domains, we have discussed various performance measures, a number of sampling and resampling methods designed to estimate the outcome of these performance measures in a reliable manner and tests allowing us to estimate the statistical significance of the observed results. Many other aspects linked to each of these steps were also surveyed along the line, such as the notion of bias and variance and the debate on the need to practice statistical significance testing. The only aspect of evaluation that was not questioned, so far, is the issue of determining an appropriate testbed for our experiments. All the components that we discussed so far in this book made the implicit assumption expressed in the second sentence of this chapter: “Given one or more fixed domains.” It is now time to expand on this issue because the application, results, and subsequent interpretation of the different components of the evaluation process depend critically on the domains on which these are assessed and quantified. Furthermore, selecting datasets for evaluating the algorithms is certainly a nontrivial issue.

One important result connected to the choice of datasets on which to evaluate learning algorithms is summarized in Wolpert's “No Free Lunch” theorems. These theorems show the importance of evaluating algorithms on a large number of problems, because, if only a small sample of problems is considered, the results could be biased.

This chapter is aimed at establishing the conceptual foundation of the relevant aspects of machine learning and statistics on which the book rests. This very brief overview is in no way exhaustive. Rather, our main aim is to elucidate the relationship of these concepts to the performance evaluation of learning algorithms. The chapter is composed of two parts. The first part discusses concepts most specific to machine learning; the second part focuses on the statistical elements. Even though these may seem like two disparate parts, they are not entirely independent. We try to highlight the relationship between the concepts discussed in one field to the problems at hand in the other. Let us start with a brief discussion of the important concepts of machine learning.

Machine Learning Overview

Learning is the human process that allows us to acquire the skills necessary to adapt to the multitude of situations that we encounter throughout our lives. As human beings, we rely on many different kinds of learning processes at different stages to acquire different functionalities. We learn a variety of different skills, e.g., motor, verbal, mathematical, and so on. Moreover, the learning process differs with the situations and time, e.g., learning how to speak as a toddler is different from learning similar skill sets in a given profession. Variations in learning are also visible in terms of contexts and the related tools, e.g., classroom learning differs from social contexts, rote learning may be more suitable for memorizing but differs from learning how to reason.

This book was started at Monash University (Melbourne, Australia) and Laval University (Quebec City, Canada) with the subsequent writing taking place at the University of Ottawa (Ottawa, Canada) and McGill University (Montreal, Canada). The main idea stemmed from the observation that while machine learning as a field is maturing, the importance of evaluation has not received due appreciation from the developers of learning systems. Although almost all studies make a case for the evaluation of the algorithms they present, we find that many (in fact a majority) demonstrate a limited understanding of the issues involved in proper evaluation, despite the best intention of their authors. We concede that optimal choices cannot always be made due to limiting circumstances, and trade-offs are inevitable. However, the methods adopted in many cases do not reflect attention to the details warranted by a proper evaluation approach (of course there are exceptions and we do not mean to generalize this observation).

Our aim here is not to present the readers with yet another recipe for evaluation that can replace the current default approach. Rather, we try to develop an understanding of and appreciation for the different concerns of importance in the practical application and deployment of learning systems. Once these concerns are well understood, the other pieces of the puzzle fall quickly in place since the researcher is not left shooting in the dark.

Technological advances in recent decades have made it possible to automate many tasks that previously required significant amounts of manual time, performing regular or repetitive activities. Certainly, computing machines have proven to be a great asset in improving human speed and efficiency as well as in reducing errors in these essentially mechanical tasks. More impressive, however, is the fact that the emergence of computing technologies has also enabled the automation of tasks that require significant understanding of intrinsically human domains that can in no way be qualified as merely mechanical. Although we humans have maintained an edge in performing some of these tasks, e.g., recognizing pictures or delineating boundaries in a given picture, we have been less successful at others, e.g., fraud or computer network attack detection, owing to the sheer volume of data involved and to the presence of nonlinear patterns to be discerned and analyzed simultaneously within these data. Machine learning and datamining, on the other hand, have heralded significant advances, both theoretical and applied, in this direction, thus getting us one step closer to realizing such goals.

Machine learning is embodied by different learning approaches, which are themselves implemented within various frameworks. Examples of some of the most prominent of these learning paradigms include supervised learning, in which the data labels are available and generally discrete; unsupervised learning, in which the data labels are unavailable; semisupervised learning, in which some, generally discrete, data labels are available, but not all; regression, in which the data labels are continuous; and reinforcement learning, in which learning is based on an agent policy optimization in a reward setting.

Tables B.1 and B.2 show the results obtained using 10-fold cross validation by c45 and NB on each instance of the labor data respectively as output by WEKA. The first column lists the instance number; the second column lists the instance label, where class 1 corresponds to class “bad” and class 2 corresponds to class “good”; the third column lists the predicted class, using the same naming convention; column 4 uses the “+” symbol to indicate whether the predicted label differs from the actual one and a blank if they are in agreement; finally, the last two values, which are complementary and add up to 1, indicate the confidence of their prediction. The first value indicates how much the classifier believes the instance to be of class 1 (bad), and the second indicates how much the classifier believes the instance to be of classs 2 (good). The dominant value is preceded by a “*” symbol and corresponds to the value of the predicted label.

Please note that the numbers denoting the instances in the first column are not sequential. After number 6 or 7 is reached, a 1–6 or 1–7 sequence is repeated. This is because every 1–6 or 1–7 sequence represents a different fold. Indeed, it can be seen that 10 different sequences are present in each classifier run, corresponding to the 10 folds of 10-fold cross-validation.

This appendix is a companion to Chapter 9. In particular, it discusses two case studies that illustrate the evaluation framework laid out in that chapter and whose details were discussed all throughout the book. The first case study focuses on a practical (albeit semiartificial) domain; the second uses datasets from the UCI Repository for Machine Learning. The two studies are now discussed in turn.

Illustrative Case Study 1

In this case study, we used the dataset generated by Health Canada for the 2008 ICDM Data Contest. The purpose of the data is to serve as a basis for construction of automated learning systems able to monitor the amount of a few particular xenon isotopes (radioxenon) released in the atmosphere in an effort to verify compliance of the global ban on nuclear tests (the Comprehensive Nuclear Test Ban Treaty or (CTBT). These isotopes, when released in some given pattern, are characteristic of nuclear explosions. What makes the problem difficult, however, is that the monitoring stations are typically not located at the site of the explosion. Instead, the isotopes are transported, over days or weeks, through various weather systems, toward these stations and, in the process, lose their characteristic pattern. This is further complicated by the fact that xenon isotopes in various quantities are present in the atmosphere at the sites of the monitoring stations. This is due to the release of such gases by perfectly legal civil nuclear plants such as medical isotope production facilities and nuclear power plants.

We saw in Chapters 3 and 4 the concerns that arise from having to choose appropriate performance measures. Once a performance measure is decided upon, the next obvious concern is to find a good method for testing the learning algorithm so as to obtain as unbiased an estimate of the chosen performance measure as possible. Also of interest is the related concern of whether the technique we use to obtain such an estimate brings us as close as possible to the true measure value.

Ideally we would have access to the entire population and test our classifiers on it. Even if the entire population were not available, if a lot of representative data from that population could be obtained, error estimation would be quite simple. It would consist of testing the algorithms on the data they were trained on. Although such an estimate, commonly known as the resubstitution error, is usually optimistically biased, as the number of instances in the dataset increases, it tends toward the true error rate. Realistically, however, we are given a significantly limited-sized sample of the population. Areliable alternative thus consists of testing the algorithm on a large set of unseen data points. This approach is commonly known as the holdout method. Unfortunately, such an approach still requires quite a lot of data for testing the algorithm's performance, which is relatively rare in most practical situation.

Our discussion in the last chapter focused on performance measures that relied solely on the information obtained from the confusion matrix. Consequently it did not take into consideration measures that either incorporate information in addition to that conveyed by the confusion matrix or account for classifiers that are not discrete. In this chapter, we extend our discussion to incorporate some of these measures. In particular, we focus on measures associated with scoring classifiers. A scoring classifier typically outputs a real-valued score on each instance. This real-valued score need not necessarily be the likelihood of the test instance over a class, although such probabilistic classifiers can be considered to be a special case of scoring classifiers. The scores output by the classifiers over the test instances can then be thresholded to obtain class memberships for instances (e.g., all examples with scores above the threshold are labeled as positive, whereas those with scores below it are labeled as negative). Graphical analysis methods and the associated performance measures have proven to be very effective tools in studying both the behavior and the performance of such scoring classifiers. Among these, the receiver operating characteristic (ROC) analysis has shown significant promise and hence has gained considerable popularity as a graphical measure of choice. We discuss ROC analysis in significant detail. We also discuss some alternative graphical measures that can be applied depending on the domain of application and assessment criterion of interest.