Skip to main content Accessibility help
×
  1. Home
  2. Browse subjects
  3. Computer Science
  4. Pattern Recognition and Machine Learning
  5. Content listing

Refine search

Refine search

Access:
Content type:
Publication date:
Apply   From and To filters
Subject: Show more
Journals Show more
Publishers: Show more
Societies Show more
Series: Show more
Collections: Show more

Actions for selected content:

Select all | Deselect all
  • View selected items
  • Export citations
  • Download PDF (zip)
  • Save to Kindle
  • Save to Dropbox
  • Save to Google Drive

Save Search

You can save your searches here and later view and run them again in "My saved searches".

Please provide a title, maximum of 40 characters.
Cancel
×

2275 results in Pattern Recognition and Machine Learning

Page 77 of 114

  • 9 - Dimension reduction techniques

  • from Part III - Methods for large scale machine learning
  • Jeremy Watt, Northwestern University, Illinois, Reza Borhani, Northwestern University, Illinois, Aggelos K. Katsaggelos, Northwestern University, Illinois
  • Book:
    Machine Learning Refined
    Published online:
    05 September 2016
    Print publication:
    08 September 2016, pp 245-262

  • Summary

    Large datasets, as well as data consisting of a large number of features, present computational problems in the training of predictive models. In this chapter we discuss several useful techniques for reducing the dimension of a given dataset, that is reducing the number of data points or number of features, often employed in order to make predictive learning methods scale to larger datasets. More specifically, we discuss widely used methods for reducing the data dimension, that is the number of data points, of a dataset including random subsampling and K-means clustering. We then detail a common way of reducing the feature dimension, or number features, of a dataset as explained in Fig. 9.1. A classical approach for feature dimension reduction, principal component analysis (PCA), while often used for general data analysis is a relatively poor tool for reducing the feature dimension of predictive modeling data. However, PCA presents a fundamental mathematical archetype, the matrix factorization, that provides a very useful way of organizing our thinking about a wide array of important learning models (including linear regression, K-means, recommender systems – introduced after detailing PCA in this chapter), all of which may be thought of as variations of the simple theme of matrix factorization.

    Techniques for data dimension reduction

    In this section we detail two common ways of reducing the data dimension of a dataset: random subsampling and K-means clustering.

    Random subsampling

    Random subsampling is a simple and intuitive way of reducing the data dimension of a dataset, and is often the first approach employed when performing regression/ classification on datasets too large for available computational resources. Given a set of P points we keep a random subsample of S < P of the entire set. Clearly the smaller we choose S the larger the chance we may loose an important structural characteristic of the underlying dataset (for example the geometry of the separating boundary between two classes of data). While there is no formula or hard rule for how large S should be, a simple guideline used in practice is to choose S as large as possible given the computational resources available so as to minimize this risk.

  • Part I - Fundamental tools and concepts

  • Jeremy Watt, Northwestern University, Illinois, Reza Borhani, Northwestern University, Illinois, Aggelos K. Katsaggelos, Northwestern University, Illinois
  • Book:
    Machine Learning Refined
    Published online:
    05 September 2016
    Print publication:
    08 September 2016, pp 19-20

  • Summary

    In the three chapters that follow we describe in significant detail the basic concepts of machine learning introduced in Chapter 1, beginning with an introduction to several fundamental tools of numerical optimization in Chapter 2. This includes a thorough description of calculus-defined optimality, as well as the widely used gradient descent and Newton's method algorithms. Discussing these essential tools first will enable us to immediately and effectively deal with all of the formal learning problems we will see throughout the entirety of the text. Chapters 3 and 4 then introduce linear regression and classification respectively, the two predictive learning problems which form the bedrock of modern machine learning. We motivate both problems naturally, letting illustrations and geometric intuition guide our formal derivations, and in each case describe (through a variety of examples) how knowledge is used to forge effective features.

Machine Learning Refined
  • Foundations, Algorithms, and Applications
  • Jeremy Watt, Reza Borhani, Aggelos K. Katsaggelos
  • Published online:
    05 September 2016
    Print publication:
    08 September 2016
  • Providing a unique approach to machine learning, this text contains fresh and intuitive, yet rigorous, descriptions of all fundamental concepts necessary to conduct research, build products, tinker, and play. By prioritizing geometric intuition, algorithmic thinking, and practical real world applications in disciplines including computer vision, natural language processing, economics, neuroscience, recommender systems, physics, and biology, this text provides readers with both a lucid understanding of foundational material as well as the practical tools needed to solve real-world problems. With in-depth Python and MATLAB/OCTAVE-based computational exercises and a complete treatment of cutting edge numerical optimization techniques, this is an essential resource for students and an ideal reference for researchers and practitioners working in machine learning, computer science, electrical engineering, signal processing, and numerical optimization.

  • 13 - Objective Bayes Inference and MCMC

  • from Part II - Early Computer-Age Methods
  • Bradley Efron, Stanford University, California, Trevor Hastie, Stanford University, California
  • Book:
    Computer Age Statistical Inference
    Published online:
    05 July 2016
    Print publication:
    21 July 2016, pp 233-263

  • Summary

    From its very beginnings, Bayesian inference exerted a powerful influence on statistical thinking. The notion of a single coherent methodology employing only the rules of probability to go from assumption to conclusion was and is immensely attractive. For 200 years, however, two impediments stood between Bayesian theory's philosophical attraction and its practical application.

  • In the absence of relevant past experience, the choice of a prior distribution introduces an unwanted subjective element into scientific inference.

  • Bayes’ rule (3.5) looks simple enough, but carrying out the numerical calculation of a posterior distribution often involves intricate higherdimensional integrals.

    • The two impediments fit neatly into the dichotomy of Chapter 1, the first being inferential and the second algorithmic.

    A renewed cycle of Bayesian enthusiasm took hold in the 1960s, at first concerned mainly with coherent inference. Building on work by Bruno de Finetti and L. J. Savage, a principled theory of subjective probability was constructed: the Bayesian statistician, by the careful elicitation of prior knowledge, utility, and belief, arrives at the correct subjective prior distribution for the problem at hand. Subjective Bayesianism is particularly appropriate for individual decision making, say for the business executive trying to choose the best investment in the face of uncertain information.

    It is less appropriate for scientific inference, where the sometimes skeptical world of science puts a premium on objectivity. An answer came from the school of objective Bayes inference. Following the approach of Laplace and Jeffreys, as discussed in Section 3.2, their goal was to fashion objective, or “uninformative,” prior distributions that in some sense were unbiased in their effects upon the data analysis.

    In what came as a surprise to the Bayes community, the objective school has been the most successful in bringing Bayesian ideas to bear on scientific data analysis. Of the 24 articles in the December 2014 issue of the Annals of Applied Statistics, 8 employed Bayesian analysis, predominantly based on objective priors.

    This is where electronic computation enters the story. Commencing in the 1980s, dramatic steps forward were made in the numerical calculation of high-dimensional Bayes posterior distributions. Markov chain Monte Carlo (MCMC) is the generic name for modern posterior computation algorithms.

  • 16 - Sparse Modeling and the Lasso

  • from Part III - Twenty-First-Century Topics
  • Bradley Efron, Stanford University, California, Trevor Hastie, Stanford University, California
  • Book:
    Computer Age Statistical Inference
    Published online:
    05 July 2016
    Print publication:
    21 July 2016, pp 298-323

  • Summary

    The amount of data we are faced with keeps growing. From around the late 1990s we started to see wide data sets, where the number of variables far exceeds the number of observations. This was largely due to our increasing ability to measure a large amount of information automatically. In genomics, for example, we can use a high-throughput experiment to automatically measure the expression of tens of thousands of genes in a sample in a short amount of time. Similarly, sequencing equipment allows us to genotype millions of SNPs (single-nucleotide polymorphisms) cheaply and quickly. In document retrieval and modeling, we represent a document by the presence or count of each word in the dictionary. This easily leads to a feature vector with 20,000 components, one for each distinct vocabulary word, although most would be zero for a small document. If we move to bi-grams or higher, the feature space gets really large.

    In even more modest situations, we can be faced with hundreds of variables. If these variables are to be predictors in a regression or logistic regression model, we probably do not want to use them all. It is likely that a subset will do the job well, and including all the redundant variables will degrade our fit. Hence we are often interested in identifying a good subset of variables. Note also that in these wide-data situations, even linear models are over-parametrized, so some form of reduction or regularization is essential.

    In this chapter we will discuss some of the popular methods for model selection, starting with the time-tested and worthy forward-stepwise approach. We then look at the lasso, a popular modern method that does selection and shrinkage via convex optimization. The LARs algorithm ties these two approaches together, and leads to methods that can deliver paths of solutions.

    Finally, we discuss some connections with other modern big-and widedata approaches, and mention some extensions.

    Forward Stepwise Regression

    Stepwise procedures have been around for a very long time. They were originally devised in times when data sets were quite modest in size, in particular in terms of the number of variables. Originally thought of as the poor cousins of “best-subset” selection, they had the advantage of being much cheaper to compute (and in fact possible to compute for large p).We will review best-subset regression first.

  • 17 - Random Forests and Boosting

  • from Part III - Twenty-First-Century Topics
  • Bradley Efron, Stanford University, California, Trevor Hastie, Stanford University, California
  • Book:
    Computer Age Statistical Inference
    Published online:
    05 July 2016
    Print publication:
    21 July 2016, pp 324-350

  • Summary

    In the modern world we are often faced with enormous data sets, both in terms of the number of observations n and in terms of the number of variables p. This is of course good news—we have always said the more data we have, the better predictive models we can build. Well, we are there now—we have tons of data, and must figure out how to use it.

    Although we can scale up our software to fit the collection of linear and generalized linear models to these behemoths, they are often too modest and can fall way short in terms of predictive power. A need arose for some general purpose tools that could scale well to these bigger problems, and exploit the large amount of data by fitting a much richer class of functions, almost automatically. Random forests and boosting are two relatively recent innovations that fit the bill, and have become very popular as “out-thebox” learning algorithms that enjoy good predictive performance. Random forests are somewhat more automatic than boosting, but can also suffer a small performance hit as a consequence.

    These two methods have something in common: they both represent the fitted model by a sum of regression trees. We discuss trees in some detail in Chapter 8. A single regression tree is typically a rather weak prediction model; it is rather amazing that an ensemble of trees leads to the state of the art in black-box predictors!

    We can broadly describe both these methods very simply.

    Random forest Grow many deep regression trees to randomized versions of the training data, and average them. Here “randomized” is a wideranging term, and includes bootstrap sampling and/or subsampling of the observations, as well as subsampling of the variables.

    Boosting Repeatedly grow shallow trees to the residuals, and hence build up an additive model consisting of a sum of trees.

    The basic mechanism in random forests is variance reduction by averaging. Each deep tree has a high variance, and the averaging brings the variance down. In boosting the basic mechanism is bias reduction, although different flavors include some variance reduction as well. Both methods inherit all the good attributes of trees, most notable of which is variable selection.

Page 77 of 114