Skip to main content Accessibility help
×
  1. Home
  2. Browse subjects
  3. Computer Science
  4. Distributed, Networked and Mobile Computing
  5. Content listing

Refine search

Refine search

Access:
Content type:
Publication date:
Apply   From and To filters
Subject: Show more
Journals Show more
Publishers: Show more
Societies Show more
Series: Show more
Collections: Show more

Actions for selected content:

Select all | Deselect all
  • View selected items
  • Export citations
  • Download PDF (zip)
  • Save to Kindle
  • Save to Dropbox
  • Save to Google Drive

Save Search

You can save your searches here and later view and run them again in "My saved searches".

Please provide a title, maximum of 40 characters.
Cancel
×

1983 results in Distributed, Networked and Mobile Computing

Page 44 of 100

  • 19 - Large-Scale FPGA-Based Convolutional Networks

  • from Part Four - Applications
  • Edited by Ron Bekkerman, Mikhail Bilenko, John Langford
  • Book:
    Scaling up Machine Learning
    Published online:
    05 February 2012
    Print publication:
    30 December 2011, pp 399-419

  • Summary

    Micro-robots, unmanned aerial vehicles, imaging sensor networks, wireless phones, and other embedded vision systems all require low cost and high-speed implementations of synthetic vision systems capable of recognizing and categorizing objects in a scene.

    Many successful object recognition systems use dense features extracted on regularly spaced patches over the input image. The majority of the feature extraction systems have a common structure composed of a filter bank (generally based on oriented edge detectors or 2D Gabor functions), a nonlinear operation (quantization, winner-take-all, sparsification, normalization, and/or pointwise saturation), and finally a pooling operation (max, average, or histogramming). For example, the scale-invariant feature transform (SIFT) (Lowe, 2004) operator applies oriented edge filters to a small patch and determines the dominant orientation through a winner-take-all operation. Finally, the resulting sparse vectors are added (pooled) over a larger patch to form a local orientation histogram. Some recognition systems use a single stage of feature extractors (Lazebnik, Schmid, and Ponce, 2006; Dalal and Triggs, 2005; Berg, Berg, and Malik, 2005; Pinto, Cox, and DiCarlo, 2008).

    Other models such as HMAX-type models (Serre, Wolf, and Poggio, 2005; Mutch, and Lowe, 2006) and convolutional networks use two more layers of successive feature extractors. Different training algorithms have been used for learning the parameters of convolutional networks. In LeCun et al. (1998b) and Huang and LeCun (2006), pure supervised learning is used to update the parameters. However, recent works have focused on training with an auxiliary task (Ahmed et al., 2008) or using unsupervised objectives (Ranzato et al., 2007b; Kavukcuoglu et al., 2009; Jarrett et al., 2009; Lee et al., 2009).

  • 2 - MapReduce and Its Application to Massively Parallel Learning of Decision Tree Ensembles

  • from Part One - Frameworks for Scaling Up Machine Learning
  • Edited by Ron Bekkerman, Mikhail Bilenko, John Langford
  • Book:
    Scaling up Machine Learning
    Published online:
    05 February 2012
    Print publication:
    30 December 2011, pp 23-48

  • Summary

    In this chapter we look at leveraging the MapReduce distributed computing framework (Dean and Ghemawat, 2004) for parallelizing machine learning methods of wide interest, with a specific focus on learning ensembles of classification or regression trees. Building a production-ready implementation of a distributed learning algorithm can be a complex task. With the wide and growing availability of MapReduce-capable computing infrastructures, it is natural to ask whether such infrastructures may be of use in parallelizing common data mining tasks such as tree learning. For many data mining applications, MapReduce may offer scalability as well as ease of deployment in a production setting (for reasons explained later).

    We initially give an overview of MapReduce and outline its application in a classic clustering algorithm, k-means. Subsequently, we focus on PLANET: a scalable distributed framework for learning tree models over large datasets. PLANET defines tree learning as a series of distributed computations and implements each one using the MapReduce model. We show how this framework supports scalable construction of classification and regression trees, as well as ensembles of such models. We discuss the benefits and challenges of using a MapReduce compute cluster for tree learning and demonstrate the scalability of this approach by applying it to a real-world learning task from the domain of computational advertising.

    MapReduce is a simple model for distributed computing that abstracts away many of the difficulties in parallelizing data management operations across a cluster of commodity machines.

  • 5 - Uniformly Fine-Grained Data-Parallel Computing for Machine Learning Algorithms

  • from Part One - Frameworks for Scaling Up Machine Learning
  • Edited by Ron Bekkerman, Mikhail Bilenko, John Langford
  • Book:
    Scaling up Machine Learning
    Published online:
    05 February 2012
    Print publication:
    30 December 2011, pp 89-106

  • Summary

    The graphics processing unit (GPU) of modern computers has evolved into a powerful, general-purpose, massively parallel numerical (co-)processor. The numerical computation in a number of machine learning algorithms fits well on the GPU. To help identify such algorithms, we present uniformly fine-grained data-parallel computing and illustrate it on two machine learning algorithms, clustering and regression clustering, on a GPU and central processing unit (CPU) mixed computing architecture. We discuss the key issues involved in a successful design of the algorithms, data structures, and computation partitioning between a CPU and a GPU. Performance gains on a CPU and GPU mixed architecture are compared with the performance of the regression clustering algorithm implemented completely on a CPU. Significant speedups are reported. A GPU and CPU mixed architecture also achieves better cost-performance and energy-performance ratios.

    The computing power of the CPU has increased dramatically in the past few decades, supported by both miniaturization and increasing clock frequencies. More and more electronic gates were packed onto the same area of a silicon die as miniaturization continued. Hardware-supported parallel computing, pipelining for example, further increased the computing power of CPUs. Frequency increases speeded up CPUs even more directly. However, the long-predicted physical limit of the miniaturization process was finally hit a few years ago such that increasing the frequency was no longer feasible due to the accompanied nonlinear increase in power consumption, even though miniaturization still continues.

  • 4 - IBM Parallel Machine Learning Toolbox

  • from Part One - Frameworks for Scaling Up Machine Learning
  • Edited by Ron Bekkerman, Mikhail Bilenko, John Langford
  • Book:
    Scaling up Machine Learning
    Published online:
    05 February 2012
    Print publication:
    30 December 2011, pp 69-88

  • Summary

    In many ways, the objective of the IBM Parallel Machine Learning Toolbox (PML) is similar to that of Google's MapReduce programming model (Dean and Ghemawat, 2004) and the open source Hadoop system, which is to provide Application Programming Interfaces (APIs) that enable programmers who have no prior experience in parallel and distributed systems to nevertheless implement parallel algorithms with relative ease. Like MapReduce and Hadoop, PML supports associative-commutative computations as its primary parallelization mechanism. Unlike MapReduce and Hadoop, PML fundamentally assumes that learning algorithms can be iterative in nature, requiring multiple passes over data. It also extends the associative-commutative computational model in various aspects, the most important of which are:

    1. The ability to maintain the state of each worker node between iterations, making it possible, for example, to partition and distribute data structures across workers

    2. Efficient distribution of data, including the ability for each worker to read a subset of the data, to sample the data, or to scan the entire dataset

    3. Access to both sparse and dense datasets

    4. Parallel merge operations using tree structures for efficient collection of worker results on very large clusters

    In order to make these extensions to the computational model and still address ease of use, PML provides an object-oriented API in which algorithms are objects that implement a predefined set of interface methods. The PML infrastructure then uses these interface methods to distribute algorithm objects and their computations across multiple compute nodes.

  • 1 - Scaling Up Machine Learning: Introduction

  • Edited by Ron Bekkerman, Mikhail Bilenko, John Langford
  • Book:
    Scaling up Machine Learning
    Published online:
    05 February 2012
    Print publication:
    30 December 2011, pp 1-20

  • Summary

    Distributed and parallel processing of very large datasets has been employed for decades in specialized, high-budget settings, such as financial and petroleum industry applications. Recent years have brought dramatic progress in usability, cost effectiveness, and diversity of parallel computing platforms, with their popularity growing for a broad set of data analysis and machine learning tasks.

    The current rise in interest in scaling up machine learning applications can be partially attributed to the evolution of hardware architectures and programming frameworks that make it easy to exploit the types of parallelism realizable in many learning algorithms. A number of platforms make it convenient to implement concurrent processing of data instances or their features. This allows fairly straightforward parallelization of many learning algorithms that view input as an unordered batch of examples and aggregate isolated computations over each of them.

    Increased attention to large-scale machine learning is also due to the spread of very large datasets across many modern applications. Such datasets are often accumulated on distributed storage platforms, motivating the development of learning algorithms that can be distributed appropriately. Finally, the proliferation of sensing devices that perform real-time inference based on high-dimensional, complex feature representations drives additional demand for utilizing parallelism in learning-centric applications. Examples of this trend include speech recognition and visual object detection becoming commonplace in autonomous robots and mobile devices.

  • 15 - Parallel Graph-Based Semi-Supervised Learning

  • from Part Three - Alternative Learning Settings
  • Edited by Ron Bekkerman, Mikhail Bilenko, John Langford
  • Book:
    Scaling up Machine Learning
    Published online:
    05 February 2012
    Print publication:
    30 December 2011, pp 307-330

  • Summary

    Semi-supervised learning (SSL) is the process of training decision functions using small amounts of labeled and relatively large amounts of unlabeled data. In many applications, annotating training data is time consuming and error prone. Speech recognition is the typical example, which requires large amounts of meticulously annotated speech data (Evermann et al., 2005) to produce an accurate system. In the case of document classification for internet search, it is not even feasible to accurately annotate a relatively large number of web pages for all categories of potential interest. SSL lends itself as a useful technique in many machine learning applications because one need annotate only relatively small amounts of the available data. SSL is related to the problem of transductive learning (Vapnik, 1998). In general, a learner is transductive if it is designed for prediction on only a closed dataset, where the test set is revealed at training time. In practice, however, transductive learners can be modified to handle unseen data (Sindhwani, Niyogi, and Belkin, 2005; Zhu, 2005a). Chapter 25 in Chapelle, Scholkopf, and Zien (2007) gives a full discussion on the relationship between SSL and transductive learning. In this chapter, SSL refers to the semi-supervised transductive classification problem.

    Let x ∈ X denote the input to the decision function (classifier), f, and y ∈ Y denote its output label, that is, f : X → Y. In most cases f(x) = argmaxy∈Yp(y|x).

  • 8 - Large-Scale Learning to Rank Using Boosted Decision Trees

  • from Part Two - Supervised and Unsupervised Learning Algorithms
  • Edited by Ron Bekkerman, Mikhail Bilenko, John Langford
  • Book:
    Scaling up Machine Learning
    Published online:
    05 February 2012
    Print publication:
    30 December 2011, pp 148-169

  • Summary

    The web search ranking task has become increasingly important because of the rapid growth of the internet. With the growth of the web and the number of web search users, the amount of available training data for learning web ranking models has also increased. We investigate the problem of learning to rank on a cluster using web search data composed of 140,000 queries and approximately 14 million URLs. For datasets much larger than this, distributed computing will become essential, because of both speed and memory constraints. We compare a baseline algorithm that has been carefully engineered to allow training on the full dataset using a single machine, in order to evaluate the loss or gain incurred by the distributed algorithms we consider. The underlying algorithm we use is a boosted tree ranking algorithm called LambdaMART, where a split at a given vertex in each decision tree is determined by the split criterion for a particular feature. Our contributions are twofold. First, we implement a method for improving the speed of training when the training data fits in main memory on a single machine by distributing the vertex split computations of the decision trees. The model produced is equivalent to the model produced from centralized training, but achieves faster training times. Second, we develop a training method for the case where the training data size exceeds the main memory of a single machine. Our second approach easily scales to far larger datasets, that is, billions of examples, and is based on data distribution.

  • 12 - Large-Scale Spectral Clustering with Map Reduce and MPI

  • from Part Two - Supervised and Unsupervised Learning Algorithms
  • Edited by Ron Bekkerman, Mikhail Bilenko, John Langford
  • Book:
    Scaling up Machine Learning
    Published online:
    05 February 2012
    Print publication:
    30 December 2011, pp 240-261

  • Summary

    Spectral clustering is a technique for finding group structure in data. It makes use of the spectrum of the data similarity matrix to perform dimensionality reduction for clustering in fewer dimensions. Spectral clustering algorithms have been shown to be more effective in finding clusters than traditional algorithms such as k-means. However, spectral clustering suffers from a scalability problem in both memory use and computation time when the size of a dataset is large. To perform clustering on large datasets, in this work, we parallelize both memory use and computation using MapReduce and MPI. Through an empirical study on a document set of 534,135 instances and a photo set of 2,121,863 images, we show that our parallel algorithm can effectively handle large problems.

    Clustering is one of the most important subfields of machine learning and data mining tasks. In the last decade, spectral clustering (e.g., Shi and Malik, 2000; Meila and Shi, 2000; Fowlkes et al., 2004), motivated by normalized graph cut, has attracted much attention. Unlike traditional partition-based clustering, spectral clustering exploits a pairwise data similarity matrix. It has been shown to be more effective than traditional methods such as k-means, which considers only the similarity between instances and k centroids (Ng, Jordan, and Weiss, 2001).Because of its effectiveness, spectral clustering has been widely used in several areas such as information retrieval and computer vision (e.g., Dhillon, 2001; Xu, Liu, and Gong, 2003; Shi and Malik, 2000; Yu and Shi, 2003).

  • 9 - The Transform Regression Algorithm

  • from Part Two - Supervised and Unsupervised Learning Algorithms
  • Edited by Ron Bekkerman, Mikhail Bilenko, John Langford
  • Book:
    Scaling up Machine Learning
    Published online:
    05 February 2012
    Print publication:
    30 December 2011, pp 170-189

  • Summary

    Massive training datasets, ranging in size from tens of gigabytes to several terabytes, arise in diverse machine learning applications in areas such as text mining of web corpora, multimedia analysis of image and video data, retail modeling of customer transaction data, bioinformatic analysis of genomic and microarray data, medical analysis of clinical diagnostic data such as functional magnetic resonance imaging (fMRI) images, and environmental modeling using sensor and streaming data. Provost and Kolluri (1999) in their overview of machine learning with massive datasets, emphasize the need for developing parallel algorithms and implementations for these applications.

    In this chapter, we describe the Transform Regression (TReg) algorithm (Pednault, 2006), which is a general-purpose, non-parametric methodology suitable for a wide variety of regression applications. TReg was originally created for the data mining component of the IBM InfoSphere Warehouse product, guided by a challenging set of requirements:

    1. The modeling time should be comparable to linear regression.

    2. The resulting models should be compact and efficient to apply.

    3. The model quality should be reliable without any further tuning.

    4. The model training and scoring should be parallelized for large datasets stored as partitioned tables in IBM's DB2 database systems.

    Requirements 1 and 2 were deemed necessary for a successful commercial algorithm, although this ruled out certain ensemble-based methods that produce highquality models but have high computation and storage requirements. Requirement 3 ensured that the chosen algorithm did not unduly compromise the concomitant model quality in view of requirements 1 and 2.

  • 7 - Massive SVM Parallelization Using Hardware Accelerators

  • from Part Two - Supervised and Unsupervised Learning Algorithms
  • Edited by Ron Bekkerman, Mikhail Bilenko, John Langford
  • Book:
    Scaling up Machine Learning
    Published online:
    05 February 2012
    Print publication:
    30 December 2011, pp 127-147

  • Summary

    Support Vector Machines (SVMs) are some of the most widely used classification and regression algorithms for data analysis, pattern recognition, or cognitive tasks. Yet learning problems that can be solved by SVMs are limited in size because of high computational cost and excessive storage requirements. Many variations of the original SVM algorithm were introduced that scale better to large problems. They change the SVM framework quite drastically, such as apply optimizations other than the maximum margin, or introduce different error metrics for the cost function. Such algorithms may work for some applications, but they do not have the robustness and universality that make SVMs so popular.

    The approach taken here is to maintain the SVM algorithm in its original form and scale it to large problems through parallelization. Computer performance cannot be improved anymore at the pace of the last few decades by increasing the clock frequencies. Today, significant accelerations are achieved mostly through parallel architectures, and multicore processors are commonplace nowadays. Mapping the SVM algorithm to multicore processors with shared-memory architectures is straightforward, yet this approach does not scale to a large number of processors. Here we investigate parallelization concepts that scale to hundreds and thousands of cores where, for example, cache coherence can no longer be maintained.

    A number of SVM implementations on clusters or graphics processors (GPUs) have been proposed recently. A parallel optimization algorithm based on gradient projections has been demonstrated (see Zanghirati, and Zanni, 2003; Zanni, Serafini, and Zanghirati, 2006) that uses a spectral gradient method for fast convergence while maintaining the Karush-Kuhn-Tucker (KKT) constraints.

  • 16 - Distributed Transfer Learning via Cooperative Matrix Factorization

  • from Part Three - Alternative Learning Settings
    • By Evan Xiang, Hong Kong University of Science and Technology, Nathan Liu, Hong Kong University of Science and Technology, Qiang Yang, Hong Kong University of Science and Technology
  • Edited by Ron Bekkerman, Mikhail Bilenko, John Langford
  • Book:
    Scaling up Machine Learning
    Published online:
    05 February 2012
    Print publication:
    30 December 2011, pp 331-351

  • Summary

    Machine learning and data-mining technologies have already achieved significant success in many knowledge engineering areas including web search, computational advertising, recommender systems, etc. A major challenge in machine learning is the data sparsity problem. For example, in the domain of online recommender systems, we attempt to recommend information items (e.g., movies, TV, books, news, images, web pages, etc.) that are likely to be of interest to the user. However, the item space is usually very large and the amount of user preference values is small. When the user data are too sparse, it is difficult to obtain a reliable and useful model for recommendation. Whereas large online sites like Amazon and Google can easily access huge volumes of user data, the enormous number of smaller online business sites, which collectively constitute the long tail of the web, are much more likely to have very sparse user data and have difficulty in generating accurate recommendations. One potential solution to the data sparsity problem is to transfer knowledge from other information sources (e.g., Mehta and Hofmann, 2007; Li, Yang, and Xue, 2009). Such techniques for knowledge transfer are called transfer learning (see, e.g., Pan and Yang, 2010). An additional issue is that, in reality, many small websites often attract similar users and/or provide similar items, if not the identical ones, which implies that data about such users/items could potentially be distributed across different systems. For example, Delicious and Digg are both popular online social bookmarking tools.

  • 17 - Interactive Channel Coding

  • from Part III - Multihop Networks
  • Abbas El Gamal, Young-Han Kim, University of California, San Diego
  • Book:
    Network Information Theory
    Published online:
    05 June 2012
    Print publication:
    08 December 2011, pp 427-458

  • Summary

    The network models we studied so far involve only one-way (feedforward) communication. Many communication systems are inherently interactive, allowing for cooperation through feedback and information exchange over multiway channels. In this chapter, we study the role of feedback in communication and present results on the two-way channel introduced by Shannon as the first multiuser channel. The role of multiway interaction in compression and secure communication will be studied in Chapters 20 and 22, respectively.

    As we showed in Section 3.1.1, the capacity of a memoryless point-to-point channel does not increase when noiseless causal feedback is present. Feedback can still benefit point-to-point communication, however, by simplifying coding and improving reliability. The idea is to first send the message uncoded and then to use feedback to iteratively reduce the receiver's error about the message, the error about the error, and so on. We demonstrate this iterative refinement paradigm via the Schalkwijk–Kailath coding scheme for the Gaussian channel and the Horstein and block feedback coding schemes for the binary symmetric channel. We show that the probability of error for the Schalkwijk–Kailath scheme decays double-exponentially in the block length, which is significantly faster than the single-exponential decay of the probability of error without feedback.

    We then show that feedback can enlarge the capacity region in multiuser channels. For the multiple access channel, feedback enlarges the capacity region by enabling statistical cooperation between the transmitters. We show that the capacity of the Gaussian MAC with feedback coincides with the outer bound obtained by allowing arbitrary (instead of product) joint input distributions. For the broadcast channel, feedback can enlarge the capacity region by enabling the sender to simultaneously refine both receivers’ knowledge about the messages. For the relay channel, we show that the cutset bound is achievable when noiseless causal feedback from the receiver to the relay is allowed. This is in contrast to the case without feedback in which the cutset bound is not achievable in general.

    Finally, we discuss the two-way channel, where two nodes wish to exchange their messages interactively over a shared noisy channel. The capacity region of this channel is not known in general. We first establish simple inner and outer bounds on the capacity region.

  • 13 - Multiple Description Coding

  • from Part II - Single-Hop Networks
  • Abbas El Gamal, Young-Han Kim, University of California, San Diego
  • Book:
    Network Information Theory
    Published online:
    05 June 2012
    Print publication:
    08 December 2011, pp 320-335

  • Summary

    We consider the problem of generating two descriptions of a source such that each description by itself can be used to reconstruct the source with some desired distortion and the two descriptions together can be used to reconstruct the source with a lower distortion. This problem is motivated by the need to efficiently communicate multimedia content over networks such as the Internet. Consider the following two scenarios:

  • • Path diversity: Suppose we wish to send a movie to a viewer over a network that suffers from data loss and delays. We can send multiple copies of the same description of the movie to the viewer via different paths in the network. Such replication, however, is inefficient and the viewer does not benefit from receiving more than one copy of the description. Multiple description coding provides a more efficient means to achieve such “path diversity.” We generate multiple descriptions of the movie, so that if the viewer receives only one of them, the movie can be reconstructed with some acceptable quality, and if the viewer receives two of them, the movie can be reconstructed with a higher quality and so on.

  • • Successive refinement: Suppose we wish to send a movie with different levels of quality to different viewers. We can send a separate description of the movie to each viewer. These descriptions, however, are likely to have significant overlaps. Successive refinement, which is a special case of multiple description coding, provides a more efficient way to distribute the movie. The idea is to send the lowest quality description and successive refinements of it (instead of additional full descriptions). Each viewer then uses the lowest quality description and some of the successive refinements to reconstruct the movie at her desired level of quality.

    • The optimal scheme for generating multiple descriptions is not known in general. We present the El Gamal–Cover coding scheme for generating two descriptions that are individually good but still carry additional information about the source when combined together. The proof of achievability uses the multivariate covering lemma in Section 8.4. We show that this scheme is optimal for the quadratic Gaussian case. The key to the converse is the identification of a common-information random variable. We then present an improvement on the El Gamal–Cover scheme by Zhang and Berger that involves sending an additional common description.

Page 44 of 100