Abstract

In this chapter, we show that compressed learning, learning directly in the compressed domain, is possible. In particular, we provide tight bounds demonstrating that the linear kernel SVM's classifier in the measurement domain, with high probability, has true accuracy close to the accuracy of the best linear threshold classifier in the data domain. We show that this is beneficial both from the compressed sensing and the machine learning points of view. Furthermore, we indicate that for a family of well-known compressed sensing matrices, compressed learning is provided on the fly. Finally, we support our claims with experimental results in the texture analysis application.

Introduction

In many applications, the data has a sparse representation in some basis in a much higher dimensional space. Examples are the sparse representation of images in the wavelet domain, the bag of words model of text, and the routing tables in data monitoring systems.

Compressed sensing combines measurement to reduce the dimensionality of the underlying data with reconstruction to recover sparse data from the projection in the measurement domain. However, there are many sensing applications where the objective is not full reconstruction but is instead classification with respect to some signature. Examples include radar, detection of trace chemicals, face detection [7, 8], and video streaming [9] where we might be interested in anomalies corresponding to changes in wavelet coefficients in the data domain. In all these cases our objective is pattern recognition in the measurement domain.

In this chapter, we present a comprehensive framework for tackling the classical problem of face recognition, based on theory and algorithms from sparse representation. Despite intense interest in the past several decades, traditional pattern recognition theory still stops short of providing a satisfactory solution capable of recognizing human faces in the presence of real-world nuisances such as occlusion and variabilities in pose and illumination. Our new approach, called sparse representation-based classification (SRC), is motivated by a very natural notion of sparsity, namely, one should always try to explain a query image using a small number of training images from a single subject category. This sparse representation is sought via ℓ1 minimization. We show how this core idea can be generalized and extended to account for various physical variabilities encountered in face recognition. The end result of our investigation is a full-fledged practical system aimed at security and access control applications. The system is capable of accurately recognizing subjects out of a database of several hundred subjects with state-of-the-art accuracy.

Introduction

Automatic face recognition is a classical problem in the computer vision community. The community's sustained interest in this problem is mainly due to two reasons. First, in face recognition, we encounter many of the common variabilities that plague vision systems in general: illumination, occlusion, pose, and misalignment. Inspired by the good performance of humans in recognizing familiar faces [38], we have reason to believe that effective automatic face recognition is possible, and that the quest to achieve this will tell us something about visual recognition in general.

Compressed sensing (CS) is an exciting, rapidly growing field that has attracted considerable attention in electrical engineering, applied mathematics, statistics, and computer science. Since its initial introduction several years ago, an avalanche of results have been obtained, both of theoretical and practical nature, and various conferences, workshops, and special sessions have been dedicated to this growing research field. This book provides the first comprehensive introduction to the subject, highlighting recent theoretical advances and a range of applications, as well as outlining numerous remaining research challenges.

CS offers a framework for simultaneous sensing and compression of finite-dimensional vectors, that relies on linear dimensionality reduction. Quite surprisingly, it predicts that sparse high-dimensional signals can be recovered from highly incomplete measurements by using efficient algorithms. To be more specific, let x be a length-n vector. In CS we do not measure x directly, but rather acquire m < n linear measurements of the form y = Ax using an m × n CS matrix A. Ideally, the matrix is designed to reduce the number of measurements as much as possible while allowing for recovery of a wide class of signals from their measurement vectors y. Thus, we would like to choose m ≪ n. However, this renders the matrix A rank-deficient, meaning that it has a nonempty nullspace. This implies that for any particular signal x0, an infinite number of signals x will yield the same measurements y = Ax = Ax0 for the chosen CS matrix.

This chapter surveys recent work in applying ideas from graphical models and message passing algorithms to solve large-scale regularized regression problems. In particular, the focus is on compressed sensing reconstruction via 11 penalized least-squares (known as LASSO or BPDN). We discuss how to derive fast approximate message passing algorithms to solve this problem. Surprisingly, the analysis of such algorithms allows one to prove exact high-dimensional limit results for the LASSO risk.

Introduction

The problem of reconstructing a high-dimensional vector x ∈ ℝn from a collection of observations y ∈ ℝm arises in a number of contexts, ranging from statistical learning to signal processing. It is often assumed that the measurement process is approximately linear, i.e. that

where A ∈ ℝm×n is a known measurement matrix, and w is a noise vector.

The graphical models approach to such a reconstruction problem postulates a joint probability distribution on (x, y) which takes, without loss of generality, the form

The conditional distribution p(dy|x) models the noise process, while the prior p(dx) encodes information on the vector x. In particular, within compressed sensing, it can describe its sparsity properties. Within a graphical models approach, either of these distributions (or both) factorizes according to a specific graph structure. The resulting posterior distribution p(dx|y) is used for inferring x given y.

There are many reasons to be skeptical about the idea that the joint probability distribution p(dx, dy) can be determined, and used for reconstructing x.

In this chapter the authors go beyond traditional sparse modeling, and address collaborative structured sparsity to add stability and prior information to the representation. In structured sparse modeling, instead of considering the dictionary atoms as singletons, the atoms are partitioned in groups, and a few groups are selected at a time for the signal encoding. A complementary way of adding structure, stability, and prior information to a model is via collaboration. Here, multiple signals, which are known to follow the same model, are allowed to collaborate in the coding. The first studied framework connects sparse modeling with Gaussian Mixture Models and leads to state-of-the-art image restoration. The second framework derives a hierarchical structure on top of the collaboration and is well fitted for source separation. Both models enjoy very important theoretical virtues as well.

Introduction

In traditional sparse modeling, it is assumed that a signal can be accurately represented by a sparse linear combination of atoms from a (learned) dictionary. A large class of signals, including most natural images and sounds, is well described by this model, as demonstrated by numerous state-of-the-art results in various signal processing applications.

From a data modeling point of view, sparsity can be seen as a form of regularization, that is, as a device to restrict or control the set of coefficient values which are allowed in the model to produce an estimate of the data.

Modern data are often composed of two or more morphologically distinct constituents, and one typical goal is the extraction of those components. Recently, sparsity methodologies have been successfully utilized to solve this problem, both theoretically as well as empirically. The key idea is to choose a deliberately overcomplete representation made of several frames each one providing a sparse expansion of one of the components to be extracted. The morphological difference between the components is then encoded as incoherence conditions of those frames. The decomposition principle is to minimize the ℓ1 norm of the frame coefficients. This chapter shall serve as an introduction to and a survey of this exciting area of research as well as a reference for the state of the art of this research field.

Introduction

Over the last few years, scientists have faced an ever growing deluge of data, which needs to be transmitted, analyzed, and stored. A close analysis reveals that most of these data might be classified as multimodal data, i.e., being composed of distinct subcomponents. Prominent examples are audio data, which might consist of a superposition of the sounds of different instruments, or imaging data from neurobiology, which is typically a composition of the soma of a neuron, its dendrites, and its spines. In both these exemplary situations, the data has to be separated into appropriate single components for further analysis. In the first case, separating the audio signal into the signals of the different instruments is a first step to enable the audio technician to obtain a musical score from a recording.

Parametric signals, such as streams of short pulses, appear in many applications including bio-imaging, radar, and spread-spectrum communication. The recently developed finite rate of innovation (FRI) framework has paved the way to low-rate sampling of such signals, by exploiting the fact that only a small number of parameters per unit of time are needed to fully describe them. For example, a stream of pulses can be uniquely defined by the time delays of the pulses and their amplitudes, which leads to far fewer degrees of freedom than the signal's Nyquist rate samples. This chapter provides an overview of FRI theory, algorithms, and applications. We begin by discussing theoretical results and practical algorithms allowing perfect reconstruction of FRI signals from a minimal number of samples. We then turn to treat recovery from noisy measurements. Finally, we overview a diverse set of applications of FRI theory, in areas such as super-resolution, radar, and ultrasound.

Introduction

We live in an analog world, but we would like our digital computers to interact with it. For example, sound is a continuous-time phenomenon, which can be characterized by the variations in air pressure as a function of time. For digital processing of such real-world signals to be possible, we require a sampling mechanism which converts continuous signals to discrete sequences of numbers, while preserving the information present in those signals.

In classical sampling theory, which dates back to the beginning of the twentieth century [1–3], a bandlimited signal whose maximum frequency is fmax is sampled at or above the Nyquist rate 2fmax.

Compressed sensing (CS) is an exciting, rapidly growing, field that has attracted considerable attention in signal processing, statistics, and computer science, as well as the broader scientific community. Since its initial development only a few years ago, thousands of papers have appeared in this area, and hundreds of conferences, workshops, and special sessions have been dedicated to this growing research field. In this chapter, we provide an up-to-date review of the basics of the theory underlying CS. This chapter should serve as a review to practitioners wanting to join this emerging field, and as a reference for researchers. We focus primarily on the theory and algorithms for sparse recovery in finite dimensions. In subsequent chapters of the book, we will see how the fundamentals presented in this chapter are expanded and extended in many exciting directions, including new models for describing structure in both analog and discrete-time signals, new sensing design techniques, more advanced recovery results and powerful new recovery algorithms, and emerging applications of the basic theory and its extensions.

Introduction

We are in the midst of a digital revolution that is driving the development and deployment of new kinds of sensing systems with ever-increasing fidelity and resolution. The theoretical foundation of this revolution is the pioneering work of Kotelnikov, Nyquist, Shannon, and Whittaker on sampling continuous-time bandlimited signals [162, 195, 209, 247]. Their results demonstrate that signals, images, videos, and other data can be exactly recovered from a set of uniformly spaced samples taken at the so-called Nyquist rate of twice the highest frequency present in the signal of interest.

In order to use GNSS for almost any practical application or to simulate a GNSS signal, we need to be able to define satellite coordinates at any moment of time. In this chapter we look at GNSS satellite orbits. We consider their mathematical presentations and requirements of constellation design. The subject matter of this chapter in relation to other chapters is shown on Figure 2.1

Development of models for celestial body movements from Ptolemy to Einstein

Movement of satellites around the Earth is described using mathematical instruments and models which were developed for describing the movement of the planets. The first known applicable mathematical model for orbital movement of the celestial bodies was developed by Claudius Ptolemy, who was working in the first century AD in Alexandria. He created the first accurate working model of a celestial body’s motion [1].

We thought that it would be out of keeping with the book’s style if the last word in it was a date in the last reference in the last chapter. Instead, we try to give our view on what the next step might be after the last page has been turned over.

The purpose of this book is to supply readers with a good text book for reference and lab use. What is possible to bundle with the book is limited of course to software and data. We have tried to demonstrate that most of the work can be done and often is done in a digital domain, and therefore in many cases such a digital lab will suffice.

The bundled software packages are free academic versions of the programs which are used in professional high-end applications. They are sufficient in many cases for academic purposes and for self-education. If a reader is engaged in R&D or testing, or is otherwise interested in more capabilities, these programs can be upgraded to professional versions. For those who are interested in development of their own models or algorithms, the development versions are the best solution. Development versions with an application programming interface (API) allow access to source code and use an already developed, optimized, and tested receiver and simulator framework to incorporate models or algorithms developed by the reader. That would eliminate a need to spend time and effort on developing the main receiver and simulator components and instead allow concentration on the subject of interest.

We have described how a GNSS signal is generated in Chapter 3. In Chapter 4 we examined how a GNSS signal propagates through the Earth’s atmosphere. In this chapter we discuss how a radio frequency signal is converted by a GNSS receiver front end to a digital format for further processing. The scope of this chapter is shown schematically in Figure 5.1. The purpose of this chapter is to describe a design of a receiver front end and the operations of its major components, and to analyze how the design of these components affects a GNSS signal.

Generic GNSS receiver

A generic receiver flowchart is presented in Figure 5.2. The receiver in the flowchart does not include a navigation processor. A receiver acquires a radio frequency (RF) signal coming through atmosphere from a satellite. The signal goes to a front end. On the output of the front end we have a digitized intermediate frequency signal, which can then be processed either in digital hardware or in the software. In the front end the signal is filtered by a bandpass filter and then down-converted from RF to intermediate frequency (IF). (Figure 5.3 shows an example of a simplified frequency plan for the front end.) Next the IF signal goes through an analog to digital converter (ADC), and the resulting digitized IF (DIF) signal goes from the front end to a baseband processor. The baseband processor performs signal acquisition and tracking. The acquisition and tracking are achieved by means of correlating the incoming signal in the shape of DIF with a replica signal which is generated within the receiver. The baseband processor outputs raw measurements, in particular code phase, carrier phase, Doppler, and signal-to-noise ratio (SNR) measurements. Baseband processor operation is described in detail in the next chapter.

An effect of electromagnetic scintillation has been noted in the visible frequency range with the introduction of telescopes. It is a random rapid fluctuation of signal instant amplitude and phase. Newton identified this phenomenon with the atmosphere and recommended locating telescopes on the tops of the highest mountains. Further advance in scintillation research was achieved when astronomy moved to radio frequencies. Scintillation effects had been discovered by monitoring signals from other galaxies, in particular from Cassiopeia [1]. The source of scintillation had been established by making measurements with a set of receivers located at a distance from each other and operating on various frequencies. Correlations between scintillation effects in the receivers at different locations established that the source of scintillation is in the Earth’s atmosphere. The dependence on frequency showed that the medium in which scintillation occurs is dispersive.

A GNSS signal in particular is subject to amplitude and phase scintillation caused by its propagation through the atmosphere, though the scintillation effects on GNSS from signal propagation in the troposphere are much smaller than those caused by signal propagation in the ionosphere. The effect of signal scintillation is important for navigation and geodetic applications, because it affects receiver performance to a point where it may lose a lock and stop tracking a signal. Amplitude scintillation results in signal quality degradation. Phase scintillation affects the carrier tracking loop in such a way that it requires a wider bandwidth due to higher dynamics of the carrier. In Chapter 12 we consider such effects and how they can be mitigated for some applications.

In this chapter we describe operation of a GNSS receiver baseband processor. The place which this chapter occupies in the book is schematically presented in Figure 6.1. It is one of two major components of a GNSS receiver. In the previous chapter we described the other component of the receiver – the RF front end, which takes an RF signal from an antenna, amplifies, down-converts, filters, and digitizes it. A digitized intermediate frequency signal (DIF), also described in the previous chapter, is taken from the output of the front end to the input of the baseband processor. The baseband processor processes the DIF signal and outputs all information carried by the GNSS signal, pseudoranges or code phase observations, carrier phase observations, Doppler, signal to noise ratio, navigation message, and so on.

Do we need all the receiver or just a baseband processor?

If we compare the structure of a generic receiver as it is usually presented, and as illustrated in Figure 6.2, with the flowchart of this book presentation in Figure 6.1, we can see that a navigation processor is omitted from the receiver design. This has been done on purpose. In numerous applications today the navigation processor embedded in the receiver is not actually used, though all conventional receivers have one.