In this chapter the authors go beyond traditional sparse modeling, and address collaborative structured sparsity to add stability and prior information to the representation. In structured sparse modeling, instead of considering the dictionary atoms as singletons, the atoms are partitioned in groups, and a few groups are selected at a time for the signal encoding. A complementary way of adding structure, stability, and prior information to a model is via collaboration. Here, multiple signals, which are known to follow the same model, are allowed to collaborate in the coding. The first studied framework connects sparse modeling with Gaussian Mixture Models and leads to state-of-the-art image restoration. The second framework derives a hierarchical structure on top of the collaboration and is well fitted for source separation. Both models enjoy very important theoretical virtues as well.

Introduction

In traditional sparse modeling, it is assumed that a signal can be accurately represented by a sparse linear combination of atoms from a (learned) dictionary. A large class of signals, including most natural images and sounds, is well described by this model, as demonstrated by numerous state-of-the-art results in various signal processing applications.

From a data modeling point of view, sparsity can be seen as a form of regularization, that is, as a device to restrict or control the set of coefficient values which are allowed in the model to produce an estimate of the data.

Modern data are often composed of two or more morphologically distinct constituents, and one typical goal is the extraction of those components. Recently, sparsity methodologies have been successfully utilized to solve this problem, both theoretically as well as empirically. The key idea is to choose a deliberately overcomplete representation made of several frames each one providing a sparse expansion of one of the components to be extracted. The morphological difference between the components is then encoded as incoherence conditions of those frames. The decomposition principle is to minimize the ℓ1 norm of the frame coefficients. This chapter shall serve as an introduction to and a survey of this exciting area of research as well as a reference for the state of the art of this research field.

Introduction

Over the last few years, scientists have faced an ever growing deluge of data, which needs to be transmitted, analyzed, and stored. A close analysis reveals that most of these data might be classified as multimodal data, i.e., being composed of distinct subcomponents. Prominent examples are audio data, which might consist of a superposition of the sounds of different instruments, or imaging data from neurobiology, which is typically a composition of the soma of a neuron, its dendrites, and its spines. In both these exemplary situations, the data has to be separated into appropriate single components for further analysis. In the first case, separating the audio signal into the signals of the different instruments is a first step to enable the audio technician to obtain a musical score from a recording.

Parametric signals, such as streams of short pulses, appear in many applications including bio-imaging, radar, and spread-spectrum communication. The recently developed finite rate of innovation (FRI) framework has paved the way to low-rate sampling of such signals, by exploiting the fact that only a small number of parameters per unit of time are needed to fully describe them. For example, a stream of pulses can be uniquely defined by the time delays of the pulses and their amplitudes, which leads to far fewer degrees of freedom than the signal's Nyquist rate samples. This chapter provides an overview of FRI theory, algorithms, and applications. We begin by discussing theoretical results and practical algorithms allowing perfect reconstruction of FRI signals from a minimal number of samples. We then turn to treat recovery from noisy measurements. Finally, we overview a diverse set of applications of FRI theory, in areas such as super-resolution, radar, and ultrasound.

Introduction

We live in an analog world, but we would like our digital computers to interact with it. For example, sound is a continuous-time phenomenon, which can be characterized by the variations in air pressure as a function of time. For digital processing of such real-world signals to be possible, we require a sampling mechanism which converts continuous signals to discrete sequences of numbers, while preserving the information present in those signals.

In classical sampling theory, which dates back to the beginning of the twentieth century [1–3], a bandlimited signal whose maximum frequency is fmax is sampled at or above the Nyquist rate 2fmax.

Compressed sensing (CS) is an exciting, rapidly growing, field that has attracted considerable attention in signal processing, statistics, and computer science, as well as the broader scientific community. Since its initial development only a few years ago, thousands of papers have appeared in this area, and hundreds of conferences, workshops, and special sessions have been dedicated to this growing research field. In this chapter, we provide an up-to-date review of the basics of the theory underlying CS. This chapter should serve as a review to practitioners wanting to join this emerging field, and as a reference for researchers. We focus primarily on the theory and algorithms for sparse recovery in finite dimensions. In subsequent chapters of the book, we will see how the fundamentals presented in this chapter are expanded and extended in many exciting directions, including new models for describing structure in both analog and discrete-time signals, new sensing design techniques, more advanced recovery results and powerful new recovery algorithms, and emerging applications of the basic theory and its extensions.

Introduction

We are in the midst of a digital revolution that is driving the development and deployment of new kinds of sensing systems with ever-increasing fidelity and resolution. The theoretical foundation of this revolution is the pioneering work of Kotelnikov, Nyquist, Shannon, and Whittaker on sampling continuous-time bandlimited signals [162, 195, 209, 247]. Their results demonstrate that signals, images, videos, and other data can be exactly recovered from a set of uniformly spaced samples taken at the so-called Nyquist rate of twice the highest frequency present in the signal of interest.

This book presents in a systematic manner a general nonparametric theory of statistics on manifolds with emphasis on manifolds of shapes, and with applications to diverse fields of science and engineering. There are many areas of significant application of statistics on manifolds. For example, directional statistics (statistics on the sphere S2) are used to study shifts in the Earth's magnetic poles over geological time, which have an important bearing on the subject of tectonics. Applications in morphometrics involve classification of biological species and subspecies. There are many important applications to medical diagnostics, image analysis (including scene recognition), and machine vision (e.g., robotics). We take a fresh look here in analyzing existing data pertaining to a number of such applications. It is our goal to lay the groundwork for other future applications of this exciting emerging field of nonparametric statistics.

Landmark-based shape spaces were first introduced by D. G. Kendall more than three decades ago, and pioneering statistical work on shapes with applications to morphometrics was carried out by F. Bookstein around the same time. Statistics on spheres, or directional statistics, arose even earlier, and a very substantial statistical literature on directional statistics exists, including a seminal 1953 paper by R. A. Fisher and books by Watson (1983), Mardia and Jupp (2000), Fisher et al. (1987), and others. For statistics on shape spaces, important parametric models have been developed by Kent, Dryden, Mardia, and others, and a comprehensive treatment of the literature may be found in the book by Dryden and Mardia (1998).

This chapter collects together, and describes in a simple manner, a number of applications of the theory presented in this book. The examples are based on real data, and, where possible, results of parametric inference in the literature are cited for comparison with the new nonparametric inference theory.

Data example on S1: wind and ozone

The wind direction and ozone concentration were observed at a weather station for 19 days. Table 2.1 shows the wind directions in degrees. The data are taken from Johnson and Wehrly (1977). The data viewed on the unit circle S1 are plotted in Figure 3.1. We compute the sample extrinsic and intrinsic mean directions, which come out to be 16.71 and 5.68 degrees, respectively. They are displayed in the figure. We use angular coordinates for the data in degrees lying between [0°, 360°) as in Table 2.1. An asymptotic 95% confidence region for the intrinsic mean as obtained in Section 3.7, Chapter 3, turns out to be

{(cos θ, sin θ) : - 0.434 ≤ θ ≤ 0.6324}.

The corresponding end points of this arc are also displayed in Figure 3.1.

Johnson and Wehrly (1977) computed the so-called angular–linear correlation ρAL = maxα{ρ(cos(θ - α), X)}, where X is the ozone concentration when the direction of wind is θ. Here ρ denotes the true coefficient of correlation. Based on the sample counterpart rAL, the 95% confidence interval for ρAL was found to be (0.32, 1.00).

The real projective space ℝPm is the manifold of all lines through the origin in ℝm+1. It may be identified with the sphere Sm modulo the identification of a point p with its antipodal point -p. For m = 1 or 2, this is the axial space. Extrinsic and intrinsic inferences on ℝPm are developed first in this chapter. This aids in the nonparametric analysis on the space of projective shapes, identified here, subject to a registration, with (ℝPm)k-m-2.

Introduction

Consider a k-ad picked on a planar image of an object or a scene in three dimensions. If one thinks of images or photographs obtained through a central projection (a pinhole camera is an example of this), a ray is received as a landmark on the image plane (e.g., the film of the camera). Because axes in three dimensions comprise the projective space ℝP2, the k-ad in this view is valued in ℝP2. For a k-ad in three dimensions to represent a k-ad in ℝP2, the corresponding axes must all be distinct. To have invariance with regard to camera angles, one may first look at the original noncollinear three-dimensional k-ad u and achieve affine invariance by its affine shape (i.e., by the equivalence class Au, A ∈ GL(3, ℝ)), and finally take the corresponding equivalence class of axes in ℝP2 to define the projective shape of the k-ad as the equivalence class, or orbit, with respect to projective transformations on ℝP2.

In this chapter we adapt and extend the nonparametric density estimation procedures on Euclidean spaces to general (Riemannian) manifolds.

Introduction

So far in this book we have used notions of center and spread of distributions on manifolds to identify them or to distinguish between two or more distributions. However, in certain applications, other aspects of the distribution may also be important. The reader is referred to the data in Section 14.5.3 for such an example. Also, our inference method so far has been frequentist.

In this chapter and the next, we pursue different goals and a different route. Our approach here and in the next chapter will be nonparametric Bayesian, which involves modeling the full data distribution in a flexible way that is easy to work with. The basic idea will be to represent the unknown distribution as an infinite mixture of some known parametric distribution on the manifold of interest and then set a full support prior on the mixing distribution. Hence the parameters defining the distribution are no longer finite-dimensional but reside in the infinite-dimensional space of all probabilities. By making the parameter space infinite-dimensional, we ensure a flexible model for the unknown distribution and consistency of its estimate under mild assumptions. All these will be made rigorous through the various theorems we will encounter in the subsequent sections.

For a prior on the mixing distribution, a common choice can be the Dirichlet process prior (see Ferguson, 1973, 1974). We present a simple algorithm for posterior computations in Section 13.4.

Manifolds of greatest interest in this book are spaces of shapes of k-ads in ℝm, with a k-ad being a set of k labeled points, or landmarks, on an object in ℝm. This chapter introduces these shape spaces.

Introduction

The statistical analysis of shape distributions based on random samples is important in many areas such as morphometrics, medical diagnostics, machine vision, and robotics. In this chapter and the chapters that follow, we will be interested mainly in the analysis of shapes of landmark-based data, in which each observation consists of k > m points in m dimensions, representing k landmarks on an object, called a k-ad. The choice of landmarks is generally made with expert help in the particular field of application. Depending on the way the data are collected and recorded, the appropriate shape of a k-ad is the maximal invariant specified by its orbit under a group of transformations.

For example, one may look at k-ads modulo size and Euclidean rigid body motions of translation and rotation. The analysis of this invariance class of shapes was pioneered by Kendall (1977, 1984) and Bookstein (1978). Bookstein's approach is primarily registration-based, requiring two or three landmarks to be brought into a standard position by translating, rotating and scaling the k-ad. We would prefer Kendall's more invariant view of a shape identified with the orbit under rotation (in m dimensions) of the k-ad centered at the origin and scaled to have a unit size.