Introduction

Spatial resolution of imaging devices is steadily increasing: mobile phone cameras have 5–10MP while point-and-shoot cameras have 12–18MP. As the spatial resolution of imaging devices increases, the effect of either camera or subject motion on the captured images is magnified, resulting in the acquisition of heavily blurred images. Since the motion-blur kernel is a low pass kernel, the high frequencies in the image are heavily attenuated. Deblurring the effects of such low pass blur kernels results in the introduction of significant high frequency artifacts. When the size of the blur kernel is small, these artifacts can be mitigated by the use of appropriate image regularizers that allow for the hallucination of the high frequency detail. Unfortunately there are several scenarios in which such a technique, relying primarily on image regularization, cannot be directly applied. They fall into the following three main categories.

1. Large motion

The resolutions of image sensors are rapidly increasing, while the hands of photographers are not becoming any steadier. This results in ever larger sizes of blur kernels. While image regularization allows us to handle blur kernels of a moderate size (say about 10–15 pixels), larger blur kernels test the ability of even modern deblurring algorithms, especially in regions of high texture. Such large motion blur kernels necessitate active approaches that shape the blur kernel to be invertible in order to handle these robustly.

2. Object motion

Object motion causes blur that is spatially variant. In particular, the blur kernel depends on the local velocity of the objects.

Introduction

Processing images has become an everyday practice in a wide range of applications in science and technology and we rely on our images with ever growing emphasis. Our understanding of the world is however limited by measuring devices that we use to acquire images. Inadequate measuring conditions together with technological limitations of the measuring devices result in acquired images that represent a degraded version of the “true” image.

Blur induced by camera motion is a frequent problem in photography – mainly when the light conditions are poor. As the exposure time increases, involuntary camera motion has a growing effect on the acquired image. Image stabilization (IS) devices that help to reduce the motion blur by moving the camera sensor in the opposite direction are becoming more common. However, such a hardware remedy has its limitations as it can only compensate for motion of a very small extent and speed. Deblurring the image offline using mathematical algorithms is usually the only choice we have to obtain a sharp image. Motion blur can be modeled by convolution and the deblurring process is called deconvolution, which is a well-known ill-posed problem. In general, the situation is even more complicated, since we usually have no or limited information about the blur shape.

We can divide the deconvolution methods into two categories: methods that estimate the blur and the sharp image directly from the acquired image (blind deconvolution), and methods that use information from other sensors to estimate the blur (semi-blind deconvolution).

Introduction

Digital cameras convert incident light energy into electrical signals and present them as an image after altering the signals through different processes which include sensor correction, noise reduction, scaling, gamma correction, image enhancement, color space conversion, frame-rate change, compression, and storage/transmission (Nakamura 2005). Although today's camera sensors have high quantum efficiency and high signalto-noise ratios, they inherently have an upper limit (full well capacity) for accumulation of light energy. Also, the sensor's least acquisition capacity depends on its pre-set sensitivity. The total variation in the magnitude of irradiance incident at a camera is called the dynamic range (DR) and is defined as DR = (maximum signal value)/(minimum signal value). Most digital cameras available in the market today are unable to account for the entire DR due to hardware limitations. Scenes with high dynamic range (HDR) either appear dark or become saturated. The solution for overcoming this limitation and estimating the original data is referred to as high dynamic range imaging (HDRI) (Debevec & Malik 1997, Mertens, Kautz & Van Reeth 2007, Nayar & Mitsunaga 2000).

Over the years, several algorithmic approaches have been investigated for estimation of scene irradiance (see, for example, Debevec & Malik (1997), Mann & Picard (1995), Mitsunaga & Nayar (1999)). The basic idea in these approaches is to capture multiple images of a scene with different exposure settings and algorithmically extract HDR information from these observations. By varying the exposure settings, one can control the amount of energy received by the sensors to overcome sensor bounds/limits.

Camera shake is one of the most common causes of image blur. This type of blur arises when a long exposure is required (due to low light levels, for example), and the camera is not held still.

Removing blur due to camera shake is a very active area of research. With just a single photograph as input, this blur removal is known as the blind deconvolution problem, i.e. simultaneously recovering both the blurring function (point spread function or PSF) and the deblurred, latent image. Unfortunately, the blind deconvolution problem is inherently ill-posed, as the observed blurred image provides only a partial constraint on the solution. Therefore, the problem cannot be solved without assuming constraints or priors on the blur or the deblurred image. The most common assumption is that the blur is spatially invariant, but this can handle only a limited set of camera motions. The deblurred image is typically assumed to have natural image statistics (Fergus, Singh, Hertzmann, Roweis & Freeman 2006). A key open problem is to model general camera motions, which are quite common and can cause the blur kernels to vary spatially.

Review of image deblurring methods

Image deblurring has received a lot of attention in the computer vision community. Deblurring is the combination of two tightly coupled sub-problems: PSF estimation and non-blind image deconvolution. These problems have been addressed both independently and jointly (Richardson 1972).

Introduction

Motion blur from camera egomotion is an artifact in photography caused by the relative motion between the camera and an imaged scene during exposure. Assuming a static and distant scene, and ignoring the effects of defocus and lens aberration, each point in the blurred image can be described as the convolution of the un-blurred image by a point spread function (PSF) that describes the relative motion trajectory at that point's position. The aim of image deblurring is to reverse this convolution process to recover the clear image of the scene from the captured blurry image as shown in Figure 8.1.

A common assumption in existing motion deblurring algorithms is that the motion PSF is spatially invariant. This implies that all pixels are convolved with the same motion blur kernel. However, as discussed by Levin, Weiss, Durand & Freeman (2009) the global PSF assumption is often invalid. In their experiments, images taken with camera shake exhibited notable amounts of rotation that attributed to spatially-varying motion blur within the image. Figure 8.2 shows a photograph that illustrates this effect. As a result, Levin et al. (2009) advocated the need for a better motion blur model as well as image priors to help regularize the solution space when performing deblurring. This chapter addresses the former issue by introducing a new and compact motion blur model that is able to describe spatially-varying motion blur caused by a camera undergoing egomotion.

Recovering an un-blurred image from a single motion-blurred picture has long been a fundamental research problem. If one assumes that the blur kernel – or point spread function (PSF) – is shift invariant, the problem reduces to that of image deconvolution. Image deconvolution can be further categorized as non-blind and blind.

In non-blind deconvolution, the motion blur kernel is assumed to be known or computed elsewhere; the task is to estimate the un-blurred latent image. The general problems to address in non-blind deconvolution include reducing possible unpleasant ringing artifacts that appear near strong edges, suppressing noise, and saving computation. Traditional methods such as Wiener deconvolution (Wiener 1949) and the Richardson–Lucy (RL) method (Richardson 1972, Lucy 1974) were proposed decades ago and find many variants thanks to their simplicity and efficiency. Recent developments involve new models with sparse regularization and the proposal of effective linear and non-linear optimization to improve result quality and further reduce running time.

Blind deconvolution is a much more challenging problem, since both the blur kernel and the latent image are unknown. One can regard non-blind deconvolution as an inevitable step in blind deconvolution during the course of PSF estimation or after the PSF has been computed. Both blind and non-blind deconvolution are practically very useful; they are studied and employed in a variety of disciplines including, but not limited to, image processing, computer vision, medical and astronomic imaging, and digital communication.

In this chapter we discuss modelling and removing spatially-variant blur from photographs. We describe a compact global parameterization of camera-shake blur, based on the 3D rotation of the camera during the exposure. Our model uses three-parameter homographies to connect camera motion to image motion and, by assigning weights to a set of these homographies, can be seen as a generalization of the standard, spatially-invariant convolutional model of image blur. As such we show how existing algorithms, designed for spatially-invariant deblurring, can be ‘upgraded’ in a straightforward manner to handle spatially-variant blur instead. We demonstrate this with algorithms working on real images, showing results for blind estimation of blur parameters from single images, followed by non-blind image restoration using these parameters. Finally, we introduce an efficient approximation to the global model, which significantly reduces the computational cost of modelling the spatially-variant blur. By approximating the blur as locally-uniform, we can take advantage of fast Fourier-domain convolution and deconvolution, reducing the time required for blind deblurring by an order of magnitude.

Introduction

Everybody is familiar with camera shake, since the resulting blur spoils many photos taken in low-light conditions. Camera-shake blur is caused by motion of the camera during the exposure; while the shutter is open, the camera passes through a sequence of different poses, each of which gives a different view of the scene. The sensor accumulates all of these views, summing them up to form the recorded image, which is blurred as a result. We would like to be able to deblur such images to recover the underlying sharp image, which we would have captured if the camera had not moved.

The computer vision community is witnessing a major resurgence in the area of motion deblurring spurred by the emerging ubiquity of portable imaging devices. Rapid strides are being made in handling motion blur both algorithmically and through tailor-made hardware-assisted technologies. The main goal of this book is to ensure a timely dissemination of recent findings in this very active research area. Given the flurry of activity in the last few years in tackling uniform as well as non-uniform motion blur resulting from incidental shake in hand-held consumer cameras as well as object motion, we felt that a compilation of recent and concerted efforts for restoring images degraded by motion blur was well overdue. Since no single compendium of the kind envisaged here exists, we believe that this is an opportune time for publishing a comprehensive collection of contributed chapters by leading researchers providing in-depth coverage of recently developed methodologies with excellent supporting experiments, encompassing both algorithms and architectures.

As is well known, the main cause of motion blur is the result of averaging of intensities due to relative motion between a camera and a scene during exposure time. Motion blur is normally considered a nuisance although one must not overlook the fact that some works have used blur for creating aesthetic appeal or exploited it as a valuable cue in depth recovery and image forensics. Early works were non-blind in the sense that the motion blur kernel (i.e. the point spread function (PSF)) was assumed to be of a simple form, such as those arising from uniform camera motion, and efforts were primarily directed at designing a stable estimate for the original image.

Introduction

A number of computational imaging (CI) based motion deblurring techniques have been introduced to improve image quality. These techniques use optical coding to measure a stronger signal level instead of a noisy short exposure image. However, the performance of these techniques is limited by the decoding step, which amplifies noise. While it is well understood that optical coding can increase performance at low light levels, little is known about the quantitative performance advantage of computational imaging in general settings.

In this chapter, we derive the performance bounds for various computational imaging-based motion deblurring techniques. We then discuss the implications of these bounds for several real-world scenarios. The scenarios are defined in terms of real-world lighting (e.g. moonlit night or cloudy day, indoor or outdoor), scene properties (albedo, object velocities), and sensor characteristics. The results show that computational imaging techniques do not provide a significant performance advantage when imaging with illumination brighter than typical indoor lighting. This is illustrated in Figure 13.1. These results can be readily used by practitioners to decide whether to use CI and, if so, to design the imaging system. We also study the role of image priors on the decoding steps. Our empirical results show that the use of priors reduces the performance advantage of CI techniques even further.

Scope

The analysis in this chapter focuses on techniques that use optical coding to preserve high frequencies in the blur kernels so that deblurring becomes a well-conditioned problem. These techniques assume that the blur kernel is known a priori. The analysis is limited to techniques that acquire a single image and follow a linear imaging model.

The need to recognize motion-blurred faces is vital for a wide variety of security applications ranging from maritime surveillance to road traffic policing. While much of the theory in the analysis of motion-blurred images focuses on restoration of the blurred image, we argue that this is an unnecessary and expensive step for face recognition. Instead, we adopt a direct approach based on the set-theoretic characterization of the space of motion-blurred images of a single sharp image. This set lacks the nice property of convexity that was exploited in a recent paper to achieve competitive results in real-world datasets (Vageeswaran, Mitra & Chellappa 2013). Keeping this non-convexity in mind, we propose a bank of classifiers (BoC) approach for directly recognizing motion-blurred face images. We divide the parameter space of motion blur into many different bins in such a way that the set of blurred images within each bin is a convex set. In each such bin, we learn support vector machine (SVM) classifiers that separate the convex sets associated with each person in the gallery database. Our experiments on synthetic and real datasets provide compelling evidence that this approach is a viable solution for recognition of motion-blurred face images.

Introduction

A system that can recognize motion-blurred faces can be of vital use in a wide variety of security applications, ranging from maritime surveillance to road traffic policing. Figure 12.1 shows two possible maritime surveillance scenarios: shore-to-ship (the camera is mounted on-shore and the subjects are in the ship), and ship-to-shore (the camera is on the ship and the subjects are moving on-shore).

This section describes the methods used to implement the deliberative autonomy layer that was initially described in Chapter 1. Planning concerns the question of deciding what to do, of which deciding where to go is a special case. Central to planning is the predictive model, which maps candidate actions onto their associated consequences. Equally as important is the mechanism of search because there tend to be many alternative actions to be assessed at any point in time.

Planners think about the future, employing some degree of look ahead and there is a central trade-off between the computational cost of look ahead and the cognitive performance of the system. In addition to perception, planning is where most of what impresses us about robots is located. Given a sufficiently accurate model of the environment, planning technology today can solve, in a comparative instant, problems that we humans would find quite daunting.

Introduction

Planning refers to processes that deliberate, predict, and often optimize. Respectively these actions will mean:

• Deliberate: Consider many possible sequences of future actions.

• Predict: Predict the outcomes for each sequence.

• Optimize: Pick one, perhaps based on some sense of relative merit.

Although robot arms that spot weld our cars together have been around for some time, a new class of robots, the mobile robot, has been quietly growing in significance and ability. For several decades now, behind the scenes in research laboratories throughout the world, robots have been evolving to move automatically from place to place. Mobility enables a new capacity to interact with humans while relieving us from jobs we would rather not do anyway.

Mobile robots have recently entered the public consciousness as a result of the spectacular success of the Mars rovers, television shows such as Battlebots, and the increasingly robotic toys that are becoming popular at this time.

Mobility of a robot changes everything. The mobile robot faces a different local environment every time it moves. It has the capacity to influence, and be influenced by, a much larger neighborhood than a stationary robot. More important, the world is a dangerous place, and it often cannot be engineered to suit the limitations of the robot, so mobility raises the needed intelligence level. Successfully coping with the different demands and risks of each place and each situation is a significant challenge for even biological systems.

Preface

Robotics can be a very challenging and very satisfying way to spend your time. A profound moment in the history of most roboticists is the first moment a robot performed a task under the influence of his or her software or electronics. Although a productive pursuit of the study of robotics involves aspects of engineering, mathematics, and physics, its elements do not convey the magic we all feel when interacting with a responsive semi-intelligent device of our own creation.

This book introduces the science and engineering of a particularly interesting class of robots – mobile robots. Although there are many analogs to the field of robot manipulators, mobile robots are sufficiently different to justify their treatment in an entirely separate text. Although the book concentrates on wheeled mobile robots, most of its content is independent of the specific locomotion subsystem used.

The field of mobile robots is changing rapidly. Many specialties are evolving in both the research and the commercial sectors. Any textbook offered in such an evolving field will represent only a snapshot of the field as it was understood at the time of publication. However, the rapid growth of the field, its several decades of history, and its pervasive popular appeal suggest that the time is now right to produce an early text that attempts to codify some of the fundamental ideas in a more accessible manner.

A large number of problems in mobile robotics can be reduced to a few basic problem formulations. Most problems reduce to some mixture of optimizing something, solving simultaneous linear or nonlinear equations, or integrating differential equations. Well-known numerical methods exist for all of these problems and all are accessible as black boxes in both software applications and general purpose toolboxes. Of course offline toolboxes cannot be used to control a system in real time and almost any solution benefits from exploiting the nature of the problem, so it is still very common to implement numerical algorithms from scratch for many real time systems.

The techniques described in this section will be referenced in many future places in the text. These techniques will be used to compute wheel velocities, invert dynamic models, generate trajectories, track features in an image, construct globally consistent maps, identify dynamic models, calibrate cameras, and so on.

Linearization and Optimization of Functions of Vectors

Perhaps paradoxically, linearization is the fundamental process that enables us to deal with nonlinear functions. The topics of linearization and optimization are closely linked because a local optimum of a function coincides with special properties of its linear approximation.

The equations that describe the motion of wheeled mobile robots (WMRs) are very different from those that describe manipulators because they are differential rather than algebraic, and often underactuated and constrained. Unlike manipulators, the simplest models of how mobile robots move are nonlinear differential equations. Much of the relative difficulty of mobile robot modelling can be traced to this fact.

This section explores dynamics in two different senses of the term. Dynamics in mechanics refers to the particular models of mechanical systems that are used in that discipline. These tend to be second-order equations involving forces, mass properties, and accelerations. In control theory, dynamics refers to any differential equations that describe the system of interest.

Moving Coordinate Systems

The fact that mobile robot sensors are fixed to the robot and move with them has profound implications. On the one hand, it is the source of nonlinear models in Kalman filters. On the other, it is the source of nonholonomic wheel constraints. This section concentrates on a third effect – the fact that the derivatives of many vector quantities of interest depend on the motion of the robot.