Image inpainting is a mathematically highly ill-posed process. Once something is completely lost, we can never know for sure what has been there – except if we knew the image before it was damaged or we can travel back in time and have a look. This means that image inpainting does not have a unique solution in general. Since we are intelligent human beings, we can narrow done our search by using roughly two main strategies: our natural visual perception and experience and our historical knowledge (almost like travelling back in time) about the image and its hole. Both these strategies are crucial for modelling of the desired inpainting process and will determine the inpainting methods that we will consider in subsequent chapters.

Historical knowledge of the image could be many things and depend very much on the kind of application. In art restoration, known characteristics and techniques used by the painter, intact illustrations of a similar scene by different artists (such is the case in many religious paintings from the eighteenth century), and knowledge about general painting techniques, colour mixtures and materials used at the time the painting was created have been used. In video inpainting, knowledge from earlier frames could be used to extrapolate in time into later frames where damage appears. In medical imaging applications, anatomical knowledge of a template human body is used, and similarly, there are many other application in which geometrical knowledge about the shapes of objects one is looking for can be used (compare the application on the inpainting of satellite images of roads in Section 9.2). This part of the interpolation process depends on the context of the interpolation.

Visual perception, however, gives a context-free continuation of an incomplete scene. It constitutes our – either natural or learned – ability to automatically interpolate broken or occluded structures. This automatic continuation follows certain laws.

This book is concerned with digital image processing techniques that use partial differential equations (PDEs) for the task of image inpainting. Image inpainting is an artistic term for virtual image restoration or image interpolation whereby missing or occluded parts of images are filled in based on information provided by the intact parts of the image. Computer graphic designers, artists and photographers have long used manual inpainting to digitally restore damaged paintings or manipulate photographs. Today, mathematicians apply powerful methods based on PDEs to automate this task. They operate in much the same way that trained restorers do: they propagate information from the structure around a hole into the hole to fill it in.

Virtual image restoration is an important challenge in our modern computerised society. From the reconstruction of crucial information in satellite images of the Earth to the renovation of digital photographs and ancient artwork, virtual image restoration is ubiquitous. The example in Figure 1.1 is entitled Mathematical Analysis Can Make You Fly, and it should give you a first impression of the idea of image inpainting with PDEs. The PDE model used for this example is called TV-H−1inpainting and will be discussed in great detail in Section 5.3.

Digital Image Restoration in Modern Society

Digital images are one of the main sources of information today. The vast number of images and videos that exist in digital form nowadays makes their unaided processing and interpretation by humans impossible. Automatic storage management, processing and analysis algorithms are needed to be able to retrieve only the essence of what the visual world has up its sleeve.

The purpose of this book is to provide an introduction to the use of partial differential equations (PDEs) for digital image restoration. It is a way of sharing what I have learned while studying these methods for about ten years, defending a Ph.D. thesis on PDE inpainting, teaching courses in Göttingen and Cambridge on the topic and writing a couple of research papers on the use of PDEs in image inpainting along the way. Let me say what this book is and what it is not. It is:

1. An introduction to inpainting methods that use PDEs and local variational approaches to restore lost image contents;

2. An account from an enthusiast on some state-of-the-art inpainting methods from their abilities to their limitations and reference for informed researchers in the field of digital image processing; and

3. A work targeted at readers with basic knowledge in functional analysis, PDEs, measure theory and convex optimisation. Therefore, I recommend this textbook only for students from the graduate level onwards.

This book is not:

1. A book that gives credit to the whole wealth of inpainting methods (in particular, this book will focus only on local inpainting methods and will scratch only non-local inpainting methods such as exemplar-based inpainting); or

2. A book for undergraduate students.

One final comment before we go in medias res: the more I learn and understand about image inpainting, the more the complexity of the matter and the variety of different problems, each requiring specialised methods, become apparent.

Before we start, let us emphasise here once more that in this and most subsequent chapters we focus on local structural inpainting methods. Structural image inpainting means that we fill in missing parts in images by using local structural information only. To do so, we formulate a partial differential equation (PDE) or a variational approach that picks up this information in terms of colour/grey values and image edges and propagates the information into the missing domain by means of transport and diffusion.

In this chapter we commence our presentation of variational and PDE methods for image inpainting with an axiomatic derivation of a PDE interpolator proposed by [CMS98a]. The resulting generic second-order PDE gives rise to our first three PDE inpainters: harmonic inpainting, total variation (TV) inpainting and absolutely minimising Lipschitz extensions (AMLE) inpainting. In the last section we focus on an extension of TV inpainting that leads us to the discussion of higher-order, in particular, curvature-based, PDEs for image inpainting in Chapter 5.

Throughout this chapter we use the following definitions and assumptions:

1. • The image domain Ω ⊂ ℝ2 is bounded and open with Lipschitz boundary ∂Ω.

2. • The inpainting domain D⊂ Ω lies in the interior of Ω, that is, ∂D∩∂=ø.

An Axiomatic Approach to Image Inpainting

One of the most pioneering contributions in PDE-based image processing methods certainly is the work of Alvarez et al. [AGLM93]. There the authors derive a general second-order PDE whose evolution describes a multi-scale analysis of an image, that is, a family of transforms which when applied to a given image produces a sequences of new images.

The scope of this chapter is the presentation of inpainting methods which use fourth-order (and higher!) partial differential equations (PDEs) to fill in missing image contents in gaps in the image domain. In the following section, we first motivate the use of higher-order flows for image inpainting.

Second- Versus Higher-Order Approaches

In this section we want to emphasise the difference between second-order diffusions as discussed in Chapter 4 and higher-order – in particular, fourth-order – diffusions in inpainting. As we have seen already, second-order inpainting methods (in which the order of the method is determined by the derivatives of highest order in the PDE), such as total variation (TV) inpainting, have drawbacks when it comes to the connection of edges over large distances and the smooth propagation of level lines into the damaged domain – qualities that we agreed an image interpolator which follows the good continuation principle from Chapter 3 should have. The disability, in general, of second-order methods to connect structures across the inpainting domain was demonstrated for harmonic inpainting in Figure 4.1 and for TV inpainting in Figure 4.8. An example of the lack of smoothness of interpolated level lines is given in Figure 4.7 for TV inpainting. In the case of TV inpainting, this behaviour of the interpolator is explained using the co-area formula, Theorem 4.3.6. To remind ourselves, TV inpainting seeks an interpolator whose level lines have minimal length, thus connecting level lines from the boundary of the inpainting domain via the shortest distance (linear interpolation). In [MM98, Mas98 and Mas02], Masnou and Morel propose an extension of the length penalisation in TV inpainting by an additional curvature term that should be small for interpolating level lines.