This chapter concerns models for 2D and 3D shape. The motivation for shape models is twofold. First, we may wish to identify exactly which pixels in the scene belong to a given object. One approach to this segmentation problem, is to model the outer contour of the object (i.e., the shape) explicitly. Second, the shape may provide information about the identity or other characteristics of the object: it can be used as an intermediate representation for inferring higher-level properties.

Unfortunately, modeling the shape of an object is challenging; we must account for deformations of the object, the possible absence of some parts of the object and even changes in the object topology. Furthermore, the object may be partially occluded, making it difficult to relate the shape model to the observed data.

One possible approach to establishing 2D object shape is to use a bottom-up approach; here, a set of boundary fragments are identified using an edge detector (Section 13.2.1) and the goal is to connect these fragments to form a coherent object contour. Unfortunately, achieving this goal has proved surprisingly elusive. In practice, the edge detector finds extraneous edge fragments that are not part of the object contour and misses others that are part of the true contour. Hence it is difficult to connect the edge fragments in a way that correctly reconstructs the contour of an object.

In the final part of this book, we discuss four families of models. There is very little new theoretical material; these models are straight applications of the learning and inference techniques introduced in the first nine chapters. Nonetheless, this material addresses some of the most important machine vision applications: shape modeling, face recognition, tracking, and object recognition.

In Chapter 17 we discuss models that characterize the shape of objects. This is a useful goal in itself as knowledge of shape can help localize or segment an object. Furthermore, shape models can be used in combination with models for the RGB values to provide a more accurate generative account of the observed data.

In Chapter 18 we investigate models that distinguish between the identities of objects and the style in which they are observed; a prototypical example of this type of application would be face recognition. Here the goal is to build a generative model of the data that can separate critical information about identity from the irrelevant image changes due to pose, expression and lighting.

In Chapter 19 we discuss a family of models for tracking visual objects through time sequences. These are essentially graphical models based on chains such as those discussed in Chapter 11. However, there are two main differences. First, we focus here on the case where the unknown variable is continuous rather than discrete. Second, we do not usually have the benefit of observing the full sequence; we must make a decision at each time based on information from only the past.

The genesis of this great and beautiful book spans more than 20 years. It collects and unifies many theoretical notions and results published by Bruno Courcelle and others in a large number of articles.

The concept of a language to communicate with a computer, a machine or any kind of device performing operations is at the heart of Computer Science, a field that has truly thrived with the emergence of symbolic programming languages in the 1960s. Formalizing the algorithms that enable computers to calculate an intended result, to control a machine or a robot, to search and find the relevant information in response to a query, and even to imitate the human brain in actions such as measuring risk and making decisions, is the main activity of computer scientists as well as of ordinary computer users.

The languages designed for these tasks, which number by thousands, are defined in the first place by syntactic rules that construct sets of words and to which are then attached meanings. This understanding of a language was first conceived by structural linguists, in particular Nicolaï Troubetskoï, Roman Jacobson and Noam Chomsky, and has transformed Linguistics, the study of natural languages, by giving it new directions. It has also been extended to programming languages, which are artificial languages, and to the Lambda Calculus, one of many languages devised by logicians, among whom we can cite Kurt Gödel, Alonzo Church and Alan Turing, who aspired to standardize mathematical notation and to mechanize proofs.

This chapter presents the main definitions and results of this book and their significance, with the help of a few basic examples. It is written so as to be readable independently of the other chapters. Definitions are sometimes given informally, with simplified notation, and most proofs are omitted. All definitions will be repeated with the necessary technical details in subsequent chapters.

In Section 1.1, we present the notion of equational set of an algebra by using as examples a context-free language, the set of cographs and the set of series-parallel graphs. We also introduce our algebraic definition of derivation trees.

In Section 1.2, we introduce the notion of recognizability in a concrete way, in terms of properties that can be proved or refuted, for every element of the considered algebra, by an induction on any term that defines this element. We formulate a concrete version of the Filtering Theorem saying that the intersection of an equational set and a recognizable one is equational. It follows that one can decide if a property belonging to a finite inductive set of properties is valid for every element of a given equational set. We explain the relationship between recognizability and finite automata on terms.

In Section 1.3, we show with several key examples how monadic second-order sentences can express graph properties. We recall the fundamental equivalence of monadic second-order sentences and finite automata on words and terms.

As explained in the introduction of Section 1.7, there are no appropriate finite-state automata or finite-state transducers that work “directly” on graphs. Thus, for graphs, the role of automata and transducers is taken over by monadic second-order logic: instead of being accepted by an automaton, a set of graphs is defined by a monadic second-order sentence; and instead of being computed by a transducer, a graph transduction is defined by a definition scheme consisting of monadic second-order formulas. With respect to automata, the original motivation for this approach was Theorem 1.16 (cf. Theorem 5.82): for terms and words monadic second-order logic and finite-state automata have the same expressive power.

The aim of this chapter is to show that, in a certain sense, the automata-theoretic characterization of monadic-second order logic for sets of terms and words (Theorem 1.16) can be generalized to transductions. This means of course, that the automata should produce output, i.e., that they are transducers. We will concentrate on transductions that are partial functions, and in particular on deterministic devices, i.e., parameterless definition schemes and deterministic transducers. For these, we will show that monadic second-order transductions of words correspond to two-way finite-state transducers, and that monadic second-order transductions of terms correspond to (compositions of) tree-walking transducers. These transducers are well known from Formal Language Theory.

A two-way finite-state transducer is a finite-state automaton with a two-way read-only input tape and a one-way output tape. The input word is placed on the input tape between endmarkers, and the transducer has a reading head that is positioned over a cell of the input tape, at each moment of time.

We will define graph operations that generalize the concatenation of words. Some of these operations have been used in Section 1.1 to define the cographs and the series-parallel graphs. We will define actually two signatures of graph operations, hence two graph algebras. Both algebras will be defined in such a way that their equational sets are exactly the sets defined by certain context-free graph grammars. (The notion of an equational set has been presented informally in Section 1.1.4 and will be studied in detail in Chapter 3.)

Two main types of context-free sets of graphs defined by certain graph rewriting mechanisms have emerged from the intense research conducted from around 1980 and synthesized in the handbook [*Roz] edited by Rozenberg.

We will first define the HR graph algebra that corresponds in this respect to the hyperedge replacement grammars. It turns out that the operations of this algebra yield an exact characterization (as equational sets) of the sets of graphs of tree-width bounded by fixed integers. The terms built with the operations of the HR algebra can be seen as algebraic expressions of tree-decompositions. (Tree-width and the corresponding tree-decompositions are important for the construction of efficient graph algorithms and also for the characterization of the graphs that exclude a fixed graph as a minor.)

The second algebra, called the VR algebra, is defined so that its equational sets are those generated by the (context-free) vertex replacement grammars. A new graph complexity measure called clique-width has arisen in a natural way from the definition of the VR algebra, without having (yet) any independent combinatorial characterization.

This chapter defines monadic second-order logic, and shows how it can be used to formalize the expression of graph properties. Monadic second-order formulas can also be used to express the properties of sets of vertices and/or edges in graphs. The Recognizability Theorem (Section 5.3.8) implies that a class of graphs characterized as the class of (finite) models of a monadic second-order sentence is VR-recognizable, and that it is HR-recognizable (but not necessarily VR-recognizable) if the sentence is written with edge set quantifications.

It follows that the monadic second-order satisfiability problem for the class of graphs of tree-width or of clique-width at most k is decidable for each k. Applications to the construction of fixed-parameter tractable algorithms and other algorithmic consequences will be developed in Chapter 6.

Although our first applications concern graphs, we will give the definitions and prove the main results for the general case of relational structures because proofs are not more difficult, and the corresponding results apply in a uniform manner to labeled graphs represented, in several ways, by binary relational structures. However, relational structures with n-ary relations for n ≥ 2 are actually interesting by themselves. As we have shown in Section 1.9, they are useful to formalize betweenness and cyclic ordering (and for relational databases).

In Sections 5.1 and 5.2, we review basic notions of logic, define monadic second-order logic, and give some results that delineate its expressive power.

We will leave it to the researchers of the next ten years and beyond to decide which of the results presented in this book are the most important. Here we will only indicate what are, in our opinion, the main open problems and research directions, among those that are closely related with the topics of this book.

Algorithmic applications

This topic, studied in Chapter 6, is the most difficult to discuss because of the large number of new articles published every year. Hence, the following comments have a good chance of becoming rapidly obsolete.

The algorithmic consequences of the Recognizability Theorem depend crucially on algorithms that construct graph decompositions of the relevant types or, equivalently, that construct terms over the signatures FHR, FVR which evaluate to the input graphs or relational structures. The efficiency of such algorithms, either for all graphs, but perhaps more promisingly for particular classes of graphs, is a bottleneck for applications.

Another difficulty is due to the sizes of the automata that result from the “compilation” of monadic second-order formulas. Their “cosmological” sizes (cf. [FriGro04], [StoMey], [Wey]) make their constructions intractable for general formulas. However, for certain formulas arising from concrete problems of hardware and software verification they remain manageable, as reported by Klarlund et al., who developed software for that purpose called MONA (cf. [Hen+], [BasKla]). Soguet [Sog] tried using MONA for graphs of bounded tree-width and clique-width, but even for basic graph properties such as connectivity or 3-colorability, the automata become too large to be constructed as soon as one wishes to handle graphs of clique-width more than 3.

Monadic second-order transductions are transformations of relational structures specified by monadic second-order formulas. They can be used to represent transformations of graphs and related combinatorial structures via appropriate representations of these objects by relational structures, as shown in the examples discussed in Section 1.7.1.

These transductions are important for several reasons. First, because they are useful tools for constructing monadic second-order formulas with the help of the Backwards Translation Theorem (Theorem 7.10). Second, because by means of the Equationality Theorems (Theorems 7.36 and 7.51) they yield logical characterizations of the HR- and VR-equational sets of graphs that are independent of the signatures FHR and FVR. From these characterizations, we get short proofs that certain sets of graphs have bounded, or unbounded, tree-width or clique-width.

They also play the role of transducers in formal language theory: the image of a VR-equational set of graphs under a monadic second-order transduction is VR-equational, and a similar result holds for HR-equational sets and monadic second-order transductions that transform incidence graphs. Finally, the decidability of the monadic second-order satisfiability problem for a set of structures C implies the decidability of the same problem for its image τ(C) under a monadic second-order transduction τ. Hence, monadic second-order transductions make it possible to relate decidability and undecidability results concerning monadic second-order satisfiability problems for graphs and relational structures.

Section 7.1 presents the definitions and the fundamental properties. Section 7.2 is devoted to the Equationality Theorem for the VR algebra, one of the main results of this book.