Many kinds of recursive definitions and recursive predicates appear in the descriptions of programs and programming languages. Some recursive definitions, such as list and tree data structures, are naturally covariant; these are straightforward to handle using a simple least-fixed-point method as described in Chapter 10. But some useful kinds of self-referencing definitions are not covariant. When the recursion goes through function arguments, it may be contravariant (see Ffunopt on page 64) or some mixture that is neither covariant nor contravariant. This kind of recursion requires more difficult mathematics, yet it is essential in reasoning about certain kinds of programs:

1. • Object-oriented programs in which class C has methods with a “this” or “self” parameter of type C;

2. • Functional programming languages with mutable references at higher types—such as ML;

3. • Concurrent languages with dynamically creatable locks whose resource invariants can describe other locks—a typical idiom in Pthreads concurrency;

4. • Functional languages (such as ML) where datatype recursion can go through function-parameters.

Does the C programming language have these features? Well, yes. C's type system is rather loose (with casts to void* and back). C programs that use void* in design patterns similar to objects or function closures can be perfectly correct, but proving their correctness in a program logic may need noncovariant recursion.

This chapter, and the next two chapters (predicate implication and subtyping; general recursive predicates) present the logical machinery to reason about such recursions in the VST program logics.

The Verified Software Toolchain has many components, put together in a modular way:

msl. The proof theory and semantics of separation logics and indirection theory is independent of any particular programming language, independent of the memory model, independent of particular theories of concurrency.

compcert. The CompCert verified C compiler is independent of any particular program logic (such as separation logic), of any particular theory of concurrency, and of the external-function context (such as an operating system-call setup). CompCert incorporates several programming languages, from C through C light to C minor and then (in various stages) to assembly languages for various target machines. The CompCert family may also include source languages such as C++ or ML. These various operational semantics all use the same memory model, and the same notion of external function call.

sepcomp. The theory of separate compilation explains how to specify the compilation of a programming language that may make shared-memory external function calls, shared-memory calls to an operating system, and shared-memory interaction with other threads. This depends on CompCert's memory model, but not on any particular one of the CompCert languages. Eventually, parts of the sepcomp theory will migrate into CompCert itself.

Some parts of the separate-compilation system concern modular program verifications of modular programs. We may even want to link program modules—and their verifications—written in different languages (C, ML, Java, assembly). This system requires that each language have a program logic that uses the same mpred (memory predicates) modeled using resource maps (rmap).

A program logic is sound when, if you can prove some specification (such as a Hoare triple {P} c {Q}) about a program c, then when c actually executes it will obey that specification.

What does it mean to “actually execute”? If c is written in a source language L, then we can formally specify an operational semantics for L. Then we can give a formal model for the program logic in terms of the operational semantics, and formally prove the soundness of all the inference rules of the logic.

Then one is left trying to believe that the operational semantics accurately characterizes the execution of the language L. But many source languages do not directly execute; they are compiled into lower-level languages or machine language that executes it its own operational semantics. Fortunately, at this point in the 21st century we can rely on formal compiler correctness proofs, that execution in the operational semantics of the source language corresponds to execution in the operational semantics of the machine language. And the machine languages tend to be well specified; machine language is already a formal language, and it is even possible to formally prove that the logic gates of a computer chip correctly implement the instruction set architecure (ISA), that is, the machine language.

So, we prove the program correct using the program logic, we prove the program logic sound with respect to the source-language operational semantics, and prove the compiler correct with respect to the source- and machine-language semantics.

Here we present a simple λ-calculus with references to illustrate the use of indirection theory. The λ-calculus is well understood and its type system presents no surprises, so it provides us as a nice vehicle for explaining how to apply indirection theory.

One reason this language is interesting, from our point of view, is that it was historically rather difficult to find a semantic theory for general references—that is, references that may contain data of any type, including quantified types. In contrast, the theory of references at base types (e.g., only containing integers) is much simpler. Tofte had an syntactic/operational theory of general references as early as 1990 [86], but it was not until the step-indexed model of Ahmed, Appel and Virga [4, 2] in 2003 that a semantic theory of general references was found. The model of Ahmed et al. was refined and generalized in the following years by Appel et al. [11], and then further refined by Hobor et al. [52] into the indirection theory that appears in this book.

The λ-calculus with references is a bit of a detour from our main aim in this book, which is building program logics for C. However, it provides a relatively simple, self-contained example that illustrates the techniques we will be using later in more complicated settings. In particular, we will use indirection theory to build the Hoare tuple for program logics for C along similar lines to how we construct the expression typing predicate in this chapter.

In Part III we described program verification for C: tools and techniques to demonstrate that C programs satisfy correctness properties. What we ultimately want is the correctness of a compiled machine language binary image, running on some target hardware platform. We will use a correct compiler that turns source-level programs satisfying correctness properties into machine-level programs satisfying those same properties. But defining formally the interface between a compiler correctness proof and a program logic has proven to be fraught with difficulties. Resolving these difficulties is still the object of ongoing research. Here we will explore some of the issues that have arisen and report on the current state of the integration effort.

The two issues that have caused the most headaches revolve around understanding and specifying how compiled programs interact with their environment. First, how should we reason about the execution environment when it may behave in unpredictable ways at runtime? In other words, how do we reason about program nondeterminism? Second, how do we specify correctness for programs that exhibit shared memory interactions?

The first question regarding nondeterminism is treated in detail in Dockins's dissertation [38]. Dockins develops a general theory of refinements for nondeterministic programs based on bisimulation methods. This theory gracefully handles the case where the execution environment is nondeterministic, and it has the critical feature that it allows programs to become more defined as they are compiled.

Predicates (of type A → Prop) in type theory give a model for Natural Deduction. A separation algebra gives a model for separation logic. We formalize these statements in Coq.

For a more expressive logic that permits general recursive types and quasi-self-reference, we use step-indexed models built with indirection theory. We will explain this in Part V; for now it suffices to say that indirection theory requires that the type T be ageable—elements of T must contain an approximation index. A given element of the model contains only a finite approximation to some ideal predicate; these approximations become weaker as we “age” them—which we do as the some operational semantics takes its steps.

To enforce that T is ageable we have a typeclass, ageable(T). Furthermore, when Separation is involved, the ageable mechanism must be compatible with the separating conjunction; this requirement is also expressed by a typeclass, Age_alg(T).

Theorem: Separation Algebras serve as a model of Separation Logic.

Proof. We express this theorem in Coq by saying that given type T, the function algNatDed models an instance of NatDed(pred T). Given a SepAlg over T, the function algSepLog models an instance of SepLog(pred T). The definability of algNatDed and algSepLog serve as a proof of the theorem.

What we show in this chapter is the indirection theory version (in the Coq file msl/alg_seplog.v), so ageable and Age-alg are mentioned from time to time.

Separation logics have assertions—for example P * (x ↦ y) * Q—that describe objects in some underlying model—for example “heaplets”—that separate in some way—such as “the heaplet satisfying P can join with (is disjoint from) the heaplet satisfying x ↦ y.” In this chapter we investigate the objects in the underlying models: what kinds of objects will we have, and what does it mean for them to join?

This study of join relations is the study of separation algebras. Once we know how the underlying objects join, this will explain the meaning of the * operator (and other operators), and will justify the reasoning rules for these operators.

In a typical separation logic, the state has a stack ρ for local variables and a heap m for pointers and arrays. Typically, m is a partial function from addresses to values. The key idea in separation logic is that that each assertion characterizes the domain of this function as well as the value of the function. The separating conjunction P * Q requires that P and Q operate on subheaps with disjoint domains.

In contrast, for the stack we do not often worry about separation: we may assume that both P and Q operate on the entirety of the stack ρ.

For now, let us ignore stacks ρ, and let us assume that assertions P are just predicates on heaps, so m ⊨ P is simply P(m).

For convenient application of the VST program logic for C light, we have synthetic or derived rules: lemmas built from common combinations of the primitive inference rules for C light. We also have proof automation: programs that look at proof goals and choose which rules to apply.

For example, consider the C-language statements x:=e→f; and e1→f := e2; where x is a variable, f is the name of a structure field, and e, e1, e2 are expressions. The first command is a load field statement, and the second is a store field. Proofs about these statements could be done using the general semax-load and semax-store rules—along with the mapsto operator—but these require a lot of reasoning about field l-values. It's best to define a synthetic field_mapsto predicate that can be used as if it were a primitive:

Definition field-mapsto (sh:share)(t1:type)(fld:ident)(v1 v2: val): mpred.

We do not show the definition here (see floyd/field_mapsto.v) but basically field_mapsto π τ v1v2 is a predicate meaning: τ is a struct type whose field f of type τ2 has address-offset δ from the base address of the struct; the size/signedness of f is ch, v1 is a pointer to a struct of type τ, and the heaplet contains exactly v1 + δ v2, (value v2 at address v1 + δ with permission-share π), where v2: τ2.

An important application of separation algebras is to model Hoare logics of programming languages with mutable memory. We generate an appropriate separation logic by choosing the correct semantic model, that is, the correct separation algebra. A natural choice is to simply take the program heaps as the elements of the separation algebra together with some appropriate join relation.

In most of the early work in this direction, heaps were modeled as partial functions from addresses to values. In those models, two heaps join iff their domains are disjoint, the result being the union of the two heaps. However, this simple model is too restrictive, especially when one considers concurrency. It rules out useful and interesting protocols where two or more threads agree to share read permission to an area of memory.

There are a number of different ways to do the necessary permission accounting. Bornat et al. [27] present two different methods; one based on fractional permissions, and another based on token counting. Parkinson, in chapter 5 of his thesis [74], presents a more sophisticated system capable of handling both methods. However, this model has some drawbacks, which we shall address below.

Fractional permissions are used to handle the sorts of accounting situations that arise from concurrent divide-and-conquer algorithms. In such algorithms, a worker thread has read-only permission to the dataset and it needs to divide this permission among various child threads.

Mappings are logical specifications of the relationship between schemas. In data exchange, one typically restricts the kind of dependencies allowed in mappings, either to be able to find more efficient procedures for constructing solutions and answering target queries, or to make mappings have desirable properties, such as closure under composition. These two tasks could be contradictory. For instance, the mapping language of SO tgds ensures closure under composition, but such mappings include a form of second-order quantification that can be difficult to handle in practice. Thus, it is desirable to replace an SO tgd by an equivalent set of st-tgds whenever possible.

In this chapter, we consider the problem of simplifying schema mappings by providing characterizations of the most common classes of mappings in terms of the structural properties they satisfy. The main goal for studying these properties is to isolate the features that different classes of mappings satisfy, and to understand what one can lose or gain by switching from one class of mappings to another. We present basic structural properties and then we use them to characterize the class of mappings specified by st-tgds, both generally, and in LAV and GAV scenarios. We also show that the structural characterizations can be used to derive complexity-theoretical results for testing definability of a mapping into some class of mappings.

So far we have tacitly assumed that one uses a native XML DBMS for performing data exchange tasks. However, this is not the only (and perhaps not even the most common) route: XML documents are often stored in relational DBMSs. Thus, it is natural to ask whether relational data exchange techniques, developed in PART TWO, can be used to perform XML data exchange tasks.

In XML terminology, translations from XML to relations are referred to as shredding of documents, whereas translations going the other way, from relations to XML, are referred to as publishing. Thus, to use relational technology for XML data exchange tasks, we can employ a two-step approach:

1. shred XML data into relations;

2. then apply a relational data-exchange engine (and publish the result back as an XML document if necessary).

The seems very natural, but the key question is whether it will work correctly. That is, are we guaranteed to have the same result as we would have gotten had we implemented a native XML data-exchange system? This is what we investigate in this chapter. It turns out that we need to impose restrictions on XML schema mappings to enable this approach, and the restrictions are similar to those we needed to ensure tractability of data exchange tasks in the previous chapters.

Translations and correctness

We now describe what we mean by correctness of translations that enable a relational data exchange system to perform XML data exchange tasks.