A Gaussian Hilbert space is a (complete) linear space of random variables with (centred) Gaussian distributions. This simple notion combines probability theory and Hilbert space theory into a rich and powerful structure, and Gaussian Hilbert spaces and connected notions such as the Wiener chaos decomposition and Wick products appear in several areas of probability theory and its applications, for example in stochastic processes and fields, stochastic integration, quantum field theory and limit theory for various statistics. There are also applications to non-probabilistic analysis, for example Banach space geometry and partial differential equations.

Although there are many references dealing with such applications where Gaussian spaces are treated and used, see for example Hida and Hitsuda (1976), Hida, Kuo, Potthoff and Streit (1993), Holden, Øksendal, Ubøe and Zhang (1996), Ibragimov and Rozanov (1970), Kahane (1985), Kuo (1996), Major (1981), Malliavin (1993, 1997), Meyer (1993), Neveu (1968), Nualart (1995, 1997+), Obata (1994), Pisier (1989), Simon (1974, 1979a), Watanabe (1984), there seems to be a shortage of works dealing with the basic properties of Gaussian spaces in general, without connecting them to a particular application. (One exception is the paper by Dobrushin and Minlos (1977).) This book is an attempt to fill the gap by providing a collection of the most important definitions and results for general Gaussian spaces, together with some applications to special Gaussian spaces.

In this chapter we use the theory of Gaussian Hilbert spaces to obtain the asymptotic distributions of some important random variables. In the first section, we study U-statistics. In the second section we extend the results to asymmetric statistics. In the third section we extend the results further; as a special case we obtain results for random graphs.

Note that the original variables are defined without any reference to normal variables or Gaussian Hilbert spaces; the Gaussian Hilbert space is introduced as a convenient tool to treat the asymptotic distribution.

A common theme in these results is that ‘typically’ the asymptotic distribution is normal, but in some ‘degenerate’ cases other limits occur; these other limits can be represented as variables in a Wiener chaos H:n: for some Gaussian Hilbert space H, and can for example be expressed as multiple Gaussian stochastic integrals. The explanation for this phenomenon that emerges from the proofs below is that the variable in question may be expanded as a sum, where each term converges in distribution to some chaos, and the first term is asymptotically normal. Typically, the first term dominates the sum and all others are asymptotically negligible, but in degenerate cases the first term vanishes and the sum is dominated by one or several later terms, which may converge to a higher order chaos.

The theory of Gaussian Hilbert spaces developed in this book has strong connections to stochastic integration, in particular to Itô integrals with respect to Brownian motion. We treat these Itô integrals in the first section, and some extensions and related results in the following ones: stochastic integrals over general measure spaces in Section 2, the Skorohod integral in Section 3, and complex stochastic integrals and measures in Section 4.

Our treatment is self-contained, and we do not require that the reader has any prior knowledge of stochastic integration. On the other hand, such a knowledge would certainly be useful; we treat only those parts of stochastic integration theory that are directly relevant to the subject of this book, and many important topics are not included. For example, we consider only stochastic integrals with respect to Gaussian processes. Moreover, even for Brownian motion we do not include the fundamental Itô formula.

Hence, this chapter will perhaps be best understood in connection and comparison with other, more direct and complete, treatments of stochastic integration; see for example McKean (1969) and Protter (1990).

Brownian motion and Itô integrals

In this section, we assume that Bt, 0 ≤ t < ∞, is a standard Brownian motion and consider, as in Example 1.10, the Gaussian Hilbert space H = H(B) spanned by {Bt}t≥0.

The Malliavin calculus (also known as stochastic calculus of variation) is a differential calculus for functions (i.e. random variables) defined on a space with a Gaussian measure. (In applications, the space is usually some version of the Wiener space.) In accordance with our general principle, we present here a version concentrating on the random variables without explicit mention of the underlying space.

We define in Sections 1–3 the basic derivative operators ∂ξ and ▽ for an arbitrary Gaussian Hilbert space, and in Section 9 the dual divergence operator. We also give a detailed treatment of the Sobolev spaces Dk, p in Sections 5–8; this includes a proof of the important Meyer inequalities in Section 8. Results on existence and smoothness of densities are given in Sections 4 and 10; these results are central in many applications. Finally, a connection with the Skorohod integral is established in Section 11.

The first application of Malliavin calculus (Malliavin 1978) was to study smoothness of solutions to partial differential operators. Many other applications have been developed later, for example to stochastic differential equations and stochastic integrals. We will not treat any of these applications here; for applications, other versions of the theory and further results on analysis on Wiener space we refer to for example Bell (1987), Bouleau and Hirsch (1991), Ikeda and Watanabe (1984), Malliavin (1993, 1997), Nualart (1995, 1997+), Nualart and Zakai (1986), Ocone (1987), Peters (1997+), Stroock (1981), Üstünel (1995), Watanabe (1984), Zakai (1985).