Introduction

Hierarchical models have played a central role in Bayesian inference since the time of Good (1965) with Lindley and Smith (1972) a key milestone providing the first comprehensive treatment of hierarchical priors for the parametric Bayes linear model. The Bayesian hierarchies have proved so popular because they provide a natural framework for “borrowing of strength” (a term apparently due to Tukey) or sharing partial information across components through the hierarchical structure. The Bayesian construction also provides a clear distinction from frequentist models in that the dependence structures do not have to be directly related to population random effects.

In this chapter we will look a little more closely at a couple of key issues emerging from the work of Teh and Jordan. First, we briefly step back a little and consider the fundamental notion played by de Finetti's representation theorem, reiterating the operational focus of Bayesian modeling on finite-dimensional specification of joint distributions on observables.

Introduction

The Dirichlet process has been a cornerstone in Bayesian nonparametrics since the seminal paper by T. S. Ferguson appeared in the Annals of Statistics in 1973. Its success can be partly explained by its mathematical tractability and it has grown tremendously with the development of Markov chain Monte Carlo (MCMC) techniques whose implementation allows a full Bayesian analysis of complex statistical models based on the Dirichlet process prior.

Introduction

Even though there is no physical connection between observations, there is a real and obvious reason for creating a dependence between them from a modeling perspective. The first observation, say X1, provides information about the unknown density f from which it came, which in turn provides information about the second observation X2, and so on. How a Bayesian learns is her choice but it is clear that with i.i.d. observations the order of learning should not matter and hence we enter the realms of exchangeable learning models. The mathematics is by now well known (de Finetti, 1937; Hewitt and Savage, 1955) and involves the construction of a prior distribution Π(d f) on a suitable space of density functions.

Bayesian nonparametrics

As modern statistics has developed in recent decades various dichotomies, where pairs of approaches are somehow contrasted, have become less sharp than they appeared to be in the past. That some border lines appear more blurred than a generation or two ago is also evident for the contrasting pairs “parametric versus nonparametric” and “frequentist versus Bayes.” It appears to follow that “Bayesian nonparametrics” cannot be a very well-defined body of methods.

What is it all about?

It is nevertheless an interesting exercise to delineate the regions of statistical methodology and practice implied by constructing a two-by-two table of sorts, via the two “factors” parametric–nonparametric and frequentist–Bayes; Bayesian nonparametrics would then be whatever is not found inside the other three categories.

(i) Frequentist parametrics encompasses the core of classical statistics, involving methods associated primarily with maximum likelihood, developed in the 1920s and onwards. Such methods relate to various optimum tests, with calculation of p-values, optimal estimators, confidence intervals, multiple comparisons, and so forth.

Introduction

Biomedical research has clearly evolved at a dramatic rate in the past decade, with improvements in technology leading to a fundamental shift in the way in which data are collected and analyzed. Before this paradigm shift, studies were most commonly designed to be simple and to focus on relationships among a few variables of primary interest. For example, in a clinical trial, patients may be randomized to receive either the drug or placebo, with the analysis focusing on a comparison of means between the two groups. However, with emerging biotechnology tools, scientists are increasingly interested in studying how patients vary in their response to drug therapies, and what factors predict this variability.

Introduction

In Chapter 7, Dunson introduced many interesting applications of nonparametric priors for inference in biomedical problems. The focus of the discussion was on Dirichlet process (DP) priors and variations. While the DP prior defines a probability model for a (discrete) random probability distribution G, the primary objective of inference in many recent applications is not inference on G. Instead many applications of the DP prior exploit the random partition of the Pólya urn scheme that is implied by the configuration of ties among the random draws from a discrete measure with DP prior. When the emphasis is on inference for the clustering, it is helpful to recognize the DP as a special case of more general clustering models. In particular we will review the product partition (PPM) and species sampling models (SSM). We discuss these models in Section 8.2. A definition and discussion of the SSM as a random probability measure also appears in Section 3.3.4. Another useful characterization of the DP is as a special case of the Pólya tree (PT) prior.

Introduction

Making inferences from observed data requires modeling the data-generating mechanism. Often, owing to a lack of clear knowledge about the data-generating mechanism, we can only make very general assumptions, leaving a large portion of the mechanism unspecified, in the sense that the distribution of the data is not specified by a finite number of parameters. Such nonparametric models guard against possible gross misspecification of the data-generating mechanism, and are quite popular, especially when adequate amounts of data can be collected. In such cases, the parameters can be best described by functions, or some infinite-dimensional objects, which assume the role of parameters. Examples of such infinite-dimensional parameters include the cumulative distribution function (c.d.f.), density function, nonparametric regression function, spectral density of a time series, unknown link function in a generalized linear model, transition density of a Markov chain and so on.

Introduction

Hierarchical modeling is a fundamental concept in Bayesian statistics. The basic idea is that parameters are endowed with distributions which may themselves introduce new parameters, and this construction recurses. A common motif in hierarchical modeling is that of the conditionally independent hierarchy, in which a set of parameters are coupled by making their distributions depend on a shared underlying parameter. These distributions are often taken to be identical, based on an assertion of exchangeability and an appeal to de Finetti's theorem.

Anderson (1984, p. 2) discusses the inadequacy of the ordered choice model we have examined thus far:

Generalizations of the model, e.g., Williams (2006), have been predicated on Anderson's observations, as well as some observed features in data being analyzed and the underlying data-generating processes.

It is useful to distinguish between two directions of the contemporary development of the ordered choice model. Although it hints at some subtle aspects of the model (underlying data-generating process), Anderson's argument directs attention primarily to the functional form of the model and its inadequacy in certain situations. Beginning with Terza (1985), a number of authors have focused instead on the fact that the model does not account adequately for individual heterogeneity that is likely to be present in micro-level data. This chapter will consider the first of these. Heterogeneity is examined in Chapter 7.