In this appendix, I outline and discuss selection issues that potentially affect the COIL data set and other data sets used in the international cooperation literature, more generally. Indeed, one potential criticism of the analyses presented in Parts II and III of this book centers precisely on the assumed correspondence between the population of interest, that is, the population of international agreements implicated by the theories of agreement design, and the COIL sample used in the analyses below for testing these theories.

In cases for which researchers rely on samples from available populations, and those populations represent non-random subsets of the population of interest, the conclusions drawn may suffer from what the broader literature terms “selection bias,” which is to say bias that results from using a sample selected not at random from the population of interest. Thus, COIL draws from the UNTS, but are the agreements registered with the UNTS the relevant population of interest?

Koremenos and Smith's “When to Select a Selection Model” (2014) lays out an algorithm for researchers to follow as they consider the potential for selection bias problems in particular substantive contexts. Koremenos and Smith argue that researchers and critics sometimes overstate the importance of selection as a result of misunderstanding the population of interest in particular studies. In some cases, the population of theoretical and empirical interest corresponds exactly to that from which the available data are sampled, and no further concerns about selection arise once the researcher makes this case. In other cases, the populations differ, but a compelling case can be made that the observed population represents a random subset of the population of interest from the point of view of the relationship under study, in which case empirical conclusions from a (random) sample drawn from the observed population generalize to the population of interest.

This appendix summarizes a few of the main points from Koremenos and Smith's rather technical manuscript and zeros in on examples most relevant to this book: the design of international agreements. Readers are directed to Koremenos and Smith (2014) for a fuller treatment of these issues and for examples broader than those considered here.