Statistical Modeling and Inference for Social Science

Recognizing a distinction between events that did occur and events that did not occur but might have is the point of the previous chapter. Given this distinction, we are generally uncertain, before a process unfolds, about what its outcome will be. We would like to relate the observable to the more fundamental data-generating process (DGP) behind it. But if the DGP is stochastic, we will always be uncertain about its defining features, and we need a language to express that uncertainty. Turning this around, given some stochastic DGP, we are generally uncertain about what might result from it. We need to be able to express this uncertainty explicitly and carefully.

To do so, we need some tools from the theory of probability. Probability theory is a conceptual apparatus in mathematics for expressing and evaluating uncertainty. As such, it is an important foundational component of statistical inference. But it is different from statistics. In probability theory, we start with a DGP with basic properties that we know (or assume, or pretend to know) and work out the consequences for the events that might be observed (e.g., their probabilities, how those probabilities are related to the DGP).

In (classical) statistical theory, by contrast, we start with a set of events we have observed and attempt to infer something about the properties of the stochastic DGP that generated them. In other words, probability contemplates what data will be observed from a given stochastic process.