Statistics can be more than a tool for describing data. In the social sciences we have hypotheses that move us beyond simple descriptions of populations to relationships between two or more variables. To analyze these relationships we often rely in practice on statistical inference. That is, we need to make decisions based on data collected on a small group (sample) for the larger group that we want to study (population). In Chapter 4 we introduced the concept of sampling and provided a description of the three general objectives pertaining to sampling: representativeness, size, and level of analysis. Here we move to a more focused explanation of how sampling helps researchers overcome practical limitations and what implications this has for quantitative analyses.
As we noted earlier, when it comes to sampling, larger is generally better, the aforementioned issues aside. So why sample at all? Why not collect data on the full population? In all types of research we find the same two answers: time and money. Research occurs in the real world and collecting the ideal data is often limited by how much time and money the researcher can direct to the project. Thus every research design must take into account the practical circumstances. In doing so, researchers necessarily restrict their investigation into a sample or subset of the population that they would like to study.
In general, when we make comparisons we would like to talk about more than just the observations in the sample. We would like to talk about the larger group of interest in our research, the population. How we do this is the objective of statistical inference. More specifically, statistical inference helps researchers provide statements of confidence in our ability to generalize or infer from the sample to the population. The ability to offer an estimate of relative precision is another reason why quantitative empirical research is so useful and popular.
Consider the ANES dataset that we used for the examples in Chapter 18. How many cases or individuals do we have in our sample? Obviously, the nearly 6,000 individuals in that sample is not anywhere near the size of the full voting age population of the United States, yet we would like to use this sample of data to describe that population.
This section of the book has covered a lot of ground. We have presented a variety of causal frameworks (Chapter 5), defined the problem of causal inference (Chapter 6), and introduced an itinerary of research designs, which we classified as experimental (Chapter 7), large-N observational (Chapter 8), or case study (Chapter 9).
This short chapter summarizes some of the themes discussed in this long, and complex, part of the book. We begin by reviewing the problem of confounding, and the various types of confounding that impede causal inference. Next, we review the panoply of research designs discussed in previous chapters, illustrating their interrelationship in a tree diagram, and discussing whether there is a “best” research design. We conclude by discussing the multi-method approach to research.
Identifying and Avoiding Confounders
Causal inference is possible only if confounders can be avoided or controlled in the analysis. Recall that we adopted a very broad definition of confounding, including any factor that produces a spurious (or biased) association between X and Y.
This idea may be rendered in a more concrete fashion using the language of causal graphs. A causal model will generally render a valid estimate if the “frontdoor” path from X to Y is unblocked and any existing “backdoor” paths from X to Y are blocked. Simply stated, a causal model should allow no covariation between X and Y except that which is a product of X's causal impact on Y. This is another way of describing causal comparability, i.e., the expected value of Y for a given value of X is the same for all units throughout the period of analysis. (You will see that there are many languages by which to express the same general idea.)
Five types of confounders, introduced in previous chapters, are recapitulated in Figure 10.1. In each case, Z stands for the confounder, generating a backdoor path from X to Y. Recall that in interpreting these graphs one must distinguish settings in which Z is conditioned from settings in which it is not (indicated by square brackets). In some settings, a confounder is created because a factor is not conditioned, in other settings because it is conditioned.
In this chapter we build on the concept of correlation with ordinary least squares (OLS) regression, even though the latter was invented first. If we think about our scatter plot, we could draw one line through the data that best fits all of our data points. Correlation is a statement about how close the points are to the line. The objective of regression is to determine the best fitting line for the data. Using regression, we can determine the average effect of our independent variable and make predictions about cases outside our sample. We will begin with the simplest regression model, bivariate, where the dependent variable is a function of a single independent variable, before expanding to consider multivariate models with multiple independent variables.
As social scientists we primarily want to explain why variables of interest vary and vary together. Regression allows us the ability to measure the effect of one variable on another. It tells us the effect of an independent variable on a dependent variable. Furthermore, it provides us with the degree of the effect, thereby providing more explanatory leverage than in any other technique we have discussed thus far. Not unlike correlation we can find the strength and direction of association between two variables. Here, however, we can also get at the specific nature of the relationship; i.e., how much variance in the dependent variable is “explained” by the independent variable.
In discussing correlation we implied without much specificity that one could draw a straight line that passed through the set of points in a manner that represented the overall pattern, positive or negative, steep or shallow. In addition to moving to thinking about causal relationships between independent and dependent variables (which was not required for correlation), with regression we also ask: Which linear relationship? In other words, of all the lines in Figure 22.1 that pass through the graph centroid – the point where X and Y intersect and marked in the figure by the intersection of the dotted lines – which fits the data the best?
Before delving into the math, it is useful to graphically illustrate the characteristics of the best fitting regression line, as in Figure 22.2. For any line, we can measure the vertical distance between the line and each observation, which is called a residual.
Arguments are articulated with the use of key concepts. Indeed, the argument of a study is inseparable from its key concepts since the latter are the linguistic tools with which an argument is formulated. Any study of democracy, for example, must wrestle with the problem of how to define this key term – which will guide our discussion in this chapter.
Concepts, in turn, receive empirical meaning through the indicators chosen to measure them. Any study of democracy must be concerned not only with how to define democracy but also with how to operationalize (measure) this abstract concept.
Conceptualization and measurement are thus closely linked. This is why we have chosen to present them together in this chapter, which begins with concept formation and proceeds to measurement.
The key concepts of social science are never fixed and, regrettably, not always clear. Many abstract concepts – such as democracy or social capital – are employed in a variety of ways and thus mean different things in different contexts. This is true even of more specific concepts such as worker-training programs. (Does a one-day program focusing on advice for job-hunting qualify? How about a person who enlists government support to take classes at a community college, or an apprenticeship program?)
The persistent ambiguity of key concepts makes it difficult for the reader, who may struggle to figure out what a term means in a given context and how it connects with other work (using the same or similar terms). It also makes it difficult for writers, who must identify which of several terms they should adopt in their own work and how they should define the chosen term.
Sometimes, the task of forming concepts seems highly arbitrary. And this, in turn, may prompt readers to adopt a skeptical attitude toward the subject. At the same time, the choice of concepts is never entirely arbitrary. Some choices are usually better than others, and a few are patently absurd.
In this spirit, we offer the following criteria, intended to guide the process of concept formation in the social sciences. A good concept, we shall argue, is resonant, internally coherent, externally differentiated, theoretically useful, and consistent in meaning, as summarized in Table 3.1.
Much of social science research is concerned with causal relationships. In this chapter we explore the general framework for making causal statements using different methodological approaches. With experiments as our point of reference, we revisit regression in the context of a causal treatment variable, and then introduce a technique to evaluate the causal effect of a treatment with observational data by matching treated and control units. Before doing so we lay out the specific assumptions necessary for causality and the different motivating factors for each causal model.
Assumptions and Assignment
Why causal inference? Our statistical objective thus far has been to infer associations among variables. From these associations we can estimate probabilities of events with the statistical methods introduced above, provided that the external conditions remain the same. This allows us to answer important questions, like: What is the mean number of parties in democracies? Are turnout and geographic regions related? Are greater issue dimensions associated with greater numbers of parties? On the contrary, we need causal inference when we would like to infer probabilities under different conditions. That is, when we would like to know what would happen if something else happened. When we expect probabilities to change in response to external factors we must rely on causal analysis. Thus, causal questions ask somewhat different questions: What are the effects of worker-training programs? Does smoking cause cancer? Does viewing a campaign advertisement change vote preferences? In each case we are asking a question that posits different probabilities under different conditions: attending versus not attending a worker-training program; smoking versus not smoking; viewing versus not viewing a campaign ad.
Throughout the statistics section of this book we have identified hypothesis tests that are based on covariance between suspected cause and effect. However, the tests themselves are only covariational; that is, they are not explicitly causal. As we discussed in Chapters 21 and 22, in order to make claims of causality we need more than the evidence of covariation between cause and effect that we can attain from these statistical methods. Minimally, we also need the cause to precede the effect and to be able to eliminate plausible alternative causes.
This chapter is devoted to reading and reviewing the social science literature on a topic. Here, we discuss how to distinguish social science sources from other sorts of work, how to locate and survey the literature on a chosen topic, how to read strategically, how to read critically, how to figure out complex arguments, how to construct a systematic review of the literature on a topic, and how to take notes as you go along. These are closely linked topics so you will find a good deal of overlap across these sections.
As you read social science you may be struck by the dry tone of the literature, especially in journal articles. (Books usually attempt to include some divertissement to relieve the tedium of pages and pages of prose.) Remind yourself that what is exciting about science – any science – is getting closer to the truth. The mode of exposition is secondary, and generally remains in the background. Indeed, it is important to adopt a dispassionate tone to discourage other factors from interfering with the logic of the argument. This is why authors generally avoid personal anecdotes, jokes, heavily symbolic or allegorical language, and other narrative devices.
Of course, there is a person behind the prose, and perhaps it would aid the cause of science if some personal elements – such as his/her motivations for studying a subject – were made explicit. But, for better or for worse, this is not the accepted scientific mode of communication in most social science fields. Bear with it.
In any case, new media are opening up new modes of communication, many of which are more personal in nature and less tightly structured. On virtually any given topic you can now find blogs (discussed below) or lectures and debates preserved on YouTube, as well as other multi-media presentations. The medium is changing, though the austere format of journal articles is likely to remain the workhorse of social science for the foreseeable future.
Identifying Social Science Sources
Reading social science presumes that one can identify works of social science from the mass of other sources out there in print and on the World Wide Web. This brings us back to an earlier discussion. What is it that distinguishes social science from other genres such as casual conversation, journalism, or partisan rhetoric?
Causation is the central explanatory trope by which relationships among persons and things are established – the cement of the universe, in Hume's words. Without some understanding of who is doing what to whom we cannot make sense of the world that we live in, we cannot hold people and institutions accountable for their actions, and we cannot act efficaciously in the world. Without a causal understanding of the world it is unlikely that we could navigate even the most mundane details of our lives, much less plan for the future. This is obvious in the policy world, where causal understanding undergirds any rational intervention. One must have some sense of what impact a policy is likely to have in order to support its adoption.
Even where causal understanding does not relate to future changes in the status quo we are likely to be reassured when we can order events around us into cause-and-effect relationships. “When we have such understanding,” notes Judea Pearl, “we feel ‘in control’ even if we have no practical way of controlling things.” Causality is not just a methodological preoccupation. It is also a way of relating to the world. That said, there are important differences between causal inference in everyday contexts and in social-scientific contexts.
Chapter 5 introduces a variety of causal frameworks that are widely employed in social science today. They may be viewed as the building blocks of a causal explanation. Chapter 6 defines the topic of causality, and lays out the attributes of a good causal hypothesis and the core components of causal analysis. Chapter 7 discusses experimental research designs, where the causal factor of interest is randomized across groups. Chapter 8 discusses large-N observational designs (i.e., non-experimental designs). Chapter 9 deals with case study designs, where the number of units is limited to one or several. Chapter 10 reviews and reflects on various aspects of causal inference, serving as a coda for this part of the book.
Recall (from Chapter 6) that estimating a causal effect involves comparing a factual (that which actually happened) to a counterfactual (that which might have happened). Returning to our perennial example, let us say that we wish to estimate the effect of a job-training program on the earnings of unemployed people after they have completed the program. Those who participate in the program are members of the treatment group. Non-participants are members of the control group.
Let us say that we know the earnings of participants and non-participants after completing the program. These are the factuals. What we do not know is what their earnings would have been if their roles had been reversed. These are the counterfactuals.
Although the only sure way to compare factuals with counterfactuals is to employ a time-machine, a well-designed experiment comes close to the mythical time-machine insofar as the control group exemplifies the (unobserved) counterfactual for the treatment group, and the treatment group exemplifies the (unobserved) counterfactual for the control group. Under certain conditions, this is a plausible scenario.
For present purposes, the defining criterion of an experiment is that the treatment is randomly assigned (“randomized”) across subjects. We do not care who controls the experiment – the researcher conducting the study or someone else. (Sometimes, an experiment that unfolds naturally, without intervention by a researcher, is referred to as a natural experiment, as discussed below.)
We begin this chapter with a review of the problem of confounding as it applies to experimental research. We proceed to introduce various approaches to experimental research. The final section offers a series of examples of experiments conducted on a variety of diverse topics. Together, these sections should give the reader a sense of the opportunities for, and limitations of, experimental research in the social sciences.
Experiments With and Without Confounding
In its simplest version, a single treatment (e.g., a worker-training program) is randomly assigned to members drawn from a known population (e.g., unemployed people of a given age). That is, some are chosen to participate in the program (the treatment group) and others are chosen to become part of the control group. The causal effect of the program is measured by comparing annual earnings of program participants with annual earnings of non-participants one year after the completion of the program.
The first step toward analysis is to get your data into a format that is convenient for the sort of analysis you wish to pursue. This, in turn, probably depends on the sort of data you are collecting. Here, we shall distinguish among four data types: qualitative, medium-N , large-N , and textual (though one can have data that is a combination of these; e.g., medium-N and qualitative, large-N and textual … etc.). Methods for handling these data types are continually invented and reinvented, so the reader may wish to consult other sources to obtain the most up-to-date information on these subjects. Our intent is to provide a useful overview, in any case, not to delve into the details.
Suppose that you are trying to integrate data drawn from a limited number of units with a diversity of evidence. The evidence may have been gathered with any of the techniques (or combination of techniques) discussed in Chapter 13. It might include text, photos, maps, and other media. The nature of the material might be variegated – a combination of what informants said and did, the researcher's own observations and theories, multi-media artifacts, locations, relevant articles from academic journals and/or popular media, feedback from colleagues, and so forth. Some of the evidence may apply across all studied units in the sample while some is specific to certain units. But one doesn't have systematic observations for a limited set of variables across all units in the sample. It may not even be clear what the variables are, what the sample is, or what the population of the study is. There may be – at least initially – no specific hypothesis but rather a general research question that awaits further refinement. In other words, the investigation may be more exploratory (to discover a theory or hypothesis) than confirmatory (to test a theory or hypothesis).
This setting exemplifies a good deal of work often described as qualitative, so we shall refer to it as qualitative data (as defined in Chapter 4).
There are two ways to learn about a subject. One might study many examples at once, focusing on a few selected dimensions of the phenomena. We shall refer to this as an extensive approach, as laid out in Chapters 7–8.
Alternatively, one might study a particular example, or several examples, in greater depth. We shall refer to this as an intensive, or case study, approach – the topic of this chapter.
A case connotes a spatially delimited phenomenon (a unit) observed at a single point in time or over some period of time. It may be a political unit with a defined area of semi-sovereignty (e.g., empire, nation-state, region, municipality), organization (e.g., firm, non-governmental organization, political party, school), social group (as defined, e.g., by ethnicity, race, age, class, gender, or sexuality), event (e.g., foreign policy crisis, revolution, democratic transition, decision-point), or individual (e.g., a biography, case history).
However defined, a case must comprise the type of phenomenon that an argument attempts to describe or explain. In a study about nation-states cases are comprised of nation-states (observed over time). In a study that attempts to explain the behavior of individuals, cases are comprised of individuals. And so forth.
A case study research design is an intensive study of a single case or a small number of cases that promises to shed light on a larger population of cases. The individual case(s) is viewed as a case of something broader, just as large-sample analysis is also generally viewed as exemplary of a broader phenomenon. Thus, both intensive and extensive analyses generally make inferences from a sample to a population, even though the sample sizes are very different.
Case study research may incorporate one or several cases. The latter is a defining characteristic of comparative historical analysis, associated with the work of Barrington Moore, Theda Skocpol, David Collier, and James Mahoney. However, as the sample of cases expands it becomes less and less feasible to investigate each case intensively. The case study format is thus implicitly a small-sample format.
In the previous chapter we discussed general causal frameworks, the building blocks of a causal explanation. In this chapter, we focus on specific hypotheses, where one factor is thought to generate change in another factor.
We begin by clarifying the concept of causality. In the next section, we discuss the criteria of a good (well-constructed) causal hypothesis. The rest of the chapter is devoted to causal analysis. First, we outline the criteria that all causal research designs seek to achieve. Next, we discuss the problem of reaching causal inference.
This chapter is fairly complex. A number of new terms are introduced, some of which may be unfamiliar to the reader and some of which are used in slightly different ways in different disciplines. Although the vocabulary may seem bewildering at first, try to familiarize yourself with these concepts – which you are likely to encounter in your reading and in your future work. The topics covered here are critical for understanding how evidence is used to infer causality in social-science settings. Whether you are primarily a consumer or a producer of social science the following chapters bear a close read and a good think.
A causal hypothesis involves at least two elements: a cause and an outcome. A cause may be referred to variously as a condition, covariate, exogenous variable, explanatory variable, explanans, independent variable, input, intervention, parent, predictor, right-side variable, treatment, or simply “X.” An outcome may be referred to as a dependent variable, descendant, effect, endogenous variable, explanandum, left-side variable, output, response, or “Y.”
Whatever the terminology, to say that a factor, X, is a cause of an outcome, Y, is to say that a change in X generates a change in Y relative to what Y would otherwise be (the counterfactual condition), given certain background conditions (ceteris paribus assumptions) and scope-conditions (the population of the inference).
Now, let's unpack things a bit. As an example, we shall focus on the causal role of a worker-training program. A reasonable hypothesis is that participation in the program (X) will enhance an unemployed person's subsequent earnings (Y). If the relationship is causal, her earnings should be higher than they would be if she had never participated in the program. Let us represent the treatment, X, as a binary (dichotomous) variable, which takes one of two values.
Having surveyed the craft of writing, we turn to the craft of speaking, the oral form of language. Good speaking is somewhat different from good writing. Of course, one can simply read a prepared text, in which case the difference virtually disappears. But this does not generally qualify as good speaking in an academic or professional context – unless, that is, it is carefully honed to appear as if it is extemporaneous.
People expect a “live” performance. Your presence should be real, unmediated. Naturally, it may be mediated by various technologies and it may be pre-recorded; but it should feel as if it is happening right here and right now. This is the dynamic quality of public speaking. It is a special quality that is probably hard-wired in our brains and therefore carries a resonance that cannot be simulated with prose.
Public speaking is inextricably linked to comportment, i.e., how you carry yourself. Every time you make a public intervention you convey a vision of yourself. It is this persona that people tend to remember. Thus, when we say “speech” in this chapter we intend to include all the visual cues that accompany speech – dress, gaze, posture, gestures, and so forth.
In ancient times, speech was the preeminent art of persuasion. Rhetoric meant speech, and only secondarily prose or poetry. That prioritization is easy to understand in the context of a predominantly oral culture.
Nowadays, the craft of public speaking has fallen into desuetude (though some writing courses also include a component devoted to public speaking). People still talk, but speech is no longer cultivated as a professional activity, with a few exceptions such as moot courts in law school. This is unfortunate because speech is no less important today than it was a century ago. Perhaps it is destined to become more important over time as online lectures and YouTube videos replace written texts, and video calls and video conferences replace email. The spoken word may turn out to be mightier than the written word.
In any case, whether you are a good speaker or a poor speaker is likely to affect how successfully you can get your ideas across – not to mention getting good grades, landing a job, and succeeding in your chosen profession. To be sure, college courses generally don't allocate much credit for participation.
Email your librarian or administrator to recommend adding this to your organisation's collection.