In this post Dean Knox, Will Lowe, and Jonathan Mummolo expand upon their APSR article ‘Administrative Records Mask Racially Biased Policing‘ which was recently published online.

Police killings of unarmed Black Americans have reignited debates over racial bias in the use of force. Research on policing plays a critical role in this debate, including by quantifying the severity of this problem. Careful estimation of racial bias in police violence is not merely academic: it is crucial information for crafting effective reforms of policing policy. The roughly 18,000 police departments in the U.S. may vary substantially in how, and to what extent, they exhibit racial bias. Only by accurately measuring discrimination can reformers focus their limited time and resources where the problem is most severe, as well as credibly evaluate the efficacy of reforms aimed at reducing racial inequity in policing.

In an article recently published online in the American Political Science Review, we develop an improved method for estimating racial bias in police behavior using administrative records, and in applying our technique uncover evidence of severe and previously hidden levels of discriminatory police violence in New York City.

The quantitative study of policing has historically faced substantial data challenges. Though the recent availability of several large administrative datasets on police behavior has spawned a raft of papers, some of these — quite surprisingly — find little to no evidence of racial bias in police-civilian interactions. Most of the studies in this literature go to great lengths to combat the familiar problem of omitted variable bias — the problem that arises when police encounters with white and nonwhite civilians differ on some unmeasured dimension that also causes the use of force. Omitted variable bias makes it difficult to interpret disparities in police behavior as evidence of racial bias. For example, if white civilians are more willing to attack police officers, and this feature was not accounted for, then it would be deeply misleading to interpret equal rates of violence against white and nonwhite civilians as evidence of no racial bias. Fortunately, many newly available data sets on police behavior contain numerous details that help neutralize this source of statistical bias, including the encounter time and location, officers’ reasons for engaging civilians, and descriptions of the appearance and behavior of civilians and officers.

However, even if the vexing problem of omitted variable bias is adequately addressed, a more fundamental and often overlooked source of statistical bias remains: sample selection bias in the encounters which appear in police administrative data. Consider the problem of estimating bias in police violence in pedestrian encounters using records from the New York Police Department’s (NYPD) “Stop, Question and Frisk” program, which was deployed on a massive scale in the 2000s. During a typical shift, NYPD officers may have encountered hundreds or even thousands of civilians, but most of these events left no administrative trace. For example, officers were not required to record data on civilians whom they observed but allowed to pass without further intervention. But if police are racially biased in whom they choose to stop — which directly determines whether an encounter appears in police administrative data — then typical analyses using police records to study racial bias in post-stop outcomes, like the use of force, will yield heavily distorted estimates.

To address this problem, our paper (1) formalizes this form of post-treatment bias in the study of police-civilian interactions, (2) shows this statistical bias can be bounded under minimal assumptions, and (3) applies these results to demonstrate that prominent research in this area substantially underestimates the degree of anti-minority bias in police use of force by the NYPD.

A simple thought experiment provides intuition for the problem of sample selection in this domain. Imagine analysts observe perfect as-if experimental conditions: a set of police-civilian encounters (i.e., sightings of civilians by police) that are identical in all respects but for the race of civilians. In other words, imagine the problem of omitted variable bias was somehow solved. Further suppose that police exhibit racial bias in the decision to stop civilians, such that white civilians are only stopped if caught committing serious violent crimes, but black civilians are stopped regardless of their behavior. Now suppose that rather than analyzing this full dataset — containing all civilians observed by police — we instead throw away all the data on non-stopped people. We would then be left with a set of incomparable encounters: those involving white civilians committing violent crimes, and black civilians, many of whom were stopped for no valid reason and posed no threat to officers.

Subsequent naive comparisons of these encounters would not yield valid causal estimates of racial bias. In fact, if officers used force at the same rates against stopped minority and white civilians, naive analyses could falsely indicate that no bias exists at all, even though much police violence in this scenario would be against unjustly detained minorities. Thus, by subsetting to encounters involving stops, we have ruined this otherwise perfect experiment. As we show in our paper, the sources of confounding produced by this type of post-treatment conditioning are nearly impossible to correct by merely controlling for observable features of the encounters.

Given that available police administrative data omits non-stop encounters, creating precisely this problem, what can be done? To address this, we formalize the problem of sample selection in police administrative data as one of causal mediation. That is, we outline a framework in which civilian race affects both the decision of officers to stop a civilian, as well as the subsequent decision to use force. We then derive the precise statistical bias in naive estimates (i.e., ignoring selection issues) of the average causal effect of civilian race on the use of force: the proportion of encounters in which force was used against minority civilians but would not have been used against white civilians, all else equal. We show that under minimal assumptions, this bias can be bounded. By doing so, we devise a technique for identifying the range of possible racial-bias estimates that are consistent with the observed data, after accounting for bias in stopping, and using only data on police stops. Importantly, our technique is nonparametric, meaning we make no assumptions about the form of the process that governs police behavior (e.g., we do not assume this process can be well-represented by simplistic linear regression models). These bounds are also sharp, guaranteeing that nothing more can be said about this estimand without additional information or assumptions. This feature makes our approach highly general, and it leaves analysts the option to impose additional structure when justified.

After deriving these results, we then apply them by replicating and extending a recent prominent study, Fryer (2019), which naively examined several police data sets ignoring the issue of sample selection. The study concluded that there was some racial bias in the use of sub-lethal force, but no evidence of bias in the use of lethal force. Using millions of “Stop, Question and Frisk’’ records collected by the New York Police Department (NYPD), we reexamine Fryer’s (2019) analysis of non-lethal force. Although Fryer’s analysis adjusted for myriad potential confounding factors using the rich set of variables contained in these data, we show that by failing to account for racial bias in stopping, Fryer (2019) substantially underestimates the extent of racially discriminative force by the NYPD.

For example, using the approach in Fryer (2019) — a logistic regression of indicators for the use of force on indicators for civilian race, plus a host of controls for features of observed encounters — we would conclude there were roughly 75,000 instances in which NYPD officers laid hands on black and Hispanic civilians between 2003 and 2013, but would not have done so had those civilians been white. Our bias-corrected estimate shows the true number is approximately 307,000, meaning the naive approach masks 232,000 such incidents. Similarly, the naive approach indicates roughly 3,400 racially discriminatory instances in which officers pointed a weapon at a black or Hispanic civilian, whereas the bias-corrected estimate shows the true number is almost five times as large. In some cases, the naive approach masks discrimination entirely by producing the illusion of no race effect. For example, while the approach in Fryer (2019) suggests that officers discriminatorily handcuffed Hispanic civilians at a statistically insignificant rate of 0.4 instances per 1,000 encounters, our corrected bounds show the true number is at least 13 — an estimate that is both statistically significant and at least 32 times larger.

While we are hopeful that this improved technique offers a way for analysts to better study racial discrimination using available policing data, we believe the best estimates will only be obtained by the collection of data that police agencies currently do not maintain: namely, records that allow us to characterize all encounters, not just those in which police stop and engage civilians. Such data collection may be more readily feasible for the study of traffic stops, where highway cameras in many jurisdictions already passively collect data on every passing car. (However, if such widespread data collection raises ethical concerns in a particular setting, we note that our statistical correction can always be employed with existing data.) Researchers should also endeavor to identify natural experiments in which police come into contact with civilians in a “race-blind” manner, such as random stops of cars during DUI checkpoints (though this approach will necessarily limit the scope of police behaviors that can be studied).

Regardless of which approach scholars pursue, this article highlights the need for further careful research into the initial, unseen stage of police-civilian interactions — that is, the process by which officers decide whether or not to stop and investigate an individual for a crime. Without serious consideration of the role of race in each stage of complex police-civilian interactions, the benefits of data-driven reforms will be stunted, as will our collective understanding of the politics of policing.

– Dean Knox, Princeton University; Will Lowe, Hertie School of Governance; and Jonathan Mummolo, Princeton University

– The authors’ APSR article is currently free access