Experimental design

We began with a study in New Jersey involving 462 agencies, paired with newly released detailed data on the use of force (nj.com 2019). During April and May of 2019, we contacted police chiefs offering to collaborate on research that “aims to make both citizens and officers safer by reducing the frequency of violence during police-citizen interactions” (see Online Appendix section B2 for full text). We relied on a custom Python script to prepare and send our messages from a dedicated institutional email. These messages contained no deception; offers to discuss collaborations were sincere. We offered to work pro-bono and cover all research costs and added “We are not asking for a firm commitment now” but are simply asking whether the recipient is “interested in discussing a potential collaboration further.” Respondents could answer (via links in email and a URL provided in print letters) yes, no, or “I am not sure, but I would like more information.” Our primary outcome is a binary indicator of answering “yes,” with all other responses and non-responses coded as negative responses. If we received no response after three email attempts – spaced eight days apart – we sent a posted letter one week after the final email.

Agencies in the N.J. study were randomly assigned to one of four conditions. ^{Footnote 1} All agencies received the information above, which served as the full text for those in the control condition. Three treatment conditions included language aimed at testing how common features of research collaboration requests affect agency responses. One treatment cell included a promise of confidentiality in any publication that resulted from a research partnership, which is a common practice in such settings and which we hypothesized would increase affirmative responses. A second “ranking” condition included mention of agency performance: the agency’s rank on uses of force per officer among contacted agencies. A third condition combined both the confidentiality and ranking treatments (with the order of the two treatments randomized within the text of the email across recipients).

Following the N.J. study, we deployed a second pre-registered experiment in 47 additional states during September and October of 2019, in which we attempted to contact approximately 2,500 local police and sheriff’s departments, a sample size we chose based on a power analysis in order to detect a possible interaction between the performance treatment and agency rank. We randomly drew our sample from a population of roughly 7,700 agencies that consistently report crime data to the FBI and for whom we could ascertain reliable contact information (these criteria excluded Alaska and Illinois; see Appendix section A1 for sampling details). Roughly 60% of the US population resides in these agencies’ jurisdictions, according to FBI data. While we would ideally wish to sample from the entire USA, the population of agencies that remain after applying these filtering criteria are those with whom a productive research collaboration might plausibly occur. It would be difficult to form collaborations with agencies that do not regularly report basic crime data or publicize reliable contact information. Our sample is therefore a relevant one for applied researchers.

The design of this experiment was highly similar to the N.J. study with some exceptions. Two changes were aimed at maximizing statistical power. First, we retained only the ranking treatment and control conditions. Second, we employed a matched pair design (Gerber and Green Reference Gerber and Green2012), in which agencies in the same state serving roughly the same population size were paired, and one agency was randomly assigned to treatment. Specifically, treated agencies were told how they ranked among the roughly 2,500 agencies sampled on the share of violent crimes “cleared” between 2013 and 2017 (crimes where an arrest was made and charge was filed) – a salient statistic for police agencies, and one on which journalists often focus (e.g. Madhani Reference Madhani2018). The use of two different performance metrics across these experiments helps to ensure the robustness of any observed treatment effects.

We hypothesized that mentions of agency performance would filter out “cheap talk” and depress affirmative responses on average, since making performance evaluations salient could cause agencies to consider the political risks associated with research partnerships. However, we anticipated that this negative effect would attenuate with agency rank, since agencies informed they were performing well relative to peers may be less likely to recoil at the specter of performance evaluations. Following both experiments, all contacted agencies were sent a debrief message informing them of the purpose of the experiment and reinforcing that our messages contained no deception. ^{Footnote 2}

Combined, contacted agencies serve jurisdictions that are home to close to 80 million people according to FBI data (see Figure 1), approximately one-quarter of the US population. These include large metropolitan police forces, mid-sized agencies, and small rural departments employing just a handful of officers. ^{Footnote 3}

Figure 1.

Coverage of Field Experiments. Number of contacted agencies in each US state. Over 400 agencies were contacted in the N.J. study.

Little evidence collaborating agencies are unrepresentative

To test whether willingness to collaborate systematically varies with agency attributes, we merged agency-level data on crime, fatal officer-involved shootings between 2015 and 2018, personnel, and jurisdiction demographics. ^{Footnote 4} In total, 319 agencies indicated willingness to discuss a potential research collaboration – approximately 11% of our combined sample of 2,944 agencies across the two experiments – 238 agencies responded negatively to our message, and 2,387 agencies did not reply at all.

We estimated separate bivariate linear regressions predicting affirmative response as a function of each covariate, weighted by jurisdiction population. ^{Footnote 5} We correct resulting p-values on all regression coefficients using the Benjamini-Hochberg method (Benjamini and Hochberg Reference Benjamini and Hochberg1995), though this adjustment makes little difference to our overall conclusions. To avoid conflating the predictive value of a regressor with the effect of our randomized interventions, we confine this analysis to the roughly 1,400 observations assigned to control, of which 201 agencies responded affirmatively. ^{Footnote 6}

Figure 2 displays the predicted change in the probability of an affirmative response when an agency is above (relative to below) the median on each trait. Across the 44 test results in this figure, only one covariate, population of the jurisdiction, was significantly associated with responses in either the control or treatment groups. Specifically, moving from below to above the median jurisdiction population in our sample is associated with a 14.84 percentage point decrease in the probability of an affirmative response (BH-corrected p-value = 0.01). However, none of the numerous measures of performance significantly predict responses. Overall, our analysis indicates selection bias – at least at the initial point of contact from researchers – poses a minimal risk in this setting.

Figure 2.

Agency and jurisdiction attributes do not predict affirmative responses.

Some may question whether we are missing meaningful associations in this analysis due to a lack of statistical power. But while additional data may allow us to detect correlations, we note that several features related to agency performance generate opposing signs, for example assaults on officers and a host of crime measures including murders and rapes per capita. While larger jurisdictions are less likely to show interest in research collaborations – an important limitation that could stymie partnerships in major departments – the overall pattern of results does not indicate selection related to agency performance, suggesting agencies arguably in most need of reform are not systematically declining to collaborate.

Mentions of performance evaluations inhibit collaborations

We now turn to assess the impact of our experimental interventions. Figure 3 displays the average effect of each treatment relative to the control condition estimated via linear regression. Because we cannot guarantee all messages were reviewed, these represent Intention-to-Treat effects (ITTs), understating the effect of universally received similar messages. ^{Footnote 7} In the national experiment, our models include indicators for all matched pairs, with standard errors clustered by matched pair.

Figure 3.

Mentioning agency performance lowers affirmative responses.

Turning first to the N.J. experiment, we find randomized offers to keep the identity of collaborating agencies confidential, including one version where a performance cue was also supplied, had no detectable effect on response rates ( $\beta = - 0.02,\,\,SE = 0.04$ and $\beta = - 0.04,\,\,SE = 0.04$ respectively). This was surprising, as such confidentiality offers are often made to convey a sense of security and thereby increase the likelihood of collaborations. However, because such offers still rely on academic collaborators to keep their word and effectively safeguard agency identities, this promise may ring hollow, and additional assurances may be required before agencies will consider collaborations. However, recipients told their statewide rank on mean uses of force per officer (“Ranking Condition”) were roughly 9 percentage points ( $SE = 0.04$ ) less likely to respond affirmatively than agencies assigned to control where about 13% of agencies agreed. Strikingly, this effect was precisely replicated in the nationwide experiment: agencies told their rank on violent crime clearance rates were about 8 percentage points ( $SE = 0.01$ ) less likely to say they would discuss a potential collaboration.

Contrary to our expectations, additional tests interacting treatment assignment with agency rank show that these negative effects persist even among top-performing agencies (see Online Appendix Figure G1). This result is consistent with police agencies having a strong aversion to outside evaluation and suggests a powerful impediment to the formation of research partnerships. While many agencies indicate openness to collaboration, a large share recoils once the topic of agency performance is inevitably broached. This may be because agencies that performed well on a given metric in the past have no guarantee of positive results in the future, especially once outside scrutiny is allowed.

Of course, other mechanisms are also possible, and we have a limited ability to adjudicate between them with the data at hand. Prior work on collaborations with law enforcement agencies emphasizes the importance of researchers spending time with agency officials to educate themselves about the particulars of the institution and personnel (Glaser and Charbonneau Reference Glaser and Charbonneau2018; Levine Reference Levine2020). Conveying a measure of performance in an initial outreach message may have inadvertently sent the signal that the researchers had “jumped the gun” and evaluated the agency before doing proper due diligence. The specific metrics we conveyed in our interventions may also have depressed responses. Agencies may have felt other measures more accurately conveyed their level of performance and thus inferred the researchers contacting them were ill-equipped to assist them. ^{Footnote 8}

However, regardless of the mechanism at play, the end result is the same: mentions of performance evaluations in outreach messages inhibit collaborations. Since evaluating the efficacy of reforms on agency performance is a central goal of these collaborations, we interpret these effects as evidence of a substantial barrier to policy experimentation, but one that may be overcome with a more measured approach to solicitation that works to establish trust with police administrators before discussing performance metrics.