Skip to main content

Avoiding Post-Treatment Bias in Audit Experiments

  • Alexander Coppock (a1)

Audit experiments are used to measure discrimination in a large number of domains (Employment: Bertrand et al. (2004); Legislator responsiveness: Butler et al. (2011); Housing: Fang et al. (2018)). Audit studies all have in common that they estimate the average difference in response rates depending on randomly varied characteristics (such as the race or gender) of a requester. Scholars conducting audit experiments often seek to extend their analyses beyond the effect on response to the effects on the quality of the response. Response is a consequence of treatment; answering these important questions well is complicated by post-treatment bias (Montgomery et al., 2018). In this note, I consider a common form of post-treatment bias that occurs in audit experiments.

Hide All
Aronow, Peter M., Baron, Jonathon and Pinson, Lauren 2018. “A Note on Dropping Experimental Subjects who Fail a Manipulation Check.” Political Analysis. In press.
Bendick, Marc, Jackson, Charles W. and Reinoso, Victor A. 1994. “Measuring Employment Discrimination through Controlled Experiments.” The Review of Black Political Economy 23 (1): 2548.
Bertrand, Marianne and Mullainathan, Sendhil 2004. “Are Emily and Greg more employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination.” The American Economic Review 94 (4): 9911013.
Björkman, Martina and Svensson, Jakob 2009. “Power to the People: Evidence from a Randomized Field Experiment of a Community-Based Monitoring Project in Uganda.” Quarterly Journal of Economics 124 (2): 735–69.
Butler, Daniel M. and Broockman, David E. 2011. “Do Politicians Racially Discriminate Against Constituents? A Field Experiment on State Legislators.” American Journal of Political Science 55 (3): 463–77.
Coppock, Alexander 2018. Replication Data for: Avoiding Post-Treatment Bias in Audit Experiments. Harvard Dataverse, v. 4.8.4. doi:10.7910/DVN/6NVI9C.
Costa, Mia 2017. “How Responsive are Political Elites? A Meta-Analysis of Experiments on Public Officials.” Journal of Experimental Political Science 4 (3): 241–54.
Fang, Albert H., Guess, Andrew M. and Humphreys, Macartan 2018. “Can the Government Deter Discrimination? Evidence from a Randomized Intervention in New York City.” Journal of Politics. In press.
Kalla, Joshua, Rosenbluth, Frances and Teele, Dawn Langan 2018. “Are You My Mentor? A Field Experiment on Gender, Ethnicity, and Political Self-Starters.” The Journal of Politics 80 (1): 337–41.
Montgomery, Jacob M., Nyhan, Brendan and Torres, Michelle 2018. “How Conditioning on Post-treatment Variables Can Ruin Your Experiment and What to Do About It.” American Journal of Political Science. In press.
Quillian, Lincoln, Pager, Devah, Hexel, Ole and Midtbøen, Arnfinn H. 2017. “Meta-analysis of Field Experiments Shows No Change in Racial Discrimination in Hiring Over Time.” Proceedings of the National Academy of Sciences 114 (41): 10870–5.
Rondeau, Daniel and List, John A. 2008. “Matching and Challenge Gifts to Charity: Evidence from Laboratory and Natural Field Experiments.” Experimental Economics 11 (3): 253–67.
White, Ariel R., Nathan, Noah L. and Faller, Julie K. 2015. “What Do I Need to Vote? Bureaucratic Discretion and Discrimination by Local Election Officials.” American Political Science Review 109 (1): 129–42.
Zhang, Junni L. and Rubin, Donald B. 2003. “Estimation of Causal Effects via Principal Stratification When Some Outcomes are Truncated by “Death”.” Journal of Educational and Behavioral Statistics 28 (4): 353–68.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Journal of Experimental Political Science
  • ISSN: 2052-2630
  • EISSN: 2052-2649
  • URL: /core/journals/journal-of-experimental-political-science
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
Type Description Title
Supplementary materials

Coppock Dataset



Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed