Skip to main content Accessibility help
×
Hostname: page-component-848d4c4894-4hhp2 Total loading time: 0 Render date: 2024-06-01T05:09:28.682Z Has data issue: false hasContentIssue false

32 - Making Effects Manifest in Randomized Experiments

Published online by Cambridge University Press:  05 June 2012

Jake Bowers
Affiliation:
University of Illinois at Urbana–Champaign
James N. Druckman
Affiliation:
Northwestern University, Illinois
Donald P. Greene
Affiliation:
Yale University, Connecticut
James H. Kuklinski
Affiliation:
University of Illinois, Urbana-Champaign
Arthur Lupia
Affiliation:
University of Michigan, Ann Arbor
Get access

Summary

Experimentalists want precise estimates of treatment effects and nearly always care about how treatment effects may differ across subgroups. After data collection, concern may turn to random imbalance between treatment groups on substantively important variables. Pursuit of these three goals – enhanced precision, understanding treatment effect heterogeneity, and imbalance adjustment – requires background information about experimental units. For example, one may group similar observations on the basis of such variables and then assign treatment within those blocks. Use of covariates after data have been collected raises extra concerns and requires special justification. For example, standard regression tables only approximate the statistical inference that experimentalists desire. The standard linear model may also mislead via extrapolation. After providing some general background about how covariates may, in principle, enable pursuit of precision and statistical adjustment, this chapter presents two alternative approaches to covariance adjustment: one using modern matching techniques and another using the linear model – both use randomization as the basis for statistical inference.

What Is a Manifest Effect?

A manifest effect is one we can distinguish from zero. Of course, we cannot talk formally about the effects of an experimental treatment as manifest without referring to probability: a scientist asks, “Could this result have occurred merely through chance?” or “If the true effect were zero, what is the chance that we'd observe an effect as large as this?”

Type
Chapter
Information
Publisher: Cambridge University Press
Print publication year: 2011

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Barnard, John, Frangakis, Constantino E., Hill, Jennifer L., and Rubin, Donald B.. 2003. “Principal Stratification Approach to Broken Randomized Experiments: A Case Study of School Choice Vouchers in New York City.” Journal of the American Statistical Association 98 (462): 299–324.CrossRefGoogle Scholar
Bowers, Jake, Fredrickson, Mark, and Hansen, Ben. 2009. RItools: Randomization Inference Tools. R package version 0.1-6 ed.
Bowers, Jake, Hansen, Ben B., and Fredrickson, Mark. 2008. Reproduction achive for: ‘Attributing Effects to a Cluster Randomized Get-Out-the-Vote Campaign’. Technical Report, University of Illinois and University of Michigan.
Bowers, Jake, and Panagopoulos, Costas. 2009. ‘“Probability of What?’: A Randomization-Based Method for Hypothesis Tests and Confidence Intervals about Treatment Effects.” Unpublished paper, University of Illinois, Urbana-Champaign. Retrieved January 24, 2011 from http://www.jakebowers.org/PAPERS/BowPanEgap2009.pdf.
Brady, Henry E. 2008. “Causation and Explanation in Social Science.” In Oxford Handbook of Political Methodology, eds. Box-Steffensmeier, Janet M., Brady, Henry E., and Collier, David. New York: Oxford University Press, chap. 10, 217–70.Google Scholar
Cochran, William G., and Cox, Gertrude M.. 1957. Experimental Designs. New York: Wiley.Google Scholar
Cox, David R., and McCullagh, Peter. 1982. “Some Aspects of Analysis of Covariance (with Discussion).” Biometrics 38: 541–61.CrossRefGoogle Scholar
Cox, David R., and Reid, Nancy. 2000. The Theory of the Design of Experiments. Boca Raton, FL: Chapman & Hall/CRC Press.Google Scholar
Cox, David R. 1958. The Planning of Experiments. New York: Wiley.Google Scholar
Fisher, Ronald A. 1925. Statistical Methods for Research Workers. Edinburgh: Oliver & Boyd.Google Scholar
Fisher, Ronald A. 1935. The Design of Experiments. Edinburgh: Oliver & Boyd.Google Scholar
Freedman, David A. 2008a. “On Regression Adjustments in Experiments with Several Treatments.” Annals of Applied Statistics 2: 176–96.CrossRefGoogle Scholar
Freedman, David A. 2008b. “On Regression Adjustments to Experimental Data.” Advances in Applied Mathematics 40: 180–93.CrossRefGoogle Scholar
Freedman, David A. 2008c. “Randomization Does Not Justify Logistic Regression.” Statistical Science 23: 237–49.CrossRefGoogle Scholar
Gadbury, Gary L. 2001. “Randomization Inference and Bias of Standard Errors.” American Statistician 55: 310–13.CrossRefGoogle Scholar
Gelman, Andrew, and Hill, Jennifer. 2007. Data Analysis Using Regression and Multi-Level/Hierarchical Models. New York: Cambridge University Press.Google Scholar
Green, Donald P. 2009. “Regression Adjustments to Experimental Data: Do David Freedman's Concerns Apply to Political Science?” Unpublished paper, Yale University.
Hansen, Ben, and Fredrickson, Mark. 2010. Optmatch: Functions for Optimal Matching, Including Full Matching. R package version 0.6-1 ed.
Hansen, Ben B. 2004. “Full Matching in an Observational Study of Coaching for the SAT.” Journal of the American Statistical Association 99 (467): 609–18.CrossRefGoogle Scholar
Hansen, Ben B. 2008. “Comment: The Essential Role of Balance Tests in Propensity-Matched Observational Studies.” Statistics in Medicine 27: 2050–4.CrossRefGoogle ScholarPubMed
Hansen, Ben B., and Bowers, Jake. 2008. “Covariate Balance in Simple, Stratified and Clustered Comparative Studies.” Statistical Science 23: 219–36.CrossRefGoogle Scholar
Hansen, Ben B., and Bowers, Jake. 2009. “Attributing Effects to a Cluster Randomized Get-Out-the-Vote Campaign.” Journal of the American Statistical Association 104: 873–85.CrossRefGoogle Scholar
Horiuchi, Yusaku, Imai, Kosuke, and Taniguchi, Naoko. 2007. “Designing and Analyzing Randomized Experiments: Application to a Japanese Election Survey Experiment.” American Journal of Political Science 51: 669–87.CrossRefGoogle Scholar
Imai, Kosuke, King, Gary, and Nall, Clayton. 2009. “The Essential Role of Pair Matching in Cluster-Randomized Experiments, with Application to the Mexican Universal Health Insurance Evaluation.” Statistical Science 24: 29–72.CrossRefGoogle Scholar
Imai, Kosuke, King, Gary, and Stuart, Elizabeth. 2008. “Misunderstandings among Experimentalists and Observationalists about Causal Inference.” Journal of the Royal Statistical Society, Series A 111: 1–22.Google Scholar
Keele, Luke J., McConnaughy, Corrine, and White, Ismail K.. 2008. “Adjusting Experimental Data.” Unpublished Manuscript, The Ohio State University.
Leisch, Friedrich. 2002. “Dynamic Generation of Statistical Reports Using Literate Data Analysis.” In Compstat 2002 – Proceedings in Computational Statistics, eds. Haerdle, Wolfgang and Roenz, Bernd. Heidelberg, Germany: Physika Verlag.Google Scholar
Leisch, Friedrich. 2005. Sweave User Manual. Retrieved from http://www.stat.uni-muenchen.de/~leisch/Sweave/Sweave-manual.pdf.
Neyman, Jerzy. 1990. “On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9 (1923).” Statistical Science 5: 463–80.Google Scholar
Nickerson, David W. 2005. “Scalable Protocols Offer Efficient Design for Field Experiments.” Political Analysis 13: 233–52.CrossRefGoogle Scholar
Panagopoulos, Costas. 2006. “The Impact of Newspaper Advertising on Voter Turnout: Evidence from a Field Experiment.” Unpublished Manuscript, Fordhan University.
Robins, James M. 2002. “Covariance Adjustment in Randomized Experiments and Observational Studies: Comment.” Statistical Science 17: 309–21.Google Scholar
Rosenbaum, Paul R. 2002a. “Covariance Adjustment in Randomized Experiments and Observational Studies.” Statistical Science 17: 286–327.Google Scholar
Rosenbaum, Paul R. 2002b. Observational Studies. 2nd ed. New York: Springer-Verlag.CrossRefGoogle Scholar
Rosenbaum, Paul R. 2002c. “Rejoinder.” Statistical Science 17: 321–27.Google Scholar
Rosenbaum, Paul R. 2010. Design of Observational Studies. New York: Springer.CrossRefGoogle ScholarPubMed
Rubin, Donald B. 1974. “Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies.” Journal of Educational Psychology 66: 688–701.CrossRefGoogle Scholar
Rubin, Donald B. 1990. “Comment: Neyman (1923) and Causal Inference in Experiments and Observational Studies.” Statistical Science 5: 472–80.CrossRefGoogle Scholar
Schochet, Peter. 2009. “Is Regression Adjustment Supported by the Neyman Model for Causal Inference?Journal of Statistical Planning and Inference 140: 246–59.CrossRefGoogle Scholar
Sekhon, Jasjeet S. 2008. “The Neyman-Rubin Model of Causal Inference and Estimation via Matching Methods.” In Oxford Handbook of Political Methodology, eds. Box-Steffensmeier, Janet M., Brady, Henry E., and Collier, David. New York: Oxford University Press, chap. 11, 271–99.Google Scholar
Senn, Stephen J. 1994. “Testing for Baseline Balance in Clinical Trials.” Statistics in Medicine 13: 1715–26.CrossRefGoogle ScholarPubMed

Save book to Kindle

To save this book to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×