Book contents
- Frontmatter
- Contents
- Prologue
- Acknowledgments
- 1 Empirical Bayes and the James—Stein Estimator
- 2 Large-Scale Hypothesis Testing
- 3 Significance Testing Algorithms
- 4 False Discovery Rate Control
- 5 Local False Discovery Rates
- 6 Theoretical, Permutation, and Empirical Null Distributions
- 7 Estimation Accuracy
- 8 Correlation Questions
- 9 Sets of Cases (Enrichment)
- 10 Combination, Relevance, and Comparability
- 11 Prediction and Effect Size Estimation
- Appendix A Exponential Families
- Appendix B Data Sets and Programs
- References
- Index
6 - Theoretical, Permutation, and Empirical Null Distributions
Published online by Cambridge University Press: 05 September 2013
- Frontmatter
- Contents
- Prologue
- Acknowledgments
- 1 Empirical Bayes and the James—Stein Estimator
- 2 Large-Scale Hypothesis Testing
- 3 Significance Testing Algorithms
- 4 False Discovery Rate Control
- 5 Local False Discovery Rates
- 6 Theoretical, Permutation, and Empirical Null Distributions
- 7 Estimation Accuracy
- 8 Correlation Questions
- 9 Sets of Cases (Enrichment)
- 10 Combination, Relevance, and Comparability
- 11 Prediction and Effect Size Estimation
- Appendix A Exponential Families
- Appendix B Data Sets and Programs
- References
- Index
Summary
In classical significance testing, the null distribution plays the role of devil's advocate: a standard that the observed data must exceed in order to convince the scientific world that something interesting has occurred. We observe, say, z = 2, and note that in a hypothetical “long run” of observations from a N(0, 1) distribution less than 2.5% of the draws would exceed 2, thereby discrediting the uninteresting null distribution as an explanation.
Considerable effort has been expended trying to maintain the classical model in large-scale testing situations, as seen in Chapter 3, but there are important differences that affect the role of the null distribution when the number of cases N is large:
• With N = 10 000 for example, the statistician has his or her own “long run” in hand. This diminishes the importance of theoretical null calculations based on mathematical models. In particular, it may become clear that the classical null distribution appropriate for a single-test application is in fact wrong for the current situation.
• Scientific applications of single-test theory most often suppose, or hope for, rejection of the null hypothesis, perhaps with power = 0.80. Largescale studies are usually carried out with the expectation that most of the N cases will accept the null hypothesis, leaving only a small number of interesting prospects for more intensive investigation.
• Sharp null hypotheses, such as H0 : μ = 0 for z ˜ N(μ, 1), are less important in large-scale studies. […]
- Type
- Chapter
- Information
- Large-Scale InferenceEmpirical Bayes Methods for Estimation, Testing, and Prediction, pp. 89 - 112Publisher: Cambridge University PressPrint publication year: 2010