Statistical Inference as Severe Testing

Mounting failures of replication in social and biological sciences give a new urgency to critically appraising proposed reforms. This book pulls back the cover on disagreements between experts charged with restoring integrity to science. It denies two pervasive views of the role of probability in inference: to assign degrees of belief, and to control error rates in a long run. If statistical consumers are unaware of assumptions behind rival evidence reforms, they can't scrutinize the consequences that affect them (in personalized medicine, psychology, etc.). The book sets sail with a simple tool: if little has been done to rule out flaws in inferring a claim, then it has not passed a severe test. Many methods advocated by data experts do not stand up to severe scrutiny and are in tension with successful strategies for blocking or accounting for cherry picking and selective reporting. Through a series of excursions and exhibits, the philosophy and history of inductive inference come alive. Philosophical tools are put to work to solve problems about science and pseudoscience, induction and falsification.

'Deborah G. Mayo argues forcefully for a frequentist position on statistical inference, and it is a pleasure to see how passionately she treats the various issues analyzed. Her writing style is highly engaging and conversational: in her frequent recourses to the first person, one can almost hear the dialogue between herself and the various people with whom she debates. The book is at the same time of highest scientific quality. It may qualify as one of the liveliest books on the philosophy of statistical inference.'

Gerd Gigerenzer - Max Planck Institute for Human Development

'Written as a series of tours and excursions, Deborah G. Mayo's lively book revisits the foundations of statistical inference from a simple and clear premise: only trust results that pass `severe tests'. Her ideas can be thought of as a modern, more complete version of Popper's notion of falsifiability. She goes beyond the usual Bayesian versus frequentist controversy and deals with pressing practical issues such as the crisis in scientific reproducibility. Whether you agree or disagree with her ideas, you will find the journey entertaining and thought provoking.'

Larry Wasserman - Carnegie Mellon University, Pennsylvania

'An extraordinary and enlightening grand tour through centuries of philosophical discourse underpinning modern statistics. Mayo's important contribution to this discourse, the severity principle, offers clarifying insight to several of the statistical conundrums all too often confounding even the brightest of modern data analysts and statistical theorists. I look forward to severity calculations eventually appearing alongside confidence intervals in statistical computer programs and journal discussions of findings.'

Steven McKinney - British Columbia Cancer Research Centre

'Whether or not you agree with her basic stance on statistical inference, if you are interested in the subject, and all scientists ought to be, Deborah G. Mayo’s writings are a must. Her views on inference are all the more valuable for being contrary to much current consensus. Her latest book will delight many and infuriate others but force all who are serious about these issues to think. Her capacity to jolt the complacent is second to none.'

Stephen Senn - author of Dicing with Death

'Deborah G. Mayo’s insights into the philosophical dimensions of these problems are unsurpassed in their originality, their importance, and the breadth of understanding on which they are based. Here she combines perspectives from philosophy of science and the foundations of statistics to eliminate mirages produced by misunderstandings both philosophical and statistical, while putting into focus the ways in which her error-statistical approach is relevant to current problems of scientific inquiry in various disciplines.'

Kent Staley - Saint Louis University, Missouri

'In this new book that reviews several competing paradigms for the philosophy of statistics in science, Deborah G. Mayo continues her project of untangling and delineating methods, models, assumptions, and goals, with an aim of moving toward pragmatic modes of inference that go beyond wishful thinking.'

Andrew Gelman - Columbia University, New York

'In this ground-breaking volume, Deborah G. Mayo cuts through the thicket of confusion surrounding debates on statistical inference, debunking the many widespread misconceptions about statistical tests and developing the theory of error statistics and severe testing. The book should be read by all practicing statisticians, and indeed by all scientists and others trying to extract meaning from data.'

David J. Hand - Imperial College, London

'This book is a detailed elaboration of the idea that statistical inferences are well-founded only if the possible ways in which they could be erroneous have been identified and responded to. The ramifications of this standpoint lead Mayo to solutions of long-standing problems in the philosophy of statistics, shows the way towards reforms in common but dubious statistical practices and helps the non-expert make informed judgements about such matters. This book is destined to become a definitive and frequently consulted resource.'

Alan Chalmers - University of Sydney

'The book by Deborah G. Mayo is a timely examination of the use of statistics in science. Her severity requirement demands that the scientist provide a sharp question and related data. Absent that, the observer should withhold judgement or outright reject. It is time to get tough. Funding agencies should take note.'

S. Stanley Young - FASA, FAAAS

'In this lively, witty, and intellectually engaging book, Deborah G. Mayo returns to first principles to make sense of statistics. She takes us beyond statistical formalism and recipes, and asks us to think philosophically about the enterprise of statistical inference itself. Her contribution will be a welcomed addition to statistical learning. Mayo’s timely book will shrink enlarged posteriors and overinflated significance, by focusing on whether our inferences have been severely tested, which is where we should be focused.'

Nathan A. Schachtman - Columbia Law