In this chapter we cover alternative approaches to multiverse analysis, such as specification curve analysis, sample splitting, and model selection.

Are female hurricanes more deadly? In this chapter we demonstrate multiverse analysis using analytical inputs from many scholars in a high-profile empirical debate. In results from more than 10,000 model specifications, only 12 percent of estimates are statistically significant and 99 percent are smaller in magnitude than what the original authors reported. Multiverse analysis shows that some published findings are extremely weak and nonrobust.

Control variable strategies can go wrong and controls can make estimates worse rather than better. In this chapter, we discuss why control variables always deserve skepticism and require specific justification. We discuss criteria for plausible controls, bad controls, and why one should not control for everything we can measure (aka kitchen sink models). There are common conditions in which controlling for a variable causes bias, and therefore when excluding that control variable reduces bias.

Raw data require a great deal of cleaning, coding, and categorizing of observations. Vague standards for this data work can make it troublingly ad hoc, with much opportunity and temptation to influence the final results. Preprocessing rules and assumptions are not often seen as part of the model, but they can influence the result just as much as control variables or functional form assumptions. In this chapter, we discuss the main data processing decisions that analysts often face and how they can affect the results: coding and classifying of variables, processing anomalous and outlier observations, and the use of sample weights.

Having developed the multiverse framework in detail, let’s take it for empirical road tests. Are banks biased in mortgage lending? Do job training programs lead to higher wages? How much do the answers to these questions depend on modeling assumptions? Sometimes “significant” results are very stable and robust across models, while other results are mostly null, supported in one in a hundred credible models. In this chapter we demonstrate how to conduct and interpret a basic multiverse analysis, and we cover basic multiverse commands in Stata.

A dataset does not speak for itself, and model assumptions can drive results just as much as the data. Limited transparency about model assumptions creates a problem of asymmetric information between analyst and reader. This chapter shows how we need better methods for robust results.

Multiverse analysis is not simply a computational method but also a philosophy of science. In this chapter we explore its core tenets and historical foundations. We discuss the foundational principle of transparency in the history of science and argue that multiverse analysis brings social science back into alignment with this core founding ideal. We make connections between this framework and multiverse concepts developed in cosmology and quantum physics.

Why do different models give different results? Which modeling assumptions matter most? These are questions of model influence. Standard regression results fail to address simple questions like, which control variables are important for getting this result? In this chapter we lay out a framework for thinking about influence and draw on empirical examples to illustrate. When a result is not fully robust, the influence analysis provides methodological explanations for the failure of robustness. These explanations can be considered methodological scope conditions – they explain why a hypothesis can be supported in some cases but not in others. We also show how multiverse results can help inform the method of sensitivity analysis

This chapter advocates a simple principle: Good analysis should be easier to publish than bad analysis. Multiverse methods promote transparency over asymmetric information and emphasize robustness, countering the fragility inherent in single-path analysis. In an era when the credibility of scientific results is often challenged, the use of multiverse analysis is crucial for bolstering both the credibility and persuasiveness of research findings.

Are some models better than others? Yes. But can we weight models by the probability that they are true? That is harder than it sounds. In this chapter we cover various methods for weighting the models in a multiverse and assess their strengths and weaknesses using a dataset on how air pollution near schools can affect student learning. Weighting models creates a tension between model selection and model robustness, and authors must be clear about how model weights change the distribution of results. We recommend uniform weights as a transparent default, and if further weighting is desired, either double lasso or influence weighting appears best for inference.