This chapter covers two-sample studies when the responses are either ordinal or numeric. We present different estimands and their associated tests and confidence intervals. For differences in means, we describe several approaches. Permutation tests and associated confidence intervals need a location shift assumption. We compare several t-tests (Student’s t-test, Welch’s t-test, and the Behrens–Fisher test) focusing on robustness and relaxing normality assumptions. For differences or ratios of medians, we recommend the melding approach because it does not require large samples nor a location shift assumption. Several confidence interval methods related to the Mann–Whitney parameter are discussed, including one compatible with the Wilcoxon–Mann–Whitney test requiring proportional odds assumptions, and the Brunner–Munzel one that is less restrictive. Causal interpretation of the Mann–Whitney parameter is compared with a related parameter, the proportion that benefit on treatment parameter. When power is the driving concern, we compare several tests, and show that the Wilcoxon–Mann–Whitney test is often more powerful than a t-test, even when the data appear approximately normal.

This short chapter formally defines hypothesis tests in terms of decision rules paired with assumptions. It defines when one set of assumptions is more restrictive than another set. It further defines a multiple perspective decision rule, where one decision rule (and hence its p-value function) can be applied under different sets of assumptions, called perspectives.For example, the one sample t-test p-value may be interpreted as testing a mean restricted to the class of normal distributions, but that same p-value is asymptotically valid under an expanded class of distributions that does not require the normality assumption. This multiple perspective decision rule formulation may allow an appropriate interpretation the t-test p-value even when the data are clearly not normally distributed. Another example is presented, giving the Wilcoxon–Mann–Whitney decision rule under two different perspectives.

This chapter provides a brief review of several important ideas in causality. We define potential outcomes, a pair of outcomes for each individual denoting their response if they had gotten treatment and their response if they had gotten control. Typically, we only observe one of the potential outcomes. We define some causal estimands, such as the average causal difference, vaccine efficacy, and the Mann–Whitney parameter. We discuss estimation of the average causal difference from a matched experiment and a randomized study. Using a hypothetical vaccine study, we discuss why causal inference requires more care and assumptions for observational studies than for experiments. We work through a study to estimate the average causal effect on compliers from a randomized study with imperfect compliance. We define principled adjustments for randomized studies. We discuss interference in causality. We review causal analysis with propensity scores for observational studies. We define directed acyclic graphs (DAGs) and show how they can be used to define the backdoor criterion and confounders. Finally, we discuss instrumental variables analysis.

The chapter focuses on inferences on the mean of n independent binary responses. For most applied problems the exact one-sided p-values and the exact central confidence interval (also called the Clopper–Pearson interval) are appropriate. Less common exact confidence intervals (by Sterne or Blaker) may have smaller width at the cost of giving up the centrality (equal error bounds on both sides of the interval). We also discuss mid-p tests and confidence intervals that do not have guaranteed coverage, but instead have coverage that is approximately the nominal level “on average.” Asymptotic methods are briefly described. We discuss three different ways of determining sample size for a one sample study with binary responses: pick the sample size that (1) gives appropriate power to reject the null hypothesis for a particular alternative, (2) gives appropriate power to observe at least one event, or (3) bounds the expected 95% confidence interval width.