Commentaries on the target article offer diverse perspectives on integrative experiment design. Our responses engage three themes: (1) Disputes of our characterization of the problem, (2) skepticism toward our proposed solution, and (3) endorsement of the solution, with accompanying discussions of its implementation in existing work and its potential for other domains. Collectively, the commentaries enhance our confidence in the promise and viability of integrative experiment design, while highlighting important considerations about how it is used.

Sipping, an early form of alcohol initiation, is associated with aspects of psychopathology and personality that reflect long-term risk for harmful alcohol use. In the Adolescent Brain and Cognitive Development cohort (N = 11,872), sipping by age 9–10 was concurrently associated with impulsivity, other aspects of externalizing, and prodromal schizophrenia symptoms. Still, these associations were cross-sectional in nature, leaving open the possibility that these features of psychopathology and personality might not reflect long-term risk for alcohol consumption and related harm across development. Here, we attempted to replicate baseline concurrent associations across three waves of data to extend concurrent associations to prospective ones. Most cross-sectional associations replicated across waves, such that impulsivity, other aspects of externalizing, reward sensitivity (e.g., surgency, sensation seeking), and prodromal schizophrenia symptoms were associated with increased odds of having sipped alcohol by the age of 12. Nevertheless, not all concurrent associations replicated prospectively; impulsigenic features did not reflect long-term risk for sipping. Thus, some psychopathology features appeared to reflect stable risk factors, whereas others appeared to reflect state-dependent risk factors. All told, sipping might not reflect long-term risk for harmful alcohol use, and the nature of sipping may change across development.

The dominant paradigm of experiments in the social and behavioral sciences views an experiment as a test of a theory, where the theory is assumed to generalize beyond the experiment's specific conditions. According to this view, which Alan Newell once characterized as “playing twenty questions with nature,” theory is advanced one experiment at a time, and the integration of disparate findings is assumed to happen via the scientific publishing process. In this article, we argue that the process of integration is at best inefficient, and at worst it does not, in fact, occur. We further show that the challenge of integration cannot be adequately addressed by recently proposed reforms that focus on the reliability and replicability of individual findings, nor simply by conducting more or larger experiments. Rather, the problem arises from the imprecise nature of social and behavioral theories and, consequently, a lack of commensurability across experiments conducted under different conditions. Therefore, researchers must fundamentally rethink how they design experiments and how the experiments relate to theory. We specifically describe an alternative framework, integrative experiment design, which intrinsically promotes commensurability and continuous integration of knowledge. In this paradigm, researchers explicitly map the design space of possible experiments associated with a given research question, embracing many potentially relevant theories rather than focusing on just one. Researchers then iteratively generate theories and test them with experiments explicitly sampled from the design space, allowing results to be integrated across experiments. Given recent methodological and technological developments, we conclude that this approach is feasible and would generate more-reliable, more-cumulative empirical and theoretical knowledge than the current paradigm – and with far greater efficiency.

Rapid whole genome sequencing (rapid WGS) is a powerful diagnostic tool that is becoming increasingly practical for widespread clinical use. However, protocols for its use are challenging to implement. A significant obstacle to clinical adoption is that laboratory certification requires an initial research development phase, which is constrained by regulations from returning results. Regulations preventing return of results have ethical implications in cases which might impact patient outcomes. Here, we describe our experience with the development of a rapid WGS research protocol, that balanced the requirements for laboratory-validated test development with the ethical needs of clinically relevant return of results.

A short-pulse, long-wavelength radio-echo sounder has successfully measured the ice depth on the South Cascade Glacier. Depths up to 250 m were determined with resolution of about 5%. Bottom returns were clear and almost never ambiguous. Their accuracy was confirmed by comparison with hot-point drilling results. The secret for successful sounding in temperate glaciers is the use of a sufficiently low center frequency. Five megahertz was most successful. Tests at 15 MHz indicated an increase in coherent clutter which rendered the bottom return observable only with prior knowledge of its location. The cause of the clutter is probably water-filled voids in the ice which behave as Rayleigh scatterers.

The sounding system consists of an avalanche-transistor transmitter, which delivers a pulse to an acute-angle crossed-wire antenna. The pulse is shaped and given its center-frequency characteristics by the resonant properties of the antenna. The transmitting and receiving antennas are identical, consisting of wires and lumped resistors. The resistors reduce antennas ringing, thereby maintaining as short a pulse as possible. The receiver consists of an oscilloscope and a Polaroid camera. No preamplification is required for depths up to 250 m, but may be necessary for deeper glaciers.