Concept formation is critical for many social scientific goals, yet it often appears neglected. This chapter underscores the importance of concepts in empirical work oriented toward causal inference, including experiments. It explores the role of conceptual hierarchies, typologies, and dichotomies both for causal attribution and for assessing generalizability. Using an example from the Metaketa Initiative, this chapter highlights the value of Sartori’s ladder of abstraction for fostering cumulative learning from experimental research. Wider use of the tools of concept formation can aid assessment of both the internal and external validity of causal claims.

This chapter focuses on the foundations of study design and statistical analysis in psychological research. It explores strategies for ensuring internal validity, such as randomization, control groups, and large sample sizes. Additionally, it addresses the complexity of human behavior by exploring multivariate experiments and the use of artificial intelligence and machine learning in neuroscience. The chapter also discusses the replication crisis and the emergence of open science practices, encouraging students to think critically about isolated scientific findings and offering tools for identifying credible research. Lastly, it critiques null hypothesis significance testing and p-values while providing an overview of key statistical topics like correlation coefficients, standardized mean differences, and regression.

This chapter explores the issue of validity in quantitative research. It covers various types of validity, such as face, content, construct, criterion, and predictive validity, and explains how each can be measured. The chapter also discusses internal and external validity, emphasizing their importance and potential threats, including selection bias, maturation, and history effects. You will learn effective strategies for mitigating these threats to ensure the robustness of your research. Key techniques such as careful sampling, appropriate recruitment procedures, and the use of statistical methods are explored to help you establish and maintain study validity. By the end, you will be equipped to design and conduct rigorous and valid research in applied linguistics.

In hierarchies of research evidence, the randomized controlled trial (RCT) usually appears near the top of the pyramid. RCTs are usually considered to be free of bias; systematic reviews of the literature may exclude studies that are not RCTs. Although controlled trials are excellent methods for establishing causation, they do not assure freedom from systematic bias. This chapter explores biases that are common among RCTs. In particular, we report on the practice of systematically excluding study participants who do not meet specific criteria. It is not uncommon for 90 percent of potential volunteers to be turned away. Common grounds for exclusion include comorbidities, even though living with multiple chronic conditions is almost universal among older adults. The selection of the control group can also increase bias. Control groups might be selected specifically because they increase the likely difference between treated and control conditions. The implications of biases in our RCTs are discussed.

The validity of conclusions drawn from specific research studies must be evaluated in light of the purposes for which the research was undertaken. We distinguish four general types of research: description and point estimation, correlation and prediction, causal inference, and explanation. For causal and explanatory research, internal validity is critical – the extent to which a causal relationship can be inferred from the results of variation in the independent and dependent variables of an experiment. Random assignment is discussed as the key to avoiding threats to internal validity. Internal validity is distinguished from construct validity (the relationship between a theoretical construct and the methods used to operationalize that concept) and external validity (the extent to which the results of a research study can be generalized to other contexts). Construct validity is discussed in terms of multiple operations and discriminant and convergent validity assessment. External validity is discussed in terms of replicability, robustness, and relevance of specific research findings.

The overarching goal of public health is to maximise the health of the population, and to achieve this we need evidence about what works and what does not work. Good studies are difficult to design and implement, and interpretation of their results and conclusions is not always as straightforward as we might hope. How, then, can we make the best use of this information? In the next three chapters we look at ways to identify, appraise, integrate and interpret the literature to generate the evidence we need to inform policy and practice. In this chapter we focus on interpreting the results from a single study, because if they are not valid they will be of limited value. The central question we have to answer when we read a study report is, ‘Are the results of the study valid?’

Peer review supports decisions related to publications, grant proposals, awards, or personnel selection. Independent of the specific occasion, we propose validity as a chief evaluation criterion for reviews. While applicable to all occasions, the principles of validity-oriented quality control are particularly suited for journal reviews. Beyond evaluating validity and the scientific potential of a given piece of research, we address how peer reviewing serves important functions and is accountable for the growth of science at a more superordinate level. We also provide guidelines and concrete recommendations for how a good peer review may serve these functions. Good peer review, thereby, fosters both the advancement of scientific research and the quality, precision, and sincerity of the scientific literature. The end of the chapter is devoted to a core set of good reviewer practices, conceived as an essential feature of academic culture.

Like Chapter 7, this chapter also discusses the sequencing of case study designs. Here we particularly focus on the deductive–inductive sequencing. Using an exemplar case study, we discuss how prediction outliers (deviant cases) identified during the initial study can guide the sequencing of designs in further stages. In the context of the example, we discuss the research question, theoretical sampling, cases, levels of analysis, and the potential requirement for additional data collection. Furthermore, we discuss the issue of omitted variable bias and internal validity in the context of sequenced case study designs. We end the chapter with a discussion on how to report sequenced case studies following deductive–inductive reasoning.

We discuss the single embedded case study design in this chapter. We deliberate how this design is different from multiple and single holistic designs in terms of the levels of analysis and the nature of replication. The selection rationale and sampling are discussed next. Afterwards, we move on to the longitudinal and/or cross-sectional single embedded designs. The strengths and the weaknesses of the design in terms of internal validity, external validity, and the number of variables are discussed subsequently. This chapter also discusses the (mis)conception regarding longitudinal designs and temporal embedded units.

We introduce and define the single holistic case study design in this chapter. The strengths of the design are discussed in detail, with examples. In particular, we discuss the potential of single holistic design in providing a detailed explanation of processes. Single holistic case studies also explore the theorizing potential of unique cases which hold the potential to reveal new dimensions of a phenomenon or falsify/refute an existing theory. Relatively high data access, construct validity, potential to include an unlimited number of variables, etc., are some other strengths that we discuss. The weaknesses of the design (i.e. low internal and external validity) are discussed afterwards. The chapter also addresses some common (mis)conceptions regarding single holistic designs and their external validity.

Here we discuss the multiple embedded case study design and the levels and cases involved in it. The design’s capacity to undertake between- and within-case comparative analyses are deliberated upon afterwards. Multiple embedded designs offer a variety of design possibilities, which we discuss in this chapter. The potential of the design for high internal validity is considered afterwards. Lastly, the design’s weaknesses, including the lack of contextual depth, the smaller number of explanatory variables, and the access difficulties to the cases/embedded units, are detailed.

The final chapter in this book discusses some methodological considerations and debates surrounding case study research and its quality. In particular, we revisit the topic of research paradigms (i.e. positivism and interpretivism). Relatedly, we discuss different quality criteria as proposed by prior researchers from both paradigmatic camps. In particular, we focus on the rigor versus trustworthiness discussion and the internal versus external validity debate. Afterwards, we briefly discuss the iterative cycles of data collection and analysis one would encounter during a qualitative case study research process. We end the chapter (and subsequently the book) with a guiding framework which will help researchers in sequencing case study designs by acknowledging the weaknesses of individual designs and leveraging their strengths. The framework can be adopted and adapted to suit the specific research objectives of the study in hand.

We discuss multiple case studies in this chapter. We start off with a discussion of theoretical sampling and replication logic. We specifically discuss literal and theoretical replication (LR and TR) in connection with multiple case studies. The strengths and limitations of LR and TR are discussed thereafter. In particular, we deliberate upon the potential of TR to enhance the internal and external validity of a case study. Henceforth, we address some common (mis)conceptions regarding replication logic, internal validity, external validity (generalizability), and reliability. We also discuss how multiple case studies might need to sacrifice the depth of observation for breadth. Other potential weaknesses, such as the smaller number of independent variables and the difficulty in controlling context, are also discussed thereafter.

This chapter provides a comprehensive overview of the verbal-guise technique (henceforth VGT), a variant of the speaker evaluation paradigm in which guises representing different language varieties are produced by different speakers, each speaking in their habitual language variety. First, key features of the VGT are discussed. Second, a brief historical sketch of the technique’s introduction, development, and proliferation in language attitudes research is offered. Third, key advantages (e.g. speaker authenticity, ease of implementation) and disadvantages (e.g. lack of full experimental control) of the technique are reviewed. Fourth, various practical considerations surrounding research planning and design are described and several recommendations are offered, including: (a) matching speakers on various demographic factors (e.g. sex, age), (b) matching speakers on extraneous vocal characteristics not of interest in the study (e.g. pitch), and (c) using multiple speakers to represent each variety of interest. Fifth, the main concerns surrounding the analysis and interpretation of data obtained using the VGT are discussed. Finally, a brief sketch is provided of the methodological considerations that were involved in designing a recent study utilising the VGT, which examined Americans’ attitudes toward standard American English and nine foreign accents: Arabic, Farsi, French, German, Hindi, Hispanic, Mandarin, Russian, and Vietnamese.

Chapter 3 focuses on how to think when evaluating experiments. This includes a discussion of realism, particularly why mundane realism or resemblance to the “real world” receives far too much attention, as well as an overview of how to design experimental treatments. The chapter then turns to validity issues, offering a new way to think about external validity in assessing experiments. This includes a detailed discussion of sampling and why the onus should be more on critics of an experimental sample than on the experimentalist him/herself (i.e., to justify a sample).

Experimental approaches are gaining in popularity across disciplines, ranging from behavioural sciences to economics. In this chapter, we discuss the advantages and disadvantages of field experiments and review their use by scholars to study routine dynamics. Based on these, we suggest that field experiments hold further promise to study routines given their potential to develop and test theory, while achieving internal and external validity. To further the adoption of field experiments to study routines, we outline a five-step procedure, including research questions and hypotheses, context and research setting, treatment and design, measurement and statistical tests, and managing field experiments. We conclude by discussing potential research questions and contexts suitable for field experiments.