When the practice of knowledge … became experimental, knowing became preoccupied with changes and the test of knowledge became the ability to bring about certain changes. Knowing, for the experimental sciences, means a certain kind of intelligently conducted doing; it ceases to be contemplative and becomes in a true sense practical.
In Chapter 5, we examined how data are elicited and transformed. These transformations foregrounded the potential of moving between unstructured and structured states of the same data. Such recursive data restructuring challenges oversimplistic distinctions between data collection and data analysis. While traditional research reporting draws a clear distinction between the two, conceptualizing data as a process prompts a deeper understanding of all the analytical tools available to us in order to make the most of the unstructured–structured continuum. The gap between data and analysis is widening due to a series of technical and societal developments, particularly the rise in qualitative big data (Adnan et al., Reference Adnan, Akbar, Khor and Ali2020). This means that these new datasets challenge the coupling of data and analysis (e.g., between experiments and between group statistics, or between interviews and thematic analysis). Large unstructured data, especially if unprompted and uncurated (see Chapter 5), require us to reconceptualize the purposes of qualitative and quantitative research and their mixing (Lieber, Reference Lieber2009).
The guiding proposal for this chapter is that qualitative and quantitative methods are synergistic. Qualitative and quantitative methods can be integrated to produce insights that are not reducible to either method. But exactly how this mixing produces outcomes that are more than the sum of the parts remains elusive. This “integration challenge” addresses the core promise of mixed methods research, namely, that mixing methods produces added value (Fetters & Freshwater, Reference Fetters and Freshwater2015a, p. 115; Guetterman et al., Reference Guetterman, Molina-Azorin and Fetters2020). This challenge is evident whenever qualitative and quantitative findings are presented side by side, with little more than a shared topic (Feilzer, Reference Feilzer2010; Seawright, Reference Seawright2016), thus failing to leverage any integrative synergy.
Addressing this integration challenge is complex because of the bewildering variety of research projects, each with multiple dimensions of possible integration (Fetters & Molina-Azorin, Reference Fetters and Molina-Azorin2017b). Ideally, there should be a clear rationale for mixing methods and explicitly formulated “integration strategies” that specify how synergies will be produced (Åkerblad et al., Reference Åkerblad, Seppänen-Järvelä and Haapakoski2021, p. 152). To address this challenge, we first conceptualize how qualitative and quantitative research purposes can be differentiated and then theorize how these purposes can be integrated to yield insights that are more than the sum of the parts.
Our approach is to focus on mixing qualitative and quantitative research purposes (see Chapter 4) within a pragmatist epistemology (see Chapter 2). Pragmatism considers research to be a human activity and, as any activity, it is goal-oriented, culturally mediated, and embedded within wider societal networks of norms and values. This makes research both meaningful and purposeful. The notion of purpose is thus wider than that of research questions or hypotheses and even research aims or objectives (Shehzad, Reference Shehzad2011). The purpose of using qualitative and quantitative methods is to generate new knowledge, and this knowledge generation can be achieved in multiple ways. We use Charles Sanders Peirce’s (Reference Peirce1955) distinction between induction, deduction, and abduction to differentiate three qualitative purposes (describing phenomena, theoretical framing, generating theory) and three quantitative purposes (measuring phenomena, testing hypotheses, exploring explanations), thereby enabling an analysis of how these purposes can be productively integrated. This typology was introduced in Chapter 4 (see Table 4.1), on research questions, to outline the differences between inductive, deductive, and abductive research. In the spirit of pragmatism, our approach is not prescriptive; it does not aim to promote specific combinations of methods as superior. Pragmatism eschews such absolutist claims and instead focuses on each method’s contribution to the problem at hand (Morgan, Reference Morgan2007).
Our guiding metaphor is tool use during carpentry. One does not ask whether the mallet is better than the saw in absolute terms; instead, the focus is on what each instrument does and specifically how these purposes can be combined synergistically (e.g., first sawing, then hammering joins to make a chair). In a similar vein, research methods are multifaceted. Some of them are designed to address specific purposes. For example, experiments are ideal for hypothesis testing, while interviews are typically used to explore experiences. And yet the same method can be used for multiple purposes (e.g., surveys can be used to explore associations and test hypotheses), and when used in combination, purposes can be integrated to produce outcomes irreducible to either method. For example, the carpenter’s chair cannot be produced by either the saw or the mallet in isolation.
This chapter is structured in four parts. First, we review current approaches to the integration challenge and make a case for a pragmatist approach. This is intended not as a comprehensive review, especially since the field of mixed methods is rapidly expanding, but as an overview guided by the quest for methodological synergies. Second, we use pragmatism to differentiate qualitative and quantitative research purposes (see also Chapter 4) and show how these purposes can be integrated to produce a more granular conceptualization of the synergies within simultaneous, sequential, and recursive designs. Third, we consider the question of creativity in mixed methods designs as a consequence of adopting a pragmatist standpoint; if there is no one-to-one relationship between research purpose, method, and the problem at hand, but a one-to-multiple relation, then we have the scope and necessity to remix methods and, in doing so, foster new synergies. We end with implications for mixed methods research and prepare the ground for our own proposal in this area, discussed in Chapter 7.
6.1 The Integration Challenge: A Pragmatist Approach
The integration challenge refers to the problem of conceptualizing how quantitative and qualitative methods can be integrated to produce insights that are not reducible to either method (Fetters & Freshwater, Reference Fetters and Freshwater2015a). Addressing this challenge is essential for legitimizing mixed methods research as a third type of research (Johnson et al., Reference Johnson, Onwuegbuzie and Turner2007) and aiding researchers to leverage potential synergies. Following the pragmatist proposition of creative synergies, this challenge can be translated in terms of discovering the multiplicity of purposes research methods can serve, particularly when mixed.
One of the earliest attempts to theorize method integration was the metaphor of triangulation. In navigation and geographic survey work, triangulation refers to identifying an unknown point by drawing a triangle with two known points and then using trigonometry to calculate the unknown location. The term was originally used in the social sciences to conceptualize measurement validation (Johnson et al., Reference Johnson, Onwuegbuzie and Turner2007). Subsequently, it was appropriated within qualitative research to theorize analysis enrichment (Denzin, Reference Denzin2012) and it remains a key criterion for assessing quality in qualitative studies (Bauer & Gaskell, Reference Bauer and Gaskell2000). While triangulation for validation stays close to the original metaphor (achieving validity through overlapping measurements), triangulation for enrichment departs from the metaphor (nonoverlapping findings reveal different aspects of the phenomena and are equally valid). Given this confusion, we use the less metaphorically loaded term “integration,” which is increasingly preferred (Fetters & Molina-Azorin, Reference Fetters and Molina-Azorin2017a).
Reviews of research practice have revealed various rationales for mixing methods (Greene et al., Reference Greene, Caracelli and Graham1989). In a review of 232 mixed methods articles, Bryman (Reference Bryman2006) identified 17 rationales, with the most common being validating, obtaining completeness, explaining, developing a measure, identifying a sample, illustrating, enhancing the analysis, and including a diversity of views. These not only reflect the strength of qualitative or quantitative methods taken separately (e.g., developing a measure for quantitative and illustrating for qualitative) but, most of all, point to the benefits of integration (e.g., obtaining completeness, enhancing the analysis). Using mixed methods to ensure that a diversity of voices are represented, in particular, points to a rationale far beyond validation and enrichment (i.e., social justice; Mertens, Reference Mertens2007), one that is in line with the pragmatist ethos of empowering action through research (see Chapter 9 for an analysis of the relationship between human interests, research, and possibility).
These studies of mixed methods research practice have also emphasized theory creation as a valuable rationale. For example, Boeije and colleagues (Reference Boeije, Slagt and van Wesel2013) found that, in addition to validation (e.g., for instrument development) and enrichment (e.g., providing illustrations and nuance), mixed methods studies often enabled speculating about underlying mechanisms and generating plausible theories. In other words, mixed methods research often facilitates abduction (see Chapter 4) to complement and leverage the insights gained from deduction and induction. Specifically, it has been argued that discovering contradictions (Greene et al., Reference Greene, Caracelli and Graham1989) and puzzling discrepancies (Bryman, Reference Bryman2006) can spur interpretation, reflecting the pragmatist insight that thought itself originates in confronting obstacles (Mead, Reference Mead1964a).
Across the diverse rationales for mixed methods research, there is the underlying idea that integration should add value beyond what either method can contribute alone (Fetters & Freshwater, Reference Fetters and Freshwater2015a). The challenge is to specify the relationships between the methods, data, and findings so that the synergy is more than accidental (Moran-Ellis et al., Reference Moran-Ellis, Alexander and Cronin2006) and can be deliberately enhanced in research. To this end, Fetters and Molina-Azorin (Reference Fetters and Molina-Azorin2017b) identified fifteen dimensions of possible integration, including philosophical, theoretical, researcher, team, literature, sampling, design, research aims, data collection, analysis, and interpretation (see also Schoonenboom & Johnson, Reference Schoonenboom and Johnson2017). These insights direct attention toward the nodal points at which integration occurs, thus potentially isolating how integration is more than the sum of the parts (Åkerblad et al., Reference Åkerblad, Seppänen-Järvelä and Haapakoski2021).
A pragmatist approach to mixing methods focuses on the nodal point of purposes to advance the integration challenge. While other paradigms bring into focus social justice (the transformative paradigm; Mertens, Reference Mertens2007) and compatibility (critical realism; Shannon-Baker, Reference Shannon-Baker2016), the pragmatist paradigm emphasizes the purpose of methods (i.e., what they actually achieve; Chapter 4) and thus helps us consider multiple methods holistically in terms of what they individually and collectively contribute to the problem at hand. The term “research purpose” subsumes research questions, hypotheses, aims, goals, and objectives and also points our attention to the articulation between what methods are intended to do, what they do, and how they do what they do. Differentiating qualitative and quantitative research purposes, we argue, provides a basis for revealing synergistic combinations of research purposes. This is because integrative synergies require a solid understanding of differences in order to understand grasp how methodological combinations come about and how differences in purpose can lead to creative novelty (for a broader argument about differences and creativity see Glӑveanu & Gillespie, Reference Glӑveanu, Gillespie, Glӑveanu, Gillespie and Valsiner2014).
Pragmatism reconceptualizes abstract concepts by focusing on their consequences (Peirce, Reference Peirce1878). It recasts debates about the meaning of truth, beauty, God, and so on in terms of what these concepts “do”. Instead of relying upon axioms, first principles, or exhaustive logically consistent definitions, pragmatism grounds the meaning of concepts in human activity. From a pragmatist standpoint, all theories, beliefs, and ideas are tools for action (Cornish & Gillespie, Reference Cornish and Gillespie2009). Simply put, meaning lies in consequences (see Chapter 1).
Pragmatism is particularly suited to mixed methods research because it values each method for its contribution (Morgan, Reference Morgan2007). Thus, it offers an alternative to postpositivism or constructionism (Feilzer, Reference Feilzer2010). It rejects the purist assumption that some methods are “better” than others in absolute terms. Pragmatism is inclusive because, in the words of James (Reference James1907, p. 31), it has “no obstructive dogmas, no rigid canons of what shall count as proof” and “will consider any evidence.” This does not imply “anything goes” relativism (see Chapter 2), in which the differences between methods are ignored. On the contrary, the differences between methods are leveraged to increase synergy. And these differences are reflected upon, from the epistemological basis of methods to their analytical steps. However, instead of asking whether methods are epistemologically commensurable or analytically compatible, pragmatism asks what each method contributes to the problem at hand. “Pragmatists,” Feilzer (Reference Feilzer2010, p. 14) writes, “do not ‘care’ which methods they use as long as the methods chosen have the potential of answering what it is one wants to know.”
The pragmatist approach addresses the integration challenge by specifying how qualitative and quantitative research purposes can be combined to achieve particular chains of inquiry. Instead of general and high-level rationales for mixed methods research (i.e., validating, enriching, developing, explaining), the pragmatist approach directs attention to the underlying qualitative and quantitative purposes and, specifically, to how these are being combined and what they help us achieve. Thus, in contrast to the many typologies that differentiate qualitative and quantitative methods based on predetermined characteristics (Coxon, Reference Coxon2005; Sale et al., Reference Sale, Lohfeld and Brazil2002), we focus on what these methods are used for (see also Denscombe, Reference Denscombe2021). Our aim is not to characterize qualitative and quantitative methods, in general, or delineate family resemblances (Morgan, Reference Morgan2018) but rather to advance a rigorously pragmatist approach grounded in research purposes.
Synergy entails both differentiation and integration; each component must remain distinct enough to add value, while contributing to an outcome that it could not achieve alone (Moran-Ellis et al., Reference Moran-Ellis, Alexander and Cronin2006). We have already differentiated the purposes of qualitative and quantitative research methods in Chapter 4. Accordingly, we now focus on how these purposes can be integrated synergistically.
6.2 Integrating Qualitative and Quantitative Research Purposes
In Chapter 4 (Table 4.1) we distinguished six research purposes. Qualitative research is suited to describing phenomena (induction), theoretical framing (deduction), and generating explanations (abduction). Quantitative research is suited to measuring phenomena (induction), testing hypotheses (deduction), and exploring explanations (abduction). Now we will use this pragmatist typology to specify more precisely how the six purposes can be synergistically combined by considering how they are “chained” together into logical sequences of investigation.
In the following subsections, we review pairings of purposes in simultaneous, qualitative first, quantitative first, and recursive designs (Creswell & Creswell, Reference Creswell and Creswell2018). Our focus is only on mixed methods pairings (i.e., we are excluding within-method pairings). For concision, we abbreviate the six purposes and adapt the notation introduced by Morse (Reference Morse1991) and refined by others (Nastasi et al., Reference Nastasi, Hitchcock and Sarkar2007; Schoonenboom & Johnson, Reference Schoonenboom and Johnson2017) such that “+” denotes simultaneous purposes, “→” denotes sequential purposes, “→←” denotes recursive purposes, and “[]” is used to group purposes within a larger design.
6.2.1 Simultaneous Designs
In simultaneous designs, qualitative and quantitative methods are used independently. Each purpose could be pursued in isolation, but when used together they converge on a phenomenon to either validate findings or enrich each other.
Measuring + describing is a common simultaneous design for both validation and enrichment. For example, to evaluate automated techniques for modeling topics within texts, Leeson and colleagues (Reference Leeson, Resnick, Alexander and Rovers2019) validated automated topic modeling (measuring) of interview transcripts by comparing it with qualitative thematic analysis (describing) and found good convergence. Hagan and colleagues (Reference Hagan, Rosenzweig, Zorn, van Londen and Donovan2017) surveyed cancer survivors and used open-ended questions to enrich and add nuance to the survey findings. If the qualitative study is guided by theory, it becomes measuring + framing, as in Emadian and colleagues’ (Reference Emadian, England and Thompson2017) validation of a diabetes questionnaire. They administered the questionnaire to a novel population and then, guided by knowledge about diet and diabetes, used interviews to evaluate suitability.
Testing + framing is another simultaneous design used for validation and enrichment. An example is Glӑveanu’s (Reference Glӑveanu2014) study of the “art bias” in lay conceptions of creativity. This bias involves favoring artistic understandings of creativity to the point at which everything art-related is automatically considered creative, and nothing is viewed as truly creative if it is not artistic. This research was survey-based, conducted online, and included two parts. The first was qualitative and required participants to think of questions they would ask to determine whether an object was creative. This helped make explicit the criteria people use to evaluate creativity and provided qualitative data. The quantitative part entailed participants evaluating sixteen professions as to whether creativity was necessary for success (with reaction times recorded). The findings showed mild support for the art bias. Art-related professions scored highly (and fast), but some other professions also showed the same pattern (particularly in everyday domains). The thematic analysis of the qualitative data showed, on the contrary, that while art-based criteria are important, they can be overshadowed by utility concerns (i.e., is it practical to use?). Methodologically, this study illustrates the simultaneous application of qualitative and quantitative methods to test an expectation derived from the literature.
The defining feature of simultaneous designs, when conceptualized in terms of qualitative and quantitative purposes, is that neither purpose grows out of the findings of the other method. Both purposes originate outside of the research process and neither finding feeds into the other method. Each analysis is conducted separately, and then the findings are compared. Furthermore, although the purposes cannot be identical, they should be similar enough to enable either validation or enrichment.
6.2.2 Qualitative First Designs
Qualitative first sequences include exploratory sequence designs (Creswell & Creswell, Reference Creswell and Creswell2018). These sequences begin with describing phenomena, theoretical framing, or generating explanations that feed forward into measurement and testing.
Describing → measuring occurs when creating a measure based on a qualitative description. A typical scenario is creating a survey in an unknown domain. For example, Senteio and colleagues (Reference Senteio, Adler-Milstein, Richardson and Veinot2019) used exploratory interviews to create survey items to measure how physicians incorporated psychosocial information into treatments. Another scenario is when a description leads to questions about prevalence. For example, Feltham-King and Macleod (Reference Feltham-King and Macleod2016) began by describing the discourses used to talk about women in South African newspapers, and then they used quantitative content analysis to measure the changing frequency of these discourses.
Another illustration of this scenario is Glӑveanu’s (Reference Glӑveanu2011) study of social representations of creativity in the United States and the United Kingdom. The focus was on how ideas about creativity are anchored in symbols. A survey was used to assess how participants would rate common symbols of creativity (quantitative) and explain their rating (qualitative). This survey, combining open and closed questions, illustrates a simultaneous design. The “qualitative first” aspect was in the creation of the symbols to be rated. An initial qualitative analysis of the first 500 images in a Google search for creativity identified the key symbols (e.g., lightbulb, brain, paintbrush and colors, computer, toy, musical note, children’s drawings, and jigsaw puzzle). The subsequent survey, built around these findings, showed that the symbol most indicative of creativity was a paintbrush and colors, closely followed by children’s drawings. The point, however, is that these quantitative findings were based on an initial open-ended qualitative analysis of symbols of creativity.
Framing → measuring is like describing + measuring, except it has greater theoretical motivation. This arises when creating a measurement tool for a predefined concept. In such a case, qualitative research is often used to provide theoretically framed illustrative data that inform the creation of survey items (e.g., Mearns et al., Reference Mearns, Kirwan and Reader2013). A similar approach is used when creating textual measures. For example, Graham and colleagues (Reference Graham, Haidt and Nosek2009) used moral foundations theory to identify candidate words in context, qualitatively assessing whether each indicated the desired concept. They subsequently added selected words to their dictionary measure, which they then used to score moral foundations in liberal and conservative speeches.
Framing → testing implies a theoretically motivated qualitative study that feeds forward into a quantitative test. For example, Obradović and Sheehy-Skeffington (Reference Obradović and Sheehy-Skeffington2020) used this sequential design to examine EU integration in Serbia. First, a qualitative analysis of interview transcripts, guided by theory, identified perceived power imbalances as a barrier to integration. Second, a survey provided a quantitative test, showing that participants who perceived Serbia to be powerless identified less with the European Union.
Generating → testing begins with a qualitative study motivated by something that requires explanation. For example, Festinger’s (Reference Festinger1956) qualitative case study of a cult that did not dissolve after their end-of-the-world prophecy failed led to the idea of cognitive dissonance, which was subsequently tested experimentally (Festinger, Reference Festinger1957). A more recent example is Haugestad and colleagues (Reference Haugestad, Skauge, Kunst and Power2021), who investigated climate activism in Norway, guided by the paradox that Norwegians have benefited hugely from oil. Interviews were used to generate an explanation that was subsequently tested using surveys.
Qualitative first designs are distinguished by the second quantitative purpose arising out of the findings from the first qualitative purpose. Uncovering the underlying purposes reveals an otherwise opaque heterogeneity. For example, the differences between describing → measuring (e.g., creating a measure) and generating → testing (e.g., creating and testing a theory) are lost within the more general qualitative → quantitative conceptualization. Specifying the underlying purposes reveals how these purposes synergistically combine into superordinate purposes (e.g., creating a measure, establishing a new theory, or putting existing findings to the test).
6.2.3 Quantitative First Designs
Quantitative first sequences include explanatory sequence designs. In such designs, the findings for any quantitative purpose (e.g., measuring, testing, exploring) feed forward into a qualitative purpose.
Measuring → describing refers to a second qualitative study that provides descriptions and illustrations for initial quantitative measurements. This sequence of purposes occurs in natural language processing when quantitative topic modeling techniques are used (e.g., to cluster Tweets or other texts) and then the qualitative analysis describes the themes within each cluster, usually with illustrative quotes (Hamad et al., Reference Hamad, Savundranayagam, Holmes, Kinsella and Johnson2016). In another example, Van Huizen and colleagues (Reference van Huizen, Dijkstra and Halmos2019) quantitatively counted the number of multidisciplinary meetings that led to recommendations and then conducted interviews to document the benefits and limitations of the multidisciplinary meetings. If they had sought to explain the observed frequency, it would have been a measuring → generating sequence.
Measuring → generating entails a quantitative measure yielding a finding that feeds forward into qualitative speculation about possible explanations. For example, studies have begun with measures revealing physician over-prescribing (Voigt et al., Reference Voigt, Gottschall and Köberlein-Neu2016) and low female participation in a swim safety program (Gupta et al., Reference Gupta, Rahman and Dutta2020); then, qualitative methods were introduced to address the question of “Why?” These studies typically conclude with proposed explanations. However, if these explanations are tested in subsequent research, then the chain of investigation would become measuring → generating → testing.
Testing → generating is similar to measuring → generating, except it starts with a deductive test and then qualitative research is used to explain the findings. For example, Mannell and colleagues (Reference Mannell, Davis and Akter2021) began with a randomized controlled trial of an intervention and then used a visual participatory method to aid the interpretation of the trial findings. Generating explanations often follows testing that yields surprising results. For example, Wee and colleagues (Reference Wee, Cher, Sin, Li and Koh2016) tested their assumption that distance to a primary care clinic was a barrier to cancer screening in Singapore. However, inconclusive findings fed forward into interviews that revealed distrust and embarrassment as additional barriers to screening.
Another example of testing → generating is Glӑveanu and colleagues’ (Reference Glӑveanu, Gillespie and Karwowski2019) study of divergent thinking of individuals and dyads. This research was conceived as an experiment with individuals and dyads brainstorming. The aim was to understand the differences in originality, practicality, surprise, and overall creativity between the two conditions (individual and social). However, these outcome-based quantitative analyses were inconclusive. Accordingly, the authors conducted a qualitative study of the video recordings of dyads working together in order to generate ideas about what was occurring during the task. This analysis related the nature of the proposed idea (e.g., original or not, practical or not) with the response it received (positive, negative, indifference). The finding was that in the social condition, more practical ideas are better received. This “practicality effect,” as named by the authors, was supported by a temporal quantitative analysis that found originality being high for initial ideas and then gradually reduced while practicality remained valued throughout. This study then suggests that working together does not reduce the creativity of ideas as much as it steers creative ideas toward the practical rather than wildly original. The design was testing → generating in which what initially seemed like an experimental nonfinding was reinterpreted in light of qualitative results.
Testing → framing entails testing a model quantitatively and then using the model as a frame for a qualitative analysis. This sequence has been refined by Seawright (Reference Seawright2016), who advocates quantitative testing of causal pathways followed by qualitative assessment of the extent to which these causal pathways are evident in particular cases. For example, Kreuzer (Reference Kreuzer2010) investigated the social conditions conducive to the emergence of proportional representation. He used a regression analysis to develop a model and then proceeded with a case-by-case examination to assess whether each country evidenced the model.
Exploring → generating arises when quantitative exploratory data analysis yields findings that require an explanation (Tukey, Reference Tukey1977). For example, Moore and Gillespie (Reference Moore and Gillespie2014) explored misunderstandings between people with brain injury and their caregivers using a survey. They found that people with brain injury tended to overestimate their abilities relative to the views of their caregivers. Qualitative analysis of verbal comments while filling out the survey indicated that caregivers actively encouraged their partners with brain injury to feel more capable and thus less of a burden than they were actually perceived to be.
Quantitative first sequences begin with a quantitative finding (a measurement, a test result, or an exploratory finding) that a qualitative analysis then describes, frames, or explains. Identifying the underlying sequence of questions enables more granular distinctions. For example, using a more general quantitative → qualitative conceptualization obfuscates the differences between measuring → describing (e.g., illustrating a quantitative measure) and testing → generating (e.g., explaining a failed experiment).
6.2.4 Recursive Designs
Recursive designs entail an analysis that moves back and forth between methods. This design is pervasive in qualitative research (e.g., alternating between inductive describing and trying various theoretical frames), and it also occurs in quantitative research (e.g., moving back and forth between measuring and reliability testing when developing a measure). However, recursive designs are rare in mixed methods research, perhaps because each method tends to have separate datasets, which inhibits moving between methods.
Describing →← measuring can occur in intervention research, where the focus is on creating change by iteratively describing the situation, intervening, and then measuring the impact. A good example is provided by Nastasi and colleagues (Reference Nastasi, Hitchcock and Sarkar2007), who developed a mental health improvement project in Sri Lanka. They designed an intervention and then iteratively modified it based on qualitative inductive descriptions of local responses and outcome measurement.
Framing →← measuring can occur during the development of a theoretically motivated measure, such as when qualitative data are being quantified. This is common in natural language processing when developing a textual measure indicative of a concept; the researchers recursively examine words in context and the properties of the overall measure (Boyd & Schwartz, Reference Boyd and Schwartz2021). The aim is to keep the emerging measure grounded in the particulars of the phenomenon.
Another example of framing →← measuring is Gillespie and Reader’s (Reference Gillespie and Reader2016) development of a publicly available tool for measuring the nature and severity of patient complaints about healthcare services. Based on a systematic review of the literature (Reader et al., Reference Reader, Gillespie and Roberts2014), they identified several common categories of complaint. Starting with this framing from the literature, they then used iterative qualitative coding of samples of complaints to refine the coding scheme, recursively checking the qualitative validity and interrater reliability with each round of development. The emergent tool was subsequently used in quantitative and qualitative research to show the validity of patient-reported insights about healthcare services (Gillespie & Reader, Reference Gillespie and Reader2018; Reader & Gillespie, Reference Reader and Gillespie2021). In this example, the framing based on the literature, the qualitative and quantitative development, and the findings were all presented in separate research articles.
Exploring →← generating sometimes occurs when modeling big qualitative datasets, when the research alternates between exploring quantitative associations and generating explanations by examining qualitative particulars. Ideas generated qualitatively can be tested quantitatively in an exploratory mode, and quantitative relationships can be validated and explained by deep dives into the qualitative data. However, caution is required to prevent overfitting – picking up spurious correlations qualitatively and subsequently testing them (Calude & Longo, Reference Calude and Longo2017). Confirmatory hypothesis testing requires a separate dataset (e.g., a holdout sample of the data). Any recursive design that involves hypothesis testing must therefore be embedded within a sequential design (i.e., a [generating →← exploring] → testing design).
Recursive designs have the maximum potential for synergy. Instead of relying on one-shot mixing (e.g., qualitative → quantitative), recursive designs can leverage synergies with each back-and-forth movement. This dynamic process is difficult to formalize, reproduce, and write up. The social world and the practice of social science are messier than the methods of social science (Feyerabend, Reference Feyerabend2001; Law, Reference Law2004), a misperception perpetuated by overly neat write-ups. Indeed, the replication crisis stems in part from recursive practices with many degrees of freedom being written up as single-shot confirmatory tests (Wicherts et al., Reference Wicherts, Veldkamp and Augusteijn2016). Recursive designs have creative potential, but they must be conceptualized, formalized, and written up appropriately.
6.3 Integration Synergies
Conceptualizing integration in terms of research purposes gives specificity to the description of research designs, enabling a richer description of mixed methods research. For example, instead of describing Nisbett and Cohen’s (Reference Nisbett and Cohen1996) classic investigation of honor culture in the southern United States as alternating between qualitative and quantitative methods, we can be more precise about the underlying logic and the interrelation of different research purposes. Nisbett and Cohen began by recursively moving between ethnographic observation (e.g., violent children’s games, no holds barred fist fighting) and descriptive statistics (e.g., homicide rates, participation in the military). Their abductive leap was to explain these qualitative descriptions and measurements in terms of an honor culture, that is, individuals’ willingness to protect their reputation using violence. They then tested this theory using both surveys (southerners showed more endorsement of violence in response to an insult) and experiments (southerners showed more aggression when insulted). Accordingly, this design can be described as [describing →← measuring →← generating] → [testing (survey) + testing (experiment)].
Being specific about the research purposes helps to distinguish types of synergy. The first phase of Nisbett and Cohen’s (Reference Nisbett and Cohen1996) research leverages a synergy of recursive generativity. Questions arising from both description and measurement spur speculation and the search for additional descriptions and measurements – like a police investigation in search of a pattern. The second phase of the research leveraged a synergy of consequent inquiry, where the speculations arising from the first phase form a cumulative base that is built upon in the second testing phase. Moreover, this second testing phase used both surveys and laboratory experiments, thus demonstrating a synergy of convergence, where independent methods provide mutually supporting evidence that converges upon an interpretation. Conceptualizing integration in terms of research purposes brings the underlying logic of these synergies into clearer focus and helps us discuss both specific studies and much bigger long-term projects.
6.4 Creating Mixed Methods
An often-overlooked mark of mixed methods designs is creativity. As a methodological approach based on integrating methods and cultivating synergies between them, its outcomes should bear the creative marks of novelty and appropriateness (Stein, Reference Stein1953). Yet any research design can become conventional and be used unreflectively. Especially in mixing methods, there are common combinations (e.g., surveys preceded or followed by interviews, experiments being also videotaped) that tend to be used unimaginatively. This is why recent discussions about innovation in mixed methods research are timely (e.g., Poth, Reference Poth2018).
The pragmatist proposition of mixing methods to produce synergies is grounded in the idea that methods are constantly created and recreated to adapt to specific research purposes. The etymological root of the term “method” is a way or path; as such, methods can potentially provide many diverse paths for addressing research questions. To understand the potential creativity of these multiple paths, it is necessary to decouple method from purpose in the sense that a purpose can potentially be achieved by using multiple methods and a method could, when used in the right kinds of combinations, serve multiple purposes. This new form of flexibility helps us overcome functional fixedness (Duncker, Reference Duncker1945) when it comes to using methods for addressing practical problems and creating new knowledge. Also, it allows us to theorize the combinatorial dynamic of mixed methods by considering how each purpose relates to the other and becomes shaped by it (particularly in recursive and integrative designs).
The pragmatist approach to mixing methods supports calls for dialogue about mixed methods by Hesse-Biber (Reference Hesse-Biber2010, pp. 417–418). Mixed methods dialoguing is not about winning but promoting conditions for dialogue, which will require several ingredients: (1) It is important to bring all stakeholders with an interest in this field to the dialogue table; we must also (2) confront our methodological and methods assumptions; (3) suspend immediate judgments; (4) embrace our differences; and (5) practice reflexivity by listening across our differences as a means toward building a new set of shared assumptions and, if not, at least a willingness to remain open to different points of view.
We should also not lose sight of the fact that different purposes and methods are born out of different human interests (Chapter 9), and often, the use of mixed methods comes out of a desire to create change, address pressing challenges, and drive processes of personal and societal transformation (Mertens, Reference Mertens2007; Poth, Reference Poth2018). As such, creativity is called on to help researchers find new and innovative ways of aligning their purpose with their methodological tools and the kind of impact they want to create in the world. Mixing methods is particularly suited to world-making research, that is, research that engages with the world, is open to being challenged by the world, and aims to feed forward into a better world (Power et al., Reference Power, Zittoun and Akkerman2023). Mixing qualitative and quantitative methods offers good opportunities for creativity, from new ways to display data (McCrudden et al., Reference McCrudden, Marchand and Schutz2021) to new designs. In Chapter 7, we will introduce new ways of displaying mixed methods data and propose a new research design based on the recursive restructuring of data between qualitative and quantitative forms.
6.5 The Contributions of a Pragmatist Approach
A pragmatist approach to the integration challenge starts with the insight that methods, just like theories (see Chapter 3), are tools for action and that research itself is a human activity, situated in material, social, and cultural terms. Instead of distinguishing methods from first principles or following a narrow understanding of quantitative versus qualitative research, a pragmatist approach differentiates methods in terms of their research purposes or what they help researchers “do.” This clear differentiation enables theorization about how these different purposes are integrated. In this chapter, we have mapped out common ways of mixing qualitative purposes (describing, framing, generating) and quantitative purposes (measuring, testing, exploring) to produce synergies. This pragmatist approach to mixing methods makes six contributions.
First, the typology of purposes (Table 4.1) provides guidance on when to use qualitative, quantitative, and mixed methods. Scholars have characterized differences in epistemology (Denzin, Reference Denzin2012) and subject matter (Shweder, Reference Shweder, Jessor, Colby and Shweder1996) and distinguished family resemblances (Coxon, Reference Coxon2005; Morgan, Reference Morgan2018). But what is needed is a contingency theory of when to use qualitative, quantitative, and mixed methods (Johnson et al., Reference Johnson, Onwuegbuzie and Turner2007). To this end, we have argued that qualitative methods are suited to describing phenomena, theoretical framing, and generating explanations, whereas quantitative methods are suited to measuring phenomena, testing hypotheses, and exploring explanations. Differentiating these purposes can guide researchers in selecting the correct methods for their problem and aid mixed methods researchers in specifying their integrative strategy (Åkerblad et al., Reference Åkerblad, Seppänen-Järvelä and Haapakoski2021).
Second, having clearly defined purposes for qualitative and quantitative research gives each approach separate domains of legitimacy. The paradigm wars encouraged researchers to choose between qualitative and quantitative research methods (Bryman, Reference Bryman, Alasuutari, Brannen and Bickman2008), creating ontological uncertainty, with each group fearing being supplanted by the other. Grounding methods in different purposes gives each method clearly defined domains of legitimacy. This enables each approach to confidently focus on and refine what it does best – without fearing supplantation. Moreover, this creates a clear domain of legitimacy for mixed methods research, which arises whenever a research problem would benefit from mixing qualitative and quantitative purposes.
Third, differentiating qualitative and quantitative research purposes provides a more granular understanding of the integration challenge. The terms “qualitative” and “quantitative” are routinely used to conceptualize integration in terms of simultaneous (qualitative + quantitative), sequential (qualitative ←/→ quantitative), and recursive (qualitative →← quantitative) designs (Creswell & Creswell, Reference Creswell and Creswell2018; Morse, Reference Morse1991; Nastasi et al., Reference Nastasi, Hitchcock and Sarkar2007). But these terms are problematic because they encompass diverse research types and purposes (Coxon, Reference Coxon2005; Krippendorff, Reference Krippendorff2019), often without a proper reflection on how they relate to each other. Unpacking the underlying purposes enables a more precise specification of how methods combine to become more than the sum of the parts.
Fourth, the existing rationales for mixed methods research (validation, illustration, explanation, enrichment, etc.; Bryman, Reference Bryman2006; Clark & Ivankova, Reference Clark, Ivankova, Clark and Ivankova2016; Greene et al., Reference Greene, Caracelli and Graham1989) can be specified in greater detail. For example, instrument development entails both qualitative framing and quantitative measuring; illustration entails first measuring and then describing; and explaining starts with measuring or testing and then uses qualitative methods to generate explanations. Thus, using pragmatism to differentiate underlying research purposes contributes theoretical underpinnings to the existing and already nuanced rationales for mixed methods research.
Fifth, specifying the purposes underlying simultaneous, sequential, and recursive mixed methods designs (Creswell & Creswell, Reference Creswell and Creswell2018; Nastasi et al., Reference Nastasi, Hitchcock and Sarkar2007) reveals three distinct sources of synergy. In simultaneous designs, the purposes allow similar or partially overlapping research questions. This leverages a synergy of convergence, either for validation or for enrichment. In sequential designs, the purposes arise in sequence, with the purpose of the second study growing out of the findings of the first study. This leverages a synergy of consequent inquiry, because the findings of the second study are dependent upon the purpose of the first study. Finally, in recursive designs, both purposes operate together, with each shaping the other, such that the questions being addressed by each study evolve in response to the other study. This synergy of recursive generativity arises from the rapid alternation between purposes and the openness of each method to the findings of the other method.
Finally, conceptualizing mixed methods in terms of chaining together qualitative and quantitative research purposes fosters methodological innovation. Traditional methodologies, particularly within the quantitative tradition, have always been keen to standardize their steps and procedures (see, for instance, discussions of “best practices” in Anguera et al., Reference Anguera, Blanco-Villaseñor, Jonsson, Losada and Portell2020) which, on the one hand, allowed researchers to claim “objectivity” but, on the other, made many of the established methods too rigid to be used without modification in the dynamic data landscape of today (see Chapter 5). Qualitative methods are traditionally more flexible, with considerable ongoing discussions precisely about the role of creativity in research (see Wegener et al., Reference Wegener, Meier and Maslo2018). The challenge for mixed methods is to navigate this path between standardization and methodological innovation, an issue the field has engaged with seriously since its inception (after all, mixing methods and fostering synergies are creative acts). The pragmatist approach sharpens this innovative potential because it does not restrict methodological choices; instead, it focuses on the overarching aim of the research and then encourages researchers to work backward in terms of what tools are best suited for the given aim. It embraces any method that advances the guiding interest.
Alongside these contributions, there are also important limitations to consider. The terminology used to specify the six purposes is blunt. We have tried to balance accuracy, parsimony, and existing usage. Being primarily an analytical tool, the typology of purposes seems to create a discontinuity between qualitative and quantitative methods, when in research practice there is often a continuum (Johnson et al., Reference Johnson, Onwuegbuzie and Turner2007; Onwuegbuzie & Leech, Reference Onwuegbuzie and Leech2005). The typology should be interpreted as “typifying” the extremes of both qualitative and quantitative purposes and also of inductive, deductive, and abductive purposes. Additionally, all paradigms both hide and reveal (Shannon-Baker, Reference Shannon-Baker2016), and using a transformative or critical realist paradigm would have focused on other nodes of integration (i.e., representation or causation). The present analysis uses a pragmatist paradigm to examine the role of research purposes in mixed methods research in-depth, and this approach foregrounds the relation between method and action, emergence, and creativity.
In conclusion, qualitative and quantitative methods can be integrated to produce insights irreducible to either method because these methods serve different purposes that can be synergistically combined. Returning to the metaphor of the carpenter’s toolbox, each tool must be evaluated in terms of what it does (e.g., the purposes of sawing, sanding, hammering). Differentiating these purposes reveals integrative synergies, with superordinate outcomes irreducible to either tool (e.g., sawing → hammering = chair). Similarly, the proposed typology of research purposes provides a framework for deciding when to use qualitative and quantitative methods, for conceptualizing how these purposes can be combined in synergistic mixed methods designs, and for adding nuance to descriptions of mixed methods research.
Open access