1 Introduction
The sole purpose the enhanced standard analysis (ESA) has been designed for is to avoid so-called untenable assumptions. One source of such assumptions can be statements of necessity.Footnote
1
A researcher making the necessity claim
$X\leftarrow Y$
cannot make the simplifying assumption
$\neg X\rightarrow Y$
during the analysis of sufficiency. ESA is agnostic as to whether
$X$
stands for an atomic condition, a conjunction, a disjunction, or any combination of those. The Qualitative Comparative Analysis (QCA) community has elaborated a set of criteria, though, that define what constitutes a meaningful claim of necessity. ESA, together with existing standards, is a tool for combining the analysis of necessary conditions with that of sufficient conditions under the constraint of often noisy social science data.
Thiem (Reference Thiem2017) does not question the appropriateness of the core purpose of ESA: to avoid untenable assumptions. What he dismisses are the criteria that distinguish meaningful necessary conditions from meaningless supersets. By trivializing the disciplined search for meaningful necessary conditions into mindless data mining for supersets, Thiem (Reference Thiem2017) finds plenty of necessary conditions everywhere. This, in turn, produces the alleged CONSOL effect of ESA: once all supersets are allowed to be seen as necessary conditions, usually all simplifying assumptions are classified as untenable by ESA and only the conservative solution can be obtained.
In Section 2 I spell out existing standards for the analysis of necessary conditions and in Section 3 I empirically demonstrate that no CONSOL effect exists once these standards are applied. In an attempt to preempt a predictable objection to my emphasis on existing conventions in the QCA community, I introduce the notions of QCA realists and idealists. QCA idealists deem separate searches for necessary conditions as misguided. Instead, researchers simply need to confine themselves to producing the most parsimonious solution from their truth table, because this will automatically identify all minimally sufficient conjunctions that together form a minimally necessary disjunction for the outcome (e.g. Baumgartner Reference Baumgartner2015). While formally correct within a regularity theory of causation framework, in Section 3 I address several unresolved issues that arise when this position is applied to social science theories and real data. Section 4 concludes with some broader reflections on the split between QCA idealists and realists.
2 Criteria for Necessary Conditions
Thiem’s (Reference Thiem2017) crucial claim for producing the CONSOL effect is that scholars must declare as necessary any superset (be it a single set or a disjunctionFootnote 2 ) of the outcome if only it passes a predefined consistency threshold. Contrary to that, QCA realists have formulated several criteria that need to be applied during the search for necessary conditions: empirical consistency, empirical relevance, and conceptual meaningfulness. Thiem chooses to dismiss all but empirical consistency without providing convincing, or sometimes even any, arguments. In my view,—and notwithstanding Thiem’s (Reference Thiem2017) claim to the contrary (fn 3)—the only coherent critique at ESA would be to take the position of a QCA idealist and argue that separate tests of necessity are meaningless. Since Thiem (Reference Thiem2017) does not take a coherent idealist position and instead accepts the realist strategy of separate analyses for necessity and sufficiency, it follows that his argument must be evaluated by the rules and practices of QCA realists.
2.1 Empirical consistency
Thiem, against current practice, chooses 0.75 as the consistency threshold for necessary conditions, thus making the occurrence of supersets much more likely. Justifying his “harmonization” (p. 482), Thiem (Reference Thiem2017) misleadingly refers to Ragin (Reference Ragin2008) and Schneider and Wagemann (Reference Schneider and Wagemann2012). These authors explicitly argue that the consistency threshold should be considerably higher (at least 0.9) for necessity relations. Nowhere in Thiem (Reference Thiem2017) is it argued in which sense prevailing reasoning about choosing high consistency thresholds for necessary conditions is flawed and in need of being changed. Even from a QCA idealist position it sounds implausible to lower the consistency threshold to 0.75.
Even if one followed Thiem (Reference Thiem2017) in arguing that the sole decision-making criterion for identifying necessary conditions consists in whether they are supersets of the outcome, QCA realists argue that one should rely on additional information in order to assess the consistency of the set relation. For instance, Haesebrouck (Reference Haesebrouck2015) proposes a more conservative consistency measure. Likewise, if a set relation displays too many deviant cases consistency in kind (Schneider and Rohlfing Reference Schneider and Rohlfing2013), it should not be considered a necessary condition, even if it passes the consistency threshold.
In sum, QCA realists adhere to tougher consistency criteria than Thiem (Reference Thiem2017) and therefore tend to identify fewer necessary conditions. This, in turn, makes the alleged CONSOL effect less likely to occur.
2.2 Empirical relevance
Thiem (Reference Thiem2017) dismisses the so-called coverage, or relevance, score and only reports it (selectively) “for completeness” (p. 481). QCA realists, among them leading scholars such as Goertz (Reference Goertz2006), Ragin (Reference Ragin2006), or Mahoney, Kimball, and Koivu (Reference Mahoney, Kimball and Koivu2009) in turn, deem it an important ingredient for any set relational assessment. Empirically consistent supersets can be—and in applied QCA often are—empirically trivial when interpreted as necessary conditions. This can either be because the superset
$X$
is so much bigger than the outcome set
$Y$
and/or because
$X$
is so big that there is hardly any instance of
$\neg X$
, i.e.
$X$
is close to being a constant. The former source of trivialness is taken care of by the coverage score (Ragin Reference Ragin2006). The latter can be assessed by the Relevance of Necessity (RoN) score (Schneider and Wagemann Reference Schneider and Wagemann2012) and is of particular importance when disjunctions are tested. They tend to be big sets in which most (and sometimes all) cases under study are members of. It, of course, amounts to a trivial claim to argue that a (near constant)
$X$
is a meaningful necessary condition for any outcome
$Y$
—not least because that
$X$
would also qualify as a consistent enough superset of outcome
$\neg Y$
.
For illustration, consider Thiem’s (Reference Thiem2017) Lipset example. In his Table 2,
$D+\neg U$
is listed as a necessary disjunction for outcome
$S$
, with consistency 0.96 and coverage 0.51. However, that very same disjunction is also a superset of outcome
$\neg S$
, with consistency 0.95 and coverage 0.55.Footnote
3
In the presence of such simultaneous superset relations, Thiem’s position strikes me as untenable, according to which the same condition
$X$
has to be declared necessary for outcome
$Y$
and also outcome
$\neg Y$
. Such a fallacy is avoided if existing QCA realist standards are adhered to that recommend taking the RoN score into account. Expression
$D+\neg U$
has very low RoN scores both for
$S$
and
$\neg S$
and therefore should be rejected as a necessary condition for either outcome.
In sum, QCA realists pay attention to empirical trivialness and therefore impose tougher criteria for necessary conditions than Thiem (Reference Thiem2017). This, in turn, makes the CONSOL effect less likely to occur.
2.3 Conceptual meaningfulness
Thiem (Reference Thiem2017) does revert to an idealist position when he states that the identification of necessary conditions is a matter of Boolean algebra rather than social science reasoning: “[…] the formal definition of the Boolean operation of implication underlying every relation of necessity,
$P\leftarrow Q\equiv P+\neg Q\equiv \neg [\neg PQ]$
, neither entails anything about the structure of
$P$
, nor that some higher-order concept needs to be identified by the researcher that causally connects all disjuncts
$p_{1},p_{2},\ldots ,p_{j}$
of
$P$
to
$Q$
, […]” (p. 480). QCA realists do not follow this particular notion of necessity and, instead, argue that each single disjunct is a SUIN condition, that is, “[…] a sufficient but unnecessary part of a factor that is insufficient but necessary for an outcome” (Mahoney et al.
Reference Mahoney, Kimball and Koivu2009, p. 126). This means that without naming what that “factor” is, no meaningful claim of necessity can be made. QCA realists, in other words, argue that the necessary condition
$P$
is a higher-order concept and usually not known prior to the analysis. Rather it needs to be conceptualized by the researcher after empirically finding that disjuncts
$p_{1},p_{2},\ldots ,p_{j}$
form a superset of
$Q$
. Further, each
$p_{j}$
is conceptually connected to
$Q$
, not causally; they are definitional attributes of
$P$
.
Thiem (Reference Thiem2017) also dismisses the need for a theoretically meaningful higher-order concept by taking the QCA idealist position that “[…] in every QCA solution that exhibits perfect consistency and coverage, the disjunction of minimally sufficient conditions for the outcome is also minimally necessary for that outcome, without a unifying higher-order concept being required.” (p. 480) This formalistic argument is problematic when dealing with real social science theories and data. First, while some examples of compound necessary conditions exist,Footnote 4 I am not aware of any social science writing in which such a complex expression as a disjunction of minimally sufficient conjunctions is interpreted as a substantively meaningful necessary condition. Second, since the introduction of parameters of fit, very few applied QCA produce solution formulas that exhibit perfect consistency and coverage and applied QCA researchers therefore end up with disjunctions of sufficient conditions that do not qualify as necessary conditions (see Section 3 for empirical evidence). QCA realists have learned to live with this whereas the position of QCA idealists on this empirical fact is yet to be spelled out.
In sum, for QCA realists “[t]he idea that the researcher must ‘make sense’ of causal conditions identified as necessary […] is very important.” (Ragin Reference Ragin2000, p. 209). Because QCA realists only accept conceptually meaningful supersets as necessary conditions—be it single sets, disjunctions, or conjunctions—the CONSOL effect is less likely to occur.
2.4 Correctly interpreting the superSubset output
A point related to the conceptual meaningfulness of necessity claims is the way in which the output of the superSubset function is interpreted in Thiem (Reference Thiem2017). Thiem erroneously treats the output of the superSubset function as the product of sums when, instead, it simply lists alternative supersets of the outcome. Whenever superSubset returns something like
$A+B\leftarrow Y$
and also
$C+D\leftarrow Y$
, Thiem automatically accepts both disjunctions as necessary, thus de facto treating it as
$(A+B)\ast (C+D)\leftarrow Y$
. The negation of this expression—
$(\neg A\neg B)+(\neg C\neg D)$
—is a bigger set than just negating one of the two disjunctions. This, in turn, means more remainders need to be blocked from the analysis of sufficiency, which, in turn, means that the CONSOL effect is more likely to occur. This is a crucial, but incorrect, move in order to manufacture the alleged CONSOL effect. From an idealist position, it is incorrect simply because the very idea of exclusively looking for non-minimal supersets makes no sense.
From a QCA realist position several issues arise. Apart from the lack of conceptual meaningfulness, consider that for virtually all of the examples used in Thiem (Reference Thiem2017), it holds that the expression derived by first rewriting the expressions returned by function superSubset as a product of sums (as de facto done by Thiem) and then minimizing this to the simplest equivalent logical expression (not done by Thiem), yields an expression that most of the time is not a superset of the outcome anymore. The perhaps most glaring example comes from Thiem’s (Reference Thiem2017) analysis of the data from Fischer (Reference Fischer2015) with consistency set to 0.75. The expression formed by the eight disjunctionsFootnote
5
rewritten as the product of sums and then minimized simply yields the empty set
$\emptyset$
.
In sum, interpreting the output of the superSubset function as if all disjunctions needed to be retained and de facto combined by logical AND is surely wrong from a QCA realist perspective (and futile from a QCA idealist perspective). Once only meaningful supersets are postulated as necessary conditions, the CONSOL effect is less likely to occur.
3 Idealists and Realists in Practice
Now that the QCA realists’ standards for analyzing necessary conditions are put in place, how would the analysis of the Lipset data and the other examples look like? Would the CONSOL effect occur, i.e. would the use of ESA unavoidably lead to the conservative solution as the only possible viable solution? And, do QCA idealists propose an alternative that works?
A QCA idealist would not perform separate analyses of necessity and sufficiency because the most parsimonious solution derived from the logical minimization of the truth table reveals a minimally necessary disjunction of the minimally sufficient conjunctions. Does this work with the examples used in Thiem (Reference Thiem2017)?
Table 1 displays the most parsimonious solution for the Lipset example. Unlike Thiem (Reference Thiem2017), I also report the parameters of fit, consistency and coverage. As can be seen, while each conjunction passes the consistency threshold of 0.8, the coverage value for the solution formula is only 0.714. Since coverage sufficiency equals consistency necessity (Ragin Reference Ragin2006), this means that the consistency of the solution formula as a necessary disjunction for outcome
$S$
is 0.714. This is certainly too low for QCA realists—and QCA idealists should probably be even be stricter when it comes to inconsistencies. Whereas QCA realists conclude that there are sufficient, but no necessary conditions, the stance of QCA idealists vis-à-vis such common situations in applied QCA is not clear yet.
Most parsimonious solution, outcome
$S$
.
The Lipset example is not an exception. Using the same 21 data sets as in Thiem (Reference Thiem2017), I derive the most parsimonious solution formula for the outcome and for its negation with consistency thresholds of 0.9 and 0.75, respectively. This yields 84 different analyses. In order to fulfill the idealist’s vision in which a separate analysis of necessity is superfluous, a causally interpretable solution must fulfill at least three criteria: the disjunction must pass the necessity consistency threshold; each conjunction must pass the sufficiency consistency threshold; and there cannot be model ambiguity (Baumgartner and Thiem Reference Baumgartner and Thiem2015).
Out of the 84 analyses, only 17 (20%) pass all criteria.Footnote 6 This seems to show that, while from the perspective of a regularity theory of causation QCA is designed for finding a necessary disjunction of minimally sufficient conjunctions, in practice that is normally not achieved. Even for those instances in which all empirical hurdles are taken, the question still remains in which sense these complex necessity statements are supposed to relate to any existing social science theory on necessary conditions. Faced with this challenge, researcher can either abandon QCA or turn to a QCA realist approach with its separate analyses of necessity and sufficiency. This is what I am going to do now.Footnote 7
In his Table 2, Thiem (Reference Thiem2017) reports eight different supersets of the outcome that all pass the consistency threshold of 0.9.Footnote
8
QCA realists would not accept all supersets as being necessary just because they pass the consistency threshold. They would screen these supersets to see which one(s) pass the empirical and theoretical hurdles for being meaningful necessary conditions. In a first step, it is checked if any of them contains deviant cases coverage in kind. In the example at hand, it turns out that for none of the supersets
$X$
any case exists with
$X<0.5$
and
$Y>0.5$
(see replication material).
The next selection criterion consists in checking each superset’s empirical relevance as a necessary condition (RoN). As Table 2 shows, more than half of the conditions display low RoN scores (
${<}0.5$
). They are supersets of the outcome because they are very big sets. As a matter of fact, almost all disjunctions with low RoN exclusively display cases with fuzzy set membership higher than 0.5 (see replication material). Ignoring the size of a set (disjunction or atomic) is problematic. For illustration, consider the fact that disjunctions
$D+\neg U$
and
$\neg U+I$
, which Thiem (Reference Thiem2017) proposes to accept as necessary for outcome
$S$
simply because they pass his consistency threshold, also pass that consistency threshold for outcome
$\neg S$
. This phenomenon of simultaneous supersets is caused by the big size of the disjunctions in question. QCA realists would therefore reject all supersets with too low RoN.
Supersets, outcome
$S$
.
Only four expressions pass the empirical relevance test:
$G$
,
$L$
,
$LG$
and
$D+U+I$
. They still need to be scrutinized for their conceptual meaningfulness. For the sake of illustration, suppose disjunction
$D+U+I$
could stand for the higher-order concept of “modernized society”
$M$
. Assume the researcher makes the necessity claim
$M\leftarrow S$
, where
$M=D+U+I$
. In other words, higher-order concept
$M$
is the necessary condition and
$D$
,
$U$
, and
$I$
are SUIN conditions and functionally equivalent conceptual attributes of
$M$
. According to ESA, this necessity claim blocks all logical remainder rows that are subsets of
$\neg (D+U+I)=\neg D\neg U\neg I$
.
Based on this necessity claim, the ESA solution formula is
$D\neg I+UG\rightarrow S$
(Table 3). As can easily be seen, this formula is identical to the most parsimonious solution reported in Thiem (Reference Thiem2017). In other words, despite a claim of necessity involving a disjunction, ESA not only does not produce the conservative solution. In this example, it produces the most parsimonious solution.
ESA solution formulas, outcome
$S$
.
NB: conservative solution for the Lipset data:
$D\neg UL\neg IG+DULIG$
; most parsimonious solution:
$D\neg I+UG$
; SUIN and necessary conditions, respectively, in bold font.
Table 3 displays the ESA most parsimonious solutions that are obtained under each of the four necessity claims that are potentially feasible based on QCA realist standards. As can be seen, none of these ESA solutions is identical to the conservative solution—i.e., there clearly is no CONSOL effect of ESA.Footnote 9 ESA makes sure, instead, that any sufficiency solution formula contains all the postulated SUIN or atomic necessary conditionFootnote 10 and that no untenable assumptions on logical remainders are made.
On rare occasions and depending on the data at hand, in applied QCA it can, of course, happen that a QCA realist’s necessity claims only allow for the conservative solution in the analysis of sufficiency. Arguing that this is a design flaw of ESA is nothing else than beating the messenger. ESA just prevents researchers from making sufficiency claims that contradict their necessity claims—a laudable feature on which both QCA realists and idealists seem to agree.
4 Concluding Remarks—Idealists and Realists
The critique at ESA in Thiem (Reference Thiem2017) is unconvincing and incoherent both for QCA realists and idealists. From a QCA realist’s perspective, Thiem’s approach to the analysis of necessary conditions and thus his critique at ESA is unconvincing because it simply ignores virtually all of the existing standards and practices. From QCA idealist’s perspective, his critique is incoherent for the simple reason that the very search for only necessary conditions has no place in that approach.
I have demonstrated that the tools at hand for researchers who locate themselves in the realist QCA camp are working fine. A meaningful claim of necessity must be empirically consistent, empirically non-trivial, and theoretical meaningful. Constraining computer-based search functions via additional empirical and theoretical criteria is nothing unusual and certainly not a form of bias as Thiem (Reference Thiem2017) insinuates. This strategy usually does not yield the conservative solution, but if it does, then this is not a flaw of ESA but a feature of the data at hand.
QCA realists could, of course, ignore these and other intricacies of applied QCA and turn into idealists. Given the internal coherence and grounding in a regularity theory of causation, why has the idealist position not gained more traction so far? Why are most QCA researchers realists and perform separate analyses of necessity and sufficiency? Why are entire books written on necessary causes (usually single conditions) despite Thiem (Reference Thiem2017) intoning that “[…] necessity is no relational property of an object that is conferred onto it by the causal interpretation of a researcher, but only by the algebraic laws that define the operation of necessity in conjunction with the data in hand.” (p. 480)? And why do QCA scholars often base their analysis on the intermediate or conservative solution when idealists argue—convincingly, I find, from the perspective of a regularity theory of causation—that only the most parsimonious solution allows for causal inference? Idealists seem to suggest that all this is explained by the majority of QCA researchers’ incapacity to grasp what the search target of QCA is.
I, instead, believe that QCA realists are fully aware that by allowing for less-than-perfect set relations, relying on solution formulas other than the most parsimonious, by separating the analyses of necessity and sufficiency etc. they pay the prize of not exclusively relying on pure formal logic nor a pure version of a regularity theory of causation. Realists are willing to pay this prize in order to make QCA applicable to, and useful for, real social science research. It is still unclear how the idealist position in its current form is fruitfully applied to social science theories and data. Claiming that QCA both is only good for, and always manages to, identifying the minimally necessary disjunctions of minimally sufficient conjunctions seemingly strikes QCA realists as impractical, for several reasons and with various strategies that have been developed in response.
First, most researchers know that the conditions under which a truth table analysis yields a causally interpretable minimally necessary disjunction of sufficient conjunctions are rarely ever met in applied QCA. QCA idealists would need to explain what researchers ought to do if and when consistency and coverage are (far) less than perfect.Footnote 11 Second, QCA realists are probably also waiting for the idealists’ take on the glaring gap between social science theories on necessary conditions, which usually are about single conditions or simpler disjunctions, on the one hand, and the QCA idealist position that usually very complex expressions—disjunctions of conjunctions—are to be interpreted as the necessary conditions, on the other.
Third, most scholars with some experience in social science research do know that drawing causal inference based on observational data is a tall order—also, of course, in QCA. QCA idealists strongly emphasize that within a regularity theory of causation framework the redundant-free most parsimonious solution (with perfect consistency and coverage) is one necessary element for causal inference.Footnote 12 Since, however, this is hardly ever found in real data, QCA realists adopt various strategies in response to noisy data.Footnote 13 For instance, some correctly point out that a QCA-based study can serve purposes other than drawing causal inference (e.g. Berg-Schlosser, De Meur, and Rihoux Reference Berg-Schlosser, De Meur, Rihoux, Rihoux and Ragin2009). Others adhere to the goal of causal inference but point out that the logical minimization of a truth table based on observational data at the cross-case level can only be one element and is limited to establishing the presence of a causal effect. The (heavier) lifting will need to be done at the within-case level and is there to probe the presence of a causal mechanism. In the absence of the latter, no QCA solution formula, redundant-free or not, should count as causal. If the goal is to draw causal inference, it strikes me as fruitful to spell out the logic of how researchers should combine their QCA with follow-up case studies (e.g. Schneider and Rohlfing Reference Schneider and Rohlfing2013, Reference Schneider and Rohlfing2016). Since, however, QCA realists who pursue causal inference cannot fully rely on a regularity theory of causation as their common ground, they need to spell out which other theory, or theories, of causation could better undergird QCA as applied in the social sciences.Footnote 14
ESA, just like many other methodological proposals made by QCA realists, are developed based on the trivial insights that the data to be analysed comes short of the ideal scenarios of pure Boolean algebra and set theory and, more generally, that methodological tools need to align with social science theories and research questions. Unless QCA idealists come forth with a coherent set of proposals that is tested in real research rather than just talked about, applied QCA researchers seem better advised to adhere to, and to work on improving, the QCA realist standards and practices.
Supplementary material
For supplementary material accompanying this paper, please visit https://doi.org/10.1017/pan.2017.45.
