3.1 The Phenomenological Step

In the phenomenological step, researchers observe associated events, without exploring the reason for that association. At this step, it suffices to discover that $P\left[X=x,Y=y\right]\ne P\left[X=x\right]P\left[Y=y\right]$ . Further, a researcher may model the joint distribution $P\left[X=x,Y=y\right]$ , derive conditional probabilities $P[Y=y|X=x]$ , and make associational statements of the type $E[Y|X=x]=y$ (an associational prediction) with the help of machine learning tools. Exceptionally, a researcher may go as far as to produce empirical evidence of a causal effect, such as the result from an interventional study (e.g., Ohm’s law of current, Newton’s law of universal gravitation, or Coulomb’s law of electrical forces), but without providing an explanation for the relationship. The main goal of the phenomenological step is to state “a problem situation,” in the sense of describing the observed anomaly for which no scientific explanation exists (Reference PopperPopper 1994b, pp. 2–3). At this step, inference occurs by logical induction, because the problem situation rests on the conclusion that, for some unknown reason, the phenomenon will reoccur.Footnote ⁷

For instance, a researcher may observe that the bid-ask spread of stocks widens in the presence of imbalanced orderflow (i.e., when the amount of shares exchanged in trades initiated by buyers does not equal the amount of shares exchanged in trades initiated by sellers over a period of time), and that the widening of bid-ask spreads often precedes a rise in intraday volatility. This is a surprising phenomenon because under the efficient market hypothesis asset prices are expected to reflect all available information at all times, making predictions futile (Reference FamaFama 1970). The existence of orderflow imbalance, the sequential nature of these events, and their predictability point to market inefficiencies, of unclear source. Such associational observations do not constitute a theory, and they do not explain why the phenomenon occurs.

3.2 The Theoretical Step

In the theoretical step, researchers advance a possible explanation for the observed associated events. This is an exercise in logical abduction (sometimes also called retroduction): Given the observed phenomenon, the most likely explanation is inferred by elimination among competing alternatives. Observations cannot be explained by a hypothesis more extraordinary than the observations themselves, and of various hypotheses the least extraordinary must be preferred (Reference Wieten, Bex, Prakken, Renooij, Prakken, Bistarelli, Santini and TaticchiWieten et al. 2020). At this step, a researcher states that $X$ and $Y$ are associated because $X$ causes $Y$ , in the sense that $P\left[Y=y|do[X=x]\right]>P\left[Y=y\right]$ . For the explanation to be scientific, it must propose a causal mechanism that is falsifiable, that is, propose the system of structural equations along the causal path from $X$ to $Y$ , where the validity of each causal link and causal path can be tested empirically.Footnote ⁸ Physics Nobel Prize laureate Wolfgang Pauli famously remarked that there are three types of explanations: correct, wrong, and not even wrong (Reference PeierlsPeierls 1992). With “not even wrong,” Pauli referred to explanations that appear to be scientific, but use unfalsifiable premises or reasoning, which can never be affirmed nor denied.

A scientist may propose a theory with the assistance of statistical tools (see Section 4.3.1), however data and statistical tools are not enough to produce a theory. The reason is, in the theoretical step the scientist injects extra-statistical information, in the form of a subjective framework of assumptions that give meaning to the observations. These assumptions are unavoidable, because the simple action of taking and interpreting measurements introduces subjective choices, making the process of discovery a creative endeavor. If theories could be deduced directly from observations, then there would be no need for experiments that test the validity of the assumptions.

Following on the previous example, the Probability of Informed Trading (PIN) theory explains liquidity provision as the result of a sequential strategic game between market makers and informed traders (Reference Easley, Kiefer, O’Hara and PapermanEasley et al. 1996). In the absence of informed traders, the orderflow is balanced, because uninformed traders initiate buys and sells in roughly equal amounts, hence market impact is mute and the mid-price barely changes. When market makers provide liquidity to uninformed traders, they profit from the bid-ask spread (they buy at the bid price and sell at the ask price). However, the presence of informed traders imbalances the orderflow, creating market impact that changes the mid-price. When market makers provide liquidity to an informed trader, the mid-price changes before market makers are able to profit from the bid-ask spread, and they are eventually forced to realize a loss. As a protection against losses, market makers react to orderflow imbalance by charging a greater premium for selling the option to be adversely selected (that premium is the bid-ask spread). In the presence of persistent orderflow imbalance, realized losses accumulate, and market makers are forced to reduce their provision of liquidity, which results in greater volatility. Two features make the PIN theory scientific: First, it describes a precise mechanism that explains the causal link: orderflow imbalance $\to$ market impact $\to$ mid-price change $\to$ realized losses $\to$ bid-ask spread widening $\to$ reduced liquidity $\to$ greater volatility. Second, the mechanism involves measurable variables, with links that are individually testable. An unscientific explanation would not propose a mechanism, or it would propose a mechanism that is not testable.

Mathematicians use the term theory with a different meaning than scientists. A mathematical theory is an area of study derived from a set of axioms, such as number theory or group theory. Following Kant’s epistemological definitions, mathematical theories are synthetic a priori logical statements, whereas scientific theories are synthetic a posteriori logical statements. This means that mathematical theories do not admit empirical evidence to the contrary, whereas scientific theories must open themselves to falsification.

3.3 The Falsification Step

In the falsification step, researchers not involved in the formulation of the theory independently: (i) deduce key implications from the theory, such that it is impossible for the theory to be true and the implications to be false; and (ii) design and execute experiments with the purpose of proving that the implications are false. Step (i) is an exercise in logical deduction because given some theorized premises, a falsifiable conclusion is reached reductively (Reference GenslerGensler 2010, pp. 104–110). When properly done, performing step (i) demands substantial creativity and domain expertise, as it must balance the strength of the deduced implication with its testability (cost, measurement errors, reproducibility, etc.). Each experiment in step (ii) focuses on falsifying one particular link in the chain of events involved in the causal mechanism, applying the tools of mediation analysis. The conclusion that the theory is false follows the structure of a modus tollens syllogism (proof by contradiction): using standard sequent notation, if $A\Rightarrow B$ , however $\neg B$ is observed, then $\neg A$ , where $A$ stands for “the theory is true” and $B$ stands for a falsifiable key implication of the theory.

One strategy of falsification is to show that $P\left[Y=y|do[X=x]\right]=P\left[Y=y\right]$ , in which case either the association is noncausal, or there is no association (i.e., the phenomenon originally observed in step (i) was a statistical fluke). A second strategy of falsification is to deduce a causal prediction from the proposed mechanism, and to show that $E\left[Y|do[X=x]\right]\ne y$ . When that is the case, there may be a causal mechanism, however, it does not work as theorized (e.g., when the actual causal graph is more complex than the one proposed). A third strategy of falsification is to deduce from the theorized causal mechanism the existence of associations, and then apply machine learning techniques to show that those associations do not exist. Unlike the first two falsification strategies, the third one does not involve a do-operation.

Following on the previous example, a researcher may split a list of stocks randomly into two groups, send buy orders that set the level of orderflow imbalance for the first group, and measure the difference in bid-ask spread, liquidity, and volatility between the two groups (an interventional study, see Section 4.1).Footnote ⁹ In response to random spikes in orderflow imbalance, a researcher may find evidence of quote cancellation, quote size reduction, and resending quotes further away from the mid-price (a natural experiment, see Section 4.2).Footnote ¹⁰ If the experimental evidence is consistent with the proposed PIN theory, the research community concludes that the theory has (temporarily) survived falsification. Furthermore, in some cases a researcher might be able to inspect the data-generating process directly, in what I call a “field study.” A researcher may approach profitable market makers and examine whether their liquidity provision algorithms are designed to widen the bid-ask spread at which they place quotes when they observe imbalanced order flow. The same researcher may approach less profitable market makers and examine whether their liquidity provision algorithms do not react to order flow imbalance. Service providers are willing to offer this level of disclosure to key clients and regulators. This field study may confirm that market makers who do not adjust their bid-ask spread in presence of orderflow imbalance succumb to Darwinian competition, leaving as survivors those whose behavior aligns with the PIN theory.

Popper gave special significance to falsification through “risky forecasts,” that is, forecasts of outcomes $y\text{'}$ under yet unobserved interventions $x\text{'}$ (Reference Vignero and WenmackersVignero and Wenmackers 2021). Mathematically, this type of falsification is represented by the counterfactual expression $E\left[Y_{X=x^{\prime }}|X=x,Y=y\right]\ne y\text{'}$ , namely the expected value of $Y$ in an alternative universe where $X$ is set to $x\text{'}$ (a do-operation) for the subset of observations where what actually happened is $X=x$ and $Y=y$ .Footnote ¹¹ Successful theories answer questions about previously observed events, as well as never-before observed events. To come up with risky forecasts, an experiment designer scrutinizes the theory, deducing its ultimate implications under hypothetical $x\text{'}$ , and then searches or waits for them. Because the theory was developed during the theoretical step without knowledge of $\left(x\text{'},y\text{'}\right)$ , this type of analysis constitutes an instance of out-of-sample assessment. For example, the PIN theory implied the possibility of failures in the provision of liquidity approximately fourteen years before the flash crash of 2010 took place. Traders who had implemented liquidity provision models based on the PIN theory (or better, its high-frequency embodiment, VPIN) were prepared for that black-swan and profited from that event (Reference Easley, Prado and O’HaraEasley et al. 2010, Reference Easley, Prado and O’Hara2012, Reference López de PradoLópez de Prado 2018, pp. 281–300), at the expense of traders who relied on weaker microstructural theories.

3.4 Demarcation and Falsificationism in Statistics

Science is essential to human understanding in that it replaces unreliable inductive reasoning (such as “ $Y$ will follow $X$ because that is the association observed in the past”) with more reliable deductive reasoning (such as “ $Y$ will follow $X$ because $X$ causes $Y$ through a tested mechanism $M$ ”). Parsimonious theories are preferable, because they are easier to falsify, as they involve controlling for fewer variables (Occam’s razor). The most parsimonious surviving theory is not truer, however, it is better “fit” (in an evolutionary sense) to tackle more difficult problems posed by that theory. The most parsimonious surviving theory poses new problem situations, hence re-starting a new iteration of the three-step process, which will result in a better theory yet.

To appreciate the unique characteristics of the scientific method, it helps to contrast it with a dialectical predecessor. For centuries prior to the scientific revolution of the seventeenth century, academics used the Socratic method to eliminate logically inconsistent hypotheses. Like the scientific method, the Socratic method relies on three steps: (1) problem statement; (2) hypothesis formulation; and (3) elenchus (refutation), see Reference VlastosVlastos (1983, pp. 27–58). However, both methods differ in three important aspects. First, a Socratic problem statement is a definiendum (“what is $X$ ?”), not an observed empirical phenomenon (“ $X$ and $Y$ are associated”). Second, a Socratic hypothesis is a definiens (“ $X$ is …”), not a falsifiable theory (“ $X$ causes $Y$ through mechanism $M$ ”). Third, a Socratic refutation presents a counterexample that exposes implicit assumptions, where those assumptions contradict the original definition. In contrast, scientific falsification does not involve searching for contradictive implicit assumptions, since all assumptions were made explicit and coherent by a plausible causal mechanism. Instead, scientific falsification designs and executes an experiment aimed at debunking the theorized causal effect (“ $X$ does not cause $Y$ ”), or showing that the experiment’s results contradict the hypothesized mechanism (“experimental results contradict $M$ ”).Footnote ¹²

The above explanation elucidates an important fact that is often ignored or misunderstood: not all academic debate is scientific, even in empirical or mathematical subjects. A claim does not become scientific by virtue of its use of complex mathematics, its reliance on measurements, or its submission to peer review.Footnote ¹³ Philosophers of science call the challenge of separating scientific claims from pseudoscientific claims the “demarcation problem.” Popper, Kuhn, Lakatos, Musgrave, Thagard, Laudan, Lutz, and many other authors have proposed different demarcation principles. While there is no consensus on what constitutes a definitive demarcation principle across all disciplines, modern philosophers of science generally agree that, for a theory to be scientific, it must be falsifiable in some wide or narrow sense.Footnote ¹⁴

The principle of falsification is deeply ingrained in statistics and econometrics (Reference Dickson, Baird, Bandyopadhyay and ForsterDickson and Baird 2011). Frequentist statisticians routinely use Fisher’s p-values and Neyman–Pearson’s framework for falsifying a proposed hypothesis ( $H_0$ ), following a hypothetico-deductive argument of the form (using standard sequent notation):

$H_0\Rightarrow P\left[data|H_0\right]\ge \alpha ;P\left[data|H_0\right]<\alpha ⊢\neg H_0,$(1)

where $data$ denotes the observation made and $\alpha$ denotes the targeted false positive rate (Reference PerezgonzalezPerezgonzalez 2017). The above proposition is analogous to a modus tollens syllogism, with the caveat that $H_0$ is not rejected with certainty, as it would be the case in a mathematical proof. For this reason, this proposition is categorized as a stochastic proof by contradiction, where certainty is replaced by a preset confidence level (Reference ImaiImai 2013; Reference Balsubramani and RamdasBalsubramani and Ramdas 2016). Failure to reject $H_0$ does not validate $H_0$ , but rather attests that there is not sufficient empirical evidence to cast significant doubt on the truth of $H_0$ (Reference Reeves and BrewerReeves and Brewer 1980).Footnote ¹⁵ Accordingly, the logical structure of statistical hypothesis testing enforces a Popperian view of science in quantitative disciplines, whereby a hypothesis can never be accepted, but it can be rejected (i.e., falsified), see Wilkinson (2013). Popper’s influence is also palpable in Bayesian statistics, see Reference Gelman and Rohilla-ShaliziGelman and Rohilla-Shalizi (2013).

Statistical falsification can be applied to different types of claims. For the purpose of this Element, it is helpful to differentiate between the statistical falsification of: (a) associational claims; and (b) causal claims. The statistical falsification of associational claims occurs during the phenomenological step of the scientific method (e.g., when a researcher finds that “ $X$ is correlated with $Y$ ”), and it can be done on the sole basis of observational evidence. The statistical falsification of causal claims may also occur at the phenomenological step of the scientific method (e.g., when a laboratory finds that “ $X$ causes $Y$ ” in the absence of any theory to explain why), or at the falsification step of the scientific method (involving a theory, of the form “ $X$ causes $Y$ through a mechanism $M$ ”), but either way the statistical falsification of a causal claim always requires an experiment.Footnote ¹⁶ Most statisticians and econometricians are trained in the statistical falsification of associational claims and have a limited understanding of the statistical falsification of causal claims in general, and the statistical falsification of causal theories in particular. The statistical falsification of causal claims requires the careful design of experiments, and the statistical falsification of causal theories requires testing the hypothesized causal mechanism, which in turn requires testing independent effects along the causal path. The next section delves into this important topic.

4.1 Interventional Studies

In a controlled experiment, scientists assess causality by observing the effect on $Y$ of changing the values of $X$ while keeping constant all other variables in the system (a do-operation). Hasan Ibn al-Haytham (965–1040) conducted the first recorded controlled experiment in history, in which he designed a camera obscura to manipulate variables involved in vision. Through various ingenious experiments, Ibn al-Haytham showed that light travels in a straight line, and that light reflects from the observed objects to the observer’s eyes, hence falsifying the extramission theories of light by Ptolemy, Galen, and Euclid (Reference ToomerToomer 1964). This example illustrates a strong prerequisite for conducting a controlled experiment: the researcher must have direct control of all the variables involved in the data-generating process. When that is the case, the ceteris paribus condition is satisfied, and the difference in $Y$ can be attributed to the change in $X$ .

When some of the variables in the data-generating process are not under direct experimental control (e.g., the weather in the drownings example), the ceteris paribus condition cannot be guaranteed. In that case, scientists may execute a randomized controlled trial (RCT), whereby members of a population (called units or subjects) are randomly assigned either to a treatment or to a control group. Such random assignment aims to create two groups that are as comparable as possible, so that any difference in outcomes can be attributed to the treatment. In an RCT, the researcher carries out the do-operation on two random samples of units, rather than on a particular unit, hence enabling a ceteris paribus comparison. The randomization also allows the researcher to quantify the experiment’s uncertainty via Monte Carlo, by computing the standard deviation on ATEs from different subsamples. Scientists may keep secret from participants (single-blind) and researchers (double-blind) which units belong to each group, in order to further remove subject and experimenter biases. For additional information, see Reference Hernán and RobinsHernán and Robins (2020) and Reference Kohavi, Tang, Xu, Hemkens and IoannidisKohavi et al. (2020).

We can use the earlier characterization of the fundamental problem of causal inference to show how random assignment achieves its goal. Consider the situation where a researcher assigns units randomly to $X=x_0$ (control group) and $X=x_1$ (treatment group). Following with the earlier example, this is equivalent to tossing a coin at the beginning of every month, then setting $X=x_0$ (low ice cream sales) on heads and setting $X=x_1$ (high ice cream sales) on tails. Because the intervention on $X$ was decided at random, units in the treatment group are expected to be undistinguishable from units in the control group, hence

$E\left[Y_{X=x_0}|X=x_1\right]=E\left[Y_{X=x_0}|X=x_0\right]=E\left[Y_{X=x_0}\right]=E\left[Y|do\left[X=x_0\right]\right]$(4)

$E\left[Y_{X=x_1}|X=x_1\right]=E\left[Y_{X=x_1}|X=x_0\right]=E\left[Y_{X=x_1}\right]=E\left[Y|do\left[X=x_1\right]\right].$(5)

Random assignment makes $Y_{X=x_0}$ and $Y_{X=x_1}$ independent of the observed $X$ . The implication from the first equation above is that $\text{SSB}=0$ . In the drownings example, $E\left[Y_{X=x_0}|X=x_1\right]=E\left[Y_{X=x_0}|X=x_0\right]$ , because suppressing ice cream sales would have had the same expected outcome ( $E\left[Y_{X=x_0}\right]$ ) on both, high sales months and low sales months, since the monthly sales were set at random to begin with (irrespective of the weather).

In conclusion, under random assignment, the observed difference matches both ATT and ATE:

$\begin{array}{l}E\left[Y|X=x_1\right]-E\left[Y|X=x_0\right]\\ =\underset{\text{ATT}}{\underbrace{E\left[Y_{X=x_1}|X=x_1\right]-E\left[Y_{X=x_0}|X=x_1\right]}}+\underset{\text{SSB}}{\underbrace{E\left[Y_{X=x_0}|X=x_1\right]-E\left[Y_{X=x_0}|X=x_0\right]}}\\ =\underset{\text{ATE}}{\underbrace{E\left[Y|do\left[X=x_1\right]\right]-E\left[Y|do\left[X=x_0\right]\right]}}+\underset{\text{SSB}=0}{\underbrace{E\left[Y|do\left[X=x_0\right]\right]-E\left[Y|do\left[X=x_0\right]\right]}}\\ =\text{ATE}\text{.}\end{array}$(6)

4.2 Natural Experiments

Sometimes interventional studies are not possible, because they are unfeasible, unethical, or prohibitively expensive. Under those circumstances, scientists may resort to natural experiments or simulated interventions. In a natural experiment (also known as a quasi-experiment), units are assigned to the treatment and control groups determined randomly by Nature or by other factors outside the influence of scientists (Reference DunningDunning 2012). Although natural experiments are observational (as opposed to interventional, like controlled experiments and RCT) studies, the fact that the assignment of units to groups is assumed random enables the attribution of the difference in outcomes to the treatment. Put differently, Nature performs the do-operation, and the researcher’s challenge is to identify the two random groups that enable a ceteris paribus comparison. Common examples of natural experiments include (1) regression discontinuity design (RDD); (2) crossover studies (COSs); and (3) difference-in-differences (DID) studies. Case–control studies,Footnote ¹⁷ cohort studies,Footnote ¹⁸ and synthetic control studiesFootnote ¹⁹ are not proper natural experiments because there is no random assignment of units to groups.

Regression discontinuity design studies compare the outcomes of: (a) units that received treatment because the value of an assignment variable fell barely above a threshold; and (b) units that escaped treatment because the value of an assignment variable fell barely below a threshold. The critical assumption behind RDD is that groups (a) and (b) are comparable in everything but the slight difference in the assignment variable, which can be attributed to noise, hence the difference in outcomes between (a) and (b) is the treatment effect. For further reading, see Reference Imbens and LemieuxImbens and Lemieux (2008).

A COS is a longitudinal study in which the exposure of units to a treatment is randomly removed for a time, and then returned. COS assumes that the effect of confounders does not change per unit over time. When that assumption holds, COSs have two advantages over standard longitudinal studies. First, in a COS the influence of confounding variables is reduced by each unit serving as its own control. Second, COS are statistically efficient, as they can identify causal effects in smaller samples than other studies. COS may not be appropriate when the order of treatments affects the outcome (order effects). Sufficiently long wash-out periods should be observed between treatments, to avoid that past treatments confound the estimated effects of new treatments (carryover effects). COS can also have an interventional counterpart, when the random assignment is under the control of the researcher. To learn more, see Reference Jones and KenwardJones and Kenward (2003).

When factors other than the treatment influence the outcome over time, researchers may apply a pre-post with-without comparison, called a DID study. In a DID study, researchers compare two differences: (i) the before-after difference in outcomes of the treatment group; and (ii) the before-after difference in outcomes of the control group (where the random assignment of units to groups is done by Nature). By computing the difference between (i) and (ii), DID attempts to remove from the treatment effect (i) all time-varying factors captured by (ii). DID relies on the “equal-trends assumption,” namely that no time-varying differences exist between treatment and control groups. The validity of the equal-trends assumption can be assessed in a number of ways. For example, researchers may compute changes in outcomes for the treatment and control groups repeatedly before the treatment is actually administered, so as to confirm that the outcome trends move in parallel. For additional information, see Reference Angrist and PischkeAngrist and Pischke (2008, pp. 227–243).

4.3 Simulated Interventions

The previous sections explained how interventional studies and natural experiments use randomization to achieve the ceteris paribus comparisons that result in $\text{SSB}=0$ . Each approach demanded stronger assumptions than the previous one, with the corresponding cost in terms of generality of the conclusions. For instance, the conclusions from a controlled experiment are more general than the conclusions from an RCT, because in the former researchers control the variables involved in the data-generating process in such a way that ceteris paribus comparisons are clearer. Likewise, the conclusions from an RCT are more general than the conclusions from a natural experiment, because in an RCT the researcher is in control of the random assignment, and the researcher performs the do-operation.

In recent decades, the field of causal inference has added one more tool to the scientific arsenal: when interventional studies and natural experiments are not possible, researchers may still conduct an observational study that simulates a do-operation, with the help of a hypothesized causal graph. The hypothesized causal graph encodes the information needed to remove from observations the SSB introduced by confounders, under the assumption that the causal graph is correct. The price to pay is, as one might have expected, accepting stronger assumptions that make the conclusions less general, but still useful.

Simulated interventions have two main applications: First, subject to a hypothesized causal graph, a simulated intervention allows researchers to estimate the strength of a causal effect from observational studies. Second, a simulated intervention may help falsify a hypothesized causal graph, when the strength of one of the effects posited by the graph is deemed statistically insignificant (once again, a modus tollens argument, see Section 3.4).

It is important to understand the difference between establishing a causal claim and falsifying a causal claim. Through interventional studies and natural experiments, subject to some assumptions, a researcher can establish or falsify a causal claim without knowledge of the causal graph. For this reason, they are the most powerful tools in causal inference. In simulated interventions, the causal graph is part of the assumptions, and one cannot prove what one is assuming. The most a simulated intervention can achieve is to disprove a hypothesized causal graph, by finding a contradiction between an effect claimed by a graph and the effect estimated with the help of that same graph. This power of simulated interventions to falsify causal claims can be very helpful in discovering through elimination the causal structure hidden in the data.

4.3.2 Do-Calculus

Do-calculus is a complete axiomatic system that allows researchers to estimate do-operators by means of conditional probabilities, where the necessary and sufficient conditioning variables can be determined with the help of the causal graph (Reference Shpitser and PearlShpitser and Pearl 2006). The following sections review some notions of do-calculus needed to understand this Element. I encourage the reader to learn more about these important concepts in Reference PearlPearl (2009), Reference Pearl, Glymour and JewellPearl et al. (2016), and Reference NealNeal (2020).

4.3.2.1 Blocked Paths

In a graph with three variables $\left\{X,Y,Z\right\}$ , a variable $Z$ is a confounder with respect to $X$ and $Y$ when the causal relationships include a structure X ← Z → Y. A variable $Z$ is a collider with respect to $X$ and $Y$ when the causal relationships are reversed, that is, X → Z ← Y. A variable $Z$ is a mediator with respect to $X$ and $Y$ when the causal relationships include a structure X → Z → Y.Footnote ²⁰

A path is a sequence of arrows and nodes that connect two variables $X$ and $Y$ , regardless of the direction of causation. A directed path is a path where all arrows point in the same direction. In a directed path that starts in $X$ and ends in $Z$ , $X$ is an ancestor of $Z$ , and $Z$ is a descendant of $X$ . A path between $X$ and $Y$ is blocked if either: (1) the path traverses a collider, and the researcher has not conditioned on that collider or its descendants; or (2) the researcher conditions on a variable in the path between $X$ and $Y$ , where the conditioned variable is not a collider. Association flows along any paths between $X$ and $Y$ that are not blocked. Causal association flows along an unblocked directed path that starts in treatment $X$ and ends in outcome $Y$ , denoted the causal path. Association implies causation only if all noncausal paths are blocked. This is the deeper explanation of why association does not imply causation, and why causal independence does not imply statistical independence.

Two variables $X$ and $Y$ are d-separated by a (possibly empty) set of variables $S$ if, upon conditioning on $S$ , all paths between $X$ and $Y$ are blocked. The set $S$ d-separates $X$ and $Y$ if and only if $X$ and $Y$ are conditionally independent given $S$ . For a proof of this statement, see Reference Koller and FriedmanKoller and Friedman (2009, chapter 3). This important result, sometimes called the global Markov condition in Bayesian network theory,Footnote ²¹ allows researchers to assume that $\text{SSB}=0$ , and estimate ATE as

$\text{ATE}=E\left[Y|do\left[X=x_1\right]\right]-E\left[Y|do\left[X=x_0\right]\right]=E\left[E\left[Y|S,X=x_1\right]-E\left[Y|S,X=x_0\right]\right].$(7)

The catch is, deciding which variables belong in $S$ requires knowledge of the causal graph that comprises all the paths between $X$ and $Y$ . Using the above concepts, it is possible to define various specific controls for confounding variables, including: (a) the backdoor adjustment; (b) the front-door adjustment; and (c) the method of instrumental variables (Reference PearlPearl 2009). This is not a comprehensive list of adjustments, and I have selected these three adjustments in particular because I will refer to them in the sections ahead.

4.3.2.2 Backdoor Adjustment

A backdoor path between $X$ and $Y$ is an unblocked noncausal path that connects those two variables. The term backdoor is inspired by the fact that this kind of paths have an arrow pointing into the treatment ( $X$ ). For example, Figure 2 (left) contains a backdoor path (colored in red, Y ← Z → X), and a causal path (colored in green, Y → X). Backdoor paths can be blocked by conditioning on a set of variables $S$ that satisfies the backdoor criterion. The backdoor criterion is useful when controlling for observable confounders.Footnote ²²

Figure 2

Example of a causal graph that satisfies the backdoor criterion, before (left) and after (right) conditioning on Z (shaded node)

A set of variables $S$ satisfies the backdoor criterion with regards to treatment $X$ and outcome $Y$ if the following two conditions are true: (i) conditioning on $S$ blocks all backdoor paths between $X$ and $Y$ ; and (ii) $S$ does not contain any descendants of $X$ . Then, $S$ is a sufficient adjustment set, and the causal effect of $X$ on $Y$ can be estimated as:

$P\left[Y=y|do\left[X=x\right]\right]={\displaystyle \sum }_{s}P[Y=y|X=x,S=s]P\left[S=s\right].$(8)

Intuitively, condition (i) blocks all noncausal paths, while condition (ii) keeps open all causal paths. In Figure 2, the only sufficient adjustment set $S$ is $\left\{Z\right\}$ . Set $S$ is sufficient because conditioning on $Z$ blocks that backdoor path Y ← Z → X, and $Z$ is not a descendant of $X$ . The result is that the only remaining association is the one flowing through the causal path, thus adjusting the observations in a way that simulates a do-operation on $X$ . In general, there can be multiple sufficient adjustment sets that satisfy the backdoor criterion for any given graph.

4.3.2.3 Front-Door Adjustment

Sometimes researchers may not be able to condition on a variable that satisfies the backdoor criterion, for instance when that variable is latent (unobservable). In that case, under certain conditions, the front-door criterion allows researchers to estimate the causal effect with the help of a mediator.

A set of variables $S$ satisfies the front-door criterion with regards to treatment $X$ and outcome $Y$ if the following three conditions are true: (i) all causal paths from $X$ to $Y$ go through $S$ ; (ii) there is no backdoor path between $X$ and $S$ ; (iii) all backdoor paths between $S$ and $Y$ are blocked by conditioning on $X$ . Then, $S$ is a sufficient adjustment set, and the causal effect of $X$ on $Y$ can be estimated as:

$P\left[Y=y|do\left[X=x\right]\right]={\displaystyle \sum }_{s}P[S=s|X=x]{\displaystyle \sum }_{x\text{'}}P[Y=y|S=s,X=x\text{'}]P\left[X=x\text{'}\right].$(9)

Intuitively, condition (i) ensures that $S$ completely mediates the effect of $X$ on $Y$ , condition (ii) applies the backdoor criterion on X → S, and condition (iii) applies the backdoor criterion on S → Y.

Figure 3 provides an example of a causal graph with a latent variable $Z$ (represented as a dashed oval) that confounds the effect of $X$ on $Y$ . There is a backdoor path between $X$ and $Y$ (colored in red, Y ← Z → X), and a causal path (colored in green, X → M → Y). The first condition of the backdoor criterion is violated (it is not possible to condition on $Z$ ), however $S=\left\{M\right\}$ satisfies the front-door criterion, because $M$ mediates the only causal path (X → M → Y), the path between $X$ and $M$ is blocked by collider $Y$ (M → Y ← Z → X), and conditioning on $X$ blocks the backdoor path between $M$ and $Y$ (Y ← Z → X → M). The adjustment accomplishes that the only remaining association is the one flowing through the causal path.

Figure 3

Example of a causal graph that satisfies the front-door criterion, before (top) and after (bottom) adjustment

4.3.2.4 Instrumental Variables

The front-door adjustment controls for a latent confounder when a mediator exists. In the absence of a mediator, the instrumental variables method allows researchers to control for a latent confounder $Z$ , as long as researchers can find a variable $W$ that turns $X$ into a collider, thus blocking the backdoor path through $Z$ .

A variable $W$ satisfies the instrumental variable criterion relative to treatment $X$ and outcome $Y$ if the following three conditions are true: (i) there is an arrow W → X; (ii) the causal effect of $W$ on $Y$ is fully mediated by $X$ ; and (iii) there is no backdoor path between $W$ and $Y$ .

Intuitively, conditions (i) and (ii) ensure that $W$ can be used as a proxy for $X$ , whereas condition (iii) prevents the need for an additional backdoor adjustment to de-confound the effect of $W$ on $Y$ . Figure 4 provides an example of a causal graph with a latent variable $Z$ that confounds the effect of $X$ on $Y$ . There is a backdoor path between $X$ and $Y$ (colored in red, Y ← Z → X), and a causal path (colored in green, X → Y). The first condition of the backdoor criterion is violated (it is not possible to condition on $Z$ ), and the first condition of the front-door criterion is violated (there is no mediator between $X$ and $Y$ ). Variable $W$ is an instrument, because there is an arrow W → X (arrow number 4), $X$ mediates the only causal path from $W$ to $Y$ (W → X → Y), and there is no backdoor path between $W$ and $Y$ .

Figure 4

Example of a causal graph with an instrumental variable $W$ , before (top) and after (bottom) adjustment

Assuming that Figure 4 represents a linear causal model, the coefficient $\frac{cov\left[X,Y\right]}{cov\left[X,X\right]}$ provides a biased estimate of the effect X → Y, due to the confounding effect of $Z$ . To estimate the unconfounded coefficient of effect X → Y, the instrumental variables method estimates first the coefficient of the effect W → X → Y as the slope of the regression line of $Y$ on $W$ , $r_{YW}=\frac{cov\left[Y,W\right]}{cov\left[W,W\right]}$ , which is the product of coefficients of effects (3) and (4) in Figure 4. The coefficient of effect (4) can be estimated from the slope of the regression line of $X$ on $W$ , $r_{XW}=\frac{cov\left[W,X\right]}{cov\left[W,W\right]}$ . Finally, the adjusted (unconfounded) coefficient of effect X → Y can be estimated as $\frac{r_{YW}}{r_{XW}}$ . For further reading, see Reference Hernán and RobinsHernán and Robins (2020, chapter 16).

5.1 Authors often Mistake Causality for Association

First, consider the joint distribution of $\left(X,Y\right)$ , and the standard econometric model, $Y_t={\beta }_0+{\beta }_1{X}_t+{\epsilon }_t$ . Second, consider an alternative model with specification, $X_t={\gamma }_0+{\gamma }_1{Y}_t+{\zeta }_t$ . If regression parameters are characteristics of the joint distribution of $\left(X,Y\right)$ , it should be possible to recover one set of estimates from the other, namely ${\widehat{\gamma }}_0=-{\widehat{\beta }}_0/{\widehat{\beta }}_1$ , ${\widehat{\gamma }}_1=1/{\widehat{\beta }}_1$ , and $\widehat{\zeta }=-\widehat{\epsilon }/{\widehat{\beta }}_1$ , because associational relations are nondirectional. However, least-squares estimators do not have this property. The parameter estimates from one specification are inconsistent with the parameter estimates from the alternative specification, hence a least-squares model cannot be “just” a statement on the joint distribution $\left(X,Y\right)$ . If a least-squares model does not model association, what does it model? The answer comes from the definition of the error term, which implies a directed flow of information. In the first specification, $\epsilon$ represents the portion of the outcome $Y$ that cannot be attributed to $X$ . This unexplained outcome is different from $\zeta$ , which is the portion of the outcome $X$ that cannot be attributed to $Y$ . A researcher that chooses the first specification has in mind a controlled experiment where $X$ causes $Y$ , and he estimates the effect coefficient ${\beta }_1$ under the least-squares assumption that $E\left[{\epsilon }_t|X_t\right]=0$ , rather than $E\left[{\epsilon }_t|Y_t\right]=0$ . A researcher that chooses the second specification has in mind a controlled experiment where $Y$ causes $X$ , and he estimates the effect coefficient ${\gamma }_1$ under the assumption that $E\left[{\zeta }_t|Y_t\right]=0$ , rather than $E\left[{\zeta }_t|X_t\right]=0$ . The vast majority of econometric models rely on least-squares estimators, hence implying causal relationships, not associational relationships (Reference WooldridgeImbens and Wooldridge 2009; Reference Abadie and CattaneoAbadie and Cattaneo 2018).

By choosing a particular model specification and estimating its parameters through least-squares, econometricians inject extra-statistical information consistent with some causal graph. Alternatively, econometricians could have used a Deming (or orthogonal) regression, a type of errors-in-variables model that attributes errors to both $X$ and $Y$ . Figure 5 illustrates the regression lines of: (1) a least-squares model where $X$ causes $Y$ ; (2) a least-squares model where $Y$ causes $X$ ; and (3) a Deming regression. Only result (3) characterizes the joint distribution $\left(X,Y\right)$ , without injecting extra-statistical information.

Figure 5

Three regression lines on the same dataset

The realization that econometric equations model causal relationships may come as a surprise to many economics students and professionals. This surprise is understandable, because econometrics textbooks rarely mention causality, causal discovery, causal graphs, causal mechanisms, or causal inference. Economists are not trained in the estimation of Bayesian networks, design of experiments, or applications of do-calculus.Footnote ²³ They are not taught that the causal graph determines the model’s specification, not the other way around, hence the identification of a causal graph should always precede any choice of model specification. Instead, they have been taught debunked specification-searching procedures, such as the stepwise algorithm (an instance of selection bias under multiple testing, see Reference Romano and WolfRomano and Wolf 2005), the general-to-simple algorithm (see Reference GreeneGreene 2012, pp. 178–182), or model selection through trial and error, see Reference ChatfieldChatfield (1995). Section 6.4.2.3 expands on this point, in the context of factor investing.

5.2 Authors often Misunderstand the Meaning of $\mathit{\beta }$

The least-squares method estimates $\beta$ in the equation $Y=X\beta +\epsilon$ asFootnote ²⁴

$\widehat{\beta }={(X^{\prime }X)}^{-1}{X}^{\prime }Y={(X^{\prime }X)}^{-1}{X}^{\prime }\left(X\beta +\epsilon \right)=\beta +{(X^{\prime }X)}^{-1}{X}^{\prime }\epsilon .$(10)

For the estimate to be unbiased ( $E\left[\widehat{\beta }|X\right]=\beta$ ), it must occur that $E[\epsilon |X]=0$ . This is known as the exogeneity condition. There are two approaches for achieving exogeneity. The first approach, called implicit exogeneity, is to define the error term as $\epsilon \equiv Y-E[Y|X]$ , thus $E[\epsilon |X]=E\left[Y-E[Y|X]|X\right]=E[Y|X]-E[Y|X]=0$ . Under this approach, $E[Y|X]=Y-\epsilon =X\beta$ , and $\beta$ has merely a distributional (associational) interpretation, as the slope of a regression line. This is the approach adopted by most econometrics textbooks, see for example Reference GreeneGreene (2012), Reference Hill, Griffiths and LimHill et al. (2011), Reference KennedyKennedy (2008), Reference RuudRuud (2000), and Reference WooldridgeWooldridge (2009). A first flaw of this approach is that it cannot answer interventional questions, hence it is rarely useful for building theories. A second flaw is that it is inconsistent with the causal meaning of the least-squares model specification (Section 5.1).

The second approach, called explicit exogeneity, is to assume that $\epsilon$ represents all causes of $Y$ that are uncorrelated to $X$ . In this case, exogeneity is supported by a causal argument, not by an associational definition. When $X$ has been randomly assigned, as in an RCT or a natural experiment, exogeneity is a consequence of experimental design. However, in purely observational studies, the validity of this assumption is contingent on the model being correctly specified. Under this second approach, $E\left[Y|do\left[X\right]\right]=X\beta$ , and $\beta$ has a causal interpretation, as the expected value of $Y$ given an intervention that sets the value of $X$ . More formally,

$\beta =\frac{\partial E\left[Y|do\left[X\right]\right]}{\partial X}.$(11)

Defending the assumption of correct model specification requires the identification of a causal graph consistent with the observed sample. Absent this information, $\beta$ loses its causal meaning, and reverts to the simpler associational interpretation that is inadequate for building theories and inconsistent with least-squares’ causal meaning.

The ceteris paribus assumption, so popular among economists, is consistent with the causal interpretation of the estimated $\beta$ , whereby the model simulates a controlled experiment. Haavelmo (1944) was among the first to argue that most economists imply a causal meaning when they use their estimated $\beta$ . Almost 80 years later, most econometrics textbooks continue to teach an associational meaning of the estimated $\beta$ that contradicts economists’ interpretation and use. Accordingly, economists are taught to estimate $\beta$ as if it were an associational concept, without regard for causal discovery or do-calculus, while at the same time they interpret and use the estimated $\beta$ as if it were a causal concept, leading to spurious claims.

5.3 Authors Often Mistake Association for Causality

Section 5.1 explained how economists often mean causation when they write about association. Oddly, economists also often mean association when they write about causation. A case in point is the so-called Granger causality. Consider two stationary random variables $\left\{X_t\right\}$ and $\left\{Y_t\right\}$ . Reference GrangerGranger (1969, Reference Granger1980) proposed an econometric test for (linear) causality, based on the equation:

$Y_t={\beta }_0+{\displaystyle \sum _{i=1}^I}{\beta }_i{X}_{t-i}+{\displaystyle \sum _{j=1}^J}{\gamma }_j{Y}_{t-j}+{\epsilon }_t.$(12)

According to Granger, $X$ causes $Y$ if and only if at least one of the estimated coefficients in ${\left\{{\beta }_i\right\}}_{i=1,\dots ,I}$ is statistically significant. This approach was later expanded to multivariate systems, in the form of a vector autoregression specification, see Reference HamiltonHamilton (1994, section 11.2).

The term Granger causality is an unfortunate misnomer. The confusion stems from Granger’s attempt to define causality in terms of sequential association (a characteristic of the joint distribution of probability), see Reference DieboldDiebold (2007, pp. 230–231). However, sequentiality is a necessary, non-sufficient condition for causality (Section 2). Sequential association cannot establish causality, as the latter requires an interventional or natural experiment (Sections 4.1 and 4.2), and in the absence of these, a simulated intervention justified by a discovered or hypothesized causal graph (Section 4.3). For example, a Granger causality test will conclude that a rooster’s crow ( $X_{t-1}$ ) causes the sun to dawn ( $Y_t$ ), because ${\beta }_1$ is statistically significant after controlling for lags of $Y$ . And yet, it is trivial to falsify the claim that a rooster’s crow is a cause of dawn, by silencing the rooster before dawn, or by forcing it to crow at midnight (an intervention). A second problem with Granger causality is that, if both $X$ and $Y$ are caused by $Z$ (a confounder), Granger’s test will still falsely conclude that $X$ causes $Y$ (see Figure 1). Granger causality is misleading in a causally insufficient multivariate time series (Peters et al. Reference Peters, Janzing and Scholkopf2017, pp. 205–208). A third problem is that the test itself is susceptible to selection bias, because the selection of lagged variables involves multiple testing across a large number of potential specifications that are not informed by a causal graph, for example through stepwise specification-searching algorithms. A fourth problem is that it assumes that the causal relation must be linear.

Reference GrangerGranger (1969) remains one of the most-cited articles in the econometrics literature, with over 33,000 citations, and it has become Granger’s second most-cited article. As Figure 6 illustrates, that publication receives thousands of new citations each year, and that number keeps rising, with 2,294 publications referencing it in the year 2021 alone. This confusion of association for causality has led to numerous misinformed claims in the factor investing literature (see Reference Schuller, Haberl and ZaichenkovSchuller et al. 2021 for a survey of claims based on Granger causality). While Granger causality may be used as a simple tool to help decide the direction of causal flow between two unconfounded variables (rather than the existence of causal flow), the field of causal discovery has developed more sophisticated methods to that purpose (see Reference Peters, Janzing and ScholkopfPeters et al. 2017, chapter 4).

Figure 6

Citations of Reference GrangerGranger (1969).

Source: Google Scholar, as of December 1, 202225

I cannot end this section without recognizing a few remarkable economists who, defying the resistance from their peers, have fought to bring the rigor of causal inference into their field of study. Section 4.3.2.4 already discussed the method of instrumental variables, first proposed in 1928 by economist P. G. Wright. Section 5.2 mentioned Haavelmo’s 1944 paper on the meaning of $\beta$ , whose insights continue to be ignored today to a large extent (Reference PearlPearl 2015). The original idea of the DID approach first appeared in labor economics, see Reference Ashenfelter and CardAshenfelter and Card (1986). In the year 2021, Joshua Angrist and Guido Imbens received (in conjunction with David Card) the Nobel Memorial Prize in Economics in recognition “for their methodological contributions to the analysis of causal relationships” in the context of natural experiments (see Section 4.2). Several authors have recently applied the RDD approach to corporate finance, such as Reference Bronzoni and IachiniBronzoni and Iachini (2014), Reference FlammerFlammer (2015), and Reference Malenko and ShenMalenko and Shen (2016). Reference Angrist and PischkeAngrist and Pischke (2010) have called for a “credibility revolution,” urging fellow economists to improve the reliability of their empirical work through the design of interventional studies and natural experiments. These academics offer a rare but inspiring example that ought to be emulated throughout the entire field of economics. On the other hand, asset pricing remains to this day staunchly oblivious to rigorous causal reasoning. Paraphrasing Reference LeamerLeamer (1983), factor researchers have not yet taken the “con” out of econometrics, with the dire consequences described in the following section.

6.1 Causal Content

Factor researchers almost never state explicitly the causal assumptions that they had in mind when they made various modeling decisions, and yet those assumptions shape their analysis. A different set of causal assumptions would have led to different data pre-processing, choice of variables, model specification, choice of estimator, choice of tested hypotheses, interpretation of results, portfolio design, etc. Some of these causal assumptions are suggested by the data, and some are entirely extra-statistical. I denote causal content the set of causal assumptions, whether declared or undeclared, that are embedded in a factor model’s specification, estimation, interpretation, and use. Factor investing strategies reveal part of their causal content in at least four ways.

First, the causal structure assumed by the researcher determines the model specification. A factor investing strategy is built on the claim that exposure to a particular factor ( $X$ ) causes positive average returns above the market’s ( $Y$ ), and that this causal effect (X → Y, a single link in the causal graph) is strong enough to be independently monetizable through a portfolio exposed to $X$ . A researcher only interested in modelling the joint distribution $\left(X,Y\right)$ would surely use more powerful techniques from the machine learning toolbox than a least-squares estimator, such as nonparametric regression methods (e.g., random forest regression, support-vector regression, kernel regression, or regression splines). Factor researchers’ choice of least-squares, explanatory variables, and conditioning variables, is consistent with the causal structure that they wish to impose (Section 5.1).Footnote ²⁸

Second, the estimation of $\beta$ prioritizes causal interpretation over predictive power. If factor researchers prioritized predictive power, they would: (a) use estimators with lower mean-square error than least-squares, by accepting some bias in exchange for lower variance (Reference Mullainathan and SpiessMullainathan and Spiess 2017; Reference Athey and ImbensAthey and Imbens 2019). Examples of such estimators include ridge regression, LASSO, and elastic nets; or (b) use as loss function a measure of performance, such as the Sharpe ratio (for a recent example, see Reference Cong, Tang, Wang and ZhangCong et al. 2021). So not only researchers believe that $Y$ is a function of $X$ (a causal concept), but they are also willing to sacrifice as much predictive power (an associational concept) as necessary to remove all bias from $\widehat{\beta }$ . The implication is that factor researchers assume that the errors are exogenous causes of $Y$ , uncorrelated to $X$ (the explicit exogeneity assumption). Factor researchers’ choice of least-squares is consistent with their interpretation of the estimated $\beta$ as a causal effect (Section 5.2).

Third, factor researchers place strong emphasis on testing the null hypothesis of $H_0:\beta =0$ (no causal effect) against the alternative $H_1:\beta \ne 0$ (causal effect), and expressing their findings through p-values. In contrast, machine-learners are rarely interested in estimating individual p-values, because they assess the importance of a variable in predictive (associational) terms, with the help of associational concepts such as mean-decrease accuracy (MDA), mean-decrease impurity (MDI), and Shapley values (Reference López de PradoLópez de Prado 2018). Factor researchers’ use of p-values is consistent with the claim of a significant causal effect.Footnote ²⁹

Fourth, factor investors build portfolios that overweight stocks with a high exposure to $X$ and underweight stocks with a low exposure to $X$ , at the tune of one separate portfolio for each factor. A factor investor may combine those separate factor portfolios into an aggregate multifactor portfolio, however the reason behind that action is diversification, not monetizing a multifactor prediction. This approach to building portfolios stands in contrast with how other investors use predictions to form portfolios. Investors who rely on predictive models build portfolios exposed to the residual ( $\epsilon$ ) rather than portfolios exposed to a particular factor ( $X$ ), hence for them biased estimates of $\beta$ are not a concern. Factor researchers’ approach to portfolio design is consistent with the monetization of a causal claim rather than a predictive (associational) claim.

In conclusion, the objective of a factor model such as $Y=X\beta +Z\gamma +\epsilon$ is not to predict $Y$ conditioned on $X$ and $Z$ ( $E[Y|X,Z]$ ), but to estimate the causal effect of $X$ on $Y$ $\left(E\left[Y\left|do\right[X]\right]\right)$ , which can be simulated on the observed sample by controlling for confounder $Z$ . The implication is that researchers use factors as if they had assumed explicit exogeneity, and their chosen model specification $Y=X\beta +Z\gamma +\epsilon$ is consistent with a particular causal graph (see Section 5.2), of which Figure 7 is just one possibility among several. It is the responsibility of the researcher to declare and justify what particular causal graph informed the chosen specification, such that the exogeneity assumption holds true.

Figure 7

Causal graph for which the specification $Y=X\beta +Z\gamma +\epsilon$ estimates the causal effect of $X$ on $Y$ , while adjusting for the confounding effect of $Z$

6.2 Omitted Mediation Analysis

Several papers have proposed alternative explanations for various factors, which can be grouped into two broad themes: (a) investment-based explanations; and (b) production-based explanations. For example, Reference Fama and FrenchFama and French (1996) argue that stocks approaching bankruptcy experience a price correction, which in turn is reflected as high value (a high book-to-market ratio). According to this explanation, investors holding portfolios of high-value stocks demand a premium for accepting a non-diversifiable risk of bankruptcy. Reference Berk, Green and NaikBerk et al. (1999) argue that, should firms’ assets and growth options change in predictable ways, that would impart predictability to changes in a firm’s systematic risk and its expected return. Reference JohnsonJohnson (2002) explains that the momentum effect in stock returns does not necessarily imply investor irrationality, heterogeneous information, or market frictions, because simulated efficient markets for stocks exhibit price momentum when expected dividend growth rates vary over time. Reference Gomes, Kogan and ZhangGomes et al. (2003) simulate a dynamic general equilibrium economy, concluding that the size and value factors can be consistent with a single-factor conditional CAPM. Reference ZhangZhang (2005) simulates an economy that exhibits many empirical irregularities in the cross-section of returns. Reference Sagi and SeasholesSagi and Seasholes (2007) claim that backtested performance of momentum strategies is particularly good for firms with high revenue growth, low costs, or valuable growth options. Reference Liu, Whited and ZhangLiu et al. (2009), Reference Li, Livdan and ZhangLi et al. (2009), and Reference Li and ZhangLi and Zhang (2010) associate market anomalies with corporate investment levels, using Tobin’s q-ratio (the ratio between a physical asset’s market value and its replacement value). Reference Liu and ZhangLiu and Zhang (2008) study the association between momentum portfolio returns and shifts in factor loadings on the growth rate of industrial production, concluding that the growth rate of industrial production is a priced risk factor. See Reference CochraneCochrane (2005, pp. 442–453) for additional explanations of factors, some of which are highly speculative or mutually contradictory.

These explanations, in the form of plausible economic rationales, do not rise to the level of scientific theories, for three primary reasons outlined in Sections 3 and 4. First, the authors of these explanations have not declared the causal relationship hypothetically responsible for the observed phenomenon. Second, the authors have not elucidated the ideal interventional study that would capture the causal effect of interest. A Gedankenexperiment, even if unfeasible, has the benefit of communicating clearly the essence of the causal relationship, and the counterfactual implications under various scenarios. Third, when the ideal interventional study is unfeasible, the authors have not proposed a method to estimate the causal effect through observational data (a natural experiment, or a simulated intervention). Consequently, while these economic rationales are plausible, they are also experimentally unfalsifiable. Following Pauli’s criterion, the explanations proposed by factor investing researchers are “not even wrong” (Reference LiptonLipton 2016). As discussed in Section 3, scientific knowledge is built on falsifiable theories that describe the precise causal mechanism by which $X$ causes $Y$ . Value investors may truly receive a reward ( $Y$ ) for accepting an undiversifiable risk of bankruptcy ( $X$ ), but how precisely does this happen, and why is book-to-market the best proxy for bankruptcy risk? Despite of factor models’ causal content, factor researchers rarely declare the causal mechanism by which $X$ causes $Y$ . Factor papers do not explain precisely how a firm’s (or collection of firms’) exposure to a factor triggers a sequence of events that ends up impacting stock average returns; nor do those papers derive a causal structure from the observed data; nor do those papers analyze the causal structure (forks, chains, immoralities); nor do those papers make an effort to explain the role played by the declared variables (treatment, confounder, mediator, collider, etc.); nor do those papers justify their chosen model specification in terms of the identified causal structure (an instance of concealed assumptions).

6.2.1 Example of Factor Causal Mechanism

For illustrative purposes only, and without a claim of accuracy, consider the following hypothetical situation. A researcher observes the tendency of prices ( $p_t$ ) to converge toward the value implied by fundamentals ( $v_t$ ). The researcher hypothesizes that large divergences between prices and fundamental values (HML) trigger the following mechanism: (1) As investors observe HML, they place bets that the divergence will narrow, which cause orderflow imbalance (OI); (2) the persistent OI causes permanent market impact, which over some time period ( $h$ ) pushes prices toward fundamental values (PC, for price convergence).Footnote ³⁰ An investment strategy could be proposed, whereby a fund manager acts upon (1) before (2) takes place.

As stocks rally, investors are more willing to buy them, making some of them more expensive relative to fundamentals, and as stocks sell off, investors are less willing to buy them, making some of them cheaper relative to fundamentals. The researcher realizes that the $\text{HML}\to \text{OI}\to \text{PC}$ mechanism is disrupted by diverging price momentum (MOM), that is, the momentum that moves prices further away from fundamentals, thus contributing to further increases of HML. The researcher decides to add this information to the causal mechanism as follows: (3) high MOM affects future prices in a way that delays PC; and (4) aware of that delay, investors are wary of acting upon HML in the presence of high MOM (i.e., placing a price-convergence bet too early). Accordingly, MOM is a likely confounder, and the researcher must block that backdoor path $\text{HML}\leftarrow \text{MOM}\to \text{PC}$ . Fortunately, MOM is observable, thus eligible for backdoor adjustment (Section 4.3.2.2). But even if MOM were not observable, a front-door adjustment would be possible, thanks to the mediator OI (Section 4.3.2.3).

The above description is consistent with the following system of structural equations:

${\text{OI}}_t:=f_1\underset{{\text{HML}}_t}{\underbrace{\left[p_t-v_t\right]}}+{\epsilon }_{1,t}$(13)

$\underset{P{C}_{t+h}}{\underbrace{p_{t+h}-v_t}}:=f_2\left[{\text{OI}}_t\right]+f_3\left[{\text{MOM}}_t\right]+{\epsilon }_{2,t+h}$(14)

${\text{HML}}_t:=f_4\left[{\text{MOM}}_t\right]+{\epsilon }_{3,t}$(15)

where ${\left\{f_i[.]\right\}}_{i=1,2,3,4}$ are the functions associated with each causal effect (the arrows in a causal graph), and ${\left\{{\epsilon }_{i,.}\right\}}_{i=1,2,3}$ are exogenous unspecified causes. The symbol “ $:=$ ” indicates that the relationship is causal rather than associational, thus asymmetric (e.g., the right-hand side influences the left-hand side, and not the other way around). The researcher applies causal discovery tools on a representative dataset, and finds that the derived causal structure is compatible with his theorized data-generating process. Using the discovered causal graph, he estimates the effect of HML on OI, and the effect of OI on PC, with a backdoor adjustment for MOM. The empirical analysis suggests that HML causes PC, and that the effect is mediated by OI. Encouraged by these results, the researcher submits an article to a prestigious academic journal.

Upon review of the researcher’s journal submission, a referee asks why the model does not control for bid-ask spread (BAS) and market liquidity factors (LIQ). The referee argues that OI is not directly observable, and its estimation may be biased by passive traders. For instance, a large fund may decide to place passive orders at the bid for weeks, rather than lift the offers, in order to conceal their buying intentions. Those trades will be labeled as sale-initiated by the exchange, even though the persistent OI comes from the passive buyer (a problem discussed in Reference Easley, de Prado and O’HaraEasley et al. 2016). The referee argues that BAS is more directly observable, and perhaps a better proxy for the presence of informed traders. The researcher counter-argues that he agrees that (5) OI causes market makers to widen BAS, however (6) PC also forces market makers to realize losses, as prices trend, and market makers’ reaction to those losses is also the widening of BAS. Two consequences of BAS widening are (7) lower liquidity provision and (8) greater volatility. Accordingly, BAS is a collider, and controlling for it would open the noncausal path of association $\text{HML}\leftarrow \text{OI}\to \text{BAS}\leftarrow \text{PC}$ (see Section 6.4.2.2). While the referee is not convinced with the relevance of (6), he is satisfied that the researcher has clearly stated his assumptions through a causal graph. Readers may disagree with the stated assumptions, which the causal graph makes explicit, however, under the proposed causal graph everyone agrees that controlling for either BAS or LIQ or VOL would be a mistake.

The final causal path and causal graph are reflected in Figure 8. By providing this causal graph and mechanism, the researcher has opened himself to falsification. Referees and readers may propose experiments designed to challenge every link in the causal graph. For example, researchers can test link (1) through a natural experiment, by taking advantage that fundamental data is updated at random time differences between stocks. The treatment effect for link (1) may be estimated as the difference in OI over a given period between stocks where HML has been updated versus stocks where HML has not been updated yet. Links (2), (5), (6), (7), and (8) may be tested through controlled and natural experiments similar to those mentioned in Section 3.3. Link (3) is a mathematical statement that requires no empirical testing. To test link (4), a researcher may split stocks with similar HML into two groups (a cohort study, see Section 4.2): the first group is composed of stocks where MOM is increasing HML, and the second group is composed of stocks where MOM is reducing HML. Since the split is not random, the researcher must verify that the two groups are comparable in all respects other than MOM’s direction. The treatment effect may be measured as the two groups’ difference in: (a) sentiment extracted from text, such as analyst reports, financial news, social media (see Reference Das and ChenDas and Chen 2007; Reference Baker and WurglerBaker and Wurgler 2007); (b) sentiment from surveys; or (c) exposures reports in SEC 13F forms. If link (4) is true, MOM dampens investors’ appetite for HML’s contrarian bets, which is reflected in the groups’ difference.

Figure 8

Example of a hypothesized causal mechanism of HML (in the box) within a hypothesized causal graph

These experiments are by no means unique, and many alternatives exist. The opportunities for debunking this theory will only grow as more alternative datasets become available. Contrast this openness with the narrow opportunities offered by factor investing articles currently published in journals, which are essentially limited to: (a) in-sample replication of a backtest, and (b) structural break analyses for in-sample versus out-of-sample performance.

6.3 Causal Denial

Associational investment strategies do not have causal content. Examples include statistical arbitrage (Reference Rad, Low and FaffRad et al. 2016), sentiment analysis (Reference Katayama and TsudaKatayama and Tsuda 2020), or alpha capture (Reference IsichenkoIsichenko 2021, pp. 129–154). Authors of associational investment strategies state their claims in terms of distributional properties, for example, stationarity, ergodicity, normality, homoscedasticity, serial independence, and linearity. The presence of causal content sets factor investing strategies apart, because these investment strategies make causal claims. A causal claim implies knowledge of the data-generating process responsible, among other attributes, for all distributional properties claimed by associational studies. Causal claims therefore require stronger empirical evidence and level of disclosure than mere associational claims. In the context of investment strategies, this translates among other disclosures into: (i) making all causal assumptions explicit through a causal graph; (ii) stating the falsifiable causal mechanism responsible for a claimed causal effect; and (iii) providing empirical evidence in support of (i) and (ii).

Should factor researchers declare causal graphs and causal mechanisms, they would enjoy two benefits essential to scientific discovery. First, causal graphs like the one displayed in Figure 8 would allow researchers to make their causal assumptions explicit, communicate clearly the role played by each variable in the hypothesized phenomenon, and apply do-calculus rules for debiasing estimates. This information is indispensable for justifying the proposed model specification. Second, stating the causal mechanism would provide an opportunity for falsifying a factor theory without resorting to backtests. Even if a researcher p-hacked the factor model, the research community would still be able to design creative experiments aimed at testing independently the implications of every link in the theorized causal path, employing alternative datasets. Peer-reviewers’ work would not be reduced to mechanical attempts at reproducing the author’s calculations.

The omission of causal graphs and causal mechanisms highlights the logical inconsistency at the heart of the factor investing literature: on one hand, researchers inject causal content into their models, and use those models in a way consistent with a causal interpretation (Section 6.1). On the other hand, researchers almost never state a causal graph or falsifiable causal mechanism, in denial or ignorance of the causal content of factor models, hence depriving the scientific community of the opportunity to design experiments that challenge the underlying theory and assumptions (Section 6.2). Under the current state of causal confusion, researchers report the estimated $\beta$ devoid of its causal meaning (the effect on $Y$ of an intervention on $X$ ), and present p-values as if they merely conveyed the strength of associations of unknown origin (causal and noncausal combined).

The practical implication of this logical inconsistency is that the factor investing literature remains at a phenomenological stage, where spurious claims of investment factors are accepted without challenge. Put simply: without a causal mechanism, there is no investment theory; without investment theory, there is no falsification; without falsification, investing cannot be scientific.

This does not mean that investment factors do not exist; however, it means that the empirical evidence presented by factor researchers is insufficient and flawed by scientific standards. Causal denial (or ignorance) is a likely reason for the proliferation of spurious claims in the factor investing studies, and the poor performance delivered by the factor-based investment funds, for the reasons explained next.

6.4 Spurious Investment Factors

The out-of-sample performance of factor investing has been disappointing. One of the broadest factor investing indices is the Bloomberg–Goldman Sachs Asset Management US Equity Multi-Factor Index (BBG code: BGSUSEMF <Index>). It tracks the long/short performance of the momentum, value, quality, and low-risk factors in US stocks (Bloomberg 2021). Its annualized Sharpe ratio from May 2, 2007 (the inception date) to December 2, 2022 (this Element’s submission date) has been 0.29 (t-stat = 1.16, p-value = 0.12), and the average annualized return has been 1.13 percent. This performance does not include: (a) transaction costs; (b) market impact of order execution; (c) cost of borrowing stocks for shorting positions; (d) management and incentive fees. Also, this performance assigns a favorable 0 percent risk-free rate when computing the excess returns. Using the 6-month US Government bond rates (BBG code: USGG6M <Index>) as the risk-free rates, the Sharpe ratio turns negative. Figure 9 plots the performance of this broad factor index from inception, without charging for the above costs (a)–(d). After more than fifteen years of out-of-sample performance, factor investing’s Sharpe ratio is statistically insignificant at any reasonable rejection threshold.

Figure 9

Performance of the Bloomberg – Goldman Sachs Asset Management US Equity Multi-Factor Index, since index inception (base 100 on May 2, 2007)

It takes over 31 years of daily observations for an investment strategy with an annualized Sharpe ratio of 0.29 to become statistically significant at a 95 percent confidence level (see Bailey and Reference Bailey and López de PradoLópez de Prado (2012) for details of this calculation). If the present Sharpe ratio does not decay (e.g., due to overcrowding, or hedge funds preempting factor portfolio rebalances), researchers will have to wait until the year 2039 to reject the null hypothesis that factor investing is unprofitable, and even then, they will be earning a gross annual return of 1.13 percent before paying for costs (a)–(d).

There is a profound disconnect between the unwavering conviction expressed by academic authors and the underwhelming performance experienced by factor investors. A root cause of this disconnect is that factor investing studies usually make spurious claims, of two distinct types.

6.4.2 Type-B Spuriosity

An association is true if it is not type-A spurious, however that does not mean that the association is causal. I define an empirical claim to be type-B spurious when a researcher mistakes association for causation. A leading cause for type-B spuriosity is systematic biases due to misspecification errors. A model is misspecified when its functional form is incongruent with the functional form of the data-generating process, and the role played by the variables involved. Type-B spuriosity has several distinct attributes: (a) it results in type-1 errors and type-2 errors (false positives and false negatives); (b) it can occur with a single trial; (c) it has a greater probability to take place as the sample size grows, because the noncausal association can be estimated with lower error; and (d) it cannot be corrected through multiple-testing adjustments. Its correction requires the injection of extra-statistical information, in the form of a causal theory.

The expected return of a type-B spurious investment factor is misattributed, as a result of the biased estimates. Also, type-B spurious investment factors can exhibit time-varying risk premia (more on this in Section 6.4.2.1).

Type-A and type-B spuriosity are mutually exclusive. For type-B spuriosity to take place, the association must be noncausal but true, which precludes that association from being type-A spurious. While type-A spuriosity has been studied with some depth in the factor investing literature, relatively little has been written about type-B spuriosity. Next, I discuss the main reasons for type-B spuriosity in factor investing.

6.4.2.1 Under-Controlling

Consider a data-generating process where one of its equations is $Y:=X\beta +Z\gamma +u$ , such that $\gamma \ne 0$ and $u$ is white noise. The process is unknown to a researcher, who attempts to estimate the causal effect of $X$ on $Y$ by fitting the equation $Y=X\beta +\epsilon$ on a sample ${\left\{X_t,Y_t\right\}}_{t=1,\dots ,T}$ produced by the process. This incorrect specification choice makes $\epsilon =Z\gamma +u$ , and $E[\epsilon |X]=E[Z\gamma +u|X]=\gamma E[Z|X]$ . However, if $Z$ is correlated with $X$ , $E[Z|X]\ne 0$ , hence $E[\epsilon |X]\ne 0$ . This is a problem, because the least-squares method assumes $E[\epsilon |X]=0$ (the exogeneity assumption, see Section 5.2). Missing one or several relevant variables biases the estimate of $\beta$ , potentially leading to spurious claims of causality. A false positive occurs when $\left|\widehat{\beta }\right|\gg 0$ for $\beta \approx 0$ , and a false negative occurs when $\widehat{\beta }\approx 0$ for $\left|\beta \right|\gg 0$ .

Econometrics textbooks do not distinguish between types of missing variables (see, for example, Reference GreeneGreene 2012, section 4.3.2), yet not all missing variables are created equal. There are two distinct cases that researchers must consider. In the first case, the second equation of the data-generating process is $Z:=X\delta +v$ , where $\delta \ne 0$ and $v$ is white noise. In this case, $Z$ is a mediator ( $X$ causes $Z$ , and $Z$ causes $Y$ ), and the chosen specification biases the estimation of the direct effect $\widehat{\beta }$ ; however, $\widehat{\beta }$ can still be interpreted as a total causal effect (through two causal paths with the same origin and end). The causal graph for this first case is displayed at the top of Figure 10. In the second case, the second equation of the data-generating process is $X:=Z\delta +v$ , where $\delta \ne 0$ and $v$ is white noise. In this case, $Z$ is a confounder ( $Z$ causes $X$ and $Y$ ), the chosen specification also biases $\widehat{\beta }$ , and $\widehat{\beta }$ does not measure a causal effect (whether total or direct).Footnote ³² The causal graph for this second case is displayed at the bottom of Figure 10.

Figure 10

Variable $Z$ as mediator (top) and confounder (bottom)

Assuming that the white noise is Gaussian, the expression $E\left[\widehat{\beta }|X\right]$ reduces to

$E\left[\widehat{\beta }|X\right]={(X^{\prime }X)}^{-1}{X}^{\prime }E[X\beta +Z\gamma +u|X]=$

$\beta +\gamma \delta {\left(1+{\delta }^2\right)}^{-1}=\beta +\theta$(16)

where $\theta =\gamma \delta {\left(1+{\delta }^2\right)}^{-1}$ is the bias due to the missing confounder. The Appendix contains a proof of the above proposition. The intuition behind $\theta$ is that a necessary and sufficient condition for a biased estimate of $\beta$ is that $\gamma \ne 0$ and $\delta \ne 0$ , because when both parameters are nonzero, variable $Z$ is a confounder.

A first consequence of missing a confounder is incorrect performance attribution and risk management. Part of the performance experienced by the investor comes from a misattributed risk characteristic $Z$ , which should have been hedged by a correctly specified model. The investor is exposed to both, causal association (from $\beta$ ), as intended by the model’s specification, and noncausal association (from $\theta$ ), which is not intended by the model’s specification.

A second consequence of missing a confounder is time-varying risk premia. Consider the case where the market rewards exposure to $X$ and $Z$ ( $\beta >0$ , $\gamma >0$ ). Even if the two risk premia remain constant, changes over time in $\delta$ will change $\widehat{\beta }$ . In particular, for a sufficiently negative value of $\delta$ , then $\widehat{\beta }<0$ . Performance misattribution will mislead investors into believing that the market has turned to punish exposure to risk characteristic $X$ , when in reality their losses have nothing to do with changes in risk premia. The culprit is a change in the covariance between the intended exposure ( $X$ ) and the unintended exposure that should have been hedged ( $Z$ ). Authors explain time-varying risk premia as the result to changes in expected market returns (e.g., Reference EvansEvans 1994; Reference AndersonAnderson 2011; and Reference CochraneCochrane 2011), and asset managers’ marketing departments justify their underperformance in terms of temporary changes in investor or market behavior. While these explanations are plausible, they seem to ignore that time-varying risk premia is consistent with a missing confounder (an arguably more likely and parsimonious, hence preferable, explanation). For example, consider the causal graph in Figure 8, where MOM confounds the estimate of the effect of HML on PC. If an asset manager under-controls for MOM, the value investment strategy will be exposed to changes in the covariance between MOM and HML. The asset manager may tell investors that the value strategy is losing money because of a change in value’s risk premium, when the correct explanation is that the product is defective, as a result of under-controlling. Changes in the covariance between MOM and HML have nothing to do with value’s or momentum’s true risk premia, which remain unchanged (like the direct causal effects, $\text{HML}\to \text{PC}$ and $\text{MOM}\to \text{PC}$ ). This flaw of type-B spurious factor investing strategies makes them untrustworthy.

The partial correlations method allows researchers to control for observable confounders when the causal effect is linear and the random variables jointly follow an elliptical (including multivariate normal) distribution, multivariate hypergeometric distribution, multivariate negative hypergeometric distribution, multinomial distribution, or Dirichlet distribution (Reference Baba, Shibata and SibuyaBaba et al. 2004). A researcher is said to “control” for the confounding effect of $Z$ when he adds $Z$ as a regressor in an equation set to model the effect of $X$ on $Y$ . Accordingly, the new model specification for estimating the effect of $X$ on $Y$ is $Y=X\beta +Z\gamma +\epsilon$ . This is a particular application of the more general backdoor adjustment (Section 4.3.2.2), and by far the most common confounder bias correction method used in regression analysis. This adjustment method relies on a linear regression, thus inheriting its assumptions and limitations. In particular, the partial correlations method is not robust when the explanatory variables exhibit high correlation (positive or negative) with each other (multicollinearity).

6.4.2.2 Over-Controlling

The previous section explained the negative consequences of under-controlling (e.g., missing a confounder). However, over-controlling is not less pernicious. Statisticians have been trained for decades to control for any variable $Z$ associated with $Y$ that is not $X$ (Reference Pearl and MacKenziePearl and MacKenzie 2018, pp. 139, 152, 154, 163), regardless of the role of $Z$ in the causal graph (the so-called omitted variable problem). Econometrics textbooks dismiss as a harmless error the inclusion of an irrelevant variable, regardless of the variable’s role in the causal graph. For example, Reference GreeneGreene (2012, section 4.3.3) states that the only downside to adding superfluous variables is a reduction in the precision of the estimates. This grave misunderstanding has certainly led to countless type-B spurious claims in economics.

In recent years, do-calculus has revealed that some variables should not be controlled for, even if they are associated with $Y$ . Figure 11 shows two examples of causal graphs where controlling for $Z$ will lead to biased estimates of the effect of $X$ on $Y$ .

Figure 11

Variable $Z$ as controlled mediator (top) and controlled collider (bottom)

Common examples of over-controlling include controlling for variables that are mediators or colliders relative to the causal path from $X$ to $Y$ .Footnote ³³ Controlling for a collider is a mistake, as it opens a backdoor path that biases the effect’s estimation (Berkson’s fallacy, see Reference BerksonBerkson 1946). Controlling for a mediator interferes with the mediated effect ( $X\to Z\to Y$ ) and the total causal effect ( $X\to Z\to Y$ plus $X\to Y$ ) that the researcher may wish to assess, leaving only the direct effect $X\to Y$ . In the case of the top causal graph in Figure 11, a researcher could estimate the mediated effect $X\to Z\to Y$ as the difference between the total effect ( $X\to Z\to Y$ plus $X\to Y$ ) and the direct effect ( $X\to Y$ ).

Over-controlling a collider and under-controlling a confounder have the same impact on the causal graph: allowing the flow of association through a backdoor path (Section 4.3.2.2). Consequently, over-controlled models can suffer from the same conditions as under-controlled models, namely (i) biased estimates, as a result of noncompliance with the exogeneity assumption; and (ii) time-varying risk premia. Black-box investment strategies take over-controlling to the extreme. Over-controlling explains why quantitative funds that deploy black-box investment strategies routinely transition from delivering systematic profits to delivering systematic losses, and there is not much fund managers or investors can do to detect that transition until it is too late.

The only way to determine precisely which variables a researcher must control for, in order to block (or keep blocked) noncausal paths of association, is through the careful analysis of a causal graph (e.g., front-door criterion and backdoor criterion). The problem is, factor researchers almost never estimate or declare the causal graphs associated with the phenomenon under study (Section 6.2). Odds are, factor researchers have severely biased their estimates of $\beta$ by controlling for the wrong variables, which in turn has led to false positives and false negatives.

6.4.2.3 Specification-Searching

Specification-searching is the popular practice among factor researchers of choosing a model’s specification (including the selection of variables and functional forms) based on the resulting model’s explanatory power. To cite one example, consider the three-factor model introduced by Reference Fama and FrenchFama and French (1993), and the five-factor model introduced by Reference Fama and FrenchFama and French (2015). Reference Fama and FrenchFama and French (2015)’s argument for adding two factors to their initial model specification was that “the five-factor model performs better than the three-factor model when used to explain average returns.”

These authors’ line of argumentation is self-contradictory. The use of explanatory power (an associational, noncausal concept) for selecting the specification of a predictive model is consistent with the associational goal of that analysis; however, it is at odds with the causal content of a factor model. In the context of factor models, specification-searching commingles two separate and sequential stages of the causal analysis: (1) causal discovery (Section 4.3.1); and (2) control (Section 4.3.2). Stage (2) should be informed by stage (1), not the other way around. Unlike a causal graph, a coefficient of determination cannot convey the extra-statistical information needed to de-confound the estimate of a causal effect, hence the importance of keeping stages (1) and (2) separate.

Stage (1) discovers the causal graph that best explains the phenomenon as a whole, including observational evidence and extra-statistical information. In stage (2), given the discovered causal graph, the specification of a factor model should be informed exclusively by the aim to estimate one of the causal effects (one of the arrows or causal paths) declared in the causal graph, applying the tools of do-calculus. In a causal model, the correct specification is not the one that predicts $Y$ best, but the one that debiases $\widehat{\beta }$ best, for a single treatment variable, in agreement with the causal graph. Choosing a factor model’s specification based on its explanatory power incurs the risk of biasing the estimated causal effects. For example, a researcher may achieve higher explanatory power by combining multiple causes of $Y$ , at the expense of biasing the multiple parameters’ estimates due to multicollinearity or over-controlling for a collider.Footnote ³⁴ It is easy to find realistic causal structures where specification-searching leads to false positives, and misspecified factor models that misattribute risk and performance (see Section 7.3).

There are two possible counter-arguments to the above reasoning: (a) A researcher may want to combine multiple causes of $Y$ in an attempt to model an interaction effect. However, such attempt is a stage (2) analysis that should be justified with the causal graph derived from stage (1), showing that the total effect involves several variables that are observed separately, but that need to be modeled jointly; and (b) a researcher may want to show that the two causes are not mutually redundant (a multifactor explanation, see Reference Fama and FrenchFama and French 1996). However, there exist far more sophisticated tools for making that case, such as mutual information or variation of information analyses (Reference López de PradoLópez de Prado 2020, chapter 3).

While specification-searching may involve multiple testing, specification-searching is not addressed by multiple testing corrections, as it has to do with the proper modeling of causal relationships, regardless of the number of trials involved in improving the model’s explanatory power. Accordingly, specification-searching is a source of spuriosity that is distinct from p-hacking, and whose consequence is specification bias rather than selection bias. As argued in an earlier section, investors interested in predictive power should apply machine learning algorithms, which model association, not causation.

6.4.2.4 Failure to Account for Temporal Properties

In the context of time-series analysis, two independent variables may appear to be associated when: (a) their time series are nonstationary (Reference Granger and NewboldGranger and Newbold 1974); and (b) their time series are stationary, however they exhibit strong temporal properties, such as positively autocorrelated autoregressive series or long moving averages (Reference Granger, Hyung and JeonGranger et al. 2001). This occurs regardless of the sample size and for various distributions of the error terms.

Unit root and cointegration analyses help address concerns regarding the distribution of residuals, however they cannot help mitigate the risk of making type-B spurious claims. Like their cross-sectional counterparts, time-series models also require proper model specification through causal analysis, as discussed in the earlier sections. Section 5.3 exemplified one way in which econometricians mistake association for causation in time-series models.