What is triangulation?

The term triangulation originates from an approach that determines the location of a point of interest by measuring angles to it from two or more (known) points and applying trigonometry (i.e., using the properties of triangles) (Lawlor et al., Reference Lawlor, Tilling and Davey Smith2016). In scientific research, usually triangulation is interpreted as conducting two or more research methods to obtain the answer to a complex question, although it is used differently across different research fields (see Box 1). By synthesizing multiple sources of information, the overall evidence is assumed to be more reliable, and more robust conclusions can be derived. The idea, that if a finding replicates across different settings, it is more likely to be true, is also incorporated in the well-known Bradford-Hill guidelines for causality as the ‘consistency’ criterion (Fedak et al., Reference Fedak, Bernal, Capshaw and Gross2015). To some degree, it lies at the basis of scientific practice in general. Researchers usually compare findings of their empirical work to those from other studies in the discussion section of a paper or in a review, acknowledging strengths and weaknesses. Systematic reviews and particularly meta-analyses aim to retrospectively combine estimates from similar analyses to come to a more precise point estimate (Moher et al., Reference Moher, Shamseer, Clarke, Ghersi, Liberati, Petticrew, Shekelle and Stewart2015). If these estimates come from the same method, some error might be removed by combining across them, but consistent bias inherent to the method will remain.

Triangulation in the field of behavioural and psychiatric epidemiology

Here, we focus on the use of triangulation to answer aetiological questions about behavioural and psychiatric traits. In this context, a precise definition of triangulation has been proposed by Lawlor et al. (Reference Lawlor, Tilling and Davey Smith2016) as: ‘The practice of strengthening causal inferences by integrating results from several different approaches, where each approach has different (and assumed to be largely unrelated) key sources of potential bias’. Differing from retrospective synthesis of evidence (e.g. systematic reviews), the definition of triangulation is predominantly prospective (Hammerton and Munafò, Reference Hammerton and Munafò2021; Lawlor et al., Reference Lawlor, Tilling and Davey Smith2016). It entails deliberately selecting several approaches to complement one another, conducting them in unison and then synthesizing and interpreting the results together. Approaches can vary in (1) the methods applied, (2) the population from which the sample is drawn and/or (3) the measurement of the variables.

Approach 1: Triangulation using different methods

It is recommended that two or more methods with different sources of potential bias should be used to address the same underlying causal question (Lawlor et al., Reference Lawlor, Tilling and Davey Smith2016). Triangulation is most effective if the selected methods have different expected directions of bias. This means that where the bias of one method may lead to an overestimation of the true causal effect, the bias of another method may be towards the null (or even towards the opposite sign of the effect) (Lawlor et al., Reference Lawlor, Tilling and Davey Smith2016). In Supplementary Table 1, we provide a broad, but non-exhaustive overview of methods that may be used in triangulation studies, meant to serve as an inspiration for introducing breadth. Under methodological triangulation, we differentiate experimental from observational studies. Experimental studies, e.g. in the form of randomized trials, are ideal for causal inference, but usually unfeasible in behavioural and psychiatric epidemiology. This is largely due to practical and ethical reasons, but also the long timespan that it would take for a potential causal effect to arise. In some instances, however, experimental manipulation can be used to assess relevant short-term processes that may tell you something about longer-term effects. In observational research, there is a wide range of methods that try to determine whether one variable affects another (although this causal goal is not always acknowledged [Hernán, Reference Hernán2018]), the most common being analysis of longitudinal cohort data to test if changes in the exposure are followed by changes in the outcome. Certain observational methods can be considered ‘quasi-experimental’ because some type of (environmental) influence or change is leveraged to create quasi-experimental groups in a given population, e.g. a policy change (Leatherdale, Reference Leatherdale2019; Maciejewski, Reference Maciejewski2020). Current triangulation literature also incorporates many relatively novel causal methods from the field of genetic epidemiology, which has leveraged major technical advances in molecular genetics over the past two decades (Abdellaoui et al., Reference Abdellaoui, Yengo, Verweij and Visscher2023). The most notable is Mendelian randomization (MR), which uses genetic variants as instrumental variables to proxy for levels of the exposure (Davey Smith and Hemani, Reference Davey Smith and Hemani2014). Other genetically informative designs don’t use molecular genetic data, but instead rely on inferences based on familial relatedness, such as discordant twin and sibling-control studies (Pingault et al., Reference Pingault, O’reilly, Schoeler, Ploubidis, Rijsdijk and Dudbridge2018).

An example of methodological triangulation by van de Weijer et al. (Reference van de Weijer, Demange, Pelt, Bartels and Nivard2024) explored whether spending more years in education causally increases later-life mental wellbeing (van de Weijer et al., Reference van de Weijer, Demange, Pelt, Bartels and Nivard2024). They compared results across four methodological designs: (1) a natural experiment using the raising of the school leaving age policy change (2) a sibling-control design which accounts for shared familial confounding, (3) MR and (4) within-family MR, which further accounts for assortative mating, population stratification and dynastic effects (Brumpton et al., Reference Brumpton, Sanderson, Heilbron, Hartwig, Harrison, Vie, Cho, Howe, Hughes, Boomsma and Havdahl2020). An additional example of methodological triangulation comes from Esen et al. (Reference Esen, Ehrenstein, Sørensen, Laugesen and Pedersen2022). Seeking to understand whether in utero exposure to antidepressant drugs causes offspring attention-deficit/hyperactivity disorder (ADHD), Esen et al. (Reference Esen, Ehrenstein, Sørensen, Laugesen and Pedersen2022) compared results from four methodological designs: (1) regular analysis controlling for confounders, (2) a negative-control analysis comparing the association of paternal antidepressant use with that of pregnant mothers, (3) sibling-control analysis comparing siblings who were discordant for maternal antidepressant use during pregnancy and (4) a former-user analysis comparing results of design 1 with mothers who previously took antidepressants but not during pregnancy, to mitigate confounding via maternal depression status (Esen et al., Reference Esen, Ehrenstein, Sørensen, Laugesen and Pedersen2022). For both of the triangulation examples here, the authors purposefully selected designs that each help correct for different sources of bias, where converging evidence across the different designs would suggest a true causal effect.

Approach 2: Triangulation using samples from different populations

Second, let us consider triangulating across samples from different populations. This form of triangulation can help to mitigate bias from confounding and selection pressures. An overview of applications of this approach is given in Supplementary Table 1. A particularly illustrative example comes from Brion et al. (Reference Brion, lawlor, Matijasevich, Horta, Anselmi, Araújo, Menezes, Victora and Smith2011), who tested the (potentially causal) effect of breastfeeding on cognitive development in offspring. Breastfeeding is strongly associated with socio-economic position (SEP) which is also a major confounder for the breastfeeding-cognitive development relationship. Conveniently, the direction of the confounding effect of SEP is different in the United Kingdom, where breastfeeding is associated with higher SEP, than in Brazil, where breastfeeding is associated with lower SEP. This makes the underlying confounding structures of the samples different. By investigating the causal question (if breastfeeding positively affects cognitive development) in both samples, confounding by SEP as an explanation can be partly circumvented. Finding a causal effect in both samples would constitute more reliable evidence overall (Brion et al., Reference Brion, lawlor, Matijasevich, Horta, Anselmi, Araújo, Menezes, Victora and Smith2011)

As an additional example, Chartier et al. (Reference Chartier, Kendler, Ohlsson, Sundquist and Sundquist2023) explored whether immigration increases risk for alcohol use disorder (AUD) through acculturation. Acculturation refers to the assimilation of the less dominant into the more dominant culture (Chartier et al., Reference Chartier, Kendler, Ohlsson, Sundquist and Sundquist2023). Chartier et al. (Reference Chartier, Kendler, Ohlsson, Sundquist and Sundquist2023) hypothesize that the rates of AUD in the immigrant populations will become more similar to the rates in the native population over subsequent generations. They compared immigrants in Sweden from seven different geographical regions, each with different starting levels of AUD. For example, 1^st generation immigrants from Africa have lower rates of AUD than the Swedish population and Finnish 1^st generation immigrants have higher rates. Over subsequent generations, rates of AUD in both populations became closer to the Swedish average. Change in different directions makes this less likely to be due to confounding and more likely due to acculturation.

Approach 3: Triangulation using different measures

Finally, triangulation could compare different types of measures for the variable of interest, for instance taken from different sources, raters or obtained with different measurement instruments. For example, while self-report can be affected by recall or social desirability bias, registry data may be measured with error due to missingness or incorrectly recorded disease codes whilst obtaining a correct diagnosis. An illustrative example comes from Haan et al. (Reference Haan, Sallis, Ystrom, Njølstad, Andreassen, Reichborn‐Kjennerud, Munafò, Havdahl and Zuccolo2021) who explored the association between foetal alcohol exposure and ADHD risk in offspring (Haan et al., Reference Haan, Sallis, Ystrom, Njølstad, Andreassen, Reichborn‐Kjennerud, Munafò, Havdahl and Zuccolo2021). They observed different patterns of association depending on whether ADHD symptoms were mother or teacher reported, and depending on the scale used. With no single measure being free from bias, triangulating across several available measures can help strengthen conclusions. However, careful consideration of the interpretation is needed when results differ. An overview of how this approach could be applied is given in Supplementary Table 1.

Step 1: Determine the causal question

It is of upmost importance to begin with a clear causal research question. The best causal questions are informed by theory and existing literature and take into account whether or not a potential effect is restricted to a particular timing or length of exposure. For our worked example, we ask the causal question ‘Does maternal tobacco smoking during pregnancy cause offspring depression?’ (Haber et al., Reference Haber, Wood, Wieten and Breskin2022). It should be noted that the initial question that is posed at the start of a triangulation study, will translate to (slightly) different, but related, research questions across the different approaches that are subsequently chosen (see step 5 of this guide). Approaches which are testing more similar questions will be easier to integrate and interpret together. Ultimately, triangulation aims to come to the best-fitting explanation for an observed relationship, based on the totality of the data.

Step 2: Draw a directed acyclic graph

Directed acyclic graphs (DAGs) can be used as a tool for mapping out assumed relationships between relevant variables (Lipsky and Greenland, Reference Lipsky and Greenland2022). In essence, a DAG is a graph that consists of the variables of interest with arrows between the variables indicating the direction of hypothesized causal effects and a lack of an arrow between any two variables implying that there is no hypothesized causal effect. A DAG is ‘acyclic’, meaning that cycles (or feedback loops) are not allowed. In the context of triangulation, DAGs can help you clarify your assumptions, establish key sources of bias and identify appropriate covariates for statistical analysis. A DAG for our worked example is given in the Supplementary Note.

Step 3: Identify available resources and samples

With your causal question defined, the next step is to scope out the resources available. Here, the aim is to find the boundaries within which the triangulation will have to be conducted, which may represent financial, ethical or practical constraints. For many causal questions in aetiological epidemiology, there will not be sufficient resources or time to collect new data. Therefore, identification of suitable secondary data often presents the most effective way to conduct a triangulation study. When identifying possible datasets, consideration should be given to sample characteristics and selection mechanisms, so that these sources of bias can also be triangulated.

Let us take our worked example, does maternal tobacco smoking during pregnancy cause offspring depression? The timing of our question is specific to intrauterine effects. Therefore, suitable datasets are restricted to those with relevant smoking data during pregnancy and long enough follow-up to capture depression traits in offspring. As collecting new prospective longitudinal data to investigate this question would be too time-consuming and expensive (Esen et al., Reference Esen, Ehrenstein, Sørensen, Laugesen and Pedersen2022), we identify only relevant secondary data sources for this worked example (see Supplementary Note).

Step 4: Identify suitable methodological approaches

At this stage, we identify methods which are suitable to test the causal question and have different (preferably uncorrelated) main sources of bias. The aim here is to try and mitigate the most problematic sources of bias as identified in step 2. Here, we identify which methods are feasible given the available samples.

In the Supplementary Note, we expand upon four methodological approaches that would allow us to test whether there is evidence for causal effects of maternal smoking during pregnancy on offspring depression. From Supplementary Table 1, several methods were not suitable, for example, any type of randomisation to smoking during pregnancy would pose serious ethical concerns. However, there were still a multitude of study designs which could help investigate our causal question, within the scope of available resources. First, to address potential confounding by environmental factors (e.g. SEP) and by mother’s own depression, we propose a longitudinal regression model with adjustment for confounding in an existing prospective pregnancy cohort. This method can still be biased by residual confounding and doesn’t account for shared genetic liability between parents and offspring. In other words, if the mother is more likely to smoke, the offspring is more likely to smoke, and their own smoking could cause their depression. To address these biases, we would plan to conduct an intergenerational MR study. This approach uses genetic variants as instrumental variables to reduce bias from residual confounding, and adjusts for offspring genotype to isolate maternal effects (Reference Davies, Holmes and SmithDavies et al.,). Additionally, we could add a proxy GxE MR study, which tests whether maternal smoking during pregnancy is associated with offspring depression. If we observe an association between the genetic variants for maternal smoking and offspring depression, regardless of maternal smoking status, effects are likely non-specific to smoking during pregnancy. Finally, we would repeat the confounder adjustment and MR analyses with father’s smoking as a powerful negative control (since the father cannot have direct intrauterine effects on the offspring).

Statistical power is an important factor to consider here, because it will determine the likelihood for each method to pick up on a true causal effect. For new data collection, conducting an a priori power calculation is important and highly recommended. For secondary data analysis (where data collection has already occurred) there is no consensus about the usefulness of conducting power analyses (Dziak et al., Reference Dziak, Dierker and Abar2020). Where possible (i.e., if the data are already accessible), we consider a power analysis at this stage beneficial, as it can be taken into account when specifying your expectations of potential findings and how to interpret them. When key information to conduct a power analysis is not available, it may be computed post hoc to aid interpretation. While underpowered analyses should generally be avoided there are certainly cases where adding an analysis that could mitigate an important source of bias would be valuable.

Step 5: Further specify the causal question per method

After having posed the initial causal question under step 1, here, we aim to define the specific questions that can be answered with each of the approaches that we have chosen. To facilitate the integration of findings from different approaches, we recommend researchers formulate their causal research questions as specifically as possible in this step. Drawing specific DAGs for each approach may be useful during this refinement process. To illustrate, the specific causal question for the confounder adjusted analysis in the Avon Longitudinal Study of Parents and Children (ALSPAC)(see Supplementary Note) is as follows: ‘In a sample of offspring born in the early 90s in Avon (UK), is mother-reported maternal tobacco smoking during pregnancy (in any trimester) associated with offspring self-reported depressive symptoms up to age 30 years, after adjustment for measured confounders?’

Step 6: Explicate the effects of potential biases

Once the most suitable methods, samples and measures for triangulation have been chosen, and specific questions have been formulated, it is important to explicate the potential effects of biases. Of course, it is not possible for the effects of all potential biases to be known, by their very definition some are unknown (e.g., residual confounding). In Supplementary Table 1, we highlight the most notable sources of bias for a selection of approaches.

Step 7: Pre-specify expectations under causality

For triangulation to be most effective, it is important to specify a priori what results we would expect to see under true causality and what results we may expect if our association of interest is in fact not causal (Höfler, Reference Höfler2005). Most often, pre-specification of a true causal effect is straightforward – we would expect each of the methods to show statistical evidence for an effect in the same direction across the methods/cohorts/samples fulfilling broadly similar characteristics (such as age). When specifying expected causal effects, it is important to consider the power of each method. For example, van de Weijer et al. (Reference van de Weijer, Demange, Pelt, Bartels and Nivard2024) pre-specified that they would not expect the within-family MR results to pass the threshold for statistical significance given low power and that consequently, for that method they would be looking to see whether the direction of effect was consistent with the other methods (van de Weijer et al., Reference van de Weijer, Demange, Pelt, Bartels and Nivard2024). Another important consideration is the timing and duration of the exposure of interest across different methods. If for one method the influence of the exposure is relatively short (for instance in an experimental setting) while in another method it is relatively long (for instance in a MR study where the genetic variants express a lifetime influence), it is to be expected that the size of the effect would differ. This needs to be taken into consideration when pre-specifying your expectations, and later, when comparing the results. If one method finds evidence for a causal effect and another doesn’t, but this can convincingly be explained based on (pre-specified) differences in the timing, measurement, or duration of the exposure, it might not decrease the overall evidence for causality.

More difficult, but equally as important, is to pre-specify your expected results under a non-causal model. Where known (or deducible), it is useful to begin by pre-specifying the expected biases of each method, and their expected direction of effect. This can help to ensure that you have selected methods with uncorrelated biases. For example, van de Weijer et al. (Reference van de Weijer, Demange, Pelt, Bartels and Nivard2024) specified a priori that they would conclude that there was substantial confounding or bias from reverse causation if results were not significant and consistent (van de Weijer et al., Reference van de Weijer, Demange, Pelt, Bartels and Nivard2024). When defining such expectations, it is important to consider your limitations with regard to statistical power. Identifying true causal relationships requires the presence of robust outcomes with strong statistical evidence. If your analyses are underpowered then small effects might not be detected. Nonetheless, it is likely that utilizing diverse methodologies across potentially heterogeneous samples leads to outcomes of differing levels of (statistical) strength. To employ an analogy, envision a reservoir: with each piece of evidence stemming from a method or sample, water is added, its volume proportional to the method’s significance and weight. Conversely, opposing effects would deplete the reservoir’s reserves. The overall evidence for causality is reflected in the fullness of the reservoir.

Triangulation in the context of the broader literature

A well-designed triangulation study should be clearly informed by the existing literature and set up to address the most important limitations of prior studies. The results of a triangulation study should also be evaluated within the context of the body of evidence that already exists, taking into account the inherent biases of each line of evidence and their capacity to reflect reality. If we want to come to more reliable scientific evidence to answer complex causal questions in behavioural and psychiatric epidemiology, it is crucial to acknowledge which exact biases are relevant for which methods, samples and measures, both when we compare the results of analyses within one (triangulation) study as well as when we compare results across studies.

Falsification

One approach that can be particularly powerful in a triangulation framework but which is not applied widely enough is the negative-control analysis. In the context of the worked example in this review, we chose a negative-control design where we compared the effect of maternal smoking during pregnancy on offspring depression to the effect of paternal smoking during pregnancy. Father’s smoking cannot have direct causal intrauterine effects on the offspring, therefore if paternal effects have a similar magnitude of association as maternal effects, this is likely due to bias from confounding. We recommend researchers to adapt an ethos of attempting to falsify a potentially causal finding, for which negative-control analysis is very suitable. Note that either a negative-control outcome variable or a negative-control exposure variable can be tested. It is important to select valid negative controls, where it is clear or well established that there is no true causal effect but the same confounding structure. Besides detecting whether or not there is residual bias, there are also statistical methodologies that can be applied to then correct for this bias (Shi et al., Reference Shi, Miao and Tchetgen2020).

The importance of diversity of thought

While the explicit combination of different types of analytical methods, data sources and/or samples is the major strength of triangulation, it may also pose certain difficulties. These may include lack of financial resources and time to collect new data, constraints in obtaining access to the required secondary data, insufficient methodological expertise and/or insufficient cross-cultural knowledge in your research team. Some of these constraints can be counteracted by the inclusion of collaborators with relevant expertise or datasets. We recommend including diversity of thought and an interdisciplinary team approach at the idea generation stage. Incorporating broad study designs across different fields and comparing across cultural contexts are effective ways to achieve triangulation (Forscher et al., Reference Forscher, Wagenmakers, Coles, Silan, Dutra, Basnight-Brown and IJzerman2023).

Pre-registration

Finally, we highly encourage that an analysis plan is pre-registered prior to conducting analyses. Pre-registration is crucial for the validity of any study, including studies that use secondary data analysis (Baldwin et al., Reference Baldwin, Pingault, Schoeler, Sallis and Munafò2022), because it will reduce questionable research practices and the effects of processes such as ‘confirmation bias’ and ‘hindsight bias’ (Munafò et al., Reference Munafò, Nosek, Bishop, Button, Chambers, Percie du Sert, Simonsohn, Wagenmakers, Ware and Ioannidis2017). For a triangulation study, the most crucial components to formulate and pre-register are the initial causal question, the more refined causal questions per method, the key sources of bias for each method and their expected (direction of) effect, and finally, the expected results under a true causal effect and under no causal effect. Pre-registrations can be posted on platforms such as the Open Science Framework, or researchers could consider submitting to a journal which invites registered reports (Henderson and Chambers, Reference Henderson and Chambers2022).