Introduction
There is a saying from the 13th century that states “misfortunes never come singly” (Simpson & Speake, Reference Simpson and Speake2009), which can easily be applied to children with psychopathology. Symptoms of psychopathology and cognitive performance are inherent to brain function, and thus it is not surprising that they are closely related during development (Frazier et al., Reference Frazier, Demaree and Youngstrom2004; Papachristou & Flouri, Reference Papachristou and Flouri2019; Wagner et al., Reference Wagner, Müller, Helmreich, Huss and Tadić2015). Indeed, both psychopathology symptoms and cognitive deficits put children at risk for later adverse outcomes (Althoff et al., Reference Althoff, Verhulst, Rettew, Hudziak and van der Ende2010; Hofstra et al., Reference Hofstra, van der Ende and Verhulst2002; Koenen et al., Reference Koenen, Moffitt, Roberts, Martin, Kubzansky, Harrington, Poulton and Caspi2009; Meyer et al., Reference Meyer, Carlson, Youngstrom, Ronsaville, Martinez, Gold, Hakak and Radke-Yarrow2009). Relating specific types of psychopathology to specific cognitive domains provides the opportunity to build theories of the underlying neurobiology during development. Further, when psychopathology is assessed in children in clinical settings, cognitive performance is often tested concurrently. Understanding how cognitive performance is altered in those with symptoms of psychopathology can help the diagnostic process and clinical decision-making. Thus, from both an etiological and a clinical perspective, understanding which cognitive domains are impaired, either across or within domains of psychopathology, is of crucial importance.
Symptoms of behavioral problems derived from questionnaires are often divided in two broad domains; internalizing (e.g., anxiety, depression) and externalizing (e.g., aggressive behavior) symptoms. Additionally, the dysregulation profile (DP) resembles combined symptoms of anxiety, attention problems and aggressive behavior and is thus a reflection of combined internalizing and externalizing symptoms (Althoff et al., Reference Althoff, Verhulst, Rettew, Hudziak and van der Ende2010). Children with DP symptoms are of particular interest. From a clinical perspective, those children with DP symptoms have been shown to be at increased risk for continued psychopathology (Althoff et al., Reference Althoff, Verhulst, Rettew, Hudziak and van der Ende2010; Blok et al., Reference Blok, de Mol, van der Ende, Hillegers, Althoff, Shaw and White2021; Meyer et al., Reference Meyer, Carlson, Youngstrom, Ronsaville, Martinez, Gold, Hakak and Radke-Yarrow2009). In addition, cognitive performance has been shown to be considerably lower compared to children without DP symptoms (Basten et al., Reference Basten, van der Ende, Tiemeier, Althoff, Rijlaarsdam, Jaddoe, Hofman, Hudziak, Verhulst and White2014). From an etiological perspective, the DP is also a compelling phenotype to study, as the underlying neurobiology of the DP may share components with both internalizing and externalizing symptoms.
For internalizing symptoms, there is considerable evidence for associations with global cognitive performance as well as specific cognitive domains (Blanken et al., Reference Blanken, White, Mous, Basten, Muetzel, Jaddoe, Wals, van der Ende, Verhulst and Tiemeier2017; Davis et al., Reference Davis, Ollendick and Nebel-Schwalm2008; Favre et al., Reference Favre, Hughes, Emslie, Stavinoha, Kennard and Carmody2008; Matthews et al., Reference Matthews, Coghill and Rhodes2008; Toren et al., Reference Toren, Sadeh, Wolmer, Eldar, Koren, Weizman and Laor2000; Wagner et al., Reference Wagner, Müller, Helmreich, Huss and Tadić2015). In clinical populations, anxiety has been linked to a decreased full scale intelligence quotient (FSIQ), with impairments across multiple cognitive domains (Davis et al., Reference Davis, Ollendick and Nebel-Schwalm2008). Likewise, for cognitive functioning in children and adolescents with major depressive disorder (MDD) a meta-analysis has found that children with MDD had broad cognitive deficits, including lower performance in working memory, verbal fluency, and a lower FSIQ (Wagner et al., Reference Wagner, Müller, Helmreich, Huss and Tadić2015). Further, studies in clinical and population-based samples have observed impairments in the verbal performance (Blanken et al., Reference Blanken, White, Mous, Basten, Muetzel, Jaddoe, Wals, van der Ende, Verhulst and Tiemeier2017; Toren et al., Reference Toren, Sadeh, Wolmer, Eldar, Koren, Weizman and Laor2000), working memory (Blanken et al., Reference Blanken, White, Mous, Basten, Muetzel, Jaddoe, Wals, van der Ende, Verhulst and Tiemeier2017; Matthews et al., Reference Matthews, Coghill and Rhodes2008) and in processing speed (Favre et al., Reference Favre, Hughes, Emslie, Stavinoha, Kennard and Carmody2008).
In the externalizing domain, studies have observed a lower FSIQ in those with externalizing symptoms (Papachristou & Flouri, Reference Papachristou and Flouri2019) as well as more focused deficits in executive functioning (Blanken et al., Reference Blanken, White, Mous, Basten, Muetzel, Jaddoe, Wals, van der Ende, Verhulst and Tiemeier2017). An 11-point lower IQ has been observed in children with antisocial behavior as compared to typically developing children (Koenen et al., Reference Koenen, Caspi, Moffitt, Rijsdijk and Taylor2006). As cognitive deficits are proposed to be at the core of neurodevelopmental disorders (Rutter et al., Reference Rutter, Kim-Cohen and Maughan2006), many studies assessing the externalizing domain have focused on Attention Deficit Hyperativity Disorder (ADHD) and cognitive performance. Similar to studies on antisocial behavior, a meta-analysis on cognitive performance in children with ADHD found a 9 point lower IQ (Frazier et al., Reference Frazier, Demaree and Youngstrom2004). Further, worse working memory performance was observed in children with ADHD, oppositional defiant disorder and conduct disorder as compared to typically developing children (Sergeant et al., Reference Sergeant, Geurts and Oosterlaan2002).
Relatively few studies have specifically focused on children with DP symptoms. In an earlier wave of the Generation R Study, an 11-point lower nonverbal IQ was observed in children classified as having the DP compared to typically developing children (Basten et al., Reference Basten, van der Ende, Tiemeier, Althoff, Rijlaarsdam, Jaddoe, Hofman, Hudziak, Verhulst and White2014). These results are similar to those described above for children with ADHD and antisocial behavior (Frazier et al., Reference Frazier, Demaree and Youngstrom2004; Koenen et al., Reference Koenen, Caspi, Moffitt, Rijsdijk and Taylor2006) and indeed studies have failed to detect differences in cognitive performance between children with ADHD compared to those with comorbid DP symptoms (McGough et al., Reference McGough, McCracken, Cho, Castelo, Sturm, Cowen, Piacentini and Loo2013; Peyre et al., Reference Peyre, Speranza, Cortese, Wohl and Purper-Ouakil2015). However, these studies only compared those with ADHD and the DP in clinical and smaller samples. Together with earlier work in which those with the DP appeared to have a lower IQ than those with internalizing and externalizing problems (Basten et al., Reference Basten, van der Ende, Tiemeier, Althoff, Rijlaarsdam, Jaddoe, Hofman, Hudziak, Verhulst and White2014), this underlines the importance of direct comparisons of psychopathology subgroups.
Despite the amount of effort spent on understanding how psychopathology symptoms and cognitive performance are related, there are crucial questions that remain unanswered. First, while studies comparing clinical diagnoses of psychopathology with typically developing controls provide insight into the differences between cases and controls, these clinical categories may not adequately capture the underlying mechanisms. Second, earlier work has often compared different categories of psychopathology to typically developing children. However, whether cognitive performance in specific domains can distinguish between categories of psychopathology cannot be fully assessed by comparing them individually to typically developing children. Rather, it is important to directly compare cognitive performance across domains in children that show different types of psychopathology (e.g., an internalizing subgroup vs. an externalizing subgroup) in order to better understand the specificity of the relationships. Lastly, even though sex has known effects on both cognitive performance and behavioral problems (Halpern, Reference Halpern2013; Martel, Reference Martel2013), only some work has specifically tested whether associations were different for boys and girls (Van der Ende et al., Reference Van der Ende, Verhulst and Tiemeier2016). Even though this study did not find differences, given the known unequal prevalence rates for specific behavioral problems between boys and girls, coupled with the paucity of studies, sex differences warrant further exploration.
We aim to address these three knowledge gaps in two large population-based samples of children and adolescents. First, we aim to assess the relationship between psychopathology and cognitive performance within two large samples of children and adolescents drawn from the general population. In line with the recommendations of the RDoC initiative, which aims for a better understanding of the underlying mechanisms of dysfunction through assessment of mental health traits along a continuous spectrum (Insel et al., Reference Insel, Cuthbert, Garvey, Heinssen, Pine, Quinn, Sanislow and Wang2010), we use continuous measures of internalizing, externalizing, and DP symptoms. While using continuous measures of psychopathology symptoms can provide insight in the underlying mechanisms, this approach also has one potential downside. Seemingly distinct domains of psychopathology (e.g., internalizing and externalizing symptoms) are likely to be, to some extent, correlated (Achenbach et al., Reference Achenbach, Ivanova, Rescorla, Turner and Althoff2016). When assessing their relationship with cognitive performance, it could be that some of the associations observed with, for example, internalizing symptoms are actually driven by their association with externalizing symptoms. Thus, this continuous approach could mask specificity of individual domains of psychopathology in their relationship with cognitive performance. We have earlier identified more homogeneous behavioral subgroups in our sample using a data-driven approach (Blok et al., Reference Blok, de Mol, van der Ende, Hillegers, Althoff, Shaw and White2021). These homogeneous behavioral subgroups included adolescents without symptoms of psychopathology, and an internalizing, externalizing, and DP subgroups. Thus, within our first aim we incorporate both continuous and categorical approaches. Second, we aim to directly compare mutually exclusive subgroups of children and adolescents with internalizing, externalizing, and DP symptoms, rather than only comparing them to the typically developing children. This will provide insight into whether specific domains of psychopathology are related to specific alterations in cognitive performance. Third, we test whether an interaction with sex is present in the relationship between behavioral problems and cognitive performance. Based on earlier work, we hypothesized that internalizing symptoms would be associated with impaired performance on verbal and working memory tasks, that those with externalizing symptoms would show global cognitive deficits that are most pronounced in the working memory domain, and that those with the DP would have global cognitive deficits which would be significantly greater than the internalizing subgroup. Since only one study has examined sex effects in the association between psychopathology and cognition, we include the role of sex in exploratory analyses.
Methods
Participants
Participants were drawn from two large population-based cohorts, the Adolescent Brain Cognitive Development (ABCD) study (Garavan et al., Reference Garavan, Bartsch, Conway, Decastro, Goldstein, Heeringa and Zahs2018) and the Generation R Study (hereafter Generation R) (Jaddoe et al., Reference Jaddoe, van Duijn, Franco, van der Heijden, van Iizendoorn and de Jongste2012) and. The ABCD study recruited participants through elementary schools from 21 participating sites in the US, between September 2016 and August 2018. Children that participated in the cognitive and behavioral assessment at 9 to 11 years of age (n = 11,392) were eligible for inclusion. The ABCD study aimed for a sample of which approximately 50% of the participants showed early signs of internalizing or externalizing symptoms (Casey et al., Reference Casey, Cannonier, Conley, Cohen, Barch and Heitzeg2018). As a substantial proportion of siblings, twins and triplets were included, we randomly selected one child from each family to participate, resulting in 9,641 children that were eligible for inclusion.
Generation R is a longitudinal birth cohort study in Rotterdam, the Netherlands. Pregnant women with a delivery date between April 2002 and January 2006, who were living within specific zip codes in Rotterdam, were invited for participation. Since recruitment, families have participated in multiple waves of data collection. For this study, all adolescents who participated in the cognitive and behavioral assessment during the 13- to 16-year-old assessment were included (n = 4,200). There were no prespecified exclusion criteria for participation in the cognitive or behavioral assessment. The follow-up rate was high (81.7% of the initial sample was invited for participation in this wave of data collection). However, mothers from invited participants more often had a Dutch national background and a high educational level than those that discontinued participation (Kooijman et al., Reference Kooijman, Kruithof, van Duijn, Duijts, Franco and van Ijzendoorn2016). Both studies were approved by the medical ethics committee or the institutional review board. Written informed consent and assent was obtained prior to enrollment.
Measures
Cognitive performance
The ABCD study and Generation R and used a different set of tests to assess cognitive performance. The cognitive domains assessed in both studies included verbal comprehension, fluid reasoning, working memory and processing speed.
In the ABCD study, the cognitive test battery consisted of the NIH toolbox measures of cognition and additional measures that have been described in more detail previously (Luciana et al., Reference Luciana, Bjork, Nagel, Barch, Gonzalez, Nixon and Banich2018). Briefly, to assess fluid reasoning, the Matrix Reasoning subtest from the WISC-V was used. Raw scores were converted to T-scores (US WISC-V manual Table A1). Verbal comprehension was measured using the Picture Vocabulary and the oral reading recognition test, working memory was measured using the list sorting working memory test, and processing speed was assessed using a pattern comparison task. All NIH toolbox measures of cognition were included to obtain a latent variable with a cognitive composite score (CCS) that best represents the FSIQ, as described earlier (Akshoomoff et al., Reference Akshoomoff, Beaumont, Bauer, Dikmen, Gershon, Mungas, Slotkin, Tulsky, Weintraub, Zelazo and Zelazo2013).
In Generation R, a subset of the Wechsler Intelligence Scale for Children-Fifth Edition (WISC-V) was used (Kaufman et al., Reference Kaufman, Raiford and Coalson2015). The WISC-V is an instrument assessing individual cognitive functioning in 6- to 16-year-olds. In collaboration with Pearson (Pearson Clinical Assessment, San Antonio, TX, USA), four core subtests from the WISC-V were selected to assess specific cognitive domains and to derive an estimated FSIQ. All four subtest were administered by trained research assistants. The Matrix Reasoning and Coding subtests were administered via the Q-Interactive system of Pearson (Daniel et al., Reference Daniel, Wahlstrom and Zhang2014) on an iPad Air 2, which is used by the child while the examiner remained in the room. These tests were automatically scored within the Q-Interactive system. For the Vocabulary subtest, measuring verbal comprehension/reasoning, adolescents had to provide definitions for words read out loud by the examiner. Responses on this subtest were recorded with an audio recorder, which could be used for scoring of the subtest, additional to the responses written down by the examiner. Scoring of the Vocabulary subtest was performed by research assistants. In the Matrix Reasoning subtest, which measures fluid reasoning, adolescents were provided with an incomplete matrix and asked to select the completing response option. The Digit Span subtest, measuring working memory, consisted of three separate tasks. First, the adolescents were asked to repeat a sequence of numbers in the same order that the numbers were presented. Second, the adolescents were asked to repeat a sequence of numbers in reverse order that the numbers were presented. Third, the adolescents were asked to repeat a sequence ordered from low to high. Scoring of the Digit Span subtest was performed by research assistants, after which the three subtasks were equally weighted to compute a Digit Span summary score. Lastly, in the subtest Coding, a measure of processing speed, adolescents were introduced to a key with numbers and corresponding symbols. Subsequently, with the key still present on the screen, adolescents were asked to match as many numbers with the corresponding symbols within 2 min. The digitally administered Coding subtest of the WISC-V has reported issues of inflated scores following hardware and/or software updates (Pearson, 2020). Together with Pearson, we extensively checked for the presence of drift in scores over time. We did not observe this drift in the current sample, likely because hardware and software was held as constant as possible. In a small number of adolescents (n = 132), the tablet was not functioning at the time of the assessment. In these adolescents, the paper/pencil version of the Matrix Reasoning and the Coding subtest was administered.
Raw subtest scores of the Vocabulary, Matrix Reasoning, Digit Span and paper and pencil version of the Coding subtests were converted to age-standardized T-scores based on Dutch norm scores (Dutch WISC-V manual Table A1). Importantly, given that the iPad and paper and pencil version of the Matrix Reasoning task are equivalent, no distinction in scoring (i.e., digitally vs. paper and pencil) of this subtest is necessary. The Coding subtest has documented differences between the digital and paper and pencil assessment (Raiford et al., Reference Raiford, Zhang, Drozdick, Getz, Wahlstrom, Gabel, Holdnack and Daniel2016), therefore raw Coding subtest scores were converted to raw paper pencil scores and then converted to T-scores based on Dutch norm scores (personal contact with Pearson, December 2020). T-scores for all subtests ranged from 1 to 19 and were summed and converted to a four-subtest estimated FSIQ. This custom index shows high reliability and similarity with a FSIQ (average reliability across age 6–16: 0.93). The reliability by age and the conversion table (Erasmus Index) can be found in Tables S1 and S2. To make the results of the CCS and the FSIQ analyses comparable, we scaled the CCS to the mean and standard deviation observed on the FSIQ in Generation R.
Behavioral assessment
Child behavior was assessed using the Child Behavior Checklist (CBCL) version 6–18 in both the ABCD study and Generation R. This questionnaire is a reliable and valid questionnaire to assess child behavior throughout childhood and adolescence. The CBCL has 112 items, rated on a three-point Likert scale (0 = not true, 1 = somewhat true, 2 = often true). The primary caregiver was asked to complete the questionnaire and answer the questions regarding the behavior of their child in the past 6 months. Scores for eight empirically derived syndrome scales can be calculated; Anxious/Depressed, Withdrawn/Depressed, Somatic Complaints, Social Problems, Thought Problems, Attention Problems, Rule-breaking Behavior and Aggressive Behavior. These syndrome scales can be summed up to three broader behavioral constructs, namely internalizing problems (sum score of Anxious/Depressed, Withdrawn/Depressed and Somatic Complaints), externalizing problems (sum score of Rule-breaking Behavior and Aggressive Behavior) and the DP (sum score of Anxious/Depressed, Attention Problems and Aggressive Behavior) (Achenbach & Rescorla, Reference Achenbach and Rescorla2001). Crohnbach’s alphas were calculated for all CBCL scales, based on the samples included. These metrics indicated good to excellent internal consistency (range: 0.87–0.93). Full results are reported in Table S3.
In addition to these three broad constructs, latent profile analysis (LPA) was used to derive distinct behavioral subgroups from the T-scores of the eight syndrome scales, as has been described previously for the Generation R sample (Blok et al., Reference Blok, de Mol, van der Ende, Hillegers, Althoff, Shaw and White2021). Subgroups derived based on LPA were preferred over creating categorical subgroups based on subclinical cutoffs as (1) LPA can better capture the complexity and diversity in symptoms, and (2) using LPA mutually exclusive subgroups that allow for a head-to-head comparison are created. Thus, LPA allows for the creation of independent groups, while maximizing the probability that a participant is similar to others within the group. Details regarding the LPA are provided in the supplemental material. Subgroup profiles and fit indices for the ABCD study were very similar to those observed in Generation R and are provided in Table S4 and Figure S1. Four subgroups were observed in both samples, namely profiles with no problem behavior, internalizing problems, externalizing problems and the DP.
Covariates
Multiple parental and child characteristics were included as covariates. Sex and date of birth of the child were obtained via questionnaire in the ABCD study and from medical records in Generation R. Age of the child during assessment was calculated from the date of birth and the assessment date. Race was assessed with the Multigroup Ethnic Identity Measure Revised (MEIM-R) (Yoon, Reference Yoon2011) in the ABCD study, and categorized as Asian, Black, Hispanic, White, and other. Child national origin was based on the birth country of the parents in Generation R. Three categories were used for child national origin, including Dutch, other Western (American Western, Asian Western, European, Indonesian, and Oceania) and non-Western (African, American non-Western, Asian non-Western, Cape Verdean, Dutch Antilles, Moroccan, Surinamese, and Turkish). Socioeconomic status was determined based on household income and maternal education. Household income was assessed through questionnaire and harmonized across the two samples by categorizing in two levels, it was rated low in the ABCD study if they had less than 25,000 dollar a year and in Generation R when the household had less than 2,000 euro a month; above these thresholds income was classified as above low. Maternal education was assessed by questionnaire and divided into low (ABCD study: education until 6th grade, Generation R: no education, primary school), middle (ABCD study: education until 12th grade, high school diploma, associate degree, Generation R: high school, vocational training), and high (ABCD study: educational level above associate degree, Generation R: higher vocational training, university). Parental psychopathology was measured by the Adult Self Report total problems scale T-scores (Achenbach & Rescorla, Reference Achenbach and Rescorla2003), in the ABCD study. In Generation R, parental psychopathology was measured with the anxiety and depression subscales of the Brief Symptom Inventory in Generation R (De Beurs, Reference De Beurs2004). Maternal IQ was measured in Generation R only, using a computerized version of the Ravens Advanced Progressive Matrices Test, set I (Prieler, Reference Prieler2003).
Statistical analyses
Primary analyses
Our primary analyses aimed to assess the relationship between continuous and categorical measures of psychopathology (internalizing, externalizing, and DP symptoms) and distinct cognitive domains. All analyses were performed separately in the ABCD study and Generation R. Linear regression analyses were used, in which cognitive domains were entered as dependent variables, and the continuous measures of psychopathology or subgroups derived with LPA were used as independent variables. Separate regression analyses were performed for each continuous measure of psychopathology and each cognitive domain. As we used population-based samples, CBCL scales were right-skewed, thus we square root transformed all continuous CBCL scales prior to running the analyses. To increase interpretability, we standardized all cognitive and behavioral measures for our continuous analyses to a mean of 0 and a standard deviation of 1. In the categorical approach, unstandardized cognitive domains were used. The no problems subgroup was entered as the reference group. All analyses were adjusted for several covariates in two models. In the first model we included child sex and age at assessment, in the second model we additionally adjusted for child national origin/race, socioeconomic status and parental psychopathology.
Secondary analyses
In our secondary analyses we tested whether there was an effect of sex on the relationship between continuous measures of psychopathology and cognitive performance. Therefore, continuous analyses were repeated, including an interaction term for sex.
Post hoc analyses
Three types of post hoc analyses were conducted. First, following our continuous analyses, we performed linear regression analyses to assess whether specific syndrome scales underlying the assessed broadband CBCL scales (internalizing, externalizing, and DP) were associated with cognitive domains. Second, following the categorical analyses, psychopathology subgroups were directly compared; thus, in separate models, we entered the internalizing and externalizing subgroups as the reference group. Lastly, when significant interaction effects with biological sex were detected in our secondary analyses, post hoc simple slopes analyses were conducted.
Sensitivity analyses
To assess whether the relationship was independent of maternal IQ, sensitivity analyses were performed within Generation R, where maternal IQ was added as a covariate. Further, to assess whether the group of children that scored in the lowest 25% on the CCS/FSIQ measure was driving the obtained results. Continuous analyses were repeated excluding all children scoring under the first quantile (CCS: 94.8, FSIQ: 93.0). Lastly, categorical analyses were repeated, in which the presence of subclinical psychopathology traits was based on raw score cutoffs. Children and adolescents that scored above the 80th percentile on internalizing, externalizing, or DP symptoms were coded to have subclinical symptoms, children and adolescents that scored below this threshold were coded to have no symptoms.
Analyses were performed in R version 3.6.3 (R Core Team, 2013). Missing covariates were imputed using multiple imputation through chained equations (mice) (van Buuren et al., Reference van Buuren, Groothuis-Oudshoorn, Robitzsch, Vink, Doove and Jolani2015), with 30 imputed datasets and 30 iterations per dataset with a maximum of 15.8% missing. Primary and exploratory analyses were corrected for multiple testing using the FDR-Benjamini–Hochberg procedure for a total of 99 tests (Benjamini & Hochberg, Reference Benjamini and Hochberg1995).
Results
Sample characteristics
Sample characteristics, including demographics, levels of psychopathology, number of children and adolescents included in each psychopathology subgroup, and cognitive performance for both samples are shown in Table 1. In the ABCD study approximately half of the children were White (51.0%), and in Generation R, the majority of adolescents had a Dutch background (64.5%). Both samples had a high SES on average, with most parents classified as having a high education level (ABCD study: 53.0%, Generation R: 56.7%) and an above low income (ABCD study: 77.2%, Generation R: 71.0%). In Table 2, the sample characteristics split by biological sex are provided. Within sample comparisons showed small, but mostly significant differences between boys and girls. Lastly, in Generation R a nonresponse analysis was performed comparing those participants who were included at baseline but did not take part in the current study, to those included in the current study. Comparing data collected at baseline, the participants who were included in this study were more often girls (χ2 = 7.86, df = 1, p = .005), had a Dutch national background (χ2 = 446.40, df = 2, p < 2.20 × 10−16), and had mothers with a higher educational level (χ2 = 189.99, df = 2, p < 2.20 × 10−16), higher household income (χ2 = 216.50, df = 1, p < 2.20 × 10−16), lower anxiety (mean difference = 0.30, t-statistic = 3.74, df = 2108.5, p = 1.87 × 10−4), and lower depressive symptoms (mean difference = 0.36, t-statistic = 4.18, df = 2010.1, p = 3.06 × 10−5).
Table 1. Sample characteristics
Table 2. Cognitive performance and psychopathology levels stratified by sex
Note. * Indicates a significant difference between boys & girls within one sample (p < .05). T-tests and χ2-tests were performed to compare the groups for psychopathology traits and cognitive performance.
Continuous measures of psychopathology
Internalizing symptoms
For internalizing symptoms, we found worse processing speed performance in the first, but not the second model, within the ABCD study. Contrary, we observed better performance in children with higher internalizing problems for the picture vocabulary subtest (β = 0.03, SE = 0.01, p-value = .012). In the Generation R Study, worse working memory (β = −0.04, SE = 0.02, p-value = .006) and processing speed (β = −0.06, SE = 0.02, p-value < .001) performance was observed. Additionally, a lower FSIQ was observed after adjustment for child sex and age at assessment (model 1), but this association disappeared in model 2, after additional adjustment for child national origin/race, SES, and parental psychopathology (Table 3). In our second model associations became significant that did not reach statistical significance in our first models, suggesting negative confounding effects of the variables included in the second model (national origin/race, SES, and parental psychopathology). In the ABCD study higher internalizing symptoms were associated with a lower CCS (β = −0.03, SE = 0.01, p-value = .01), and in Generation R, higher internalizing symptoms were related to better performance on the vocabulary subtest (β = 0.04, SE = 0.02, p-value = .007).
Table 3. Associations between continuous symptoms of psychopathology and cognitive performance
Note. Model 1 was corrected for child sex and age at assessment, model 2 was additionally corrected for child national origin/race, socioeconomic status and parental psychopathology. Psychopathology and cognitive performance measures were standardized to obtain β coefficients. Results that are significant at p < .05 are shown in bold, * represent the results that remained significant after multiple testing correction.
Externalizing and DP symptoms
Children and adolescents with greater externalizing and DP symptoms had lower cognitive performance across all cognitive domains. One exception was the verbal domain in Generation R, in which no statistically significant association was observed with DP symptoms after correction for multiple testing (Table 3).
Categorical subgroups of psychopathology
Our categorical approach provided very similar results to those obtained in our continuous analyses (Figure 1 and Table 4). In the internalizing subgroup, worse performance in working memory, processing speed, and the FSIQ/CCS was observed in both samples. The externalizing and DP subgroups showed global worse performance in both samples. In the ABCD study, an exception was the processing speed subtest in the externalizing subgroup, which did not reach statistical significance. Additionally, the verbal domain was not significant for the DP subgroup and only reached statistical significance in the first model for the externalizing subgroup, within Generation R.
Figure 1. Cognitive performance of children in subgroups of psychopathology as compared to those with no behavioral problems.
Table 4. Associations between categorical subgroups of psychopathology and cognitive performance
Note. Model 1 was corrected for child sex and age at assessment, model 2 was additionally corrected for child national origin/race, socioeconomic status, and parental psychopathology. Results that are significant at p < .05 are shown in bold, * represent the results that remained significant after multiple testing correction.
Interaction with sex
An interaction effect of sex was observed for externalizing symptoms and the CCS (β = 0.03, SE = 0.02, p = .043) within the ABCD study. In Generation R, interaction effects with sex were observed with the working memory subtest in Generation R for internalizing symptoms (β = −0.06, SE = 0.03, p-value = .033), externalizing symptoms (β = −0.07, SE = 0.03, p-value = .016), and DP symptoms (β = −0.11, SE = 0.03, p-value < .001). Additionally, interaction effects were observed with the FSIQ for externalizing symptoms (β = −0.07, SE = 0.03, p-value = .029) and DP symptoms (β = −0.09, SE = 0.03, p-value = .003). After correction for multiple testing, the association between working memory and externalizing or DP symptoms and the FSIQ and DP symptoms remained statistically significant. Full results for interaction effects with sex are shown in Table 5.
Table 5. Interaction effect of sex on the associations between continuous symptoms of psychopathology and cognitive performance
Note. Model 1 was corrected for child sex and age at assessment, model 2 was additionally corrected for child national origin/race, socioeconomic status and parental psychopathology. Psychopathology and cognitive performance measures were standardized to obtain β coefficients. Results that are significant at p < .05 are shown in bold, * represent the results that remained significant after multiple testing correction.
Post hoc analyses
Syndrome scales underlying internalizing, externalizing, and DP scales
The first set of post hoc analyses assessed the relationship between cognitive performance and the syndrome scales underlying the included broad CBCL scales. Interestingly, most of the observed effects in the broad scales were mirrored in the underlying syndrome scales. A notable exception was the positive relationship between the vocabulary domain and internalizing symptoms. In the ABCD study, none of the individual syndrome scales was associated with higher performance on the verbal tasks, whereas in Generation R higher performance was mainly driven by the withdrawn/depressed, and to a lesser extent by the anxious/depressed symptoms. Further, regarding the internalizing domain, the lower CCS observed in the ABCD study was driven mainly by the anxious/depressed and withdrawn/depressed syndrome scales. Negative associations for working memory and processing speed were present for anxious/depressed symptoms and somatic complaints, but not for withdrawn/depressed symptoms, in Generation R. Lastly, while all syndrome scales underlying the DP showed the negative associations with cognitive performance observed for the DP, the attention problems syndrome scale was most strongly associated to cognitive performance. Full results are shown in Table S5.
Direct comparison categorical subgroups of psychopathology
In the second set of post hoc analyses, we compared the mutually exclusive subgroups of psychopathology. Direct comparison with the internalizing subgroup revealed differences in the verbal domain of the externalizing subgroup in the ABCD study (Picture vocabulary: B = −0.97, SE = 0.34, p-value = .004) and Generation R (B = −0.43, SE = 0.20, p-value = .028), with the externalizing subgroup performing worse than the internalizing subgroup. In the ABCD study, the externalizing subgroup additionally performed significantly worse on the fluid reasoning subtest (B = −0.34, SE = 0.13, p-value = .009) and had a lower CCS (B = −1.16, SE = 0.56, p-value = .038). Children in the DP subgroup performed worse on all cognitive domains except processing speed as compared to those in the internalizing subgroup, in the ABCD study. In Generation R, those in the DP subgroup performed worse compared to those in the internalizing subgroup on fluid reasoning (B = −0.84, SE = 0.27, p-value = .002) and FSIQ (B = −3.26, SE = 1.35, p-value = .016). Comparing the externalizing and DP subgroup directly, those in the DP showed worse performance in the working memory domain (B = −1.81, SE = 0.70, p-value = .009) and the CCS (B = −1.54, SE = 0.71, p = .029) in the ABCD study only. Full results for the direct comparisons are shown in Tables S6 and S7.
Simple slopes analyses
Simple slopes analyses of the significant interaction effects indicated that cognitive domains were more heavily affected in boys than in girls with comparable psychopathology symptoms. Full results for the simple slopes analyses are presented in Table 6, and plots are provided in Figure 2.
Table 6. Simple slopes analyses for significant sex interactions on the associations between continuous symptoms of psychopathology and cognitive performance
Note. Models were corrected for child sex and age at assessment, child national origin/race, socioeconomic status and parental psychopathology. Psychopathology and cognitive performance measures were standardized to obtain β coefficients.
Figure 2. Simple slopes analyses for significant sex interactions on the associations between continuous symptoms of psychopathology and cognitive performance.
Sensitivity analyses
We performed three sets of sensitivity analyses. In the first set, we additionally adjusted for maternal IQ in Generation R. Obtained results were in both the continuous and categorical analyses very comparable to those in model 2, full results are shown in Tables S8 and S9. Second, we excluded children scoring below the first quartile of the CCS/FSIQ measure. Some of the effects, most notably those observed for externalizing and DP symptoms, attenuated after excluding these children. However, the majority of the findings remained statistically significant. Full results are presented in Table S10. Finally, categorical analyses were repeated using raw score cutoffs. The results were largely consistent with the continuous analyses in both cohorts and are described in Table S11.
Discussion
We examined the relationship between broad symptom domains of psychopathology and multiple domains of cognitive performance using both continuous and categorical approaches. In the latter we also directly compared mutually exclusive subgroups of psychopathology. Lastly, we explored the effect of sex on the relationship between psychopathology and cognitive performance. Regarding internalizing symptoms, we observed worse performance in the working memory and processing speed subtests, but contrary to our hypotheses, we also found better performance in the verbal domain, both in the Generation R and ABCD studies. In line with our hypothesis, we found evidence for worse cognitive performance across domains in children and adolescents with externalizing and DP symptoms. Notably, we observed a 5.0 point lower FSIQ in the DP subgroup compared to typically developing children and adolescents, even after correction for multiple covariates. When compared to children and adolescents in the internalizing and externalizing subgroups, those in the DP subgroup also had worse cognitive performance. Lastly, we show that the relationship between cognitive performance (working memory and FSIQ) and psychopathology is stronger in adolescent boys than in girls.
Our categorical analyses showed a lower FSIQ/CCS in those with internalizing symptoms, driven by worse performance in working memory and processing speed, which is in line with earlier work in both clinical and population-based settings (Blanken et al., Reference Blanken, White, Mous, Basten, Muetzel, Jaddoe, Wals, van der Ende, Verhulst and Tiemeier2017; Davis et al., Reference Davis, Ollendick and Nebel-Schwalm2008; Favre et al., Reference Favre, Hughes, Emslie, Stavinoha, Kennard and Carmody2008; Matthews et al., Reference Matthews, Coghill and Rhodes2008; Papachristou & Flouri, Reference Papachristou and Flouri2019). The working memory and processing speed together form the cognitive proficiency index of the WISC-V, which measures how efficient information is processed (Kaufman et al., Reference Kaufman, Raiford and Coalson2015). Indeed, the attentional control theory makes a distinction between performance effectiveness (i.e., how accurate is the task performed) and performance efficiency (i.e., how many resources are spent on the task) (Eysenck et al., Reference Eysenck, Derakshan, Santos and Calvo2007). Internalizing symptoms include depressive symptoms as well as anxiety. In depression, having slowed thought processes is a key symptom (American Psychiatric Association, 2013), which may underlie the lower processing speed. Moreover, although speculative, it could be that anxious thoughts consume some cognitive capacity, which in turn reduces the resources available for other tasks. Thus, those with internalizing symptoms could perform worse on tasks with increased demand of resources, such as working memory tasks.
Contrary to our hypothesis, we observed a positive relationship between internalizing symptoms and verbal performance. In line with these results, earlier work in undergraduate students observed a positive relationship between verbal skills and ruminating, and between verbal skills and internalizing symptoms (Penney et al., Reference Penney, Miedema and Mazmanian2015). Potentially, for susceptible individuals, better verbal skills could lead to greater rumination, which can eventually lead to internalizing symptoms. However, there are also many studies that reported a negative relationship or did not find evidence for a relationship between internalizing symptoms and verbal performance (Ang et al., Reference Ang, Frontero, Belleau and Pizzagalli2020; Blanken et al., Reference Blanken, White, Mous, Basten, Muetzel, Jaddoe, Wals, van der Ende, Verhulst and Tiemeier2017; Bornstein et al., Reference Bornstein, Hahn and Suwalsky2013; Castagna et al., Reference Castagna, Calamia and Davis Iii2021; NoackLeSage et al., Reference NoackLeSage, Shaheen, Davis, Castagna, Kaskas, Ryan and Lilly2019; Toren et al., Reference Toren, Sadeh, Wolmer, Eldar, Koren, Weizman and Laor2000). Multiple explanations could underlie these differences in findings. First, disparate findings between studies could be attributed to sample differences. For example, within the ABCD study, recent work has shown that those with a current depression had a lower verbal performance than typically developing children, but no differences were observed between typically developing children and those at high-risk or those that had remitted from a depression (Ang et al., Reference Ang, Frontero, Belleau and Pizzagalli2020). In our study, we focused on children with mostly subclinical symptoms, suggesting that the positive relationship may be present for subclinical, but not current clinical symptoms. Second, both internalizing symptoms and verbal performance are multifaceted. It is possible that the positive relationship observed here is present for some, but not all of the multi-facetted elements. This idea is supported by our findings that the relationship was driven by anxiety/depressed and withdrawn/depressed symptoms, but not somatic complaints, as well as findings from earlier work. A previous study in early childhood showed that separation anxiety was related to lower reading achievement, whereas harm avoidance was related to higher achievement (Grills-Taquechel et al., Reference Grills-Taquechel, Fletcher, Vaughn, Denton and Taylor2013). Additionally, recent work showed that those with low worrying symptoms, as opposed to high worrying symptoms, in combination with high physical anxiety, had a higher verbal performance (Castagna et al., Reference Castagna, Calamia and Davis Iii2021).
Children and adolescents with externalizing and DP symptoms showed global cognitive deficits. This is in line with earlier findings in children with ADHD, antisocial behavior, and the DP (Basten et al., Reference Basten, van der Ende, Tiemeier, Althoff, Rijlaarsdam, Jaddoe, Hofman, Hudziak, Verhulst and White2014; Frazier et al., Reference Frazier, Demaree and Youngstrom2004; Koenen et al., Reference Koenen, Caspi, Moffitt, Rijsdijk and Taylor2006). Factors contributing to these global deficits could be genetic, environmental or stochastic processes (White, Reference White2019). Within the domain of modifiable factors, education plays an important role in cognitive development. Externalizing symptoms can interfere with school functioning (Moilanen et al., Reference Moilanen, Shaw and Maxwell2010; Van der Ende et al., Reference Van der Ende, Verhulst and Tiemeier2016) and worse academic performance can lead to increased externalizing symptoms (Moilanen et al., Reference Moilanen, Shaw and Maxwell2010). Potentially, breaking this vicious cycle can improve cognitive development in those with externalizing symptoms. To date only one study has assessed the relationship between DP symptoms and educational attainment, showing that in adulthood the highest educational level is lower in those with more DP symptoms (McQuillan et al., Reference McQuillan, Kultur, Bates, O’Reilly, Dodge, Lansford and Pettit2018). Future studies should assess the relationship between cognitive performance and DP symptoms using longitudinal models and compare the discrepancy between academic performance and IQ in this group to elucidate how much school performance is hampered by DP symptoms.
A qualitative comparison between the relationships observed in children (ABCD study) and adolescents (Generation R) reveals that both in childhood and adolescence externalizing and DP symptoms are related to lower cognitive performance. While effect sizes cannot be directly compared for most cognitive domains, due to the use of different measures of cognitive performance, the Matrix Reasoning subtask, which measured fluid reasoning, was used in both cohorts. Effect sizes were highly similar, indicating that this relationship was similar in childhood and adolescence. However, regarding DP symptoms, earlier work within Generation R has revealed an 10.9 point lower nonverbal IQ in early childhood after adjustment for age and sex (Basten et al., Reference Basten, van der Ende, Tiemeier, Althoff, Rijlaarsdam, Jaddoe, Hofman, Hudziak, Verhulst and White2014), whereas here we observed a 6.3 point lower IQ using the same covariates. While this difference could be explained by a multitude of environmental and methodological differences, it is likely that the magnitude of the relationship between DP symptoms and cognitive performance undergoes some change over time.
Few studies have directly compared subgroups with distinct patterns of psychopathology, and thus did not assess the specificity of cognitive performance differences between domains of psychopathology (McGough et al., Reference McGough, McCracken, Cho, Castelo, Sturm, Cowen, Piacentini and Loo2013; Peyre et al., Reference Peyre, Speranza, Cortese, Wohl and Purper-Ouakil2015). Compared to the internalizing subgroup, children in the externalizing group performed worse on vocabulary, fluid reasoning, and CCS in the ABCD study, but adolescents only performed worse on the vocabulary subtest in Generation R. Those in the DP subgroup showed a lower FSIQ than those in the internalizing subgroup, driven by worse performance on all domains in the ABCD study and on fluid reasoning in Generation R. Regarding the difference between the externalizing and DP subgroup, earlier work examining differences between ADHD and DP was unable to discriminate between the two (McGough et al., Reference McGough, McCracken, Cho, Castelo, Sturm, Cowen, Piacentini and Loo2013; Peyre et al., Reference Peyre, Speranza, Cortese, Wohl and Purper-Ouakil2015). Here, we show that children in the DP subgroup have worse working memory performance and CCS than those in the externalizing subgroup.
In Generation R, boys with externalizing and DP symptoms performed worse than girls on the working memory subtest. For DP symptoms, boys also had worse performance than girls on the FSIQ. Thus, for those domains the relationship between psychopathology and cognitive performance is stronger in boys than in girls. These findings, together with earlier evidence that boys in general have more externalizing and DP symptoms (Holtmann et al., Reference Holtmann, Goth, Wöckel, Poustka and Bölte2008), suggest that clinicians should be aware of possible poorer cognitive skills in boys as compared to girls with the same level of psychopathology.
Findings that warranted further exploration were the associations between the vocabulary subtest and the CCS, and continuous internalizing symptoms. For vocabulary and the CCS we only observed significant associations in the second model, but not in the first model, pointing toward negative confounding effects from at least one of the covariates added in the second model (Porta, Reference Porta2014). Exploration of these covariates revealed that for the verbal domain, having a non-Western national origin or having a lower SES were driving the worse performance. For the CCS, SES, and ethnic background influenced cognitive performance. This is an interesting starting point for future studies to assess why these differences arise. Notably, we assessed these differences within two Western populations. We can speculate that having a migration background can have impact on both cognitive development and psychopathology. Regarding cognitive development, it might be that those with non-native backgrounds use a different language in their home environment, by which they spend less time practicing the language from the country of residence, potentially leading to worse test performance in the verbal domain. Further, parents from non-native backgrounds could have more trouble with the school system, simply because they are less familiar with it. Lastly, it could be that parents with a lower SES have less time and capacity to help their children with homework. Regarding psychopathology, those that migrated themselves can have severe traumas and higher rates of psychopathology depending on the reason for immigration (Cénat et al., Reference Cénat, Charles and Kebedom2020). Further, for those with a migration background, racial discrimination can impact mental health outcomes (Chou et al., Reference Chou, Asnaani and Hofmann2012) and some migrant groups have more trouble accessing mental health care (Lindert et al., Reference Lindert, Schouler-Ocak, Heinz and Priebe2008). To adequately help all children and adolescents, we need to acknowledge that there are certain groups that are more vulnerable than others, and study how psychopathology can be reduced and concurrently cognitive development can be improved in these groups.
The current study has several clinical implications. First, both in a continuous and a categorical approach we observed that cognitive performance and psychopathology were closely related. These findings suggest that not only those at the clinical end of the spectrum have lower cognitive performance, but that the relationship exists along the spectrum of symptom severity. Thus, cognitive training interventions that are effective for children with clinical symptoms, may also be effective for those who have subclinical symptoms, but who may not be help seeking. Second, lower cognitive performance was observed spanning all cognitive domains, as opposed to deficits in only specific neuropsychological domains, in those with externalizing and DP symptoms. Thus, treatment strategies should be directed at improving cognition across multiple domains, rather than targeting a specific cognitive domain. Third, those in the externalizing and DP subgroups showed worse cognitive performance than those in the internalizing subgroup. Although these differences were less pronounced in adolescence, it is important to understand in clinical settings that children with externalizing and DP symptoms will, on average, have poorer cognitive performance. Thus, treatment may not only be directed toward the clinical symptoms, but these children will likely also need extra support in the classroom. This stresses the importance of implementing recommendations made over a decade ago by the American Academy of Child and Adolescent Psychiatry, stating that psychiatrists and teachers should work together to improve the learning environment for individuals with psychopathology symptoms (Walter & Berkovitz, Reference Walter and Berkovitz2005). Lastly, the interaction and simple slopes analyses revealed that the relationship between psychopathology and cognitive performance was stronger in boys than in girls, implying that an extra focus on boys is warranted.
In addition to these clinical implications, our findings can also be extended in several ways. Most importantly, longitudinal studies will be important to identify the trajectories of behavior and cognition over time, including their interplay. Further, this study has shown that cognitive performance is related to psychopathology, but an important extension would be to assess whether cognitive performance can be improved with treatment for psychopathology; and conversely, to what extent symptoms of psychopathology can be reduced by implementing cognitive training. Given that those with DP symptoms are most severely affected, this group of children provides an interesting sample for pilot studies to implement such cognitive training. Potentially fruitful future work could also be focused on interventions in the classroom to promote equal learning opportunities for all children, regardless of their level of psychopathology symptoms. For example, by designing courses directed at teachers that can better equip them to promote a more inclusive learning environment. This is especially important in countries, such as the Netherlands, in which “inclusive education” has been implemented, but where currently 25% of the teachers feel that they do not possess sufficient knowledge to adequately teach those children with more psychopathology symptoms and approximately 50% of the teachers expressed a desire for support in terms of knowledge and practical tips to improve the learning experience for children with psychopathology (Smeets et al., Reference Smeets, Ledoux, van Loon-Dikkers, Bollen, Eimers, Joosten, Weijers and Welles2019). Lastly, for internalizing symptoms specifically, the positive relationship observed with verbal performance requires further exploration and confirmation. Given the mixed literature, a promising future direction would be to dissect internalizing symptoms into multiple subdomains along a continuum, to capture the complexity and dimensionality of these symptoms and the relationship between each subdomain with dimensions of cognition.
Strengths of this study include the large sample size involving two separate studies on two different continents. The ABCD study had nearly 10,000 children and Generation R had approximately 4,000 adolescents and both participated in extensive cognitive testing. The two large studies offered the opportunity to demonstrate replication of the findings. Second, the population-based characteristics of both studies allowed for both a continuous and a categorical approach regarding psychopathology. Third, within our categorical approach, we compared subgroups of psychopathology not only to a group of typically developing children and adolescents, but also directly compared mutually exclusive subgroups of psychopathology. Despite these strengths, our results should also be interpreted in the light of some limitations. First, a different set of measures was used to assess cognitive performance in the two samples, with the exception of fluid reasoning. However, the tests, although different, did capture the same cognitive domains and the results were very similar across both samples. The similarity in the findings supports that the effects are not test-specific, but rather truly related to the underlying cognitive construct. Second, in the current study we used cross-sectional data, by which we cannot draw conclusions about the directionality of the observed associations. Future studies should use longitudinal designs to assess the temporal relationship between cognitive and behavioral development.
Taken together, our findings provide evidence for globally worse cognitive performance in children and adolescents with externalizing and DP symptoms, with those in the DP subgroup being most heavily affected. Internalizing problems are more subtly related to cognitive performance, with worse performance on working memory and processing speed, but higher performance in the verbal domain. We did not identify specific patterns of impairments of cognitive domains with distinct subtypes of psychopathology, with the exception of the increased performance in the verbal domain for those with internalizing symptoms. Rather, we provide evidence for DP symptoms as a severe behavioral phenotype that is related to impaired cognitive performance across childhood as compared to both typically developing children and adolescents, and those with other types of psychopathology.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0954579422000165.
Acknowledgments
Data used in the preparation of this article were obtained from the Adolescent Brain Cognitive DevelopmentSM (ABCD) Study (https://abcdstudy.org), held in the NIMH Data Archive (NDA). This is a multisite, longitudinal study designed to recruit more than 10,000 children age 9–10 and follow them over 10 years into early adulthood. The ABCD Study® is supported by the National Institutes of Health and additional federal partners under award numbers U01DA041048, U01DA050989, U01DA051016, U01DA041022, U01DA051018, U01DA051037, U01DA050987, U01DA041174, U01DA041106, U01DA041117, U01DA041028, U01DA041134, U01DA050988, U01DA051039, U01DA041156, U01DA041025, U01DA041120, U01DA051038, U01DA041148, U01DA041093, U01DA041089, U24DA041123, U24DA041147. A full list of supporters is available at https://abcdstudy.org/federal-partners.html. A listing of participating sites and a complete listing of the study investigators can be found at https://abcdstudy.org/consortium_members/. ABCD consortium investigators designed and implemented the study and/or provided data but did not necessarily participate in analysis or writing of this report. This manuscript reflects the views of the authors and may not reflect the opinions or views of the NIH or ABCD consortium investigators. The ABCD data repository grows and changes over time. The ABCD data used in this report came from https://doi.org/10.15154/1524469.
Funding statement
This work was supported by the Sophia Children’s Hospital Research Foundation (SSWO) Project (TW, #S18-68, #S20-48) and the Netherlands Organization for Health Research and Development (ZonMw) TOP (TW, project number 91211021). The general design of the Generation R Study is made possible by financial support from the Erasmus Medical Center, Rotterdam, ZonMw, the Netherlands Organization for Scientific Research (NWO), and the Ministry of Health, Welfare and Sport, and is conducted by the Erasmus Medical Center in close collaboration with the Faculty of Social Sciences of the Erasmus University Rotterdam, and the Stichting Trombosedienst & Artsenlaboratorium Rijnmond (STAR-MDC), Rotterdam.
Conflicts of interest
The authors declare that they have no conflict of interest.
Open access
