Autism spectrum disorder (ASD) are neurodevelopmental disorders characterised by persistent impairment in reciprocal social communication and social interaction, and restricted, repetitive patterns of behaviour, interests and activities.1 Symptoms usually present at an early age, typically during the second year of life.1 Early identification and diagnosis is assumed to be beneficial, because it facilitates access to interventions that may improve the child's long-term developmental outcomes.Reference Johnson and Myers2 It may also be helpful in obtaining immediate assistance to children and their families.Reference Johnson and Myers2 However, universal ASD screening is controversial and the evidence supporting such screening is weak. The current guidelines from the American Academy of Pediatrics state that ASD screening should be included in general developmental check-ups at ages 18 and 24 months.Reference Johnson and Myers2 The British guidelines, on the other hand, do not recommend universal ASD screening.3
Screening tools for ASD
A number of screening tools for ASD are available. For children under 30 months of age, the most widely used is the Modified Checklist for Autism in Toddlers, Revised with Follow-Up (M-CHAT-R/F).Reference Robins, Casagrande, Barton, Chen, Dumont-Mathieu and Fein4 The M-CHAT-R/F includes a standardised questionnaire and an additional follow-up interview for those who score above a defined threshold on the questionnaire. In a population-based study of 15 612 children aged 16–30 months in the USA, the M-CHAT-R/F had a positive predictive value of 47.5% for ASD and 94.6% for all types of developmental concerns and delays.Reference Robins, Casagrande, Barton, Chen, Dumont-Mathieu and Fein4
A recent review by the US Preventive Services Task Force of published studies of universal ASD screening in children below 36 months of age noted the lack of follow-up to identify false negatives.Reference McPheeters, Weitlauf, Vehorn, Taylor, Sathe and Krishnaswami5 Without this information, it has not been possible to estimate sensitivity and specificity of screening instruments, and therefore not possible to fully assess the effectiveness of ASD screening. This was also a limitation of the large study of the M-CHAT-R/F, which did not include complete follow-up of the screen-negative children.Reference Robins, Casagrande, Barton, Chen, Dumont-Mathieu and Fein4 Furthermore, it is not well-known whether screening performance is modified by other characteristics of the child and the family, and whether there are disadvantages in implementing universal screening for ASD.
Consequently, we do not know whether existing screening tools for ASD comply with the screening criteria recommended by the World Health Organization.Reference Andermann, Blancquaert, Beauchamp and Dery6 According to these criteria, health screening should not be implemented unless it has been demonstrated that the screening test is effective and that the benefits of screening outweigh the harm and the costs.
Aims
In this study, we have estimated the sensitivity and specificity of the Social Communication Questionnaire (SCQ) for ASD screening at 36 months of age.Reference Berument, Rutter, Lord, Pickles and Bailey7 The SCQ is a 40-item questionnaire that was initially designed for ASD screening in individuals aged 4 years and older.Reference Berument, Rutter, Lord, Pickles and Bailey7 Our study was conducted in the Norwegian Mother and Child Cohort (MoBa).Reference Magnus, Birke, Vejrup, Haugan, Alsaker and Daltveit8 The cohort participants have been followed for a number of years after screening to capture all individuals diagnosed with ASD.
Method
Study population
MoBa is a nationwide, prospective, population-based cohort that includes 114 552 children born from 1999 to 2009.Reference Magnus, Birke, Vejrup, Haugan, Alsaker and Daltveit8 Mothers were recruited at the ultrasound examination provided free of charge to all pregnant women around gestational week 18. Of the women invited to participate, 41% consented. Fathers were also invited to participate if the mothers had consented first. The parents completed questionnaires during pregnancy, and have continued to receive questionnaires as the children grow older. Children with ASD diagnoses have been identified and ascertained through the Autism Birth Cohort (ABC) Study, a study of ASD nested within MoBa.Reference Stoltenberg, Schjolberg, Bresnahan, Hornig, Hirtz and Dahl9
The 36-month questionnaire was distributed to MoBa participants born in 2001 and later, a total of 100 364 children. Of these, there were 58 520 (58%) whose parents returned the questionnaire. ASD case ascertainment has been conducted until 31 December 2015. By that time, the age range of the children whose parents completed the 36-month questionnaire was 6.5–14.7 years (mean 10.2 years).
The SCQ
The SCQ items are modified questions from the standardised diagnostic interview for ASD, the Autism Diagnostic Interview – Revised (ADI-R).Reference Lord, Rutter and Le Couteur10 The items were selected to detect deficits in social interaction and communication, and repetitive and stereotyped behaviours.Reference Berument, Rutter, Lord, Pickles and Bailey7 The first SCQ item is used to determine if the child has phrase speech. If so, the other 39 items are scored. A total score of ≥15 is considered screen-positive for ASD. If the child does not have phrase speech, only the 33 non-verbal items are scored.
Previous studies of the SCQ have utilised clinic-based study samples where children have already been diagnosed with ASD or referred for evaluations of ASD. The manual cut-off level of 15 is based on the initial validation study, which included participants from 4 to 32 years of age.Reference Berument, Rutter, Lord, Pickles and Bailey7 A subsequent American study showed that the SCQ had lower sensitivity in children under 8 years compared with children who were 8 years or older.Reference Corsello, Hus, Pickles, Risi, Cook and Leventhal11 To achieve a sensitivity of 80% in children under 8 years of age, the cut-off had to be lowered to ≥12 for children aged 5–7 years and ≥11 for children under 5 years. More recent studies have also supported the use of 11 as the cut-off level in preschool and elementary-school children.Reference Moody, Reyes, Ledbetter, Wiggins, DiGuiseppi and Alexander12,Reference Barnard-Brak, Brewer, Chesnut, Richman and Schaeffer13
ASD screening in the MoBa cohort
The SCQ was embedded in the questionnaire distributed to MoBa participants at age 36 months. When the MoBa questionnaire was designed, there were no available screening instruments that had been tested in 3-year-olds. The SCQ was chosen because it captures all common symptoms of autism in children, and because the initial validation study indicated that it performed well at age 4 years and older.Reference Berument, Rutter, Lord, Pickles and Bailey7 The investigators who initiated the screening procedures in 2004 decided not to use the seven verbal SCQ items for screening, because they assumed that a large proportion of children with ASD would not have developed spoken language by 3 years of age. In order to ensure high sensitivity, it was decided to use a more lenient cut-off level than 15, and screen-positivity for ASD was defined by a score of ≥12 on the 33 non-verbal SCQ items. Screen-positive children were invited to participate in a 1-day diagnostic assessment. A randomly selected group of age-matched controls was also invited. Controls were selected every 2 weeks among MoBa participants who had passed the age of 37.5 months during the 2 preceding weeks.
Even though the seven verbal SCQ items were not used for screening by the ABC Study, the data are still available. In this article, we have included all the SCQ items in the analyses. We have also estimated the sensitivity and specificity of the SCQ at different cut-off levels:
(a) the cut-off recommended by the SCQ manual: ≥15 for the 39 scored items.
(b) the cut-off recommended by more recent studies: ≥11 for the 39 scored items.
(c) the cut-off used in the ABC Study: ≥12 for the 33 non-verbal items.
Other methods of identifying potential individuals with ASD
In addition to screening, the ABC Study used several methods to detect individuals with ASD in the cohort. The following participants were considered to potentially have ASD and invited to the clinical assessments.
(a) Children referred from their physicians or parents, if there was reason to suspect ASD. Referrals were elicited through annual newsletters to MoBa participants and information on the website of the Norwegian Institute of Public Health.
(b) Children whose parents reported autism, Asperger syndrome or autistic traits in the MoBa questionnaires at ages 5 and 7 years.
(c) Children diagnosed with ASD by specialist health services. Data from all hospitals and out-patient clinics are reported to the Norwegian Patient Register (NPR), beginning in 2008.Reference Suren, Bakken, Aase, Chin, Gunnes and Lie14 Reporting is linked to the government reimbursement system. We conducted annual linkages between MoBa and the NPR from 2008 to 2015 to identify MoBa participants with ASD diagnoses. Until all MoBa children reached 36 months of age in 2012, children with ASD diagnoses in the NPR were invited to the clinical assessments. After that, the diagnoses have been confirmed through medical record reviews, as described below.
ASD case ascertainment
The clinical assessment of the ABC Study included the ADI-R and the Autism Diagnostic Observation Schedule.Reference Lord, Risi, Lambrecht, Cook, Leventhal and DiLavore15 Diagnoses were based on the DSM-IV-TR.16 The ASD case definition included codes 299.00 (autistic disorder), 299.80 (Asperger disorder) and 299.80 (pervasive developmental disorder not otherwise specified).
After 2012, case ascertainment has been conducted through medical record reviews at the clinics where the diagnoses were recorded. In Norwegian specialist health services, diagnoses are coded according to the ICD-10.17 The ASD case definition of our reviews included codes F84.0 (childhood autism), F84.1 (atypical autism), F84.5 (Asperger syndrome), F84.8 (other pervasive developmental disorder) and F84.9 (pervasive developmental disorder, unspecified).
Other factors affecting sensitivity, specificity and screening participation
To examine whether sensitivity and specificity were affected by other types of developmental delay, we stratified the individuals with ASD by the presence or absence of phrase speech (reported on the first SCQ item) and cognitive delay (IQ below 70 at cognitive testing). We examined whether screening participation and screening performance were associated with sociodemographic characteristics of the families: parental education, parental age, maternal parity and maternal living status. We also examined whether these sociodemographic characteristics were associated with children's risk of ASD.
Statistical analyses
First, we evaluated how well the individual SCQ items differentiated between individuals with and without ASD. These abilities were evaluated by likelihood ratios (LRs).Reference Hunink, Glasziou, Siegel, Weeks, Pliskin and Elstein18 An LR is the ratio between the probability of a positive score in the presence of a condition (true positivity) and the probability of a positive score in the absence of the condition (false positivity). A high LR value indicates a high precision of a screening item, whereas an LR value close to one indicates that a screening item does not discriminate between individuals with and without the condition. The ability of a test to distinguish between ‘cases’ and ‘non-cases’ is generally considered to be poor when LR values are lower than 2.Reference Soreide, Korner and Soreide19
We then determined sensitivity, specificity and positive and negative predictive values of the SCQ at the different cut-off levels described previously. We also explored the screening performance of the SCQ through receiver operating characteristic (ROC) curves. ROC curves were created for the SCQ total (manual) score and for each of the three SCQ domains, i.e. communication, social interaction and repetitive behaviours. In an ROC curve, sensitivity (the true-positive proportion) is plotted on the y-axis, whereas 1 – specificity (the false-positive proportion) is plotted on the x-axis.Reference Hunink, Glasziou, Siegel, Weeks, Pliskin and Elstein18 The area under the ROC curve (AUC) is a measure of the overall ability to differentiate between cases and non-cases, and can be interpreted as the probability that a randomly chosen pair of one case and one non-case will be correctly categorised.Reference Hunink, Glasziou, Siegel, Weeks, Pliskin and Elstein18 An AUC value of 0.9–1 is considered excellent, while 0.80–0.89 is good and 0.70–0.79 is fair.Reference Carter, Pan, Rai and Galandiuk20 Values below 0.70 are poor, and tests with values close to 0.50 are of no value.Reference Carter, Pan, Rai and Galandiuk20 Associations between screening participation, sociodemographic characteristics and ASD diagnoses were assessed by Pearson's χ2 tests.
Ethics
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008. MoBa is regulated under the Norwegian Health Registry Act. Written informed consent was obtained from mothers and fathers who participate. The consent covers linkages to health registries and reviews of medical records. The ABC Study has approval from the Regional Committee for Medical and Health Research Ethics for South-East Norway. Participation in the clinical assessments of the ABC Study was based on an additional written informed consent.
Results
By the end of follow-up, a total of 385 individuals with ASD (0.7%) had been detected among the 58 520 children with questionnaire data at 36 months. Of those with ASD, only 56 were detected through screening and clinical assessments. The others were detected through referrals (n = 20), reports of autism or Asperger syndrome in the 5- and 7-year questionnaires (n = 18) and registry linkages (n = 291).
There were 294 (76%) children with ASD who had developed phrase speech at 36 months, while 86 (22%) had not. The information about phrase speech was derived from item 1 of the SCQ. This information was missing for 5 children with ASD (1%). Absence of phrase speech indicates severe language delay, and was largely overlapping with cognitive delay (IQ<70) among the children with ASD. Consequently, we only present results of analyses stratified by the presence or absence of phrase speech.
In non-verbal children with ASD, mean SCQ total score was 13.7 (s.d. = 3.3). In children who were verbal with ASD, the distribution of SCQ scores had large degrees of overlap with those of children without ASD, both for the total score and the different SCQ domains. Mean SCQ total score for children who were verbal with ASD was 9.3 (s.d. = 4.4), whereas the mean SCQ score for children without ASD was 6.2 (s.d. = 3.3).
In Table 1, we show the proportions of positive responses, with corresponding LR estimates, for each of the 40 SCQ items. The ability to differentiate between individuals with and without ASD varied widely. The items covering repetitive language (3–6) and behaviours (7–8, 11–16) had high levels of endorsement in those with ASD and the general population alike, with correspondingly low LR values. In non-verbal children with ASD, the communication items that do not require phrase speech (20–25, 34–35) performed better, with LR values above two for all items except one. The 15 social interaction items (9–10, 19, 26–33, 36–37, 39–40) all had LR values above two for non-verbal children with ASD, and a majority of these items had LR values above 10.
Likelihood ratios of individual Social Communication Questionnaire itemsa
ASD, autism spectrum disorder; R, repetitive and restricted behaviour; C, communication; S, social interaction.
a. Includes all responders to the Norwegian Mother and Child Cohort 36-month questionnaire (n = 58 520), with 385 individuals with ASD: 86 without phrase speech at 36 months, 294 with phrase speech at 36 months and 5 with missing information about phrase speech.
In verbal children with ASD, the repetitive language items had LR values close to one, which indicates a lack of ability to distinguish these children from children without ASD. LR values were somewhat higher for the other communication items, but three out of eight were below two. The performance of the social interaction items was generally better in verbal children with ASD, with 11 out of 15 LR values above 2.
The majority of children with ASD scored below the cut-off level of ≥15 recommended in the SCQ manual (Table 2). The sensitivity was 20% (95% CI 16–24) for ASD as a whole. For non-verbal ASD it was 46% (95% CI 35–57), whereas for verbal ASD it was 13% (95% CI 9–17). Lowering the cut-off to 11 increased sensitivity to 42% (95% CI 37–47) for ASD overall, and to 69% (95% CI 58–79) for non-verbal ASD and 34% (95% CI 29–40) for verbal ASD. Specificity was then substantially reduced, from 99% (95% CI 99–99) to 89% (95% CI 89–90). The positive predictive value was reduced from 9% (95% CI 7–11) to 3% (95% CI 2–3). With the cut-off level that was originally chosen for the ABC Study (≥12 on the 33 non-verbal items), sensitivity was 64% (95% CI 52–74) for non-verbal ASD and 13% (95% CI 9–17) for verbal ASD.
Sensitivity and specificity of the Social Communication Questionnaire (SCQ) at different cut-off levels (total n = 58 520)a
ASD, Autism spectrum disorder; SCQ-33, the 33 SCQ items that do not require phrase speech.
a. Includes all responders to the Norwegian Mother and Child Cohort 36-month questionnaire (n = 58 520 including 385 individuals with ASD).
b. The SCQ manual score was considered valid if there were ≤19 missing values on the SCQ items (available for 379 children with ASD).
c. The SCQ-33 score was considered valid if there were ≤16 missing values on the 33 non-verbal SCQ items (available for 379 children with ASD).
In Fig. 1, we show ROC curves for non-verbal children with ASD. For the communication and social interaction domains, AUC estimates were above 0.9, indicating excellent performance. For the repetitive behaviour domain, on the other hand, the AUC value of 0.67 (95% CI 0.62–0.73) was poor. Still, the AUC for the total SCQ score was 0.88 (95% CI 0.84–0.92), which is considered good for a screening test.
Receiver operating characteristic (ROC) curves for the Social Communication Questionnaire (SCQ): autism spectrum disorder (ASD) without phrase speech at 36 months.
ROC curves for verbal children with ASD are shown in Fig. 2. For the total SCQ score, the AUC was 0.71 (95% CI 0.68–0.74). This is just above the level of 0.7 that separates fair performance from poor performance. There were no apparent cut-off levels that would allow for both high sensitivity and high specificity. AUC estimates were somewhat higher for the social interaction domain, with an AUC of 0.71 (95% CI 0.68–0.74), than for the communication domain, with an AUC of 0.65 (95% CI 0.62–0.68), and the repetitive behaviour domain, with an AUC of 0.62 (95% CI 0.58–0.65).
Receiver operating characteristic (ROC) curves for the Social Communication Questionnaire (SCQ): autism spectrum disorder (ASD) with phrase speech at 36 months.
The 36-month questionnaire also included questions about whether the child had already been referred for evaluations by educational services, developmental disability services or child psychiatry services. Of the 86 non-verbal children with ASD, 71 (83%) had been referred for services by age 3 years. Of the 294 verbal children with ASD, only 52 (18%) had been referred at this age.
A large proportion of the cohort (42%) did not participate in the ASD screening, since the parents did not return the 36-month questionnaire. As shown in Supplementary Table 1 available at https://doi.org/10.1192/bjo.2019.34, the ones most likely to not respond to the questionnaire were parents without a college or university education, young parents and mothers who were single. Those characteristics were also associated with a higher risk of having children with ASD. Because the sensitivity was so low for ASD overall, we have not included analyses of how screening performance may vary across sociodemographic characteristics of the children and their families.
Discussion
Interpretation of the main findings
We investigated the sensitivity and specificity of population-based ASD screening with the SCQ at age 36 months. Most of the individuals with ASD had developed phrase speech at 36 months and did not have significant cognitive delays. Their SCQ score distributions were largely overlapping with the distributions in those without ASD, and there was no obvious way of achieving higher levels of sensitivity without severely compromising specificity. Our findings suggest that it is hard to detect these children through parental report at this age. The ASD symptoms may be too subtle or not yet apparent,Reference Lai, Lombardo, Chakrabarti and Baron-Cohen21 or the symptoms may be underreported when parents are not concerned or actively seeking help.Reference Havdahl, Bishop, Suren, Oyen, Lord and Pickles22
Sensitivity was higher for the children with ASD who had not developed phrase speech and were significantly delayed in their cognitive development. The majority of these children had already been referred to ordinary health or educational services before screening was conducted, suggesting that screening would not be necessary to elicit access to services for them. Screening may still have benefits, though, because it may accelerate the diagnostic process of differentiating ASD from other developmental disorders, thus leading to earlier initiation of adequate interventions. Another benefit, which has not been assessed in this study, is that screening may also detect children with other types of developmental delay.
The SCQ was originally validated for older children. It might have performed better at older ages in this cohort, but then the benefits of early identification would be reduced. The young age at screening may explain why the SCQ questions about repetitive language and behaviours were of little use in distinguishing individuals with ASD from those without. Repetitive language and behaviours are core features of ASD, but such behaviours are also common in typically developing 3-year-olds.Reference Evans, Leckman, Carter, Reznick, Henshaw and King23 Furthermore, the fact that the mothers were not aware that they were completing a screening instrument for autism could have made them more likely to interpret the ‘pathological’ response as favourable, i.e., as signs of the child being active and inquisitive, and practising new skills.
The SCQ questions about social interaction and non-verbal communication query deficits that are expected to be present at an early age in children with ASD, such as lack of eye gaze, social smiling or joint attention. The items of these domains had high levels of endorsement for children with ASD with significant developmental delay, but not for the majority of those with ASD with phrase speech. Limiting the use of SCQ to these two domains would not substantially improve sensitivity.
In the current American guidelines, ASD screening is recommended as part of general well-child visits at ages 18 and 24 months.Reference Johnson and Myers2 The 23 items of the M-CHAT were included in the MoBa 18-month questionnaire. Previous analyses have shown that most children with ASD scored below the pre-defined cut-off levels on this instrument as well.Reference Stenberg, Bresnahan, Gunnes, Hirtz, Hornig and Lie24 This suggests that screening would not have performed any better at earlier ages, or with another screening instrument.
The challenges of autism screening
A critical challenge in autism screening is the lack of a clear boundary between the autism spectrum and the normal range of development. Autistic traits are continuously distributed in the population.Reference Constantino and Todd25 The threshold for receiving a diagnosis depends on clinical judgement, which may vary between clinicians and change over time. In screening, the symptom scores of children with ASD may also overlap with those of other children. These features of the disorder suggest that it does not easily lend itself to detection by screening, at least not for children without general developmental delay. Many researchers and clinicians will still argue that efforts should be made to identify children with ASD as early as possible, because they are likely to have a lot to gain from early intervention.Reference McPheeters, Weitlauf, Vehorn, Taylor, Sathe and Krishnaswami5 If this is to succeed, it will require novel approaches that move beyond screening with a uniform single-informant checklists at a single time point.Reference Reiersen26
The harms and costs of screening must also be considered. The currently recommended cut-off level for the SCQ score is ≥11.Reference Corsello, Hus, Pickles, Risi, Cook and Leventhal11 When we tested this cut-off in our analyses, the sensitivity was 42% for ASD overall and the specificity was 89%. If we had used this cut-off in practice, 11% of all the children would have been defined as screen-positive and required further follow-up. Facilitating follow-up for such a large number of children would be very expensive within the ordinary healthcare system. Another problem is the psychological distress inflicted on so many parents by labelling their children as ‘screen-positive for autism’ when most of them do not have the disorder.
Another challenge of universal health screening is that those who need it the most may be the least likely to participate. This was illustrated by the participation in screening and clinical assessments in our study, which was considerably lower than expected. The proportions responding to the 36-month questionnaire were considerably lower among parents with lower socioeconomic status. The fact that the SCQ was embedded in a large questionnaire may have contributed to the attrition. On the other hand, this is a cohort where most participants are supposedly motivated to participate in research. Participation might have been even lower if the screening had been conducted in a population with a higher representation of disadvantaged parents and ethnic minorities. The problems with low participation might have been mitigated if ASD screening was integrated into routine clinical practice. Nonetheless, integrating the use of the SCQ into routine clinical practice would probably not have improved the sensitivity of screening.
Strengths and limitations
The main strength of our study was its ability to capture false negatives and thereby produce reliable estimates of sensitivity and specificity of early ASD screening with the SCQ. This allowed us to remedy an acknowledged limitation of previous studies.Reference McPheeters, Weitlauf, Vehorn, Taylor, Sathe and Krishnaswami5 However, the study was not primarily designed as a screening study, and had no designated non-screened control group. This limits our ability to quantify the costs and benefits of ASD screening. Another limitation was the large proportion of non-responders to the screening questionnaire. These parents had a higher risk of having children with ASD, and the lack of data from these parents may have led to a slight underestimation of sensitivity. On the other hand, a number of individuals with ASD are still undetected among the younger children in the cohort. Most of these are likely to be verbal children with ASD with low SCQ scores at 36 months, which will probably lead to lower estimates of sensitivity when the cohort participants grow older.
In conclusion, early ASD screening with the SCQ had low sensitivity in this study. The screening captured mainly children with ASD with significant developmental delay, while missing those children with ASD who had cognitive skills in the normal range. Currently available evidence does not support universal early screening programmes for ASD.
Funding
P.S. is supported by the OAK Foundation (grant OCAY-12-326). A.S.-H. is supported by the South-East Norway Regional Health Authority (grant 2018059). She is also a member of the MRC Integrative Epidemiology Unit, which is supported by the University of Bristol and the UK Medical Research Council (MC_UU_12013/1). C. L. receives funding from R91HD081199, R01MH104423 and SFARI 345327. P.M. is supported by the Research Council of Norway through its Centers of Excellence Funding Scheme, project number 262700. MoBa receives funding from the Norwegian Ministry of Health and Care Services and the Ministry of Education and Research. The cohort has been extensively supported by the NIH/NIEHS (contract N01-ES-75558) and the NIH/NINDS (grant 1 UO1 NS 047537-01 and grant 2 UO1 NS 047537-06A1). The NIH/NINDS grants have also funded the Autism Birth Cohort Study.
Acknowledgements
We are grateful to the families participating in MoBa and the ABC Study. We thank Therese Wardenær Bakke and Kaja Schau Knatten for their efforts in facilitating the data collection.
Supplementary material
Supplementary material is available online at https://doi.org/10.1192/bjo.2019.34.
Open access
eLetters
No eLetters have been published for this article.