Screening instruments

SNAP cohort

As part of the larger SNAP study of the prevalence of autistic-spectrum disorders (Reference Baird, Simonoff and PicklesBaird et al, 2006), within a total population cohort of 56 946 children born between 1 July 1990 and 31 December 1991 all those with a current clinical diagnosis of pervasive developmental disorder (n=255) or considered at risk of having the undetected disorder by virtue of having a statement of special educational needs (n=1515) were screened using the SCQ. (In the UK a statement of special educational needs is a legal document issued by a local educational authority when children require significant additional support in school because they have learning and/or behavioural problems.) A total of 1066 SCQs were returned completed (return rate 60.2%); 31 families declined further participation, leaving 1035 (return rate 58.5%) who returned the SCQ and opted in for further assessments. Mean age at SCQ screening in the whole SNAP sample was 10.3 years (s.d.=0.4 years). We have previously reported on the screening properties of the SCQ in the total sample, finding similar discrimination between individuals with autistic-spectrum disorders and those without as in the original validation sample (sensitivity 0.88; specificity 0.72; Reference Chandler, Charman and BairdChandler et al, 2007).

A stratified subsample (by coincidence, also n=255) received a comprehensive diagnostic assessment, including standardised clinical observation (the Autism Diagnostic Observation Schedule - Generic (ADOS-G; Reference Lord, Risi and LambrechtLord et al, 2000)), parent-interview assessments of autistic symptoms (ADI-R; Reference Lord, Rutter and LeCouteurLord, et al, 1994), language and IQ tests, evaluation of psychiatric comorbidities and a medical examination. The team used ICD-10 research diagnostic criteria (World Health Organization, 1993) to derive a clinical consensus diagnosis of childhood autism, other autistic-spectrum disorders or no autistic-spectrum disorder (for details see Reference Baird, Simonoff and PicklesBaird et al, 2006). For 36 randomly selected children, project consensus diagnoses were compared with diagnoses by eight internationally recognised experts using ICD-10 criteria (two experts independently rated the ADI-R, ADOS-G, psychometric findings and a clinical vignette for each case). Agreement between the project consensus and expert diagnoses was 93% with (weighted) κ= 0.77 (for details see Reference Baird, Simonoff and PicklesBaird et al, 2006: Fig. 1).

Fig. 1

Receiver operating characteristic (ROC) curves for the Social Communication Questionnaire (SCQ), Social Responsiveness Scale (SRS) and Children's Communication Checklist (CCC).

We collated the following data for our subsample of 119 children: IQ; severity of symptoms of autism, measured by ADI-R and ADOS-G algorithm total scores; a total count of ICD-10 symptoms (0-12), systematically completed as part of the diagnostic review process of every case; parent and teacher reports of emotional and behavioural problems; and adaptive behaviour, assessed using the Vineland Adaptive Behavior Scales (VABS; Reference Sparrow, Balla and CichettiSparrow et al, 1984). The children's (n=118) IQ was measured using the Wechsler Intelligence Scale for Children (WISC-III-UK; Reference WechslerWechsler, 1992). The (weighted) mean full-scale IQ of the sample was 73.4 (s.e.=1.6) and the range was 40-136; 56% (weighted) of the children had an IQ<70. One child could not complete the WISC-III-UK and their IQ was derived using Raven's Standard Progressive Matrices (Reference Raven, Court and RavenRaven et al, 1990).

Parents of a subsample of children (n=119) completed both the SRS and the CCC in addition to the SCQ. This afforded us the opportunity to directly compare the instrument properties of the three screens in the same sample.

Parents and teachers of these 119 children also completed the Strengths and Difficulties Questionnaire (SDQ; Goodman, Reference Goodman1997, Reference Goodman2001). Each sub-scale has five questions that are rated ‘not true’, ‘somewhat true’ and ‘certainly true’ and score 0-2, with higher scores indicating greater pathology. Four sub-scales (emotional problems, peer problems, conduct problems and hyperactivity) are summed to create a total problem score (range 0-40). We considered children whose teacher-rated (⩾16) and/or parent-rated (⩾17) total problem score fell around the 10% percentile in UK norms for SDQ scores (Reference Meltzer, Gatward and GoodmanMeltzer et al, 2000) to have a high rate of behaviour problems.

Order of completion of assessments

In the larger study (Reference Baird, Simonoff and PicklesBaird et al, 2006) the SCQ was used as the initial screening instrument to identify cases for in-depth diagnostic assessment; therefore the SCQ was completed for all children before the diagnostic assessments. For our subsample of 119 children with data on all three screening instruments the mean age at SCQ screening was 10.2 years (s.e.=0.4 years; range 9.5-11). The CCC was completed by parents immediately before the diagnostic assessment (mean age of children=12.0 years (s.e.=0.1 years; range 9.8-13.9). The SRS was completed at a mean age of 12.6 years (s.e.=0.4 years; range 11.8-13.2): for 50 children this was in advance of, and for 69 this was following, the diagnostic assessment and completion of the CCC. Scores on the three screening instruments were not consulted during the consensus clinical diagnostic process (for details see Reference Baird, Simonoff and PicklesBaird et al, 2006). The 119 children in our subsample differed from the remainder of the cohort (n=136) in terms of IQ (mean=78.5 (s.e.=1.8) v. 67.4 (s.e.=2.2); ANOVA F(1,251)=15.0, P<0.001) but not symptom severity or parental education.

Statistical analysis

Stratification of the sample with autistic-spectrum disorders and special educational needs was based on whether or not a child had a locally recorded autistic-spectrum disorder diagnosis (yes/no) and one of four levels of SCQ score (low: <8; moderately low: 8-14; moderately high: 15-21; high: >22; for details see Reference Baird, Simonoff and PicklesBaird et al, 2006: Fig. 1). Weighting allowed all statistics such as means, group differences and screen performance measures to be presented as target population estimates, taking account not only of the differences in sampling proportions according to SCQ score and local diagnosis of autistic-spectrum disorder, but also the differential response to the SCQ associated with a prior local autistic-spectrum disorder diagnosis, health district and child's gender. Wald test statistics (adjusted t- and F-tests) and P-values were calculated using the linearisation version of the robust parameter covariance matrix as implemented by the svy procedures of Stata 9 (Stata, 2005). A receiver-operator-characteristic (ROC) area-under-the-curve (AUC) analysis was performed to assess and compare the discriminant power of the screening instruments in distinguishing children with autistic-spectrum disorders (including autism) from those without (Reference Hanley and McNeilHanley & McNeil, 1982; Reference DunnDunn, 2000). Application of the weights ensured that this comparison was fair despite the SCQ-stratified sample design. Confidence intervals for weighted AUC estimates and tests were obtained using the bootstrap resampling ROC procedures of Stata 9, reverse coded in the case of the CCC.

Summary of the screening properties

The performance of the SCQ was similar to the that in initial validation study (sensitivity=0.85; specificity=0.75; Reference Berument, Rutter and LordBerument et al, 1999; Reference Rutter, Bailey and LordRutter et al, 2003) and somewhat better than in several more recent studies that have included younger children and have reported reduced sensitivity (0.71: Reference Eaves, Wingert and HoEaves et al, 2006a ; 0.71: Reference Corsello, Hus and PicklesCorsello et al, 2007; 0.67: Reference Lee, David and RusyniakLee et al, 2007) or reduced specificity (0.71: Reference Eaves, Wingert and HoEaves et al, 2006b ; 0.58: Reference Allen, Silove and WilliamsAllen et al, 2007; 0.54: Reference Corsello, Hus and PicklesCorsello et al, 2007). The present study included only a restricted age range but in a large sample Corsello et al (Reference Corsello, Hus and Pickles2007: Table 3) found that the sensitivity of the SCQ increased with age, perhaps reflecting the emergence of the full range of autistic symptoms over time. This is supported by comparison of the mean SCQ scores for children with a diagnosis of childhood autism in the present (25.8) and previous studies. Although our figure is very similar to the 25.2 obtained in the Berument et al (Reference Berument, Rutter and Lord1999) study it is higher than the 20.3 obtained in the Corsello et al (Reference Corsello, Hus and Pickles2007) study, which included children 2-16 years and also the 19.2 of the Eaves et al (Reference Eaves, Wingert and Ho2006b ) study, which included children aged 2-6 years.

The SRS had a lower sensitivity in our sample than in the original validation study (0.85; Reference Constantino and GruberConstantino & Gruber, 2005) and both the SRS and CCC had reduced specificity (0.57 and 0.41 respectively) in our subsample with low IQ. The increased specificity of the SCQ compared with that of the SRS and CCC might be due to its fuller coverage of the third autism symptom domain of restricted and repetitive behaviours and interests. All three screens showed high discrimination between children with IQ⩾70 in the ASD and non-ASD groups with AUC values between 0.87 and 0.90. The CCC had a high sensitivity but low specificity, reflecting its broader cut-off for ‘pragmatic impairment’, rather than autistic-spectrum disorder itself. For some purposes, for example screening out individuals who might have autistic-spectrum disorder when characterising a non-ASD comparison group in a research study, high sensitivity is prioritised and lower specificity does not bring costs, and on the basis of the present data the CCC could be used for such a purpose. Geurts et al (Reference Geurts, Verté and Oosterlaan2004) found that the CCC discriminated well between children with autism, children with ADHD and typical controls (discriminant function classification 78% in Study 1; 77% in Study 2).

All three screening instruments had lower specificity in the subsample with elevated levels of behavioural problems as measured by the SDQ. It is likely that in response to questions on the instruments that are meant to measure symptoms of autism, some parents are endorsing items that reflect aspects of their child's emotional, hyperactivity or conduct difficulties. One previous study has reported high scores on the SCQ for children with mood and anxiety disorders in whom a clinical diagnosis of pervasive developmental disorder had been excluded (Reference Towbin, Pradella and GorrindoTowbin et al, 2005), although such disorders are unlikely to be common in our sample as they usually do not form a reason for special educational needs registration. In the Towbin et al study significantly more children fell above the autistic-spectrum disorder cutoff on the SRS and the Social Interaction Deviance Composite on the CCC-2 than above the cut-off for the disorder on the SCQ (Reference Towbin, Pradella and GorrindoTowbin et al, 2005: Fig. 1, p. 458).

In addition to the prevalence of autistic-spectrum disorders in any particular clinical setting or research study, the characteristics (e.g. clinical diagnosis, IQ, age) of those with and without the disorder, family factors (e.g. parental education, parental knowledge about autism) and methodological factors, including whether the screen was completed before or after diagnostic assessment, will also affect how a screening instrument performs. Another factor that affects an instrument's performance in relation to a clinical diagnosis of autistic-spectrum disorder is the time period of the behaviour sampled and the three instruments used in this study differed in this respect. Whereas the CCC and SRS ask parents to rate current behaviour (for the SRS over the previous 6 months; unspecified for the CCC), about half of the items on the SCQ ask about behaviour in the 4- to 5-year period when symptoms of autism may be at their most prototypical. One further consideration that our study cannot address is whether screening instruments perform differently in different countries owing to cultural interpretation of the behaviours enquired about.

Example scenarios comparing use of the three screens

The meaning of statistics such as the AUC can be hard to translate into everyday examples to guide clinicians and researchers. To illustrate the potential impact of the different screen parameters on a hypothetical research study or clinical service we will outline two scenarios that summarise the consequences of choice of screening instrument. In both scenarios assume that among 250 children to be screened, 100 are ‘true cases’ of autistic-spectrum disorders and 150 are ‘true non-cases’. Using estimates from the present analysis this translates into: the number of the 100 ‘true cases’ that are screen positive (true positives) (SCQ=86; SRS=78; CCC pragmatic composite=93); the number of the 100 ‘true cases’ that are screen negative (false negatives) (SCQ=14; SRS=22; CCC pragmatic composite=7); the number of the 150 true non-cases that are screen negative (true negatives) (SCQ=117; SRS=100.5; CCC pragmatic composite=69); and the number of the 150 true non-cases that are screen positive (false positives) (SCQ=33; SRS=49.5; CCC pragmatic composite=81). The relative importance of these figures depends on the purpose of using the screening instrument in a particular study/service.

In the first scenario, consider that the instruments is being used to estimate, for purely administrative reasons, the number of children within a special school system who have an autistic-spectrum disorder. That is, there is no consequence or cost (such as a follow-up assessment) attached to being screen positive. In this scenario, the estimated prevalence of autistic-spectrum disorder will vary by a factor of ∼50%, depending on whether one is using the SCQ (119 screen positives, i.e. children with autistic-spectrum disorders identified), the SRS (127.5 identified) or the CCC pragmatic composite (173 identified), with the latter clearly overestimating the ‘true’ prevalence.

In a second scenario, consider a clinical service screening speech and language therapy referrals to identify children who should be ‘fast tracked’ into an expensive (and for parents sometimes anxiety provoking) multidisciplinary assessment. For this aim, high sensitivity is required for the screening procedure to be clinically useful. To maximise case-finding efficiency one could consider the proportion of cases correctly identified by the screen compared with cases missed by it (SCQ=6.14; SRS=3.55; CCC pragmatic composite=13.3). However, one would also want to minimise false positives in order not to use expensive assessment resource on children who do not have autistic-spectrum disorders and to avoid provoking unnecessary anxiety in parents. Here the most relevant figure is the number of false positives (SCQ=33; SRS=49.5; CCC pragmatic composite=81). In this scenario the SRS was least efficient in terms of case-finding and the CCC least cost-effective, with the SCQ performing best. Other scenarios would produce different outcomes, and the choice of which screen to use and which criterion to adopt could depend on the relative costs of false positives and false negatives - although these costs fall on different parties (false positives tend to cost services, whereas false negatives tend to cost the child and parent). Clinicians and researchers need to estimate as best they can the implications for their service/study of which screen they use for any particular purpose.

Strengths and limitations

The strengths of our study include: testing the ability of different screening instruments to identify children with autistic-spectrum disorders in the same sample; the generalisability of the findings, owing to the population-weighting procedure; the calculation of confidence intervals around the instrument parameter estimates; and the inclusion of both low- and high-IQ children. The comprehensive diagnostic assessment and use of a clinical consensus decision-making process that was corroborated by independent expert rating (see Reference Baird, Simonoff and PicklesBaird et al, 2006) are also strengths. Furthermore, the screens were able to differentiate those with autistic-spectrum disorders from those without such disorders but with intellectual disability and language delay despite the considerable degree of symptom overlap between these conditions, especially in the area of impairments in communication.

One limitation of the study is that the age of the at-risk sample at the time of screening (9-13 years) is older than would be required for first-level screening of young children, although it is still an age at which many children are referred for possible autistic-spectrum disorders, in particular to child and adolescent mental health services (Reference Skuse, Mandy and ScourfieldSkuse et al, 2005). Second, the SCQ was derived from items on the ADI-R and information from the ADI-R was part of the information used to achieve a clinical consensus diagnosis. This might explain, in part, the higher prediction of the SCQ compared with the SRS and CCC that we found. Another limitation is the relatively modest sample size, in particular when the subsamples with high/low IQ and high/low levels of behavioural problems were examined, leading to relatively wide confidence intervals. However, the sample size compares well with the majority of published data available on the SCQ, SRS and CCC.

Clinical implications

A score on a screening instrument is not a diagnosis. For diagnostic assessment a full parental interview regarding current and past development and behaviour, and structured observation of the child, preferably including a peer-group setting, is essential. Corsello et al (Reference Corsello, Hus and Pickles2007) found that the addition of information from the ADOS-G to information from the SCQ significantly improved specificity to detect autistic-spectrum disorders. For some clinical and research purposes they suggest that a multistage assessment beginning with the SCQ, followed by the ADOS-G and then including information from the time-intensive ADI-R assessment might be appropriate. Our study demonstrates that for some clinical and research purposes the SCQ, and to a lesser extent the SRS and CCC, can all be efficient first-level screening instruments for identifying children with possible autistic-spectrum disorder for a more in-depth assessment. Child characteristics such as IQ and behavioural problems affect performance of the screens, and this, in addition to considerations regarding the consequences/costs of a screen-positive or negative result, should be borne in mind when choosing which screen to use for any particular clinical or research purpose.