Participants

The Avon Longitudinal Study of Parents And Children ^{Reference Golding, Pembrey and Jones5} is a population-based study which investigates a wide range of environmental, genetic and psychosocial influences on the health and development of children and their parents. Figure 1 illustrates participation in ALSPAC up to and including the data gathered from teachers when the children were in school year 3. The 14 541 pregnant mothers recruited into the study between April 1991 and December 1992 had 14 062 live births. At 1 year 13 988 infants were alive and 13 971 at 7 years of age. When compared with 1991 national census data, the ALSPAC sample was found to be similar to the UK population as a whole, having only a slightly higher proportion of married or cohabiting mothers who were owner–occupiers and who had a car in the household. There were also a slightly smaller proportion of mothers from ethnic minority groups. ^{Reference Golding, Pembrey and Jones5}

Fig. 1

Description of ALSPAC sample: flow chart.

At 7 years 9 months, as part of a study on disruptive behaviour disorders (attention-deficit hyperactivity disorder (ADHD) and behaviour disorders), teachers in the geographically defined study area (the old county of Avon in the UK) were asked to complete the Development and Well-Being Assessment (DAWBA) ^{Reference Goodman, Ford, Richards, Gatward and Meltzer6} on all the children in their class with a birth date between April 1991 and December 1992. From a total of 10 431 children eligible to be contacted, teachers returned questionnaires for 3975 children whose parents also participated in this survey (current ALSPAC children), and 1140 children who had participated in previous parts of the ALSPAC study but whose parents did not respond to the current survey (previous ALSPAC children) (Fig. 1). The teacher completion was thus 5115/10 431 of eligible children (49%) or 5115/13 971 of all survivors (37%). In addition, teacher data was returned for 4383 children who had never been recruited into the ALSPAC study or had moved into the area after the study had started (never ALSPAC children). The study was approved by the ALSPAC Ethics and Law Committee and local research ethics committees.

Procedures

During pregnancy, and annually since then, detailed information about the mothers and their partners has been collected via self-report questionnaire with regard to medication, symptoms, diet and lifestyle, attitudes and behaviour, and social–environmental features. ^{Reference Golding, Pembrey and Jones5} From 4 weeks after the birth of the child, mothers completed questionnaires about the child's health, development and environment (biannually on average).

When the children were 7 years and 9 months, teachers were asked to complete the Development and Well-Being Assessment (DAWBA) ^{Reference Goodman, Ford, Richards, Gatward and Meltzer6} as part of a study on disruptive behaviour disorders. The teacher version of the DAWBA is a brief structured questionnaire that covers the operationalised diagnostic criteria for the main disruptive behavioural disorders included in DSM–IV, ⁷ namely oppositional defiant disorder, conduct disorder and ADHD. Thirty-nine cases were excluded where there were insufficient data from teachers for a diagnosis to be made. ^{Reference Achenbach, Rescorla, Cichetti and Cohen8}

Data collection from teachers occurred over three academic years (1999, 2000 and 2001), with response rates varying from year to year. A minority of schools declined to participate (5%, 13% and 6% respectively) and some failed to respond to the invitation (17%, 37% and 16%), but the response rate from the schools who agreed to participate was high (80%, 99% and 80%) leading to an overall response rate of 62%, 50% and 63% for each year.

The following family-based risk factors were assessed during pregnancy: marital status (married v. single); education (any qualification v. no educational qualifications (i.e., no O-levels, professional qualifications or higher)); financial difficulties (yes v. no); family size (0–4; 5 or more children); smoking v. nonsmoking; critical partner relationship derived from the Family Adversity Index, ^{Reference Bowen, Heron, Waylen and Wolke9} (low affection and high aggression, physical or emotional cruelty, no partner social support v. not present); poor housing defects (a summary variable of three indicators: inadequacy; basic living; and defects/infestation present v. not present); crime (in trouble with police) or conviction of the mother or father (yes v. no); and psychopathology of the mother (affective disorder, suicide attempts v. none).

In addition, the child's gender and whether or not they were born prematurely (before 37 weeks gestation) was also recorded.

Statistical analysis

Simulations

A series of 36 simulations was carried out to explore the impact of selective participant drop-out on the prediction of Y (disruptive behaviour) from a predictor X. Of primary interest were simulations in which drop out and disruptive behaviour were predicted by the same factor (X) (i.e. the drop-out occurred by selection on a predictor X in regression) and the degree of selection was varied between simulations. In each simulation, we generated a sample of 5000 cases, which was then subjected to a drop-out process. Each case i was characterised by a predictor value X _i and a criterion value Y _i , such that X and Y approximated a bivariate standard normal distribution in the sample. The correlation between X and Y varied between simulations, in the range of 0.1–0.9, in steps of 0.1 (note that, because the variables were standardised, the Pearson correlation coefficient is identical to the linear regression coefficient in an ordinary least-squares model). For each correlation level, we simulated four stochastic drop-out processes, which differed in selectivity (although keeping the overall drop-out rate constant). We used the following drop-out rule:

\batchmode \documentclass[fleqn,10pt,legalpaper]{article} \usepackage{amssymb} \usepackage{amsfonts} \usepackage{amsmath} \pagestyle{empty} \begin{document} \[\ {\delta}_{i}=\frac{1}{1+\mathrm{exp}(-\mathrm{X}_{\mathrm{i}}{/}{\tau})}\ \] \end{document}

in which δ _i was the probability that case i was dropped from the sample, and τ was a scaling parameter that was manipulated between simulations. The general form of this logistic rule is shown in Fig. 2. For each value of τ, the expected proportion of dropped cases is 0.5. In all the simulations, the proportion of dropped cases was within the 0.49–0.51 range. The drop-out process was more selective (i.e. dependent on the value of X) for lower values of τ. Across a typical simulated 5000-case sample, the point-biserial correlations between X and a binary drop-out indicator were 0.10, 0.42, 0.61 and 0.78 for τ values of 5, 1, 0.5, and 0.1 respectively, confirming the high selectivity of drop-out for the lower values of τ.

Fig. 2

Probability of dropping out (δ) as a function of X, for different values of τ.

A second set of simulations was carried out to determine the effect on the regression between X and Y (disruptive behaviour) of drop-out that is selective on the criterion Y (e.g. drop-out of cases with higher scores on the criterion variable) and of drop-out that is selective on both the predictor (X) and the criterion variable (Y).

Prevalence of disruptive behaviour disorder

As shown in Table 1 our total ALSPAC group had a lower prevalence of all teacher-based disorders than the unselected never ALSPAC group, although the findings in relation to any oppositional/conduct disorder (P = 0.075) are marginal. This ‘prevalence gap’ might be explained by our missing data for some of those who dropped out and/or by selection bias that was operating even at initial recruitment. However, the prevalence of the total ALSPAC and never ALSPAC groups was closer than the current and never groups, suggesting that the initial cohort was more representative for teacher-reported disruptive behaviour disorder than after drop-out had occurred. Nevertheless, some selection had occurred over time according to the criterion, disruptive behaviour disorder.

Table 1

Prevalence of disruptive behaviour disorder diagnoses according to cohort^a

	Never ALSPAC (n = 4383)	Current ALSPAC (n = 3946)	Previous ALSPAC (n = 1130)	Total ALSPAC (current and previous) (n = 5076)	Test for 3-group differencesa (never v. current v. previous)		Test for 2-group differencesb (total v. never)
	% (n)	% (n)	% (n)	% (n)	χ2	P	χ2	P
Any ADHD	3.8 (165)	2.4 (93)	4.8 (55)	2.9 (148)	22.6	P < 0.001	5.30	P = 0.021
Inattentive ADHD	1.6 (71)	1.2 (47)	2.3 (26)	1.4 (73)	7.74	P = 0.021	0.52	P = 0.472
Hyperactive ADHD	0.6 (26)	0.3 (12)	0.8 (9)	0.4 (21)	5.84	P = 0.054	1.53	P = 0.216
Combined ADHD	1.6 (68)	0.9 (34)	1.8 (20)	1.1 (54)	10.09	P = 0.006	4.39	P = 0.036
Any oppositional or conduct disorder	3.1 (138)	2.1 (84)	4.0 (45)	2.5 (129)	14.16	P = 0.001	3.16	P = 0.075
Oppositional defiant disorder	2.0 (86)	1.3 (52)	2.3 (26)	1.5 (78)	7.48	P = 0.024	2.50	P = 0.114
Conduct disorder	1.2 (52)	0.8 (32)	1.7 (19)	1.0 (51)	6.90	P = 0.032	0.72	P = 0.396
Any disruptive behaviour disorder	5.2 (228)	3.5 (139)	6.4 (72)	4.2 (211)	21.92	P < 0.001	5.81	P = 0.016

Is drop-out selective or random?

The comparisons between the current and previous ALSPAC children are shown in Table 2. Drop-out from ALSPAC was systematically related to having a mother who was single, had no educational qualifications, encountered financial difficulties, being raised in a large family where the mother smoked, had a poor relationship with the partner, lived in poor housing, had been involved in crime and been convicted or suffered psychopathology during pregnancy. When prediction was adjusted for all other factors, being single (odds ratio (OR) = 1.45, 95% CI 1.19–1.77), family size (OR = 3.17, 95% CI 1.55–6.46), smoking (OR) = 1.41, 95% CI 1.15–1.73), no educational qualifications (OR) = 1.35, 95% CI 1.07–1.71) and financial difficulties (OR) = 1.39, 95% CI 1.07–1.81) remained significant independent predictors of drop-out.

Table 2

Prediction of drop-out (current v. previous ALSPAC participants)

		Prevalence, %a		Prediction of drop-out
				Unadjusted			Adjusted (n = 4070)
	n	Current	Previous	OR	95% CI	P	OR	95% CI	P
Child gender, male	5115	51.0	50.4	0.97	0.85-1.11	0.702	0.93	0.79-1.08	0.341
Born prematurely	5115	4.8	6.1	1.28	0.96-1.69	0.092	1.28	0.90-1.82	0.163
Marital status, single	4957	17.3	29.1	1.97	1.68-2.30	<0.001	1.45	1.19-1.77	<0.001
Education, no qualifications	4879	10.7	17.2	1.73	1.43-2.10	<0.001	1.35	1.07-1.71	0.011
Financial difficulties	4713	7.7	12.9	1.77	1.42-2.21	<0.001	1.39	1.07-1.81	0.015
Family size, >4 children	4984	0.7	2.6	3.58	2.12-6.04	<0.001	3.17	1.55-6.46	0.002
Maternal smoking	4452	15.3	25.1	1.85	1.55-2.21	<0.001	1.41	1.15-1.73	0.001
Critical partner relationship	5058	14.4	19.6	1.44	1.21-1.71	<0.001	1.08	0.86-1.34	0.512
Housing	5041	15.4	18.5	1.25	1.05-1.48	0.014	1.10	0.89-1.36	0.379
Crime and conviction	4547	1.7	3.5	2.12	1.37-3.26	0.001	1.14	0.68-1.90	0.620
Psychopathology of mother	4889	23.2	30.3	1.44	1.23-1.67	<0.001	1.18	0.98-1.42	0.081

Does drop-out reduce the validity of prediction of disruptive behaviour disorder?

Disruptive behaviour prediction with the ALSPAC data

The same variables that were related to the drop-out process were used as predictors for the disruptive behaviour disorder criterion. The individual predictors and the magnitude of prediction were very similar for the previous and current ALSPAC groups. Teacher-reported disruptive behaviour disorder in middle childhood was more likely when parents had low education, financial difficulties or critical partner relationships, when the mother had psychopathology or smoked in pregnancy, and for boys (Table 3). There were no significant interactions between group membership (previous ALSPAC v. current ALSPAC) and individual predictors (e.g. financial difficulties) when predicting the presence or absence of disruptive behaviour disorder, i.e. the same predictive model seemed to apply equally well to previous and current ALSPAC participants.

Table 3

Simple univariable prediction of disruptive behaviour disorder for the current ALSPAC and previous ALSPAC children (those who have dropped out) using factors assessed during pregnancy

		Currenta				Previousb			Interactionc
	n	OR	95% CI		P	OR	95% CI	P	OR	95% CI	P
Child gender, male	5102	3.02	2.26	4.02	<0.001	4.40	2.43-7.99	<0.001	1.46	0.75-2.83	0.263
Born prematurely	5102	1.55	0.94	2.56	0.088	1.76	0.77-4.00	0.177	1.14	0.43-2.98	0.796
Marital status, single	4944	1.72	1.28	2.31	<0.001	1.89	1.14-3.15	0.014	1.10	0.61-1.98	0.746
Education, no qualifications	4867	1.53	1.07	2.19	0.021	2.30	1.30-4.07	0.004	1.51	0.77-2.96	0.233
Financial difficulties	4702	2.35	1.62	3.41	<0.001	2.71	1.47-4.97	0.001	1.15	0.56-2.35	0.697
Family size, >4 children	4971	1.08	0.26	4.56	0.917	1.82	0.54-6.20	0.336	1.69	0.25-11.21	0.586
Maternal smoking	4442	1.98	1.45	2.71	<0.001	2.52	1.44-4.41	0.001	1.27	0.67-2.42	0.459
Critical partner relationship	5045	1.89	1.39	2.55	<0.001	2.36	1.42-3.94	0.001	1.25	0.69-2.27	0.458
Housing	5028	1.32	0.96	1.83	0.090	1.40	0.78-2.50	0.261	1.06	0.54-2.06	0.874
Crime and conviction	4537	1.98	0.89	4.40	0.094	1.69	0.50-5.72	0.400	0.85	0.20-3.67	0.831
Psychopathology of mother	4877	2.31	1.77	3.02	<0.001	2.08	1.25-3.45	0.005	0.90	0.51-1.60	0.717

The simulations

Figure 3 gives an overview of the observed correlations between X and Y before and after the drop-out process in the simulations in which drop-out was selective on X. The results show that the drop-out process related to X has an effect on the correlations between X and Y. Figure 3 shows that in all simulations the correlation between X and Y was reduced in all simulations, and that the suppression effect was somewhat larger for the more selective drop-out processes (i.e. in those simulations in which τ was small).

Fig. 3

Correlation between predictor X and criterion Y before and after drop-out, as a function of τ.

Figure 4 demonstrates the effect of the drop-out process on a simulated sample. In this example, the correlation in the original sample was high at r = 0.90, and the drop-out process was highly selective (τ = 0.1). The plot shows that the variance in the sample was reduced on both predictor (X) and criterion variable (Y). However, the non-standardised slope of the best-fitting regression line was practically unaltered by the drop-out process. The correlation (which corresponds to the standardised regression coefficient), was reduced from 0.90 to 0.78 after drop-out, as can be seen in Fig. 4.

Fig. 4

Simulated effect of selective drop-out according to the predictor variable X on least-squares linear regression model. X = predictor, Y = criterion. (a) before drop-out and (b) after drop-out.

The simulations demonstrate that selection on X in a regression has the effect of reducing the variance in X (and Y) and attenuates the correlation between X and Y. As shown here, the effects of selective drop-out in X on predictor–criterion correlation (and, by implication, regression) can be relatively small, even under a highly selective drop-out regime. Range restriction as a result of selective drop-out does not necessarily affect the validity of a regression model, although it can lead to underestimation of the criterion–predictor correlation.

It is important to note that drop-out by selection on the criterion variable (Y) can have a very different effect on the regression coefficients. Figure 5(a) shows an example, based on the same original simulated sample as in Fig. 4 (r = 0.90 before drop-out, τ = 0.1). Figure 5(a) shows that both the regression and correlation coefficient were reduced as a result of drop-out of participants with higher scores on the criterion variable (r = 0.79 after drop-out). Figure 5(b) provides an example in which there was selective drop-out on both the predictor (X) and the criterion variable (Y), with participants that scored highly on both variables more likely to drop out. Drop-out that was selective on both variables suppressed the regression coefficient (but less so than in the example in which drop-out was selective on the criterion only) and also reduced the correlation between predictor and criterion (r = 0.77 after drop-out).

Fig. 5

Simulated effect of selective drop-out (a) after drop-out according to the criterion variable y on least-squares linear regression model and (b) after drop-out according to the predictor variable x and criterion variable y on least-squares linear regression model. X = predictor; Y = criterion.

Selective drop-out and prevalence

Drop-out was considerable, with teacher returns on 37% (5115/13 971) of those believed to be alive or 49% (5119/10 431) of those eligible to be contacted. We only consider here the response to one particular assessment during the eighth year of life of the child. The participation rate is higher for any contact in a given year, whether for face-to-face assessments or other questionnaires. ^{Reference Golding, Pembrey and Jones5} Overall, the follow-up rate is similar to recent comparable large-scale longitudinal studies with repeated assessments. ^{Reference Curtin, Ingels, Wu, Heuer and Owings23,24} In general, participation rates are higher in older cohorts enrolled some decades ago, ^{Reference Ferri25,Reference Fergusson, Boden and Horwood26} for studies focused on specific high-risk samples in the first place ^{Reference Wolke and Meyer2,Reference Marlow, Wolke, Bracewell and Samara27} or for samples that were small and selective. ^{Reference Laucht, Esser, Baving, Gerhold, Hoesch and Ihle28}

The attrition from the sample we studied was systematically related to family characteristics, which supports the conclusions of previous work ^{Reference Aylward and Pfeiffer3,Reference Wolke, Söhne, Ohrt and Riegel4,Reference Marlow, Wolke, Bracewell and Samara27} that psychosocial factors are associated with attrition in longitudinal studies. The selective drop-out of participants had an impact on the prevalence of teacher-reported disruptive behaviour disorders, with the prevalence among children who were still participating being approximately half that of children who had dropped out. The factors that influenced retention in the ALSPAC sample also influenced the likelihood of disruptive behaviour disorder, i.e. the missingness was non-ignorable. ^{Reference Parzen, Lipsitz, Fitzmaurice, Ibrahim and Troxel29} Longitudinal studies are likely to underestimate the prevalence and incidence of disorders as shown here and elsewhere. ^{Reference Costello, Mustillo, Erkanli, Keeler and Angold30} Cross-sectional studies requiring only one single assessment are likely to be a more accurate in estimating prevalence. ^{Reference Meltzer, Gatward, Goodman and Ford31}

Selective drop-out and prediction

Finally, we investigated whether selective drop-out of participants does reduce the validity of prediction from longitudinal analysis. Prospective studies can only rely on the data of the individuals who continue to participate or they have to estimate missing data using sophisticated missing value substitution modelling and imputations. ^{Reference O'Hara Hines and Hines32,Reference Royston33} To our knowledge, this is the first investigation that could compare the prediction of outcomes of current and previous participants in a prospective study. We found that selective drop-out of participants according to a range of predictor variables did not invalidate the prediction of teacher-reported disruptive behaviour disorders by factors that were assessed as early as pregnancy and birth that have previously been shown to predict these difficulties. ^{Reference Counts, Nigg, Stawicki, Rappley and von Eye10–Reference Linnet, Dalsgaard, Obel, Wisborg, Henriksen and Rodriguez12} Boys were significantly more likely to develop teacher-reported disruptive behaviour disorder, as were the children of mothers who suffered psychopathology or smoked during pregnancy, who had poor partner relationships or who were single, poorly educated or suffered financial hardship. ^{Reference Moffitt and Avshalom34} These same predictions were found for those who were still participating in the ALSPAC study as well as for those who had dropped out. Despite reduction to a super-normal current ALSPAC sample, the predictive factors and their strength were about the same as for the previous participants. Contrary to common assumptions, ^{Reference Rothman and Greenland21,Reference Hernan, Hernandez-Diaz and Robins35} the presence of a substantial selection bias did not markedly attenuate the relationship between exposure and outcome in this study. Although prevalence rates do have an impact on statistical power, differences in prevalence per se did not alter prediction in this instance. Similarly, Moffitt and colleagues ^{Reference Moffitt, Caspi, Rutter and Silva13} investigated factors suspected to predict disruptive behaviour disorders in a sample of approximately 1000 children, half of them girls. They found, that despite girls being much less likely to develop disruptive behaviour disorder (low prevalence), the same factors predicted disruptive behaviour problems in both girls and boys.

We conducted simulations to explain why the effects of selective participant drop-out on predictor–criterion correlation (and regression) were relatively small in our empirical study. We found that a range of social and parental variables previously described as precursors of disruptive behaviour disorder in children affected the drop-out process. The simulations confirmed that if the selection is on X in a regression, the effect is one of reducing the variance in X (and Y), not affecting the regression but attenuating the correlation between X and Y (see Berk, ^{Reference Berk36} p. 389). Our simulations add that even under a highly selective drop-out regime related to X, the overall reduction in the correlations is small to moderate (Fig. 3). Therefore, range restriction as a result of selective drop-out according to X does not affect the internal or external validity of the regression model, ^{Reference Berk36} although the correlation coefficient after selective drop-out may underestimate the true correlation between the predictor and criterion variable.

In our empirical ALSPAC study, we see little evidence that teachers selectively underreported on children with disruptive behaviour. It seems unlikely that teachers would have been less likely to report on those with more disruptive behaviour since teachers are usually well aware of those who disturb lessons. ^{Reference Henry37} Nevertheless, we carried out a second set of simulations (examples shown in Fig. 5(a) and (b)) that showed that if selection on the criterion (Y) had occurred (i.e. if those with high disruptive behaviour disorder were less likely included in the sample), then the regression would be attenuated and the original regression line would no longer fit the data. Although confirming that the internal and external validity are weakened in these circumstances ^{Reference Berk36} and the true relationship between X and Y is systematically underestimated, our simulation also demonstrated that when drop-out is influenced by the predictor as well as the criterion variable, this only mildly reduces estimates of the slope of the true regression line.

We conclude that the regression coefficients hold for the current, previous and entire cohort due to the fact that, despite selection bias on X (and thus restricted range), the differences between the current and previous groups with disruptive behaviour disorder are small. Where the predictor variables have small to moderate (linear) associations with both, the drop-out of participants and the outcome variable, the impact on the predictor–criterion regression is small. However, if the drop-out process is dependent on the criterion variable (e.g. high scorers systematically excluded), then internal and external validity is threatened and the true relationship between predictor and criterion can no longer be estimated reliably. Particularly in cases where the selection process follows a complex pattern (e.g. with dependencies on several variables or non-monotonic dependencies; see Berk ^{Reference Berk36} for a full discussion) internal and external validity are under threat.

Limitations and conclusions

There are limitations to our study. Even fewer teachers than parents completed the diagnostic instrument in the current sample and this itself could have introduced bias. For example, teachers may have been more likely to complete the DAWBA in well-organised affluent schools. However, as these schools are also likely to have a lower prevalence of disruptive behaviour disorders, and possibly better strategies of managing them, this would be likely to lower prevalence across all three groups. Teachers would not have been aware of which children were or had been participants in ALSPAC when they completed measures on all the children in their class. Our diagnosis of a disruptive behaviour disorder was based only on teacher reports, although the limitation of having just one informant is partly offset by the fact that teacher reports are particularly informative for diagnoses for externalising disorders. ^{Reference Henry37} Nevertheless, our findings may not be applicable to diagnoses of a disruptive behaviour disorder based on parent data, self-report data or multi-informant data, or indeed to other outcomes within this or other studies.

In conclusion, participant loss in the ALSPAC cohort was systematic, with children with teacher-reported disruptive behaviour disorder being more frequently lost to follow-up. Our results suggest that longitudinal studies are likely to underestimate the prevalence and incidence of disorders, ^{Reference Wolke, Söhne, Ohrt and Riegel4} but that this might not negate findings in relation to the predictors of disorder if selection occurs according to the predictor variables. Our results need replication in relation to other cohorts and other outcomes. However, the simulations indicate that despite highly selective drop-out as a result of X and reduced range in both predictor and outcome variables, the regression parameter estimates are only mildly affected. Our demonstrations do not imply that selective drop-out is always harmless. For instance, selective drop-out effects can have significant implications if the selection is according to the outcome variable, if the drop-out process is complex or incidental ^{Reference Berk36} or there is a non-linear relation between predictor(s) and criterion. In such cases, explicit modelling of the drop-out process (e.g. Diggle & Kenward ^{Reference Diggle and Kenward38} and Little ^{Reference Little39} ) might help to clarify the implications of drop-out for model validity. Nevertheless, although everything should be done to reduce participant loss in cohort studies, ^{Reference Farrington, Gallagher, Morley, Ledger R, West, Magnusson and Bergman40,Reference Ribisl, Walton, Mowbray, Luke, Davidson and Bootsmiller41} it is reassuring to find that aetiological models from longitudinal samples can be valid and robust under specific conditions of selective loss of participants.