Participants

The data for this report are from a population-based longitudinal study of Caucasian female twins drawn from the Virginia Twin Registry (VTR). The VTR was formed from a systematic review of all birth records in the Commonwealth of Virginia (USA) after 1918. Twins were eligible to participate if they were born between 1934 and 1971 and both members had previously responded to a mailed questionnaire completed over 1987-88 (individual response rate of 64%). Data used in the present study are from the first interview wave and accompanying self-report personality measures, and the third interview wave, which will be called Time 1 interview and Time 2 interview, respectively. At Time 1 (1987-89), 92% of the eligible individuals (n=2163) were interviewed (90% face to face and the remainder by telephone). The mean age of the twins was 29.3 years (s.d.=7.7, range 17-54 years). Time 2 (1991-93) occurred on average 5.1 years (s.d.=0.4) later. Written informed consent was obtained prior to face-to-face interviews and verbal assent prior to telephone interviews.

Measures

Interviews were conducted blind to information about co-twins. Information about interviewer characteristics has been presented elsewhere (Reference Kendler, Macclean and NealeKendler et al, 1991). A narrow definition of lifetime bulimia nervosa, or one that conformed strictly to DSM-III-R (American Psychiatric Association, 1987) criteria, was used. In addition, in order to maximise statistical power in the study of a low prevalence disorder, a broad definition of lifetime bulimia nervosa was adopted where the DSM-III-R 'D' criterion was omitted because there appear to be few meaningful differences between women who binge and use associated weight-loss methods twice a week and those who engage in such behaviours less than twice a week (Reference Garfinkel, Lin and GoeringGarfinkel et al, 1995; Reference Sullivan, Bulik and KendlerSullivan et al, 1998). This broad category differs slightly from its previous use (Reference Bulik, Sullivan and KendlerBulik et al, 1998), in that it includes women with a wider range of concern about their body shape and weight, from “a lot more concerned than most women your age” to “a little bit more concerned”.

At the first interview there was one probe question (“Have you ever in your life had eating binges during which you ate a lot of food in a short period of time?”). If this was answered negatively, no further questions were asked. At the second interview, a further probe question was asked, relating to weight loss behaviours (“Have you ever made yourself throw up as a means of controlling your shape and weight?”). If these were both answered negatively, no further questions were asked.

The diagnosis of DSM-III-R major depression was made using questions from the Structured Clinical Interview for DSM-III-R (SCID) (Reference Spitzer, Williams and GibbonSpitzer et al, 1992). Numerous probe questions were used to ascertain the presence of depressive symptoms. Initially, occurrence of major depression over the past year was assessed, using a probe question for each one of the diagnostic criteria. Then major depression over the lifetime (excluding the past year) was assessed with two probe questions.

A description of the variables examined for predictive value of reliability is provided in Table 1 : all predictor variables are from the Time 1 interview period unless otherwise noted.

Table 1

Predictor variables used to examine influences on the reliability of lifetime bulimia nervosa and major depression (predictor variables are from the Time I interview period unless otherwise noted)

Variable type	Description of variable
Demographics	Years of education (highest grade of school or year of college completed), annual salary, parental education, urbanisation
	Population size of area lived in at time of interview
	Annual salary
	Parental education
Age	Age at first interview
	Age of developing bulimia nervosa recalled at both Time 1 and Time 2
Current symptomatology	20 items of the Padua Inventory (Reference SanavioSanavio, 1988) assessing levels of obsessive and compulsive symptomatology
	Total score of the Symptom Check-List (Reference DerogatisDerogatis, 1975), assessing depression, panic and agoraphobia, somatisation and anxiety, and sleep difficulties (Reference Kendler, Walters and TruettKendler et al, 1994) from the previous 30 days
Lifetime comorbidity	Structured Clinical Interview (Reference Spitzer, Williams and GibbonSpitzer et al, 1992) for :
	DSM-III-R alcohol dependence and panic disorder
	DSM-III phobias
	DSM-III-R symptoms and DSM-III-duration generalised anxiety disorder
Personality and attitudes	Extroversion and neuroticism (Reference Eysenck and EysenckEysenck & Eysenck, 1975)
	Altruism (7 items from the Interpersonal Reactivity Index; Reference DavisDavis, 1980)
	Interpersonal dependency (Reference HirschfeldHirschfeld et al, 1977)
	Mastery (reversed coding of the powerlessness sub-scale of the Alienation subtest; Reference Maddi, Kobasa and HooverMaddi et al, 1979)
	Dispositional optimism (Reference Scheier and CarnerScheier & Carner, 1985)
	Self-esteem (Reference RosenbergRosenberg, 1965)
	Locus of control (resourcefulness sub-scale of the Attributional Style Questionnaire; Reference Peterson, Semmel and vonVaeyerPeterson et al, 1982)

Statistical analyses

Agreement between Time 1 and Time 2 diagnoses were examined using the kappa coefficient (k), tetrachoric correlations, and the Yule's Y statistic. Yule's Y is less dependent on the base rate than k, which permits a more direct comparison between the higher prevalence major depression and the lower prevalence bulima nervosa. We also calculated sensited - the proportion of true cases correctly identified (risk for false-negatives) - and specificity - the proportion of true non-cases correctly identified (risk for false-positives). For the purpose of these calculations, the Time 2 assessment was chosen as the standard, as it contained more probe questions than the Time 1 assessment. One would expect sensitivity to be lower for more prevalent disorders and specficity to be higher for less prevalent disorders.

The ability of variables to predict reliability, sensitivity and specificity was then examined using logistic regression. Results are presented as odds ratios with 95% CIs. As twin pair observations are correlated, the assumption of independent sampling is violated, and we therefore used generalised estimating equation (GEE) modelling (Reference Zeger, Liang and AlbertZeger et al, 1988) to adjust standard errors for non-independent observations using the GENMOD procedure.

Finally, separate stepwise logistic regressions were used to examine the relative importance of the significant predictors for reliability, sensitivity and specificity of reporting bulimia nervosa. All analyses were carried out with SAS version 7.0 (SAS Institute, 1996).

Agreement between interviews

For the purposes of this study, women who reported lifetime bulimia nervosa at Time 2 but not Time 1, and who reported age of onset as being after Time 1, were considered to have developed bulimia nervosa between the two assessments. These women were removed from further analysis so that onsets that occurred between Time 1 and Time 2 would not be confounded with unreliable recall. This resulted in two women being removed when considering narrowly defined bulimia nervosa and 11 women being removed when considering broadly defined bulimia nervosa. Reliability of major depression has previously been considered with complete twin pairs only (Reference Foley, Neale and KendlerFoley et al, 1998). As completeness of twin pairs was irrelevant to our analyses, we considred data from all twins, thus increasing the number of women studied and producing a slightly higher k value than previously reported. When onset of first-episode major depression between Times 1 and 2 was considered, 113 woman were removed from further analysis. Taking into account the lower base rate dependent measures (tetrachoric correlations and Yule's Y), narrowly defined bulimia nervosa is the most reliable diagnosis, and the reliability of broadly defined bulimia nervosa and major depression are similar (Table 2). The bulimia nervosa diagnoses have the lowest risk for assigning false-positive cases, but the highest risk for assigning false-negative cases.

Table 2

Agreement between Time 1 and Time 2 diagnosis - lifetime bulimia nervosa, both narrowly and broadly defined, and lifetime major depression

Time 1 assessment	Time 2 assessment
	Bulimia nervosa	No bulimia nervosa
Narrowly defined bulimia nervosa 1
Bulimia nervosa	11	31
No bulimia nervosa	8	1845
Broadly defined bulimia nervosa 2
Bulimia nervosa	33	49
No bulimia nervosa	59	1745
	Major depression	No major depression
Major depression 3
Major depression	387	277
No major depression	130	988

Clinical features predicting reliability of bulimia nervosa

By far the largest category of women with unreliably reported bulimia nervosa included women who met the full criteria at one interview and gave negative replies to the probe question/s at the other interview - for narrowly defined bulimia nervosa this occurred approximately one-third of the time, for broadly defined bulimia nervosa it occurred approximately half the time. Reported use of self-induced vomiting or laxative misuse at either interview significantly predicted reliability (P=0.005, odds ratio=3.48,95% CI 1.45-8.35). The likelihood of reporting the behaviour associated with bulimia nervosa at Time 2 was dependent on the type of behavior reported at Time 1 (see Table 3). The most memorable weight loss behaviour was self-induced vomiting (with the odds of reporting vomiting at the second interview 34 times higher if vomiting was reported at the first interview) and laxative miscue (with the odds of reporting laxative abuse at the second interview 28 times higher if laxative abuse was reported at first interivew). In contrast, odds of recalling strict dieting or fasting at Time 2 were only about twice as high if such behaviour was reported at Time 1. Binge eating was less likely to be recalled than either self-induced vomiting or laxatives, but more likely to be remembered than other weight loss behaviours.

Table 3

Reliability of reporting of behaviours associated with bulimia nervosa at Time 2, given they were reported at Time 1

Weight loss behaviour	k	Odds ratio (95% C1)	P
Self-induced vomiting	0.56	34.11 (15.74-73.94)	<0.0001
Laxative misuse	0.50	28.74 (9.41-87.81)	<0.0001
Binge eating	0.37	6.13 (4.84-7.78)	<0.0001
Exercise	0.23	3.61 (1.77-7.38)	0.0003
Strict dieting	0.15	2.29 (1.12-4.67)	0.02
Fasting	0.13	2.41 (0.97-5.98)	0.05

The more detailed Time 2 data were used to investigate any differences in frequency of eating disorder behaviours between those women with reliably reported bulimia nervosa and those women with unreliably reported bulimia nervosa. Results are summarised in Table 4. The strongest association exists between reliability and frequency of binge eating. For both narrowly defined and broadly defined bulimia nervosa, a higher monthly frequency of binge eating predicted more reliable reporting.

Table 4

The impact of symptom severity on reliability of diagnosis of lifetime bulimia nervosa

Symptom	Narrow-defined bulimia nervosa		Broadly-defined bulimia nervosa
	Mean reliable	Mean unreliable	Mean reliable	Mean unreliable	Odds ratio (95% C1)
Frequency of binges each month	26.73	9.45	12.21	5.64	1.07 (1.02-1.12) **
Duration of binge eating (in weeks)	248.40	202.90	215.46	195.21	1.00 (0.99-1.01)
Frequency of vomiting each month	59.78	20.60	39.28	7.67	1.04 (0.97-1.12)
Duration of vomiting (in weeks)	145.78	58.73	91.67	32.04	1.01 (0.99-1.03)

Predictors of reliability, sensitivity and specificity of bulimia nervosa and major depression

For the remaining analyses, there was insufficient power to calculate the odds ratio for narrowly defined bulimia nervosa. Therefore, only results for broadly defined bulimia nervosa and major depression are reported here.

Multivariate contribution of predictor variables to reliability, sensitivity and specificity

The relative contributions of those predictor variables shown to significantly predict reliability of reporting of broadly defined bulimia nervosa were examined in a stepwise regression model, including reported use of either self-induced vomiting or laxatives, frequency of binge eating, years of education, educational status of parents and presence of lifetime major depression. The variables retained in the equation that predicted more reliable reporting of bulimia nervosa were decreased likelihood of lifetime major depression at either Time 1 or Time 2 (X ²=5.18, P=0.02), use of either self-induced vomiting or laxatives (X ²=4.84, P=0.03), and greater frequency of binges each month (X ²=4.28, P=0.04). Predictors of greater reliability of major depression reporting (including only those significant predictor variables) included greater likelihood of GAD (X ²=23.17, P <0.0001), a higher score on the Symptom Check-List (Reference DerogatisDerogatis, 1975) at Time 2 (X ²=7.28, P=0.007), and increased obsessionality (X ²=4.83, P=0.03).

Due to the low predictive power of the sensitivity measure, this was not examined in a multiple regression for bulimia nervosa. Of those variables that significantly predicted greater sensitivity for major depression in the univariate analyses, two were retained in the final equation, including greater likelihood of lifetime GAD (X ²=28.92, P <0.0001) and lower financial status (X ²=7.03, P=0.008). Of those variables that significantly predicted greater specificity of bulimia nervosa in the univariate analyses, the following were retained in the final equation : decreased likelihood of lifetime major depression (X ²=10.37, P=0.001) and panic disorder (X ²=5.88, P=0.02), and increased levels of mastery (X ²=6.64, P=0.01). Correspondingly, variables that best predicted major depression specificity were a lower likelihood of lifetime GAD (X ²=92.22, P <0.0001) and alcohol dependency (X ²=16.91, P <0.0001) and lower levels of altruism (X ²=5.37, P=0.02).

Salience of behavioural markers

In terms of overall reliability, we replicated the findings of Field et al (Reference Field, Colditz and Herzog1996) where the majority of unreliable cases were women who reported full symptoms of bulimia nervosa on one occasion and responded negatively to probe questions on the other. Of all the behaviours associated with bulimia nervosa reported at Time 1, it was the presence of self-induced vomiting and laxative misuse that were most likely to be remembered at Time 2. This suggests vomiting and laxatives are more salient behavioural markers than other weight loss behaviours, and thereby less vulnerable to memory decay. However, a higher monthly frequency of binge eating rather than any weight loss behaviour significantly predicts reliable reporting of lifetime bulimia nervosa. As not all women use vomiting or laxatives, the frequencies of these behaviours may have had insufficient predictive power. These findings concur with studies on the reliability of major depression, which suggest the more severe the symptomatology, the more memorable the disorder (Reference Aneshensel, Estrada and HansellAneshensel et al, 1987; Reference Foley, Neale and KendlerFoley et al, 1998).

Role of sensitivity and specificity in determining reliability

There appear to be more differences than similarities in the profiles of overall predictive reliability of bulimia nervosa and major depression. Reliability of major depression reporting appears to be affected by overall level of functioning of the individual. The less well the person, as indicated by a number of measures including personality, current symptomatology and lifetime psychopathology, the more likely they were to reliably recall having had major depression. In contrast, there was no effect of personality or attitudes on reliability of bulimia nervosa reporting, and the strongest predictor, apart from the behavioural markers, was a lower likelihood of lifetime major depression. This finding can be explained by examination of sensitivity and specificity. The presence of true cases of major depression is marked by increased problems with psychiatric and personality functioning (unfortunately our ability to detect true cases of bulimia nervosa was limited). Conversely, the detection of true non-cases of both bulimia nervosa and major depression was marked by fewer problems with psychiatric and personality functioning. This would suggest that the overall reliability of bulimia nervosa seems to be characterised more by its ability to accurately detect non-cases, whereas the overall reliability of major depression is characterised more by its ability to detect cases.

A simple comparison of general reliability measures across psychiatric diagnoses is insufficient to elucidate the nature and mechanisms of unreliability. A more useful approach is to examine specific aspects of reliability of reporting, such as sensitivity and specificity. Given that the majority of population-based epidemiological studies utilise structured clinical interviews to identify cases of bulimia nervosa similar to the ones used in this investigation, several strategies can be employed to improve reliability of reporting in the context of such interviews. Incorporating more than one occasion of measurement (Reference Bulik, Sullivan and KendlerBulik et al, 1998) and using more specialised assessment instruments (Reference Wade, Tiggemann and MartinWade et al, 1997) can improve reliability. In addition, the inclusion of a greater number of probe questions can increase the probability that memory of past disorders will more successfully be activated and accessed, thus increasing the detection of true cases.

CLINICAL IMPLICATIONS

• ▪ Individuals who self-induce vomiting or misuse laxatives have more reliable recall of bulimia nervosa.

• ▪ More frequent occasions of binge eating are associated with more reliable reporting of the disorder.

• ▪ It is wise to include several probe questions about several behaviours when assessing bulimia nervosa (e.g. binge eating, vomiting, laxative misuse) rather than using a single probe to stimulate recall.

LIMITATIONS

• ▪ Only women were assessed, therefore these results cannot be applied to men.

• ▪ We have low power to draw any conclusions about variables that predict sensitivity of bulimia nervosa.

• ▪ The use of Time 2 assessment as the standard is based on this assessment having only one more probe question than Time l assessment.