Nutrition has been established as a major determinant of chronic diseases including type 2 diabetes(Reference Hussain, Claussen, Ramachandran and Williams1), CVD(Reference Hung, Joshipura, Jiang, Hu, Hunter, Smith-Warner, Colditz, Rosner, Spiegelman and Willett2, Reference Osler3) and certain cancers(4), with strong positive and negative effects on health throughout the lifetime. These effects may be a direct consequence of the nutrients in foods or indirectly related to energy balance, as it is widely known that habitual intake of energy in excess of expenditure results in weight gain and obesity is a risk factor for many chronic diseases. For example, the recent report from the World Cancer Research Fund and American Institute for Cancer Research concluded that red and processed meats increase risk of colorectal cancer, while high intake of certain fruits and non-starchy vegetables probably reduces risk of cancers of the ovary, pharynx and larynx, oesophagus and stomach(4). Cohort studies have also shown that low intake of animal fat, moderate consumption of alcohol, particularly red wine, and high intakes of oily fish, fruits and vegetables are associated with reduced CHD risk(Reference Osler3). Overall, it has been estimated that nutrition and associated factors (physical activity and obesity) may account for 30–40 % of all cancers(4). In contrast to many other risk factors for disease, diet is potentially modifiable. It is therefore important that dietary intake be quantified through accurate and precise methods in order to determine the specific role dietary factors play in chronic disease promotion or prevention.
In large-scale epidemiological studies, diet is often assessed using a self-completed FFQ where the frequency of intake of various itemised foods is recorded. The low cost and ease of self-administration of FFQ facilitate their use in this way. FFQ are designed to measure long-term dietary intake rather than short-term intake and are commonly used to rank study participants according to their food or nutrient intake(Reference Willett5). Such ranking is adequate for most epidemiological studies, which usually assess the relative risk of disease among those with the highest compared with the lowest levels of intake. One important quality of an FFQ is its reproducibility, defined as the consistency with which study subjects answer the same questions on various occasions. In previous reproducibility studies of FFQ, correlation coefficients have typically been found to range from 0·4 to 0·7 for food items and from 0·5 to 0·7 for nutrients(Reference Willett5, Reference Cade, Burley, Warm, Thompson and Margetts6). In the present paper we report the findings of a study undertaken to assess the reproducibility of an FFQ used to collect dietary data for a series of cancer studies(Reference Ibiebele, Hughes, O’Rourke, Webb and Whiteman7–Reference Whiteman, Sadeghi, Pandeya, Smithers, Gotley, Bain, Webb and Green9) conducted in Australia.
Methods
Study subjects
The Australian Cancer Study and Australian Ovarian Cancer Study are Australia-wide population-based case–control studies investigating the risk factors for cancers of the oesophagus and ovary in adults aged 18–79 years. Control subjects were randomly sampled from the Commonwealth Electoral Roll (enrolment is compulsory in Australia and estimated to be 97 % complete) to match the age, sex and geographic distribution of the cases. The method of selection of cases and control subjects has been described in full elsewhere(Reference Merritt, Green, Nagle and Webb8, Reference Whiteman, Sadeghi, Pandeya, Smithers, Gotley, Bain, Webb and Green9).
The present study was restricted to control subjects who completed their initial FFQ (FFQ1) during the first few months of the main study (November 2003 to April 2004). Subjects with cancer (case subjects) were excluded because of concerns they might have changed their diet since their diagnosis. Of 309 potentially eligible controls, subjects were selected to give an approximately even balance of men and women covering the full age range up to 79 years; in practice, this included all eligible controls under the age of 50 years and approximately half of those over 50 years, selected at random (n 165). This group was re-contacted and asked to complete the FFQ again (FFQ2) between January and April 2005. Repeat FFQ were returned by 114 participants (69 %) and valid data were available for 100 of these after excluding five participants with implausible daily energy intake (<3360 or >168 00 kJ for men, <2100 or >147 00 kJ for women)(Reference Willett5) and nine participants who omitted responses to more than 10 % of the FFQ items.
Dietary assessment
Dietary data were collected via a 135-item self-administered semi-quantitative FFQ that was adapted for the Australian population from the validated instrument developed by Willett and colleagues(Reference Willett, Sampson, Stampfer, Rosner, Bain, Witschi, Hennekens and Speizer10). The changes included modifying names of certain foods (e.g. biscuits for cookies) and specification of portion sizes compatible with Australian measures. A similar version has been shown to have good validity in the Australian population(Reference Ashton, Marks, Battistutta and Green11–Reference McNaughton, Marks, Gaffney, Williams and Green14). Participants were asked to report their usual eating habits. For the itemised foods on the FFQ, they were asked to indicate how often, on average, they consumed the given standard serving size of food in the previous year. Serving sizes were indicated using ‘natural’ units such as one egg, apple or orange; commonly used portions such as a slice of bread or cheese, or glass of milk (8 oz); or typical serving sizes such as half a cup of cooked carrots, peas or green beans or half a glass (4 oz) of fruit juice. The nine possible response options ranged from ‘never’ to ‘4+ times a day’. Additional items not mentioned in the frequency format included types of fat used for frying and cooking, types of margarine and butter used, the amount of sugar added to food and the usual brands of cold breakfast cereal used. Prior to analysis, food items with similar nutrient content were combined and considered as a food group; for example, spinach, silver beet and lettuce were grouped as green leafy vegetables. Individual food items such as confectionery or crisps that were of no aetiological relevance to the cancers investigated were not included. In total, eighteen food groups of interest were created (see Appendix).
Frequency of consumption of itemised foods was converted into intake in g/d by multiplying the specified serving size of each food item (in grams) by the following values for each frequency option reported: never = 0, less than once per month = 0·02, 1–3 times per month = 0·07, once per week = 0·14, 2–4 times per week = 0·43, 5–6 times per week = 0·79, once per day = 1·0, 2–3 times per day = 2·5 and 4+ times per day = 4. Seasonal foods such as fruits and vegetables were weighted according to the proportion of the year the food was available. The food composition tables of Australia (NUTTAB95(Reference Lewis, Milligan and Hunt15) and, for lactose and galactose, NUTTAB90(16)) were used to calculate daily nutrient intakes. Lactose and galactose were included because these have previously been linked with ovarian cancer risk(Reference Webb, Bain, Purdie, Harvey and Green17).
In addition to the itemised foods, three summary questions on total consumption of vegetables (excluding potatoes), fruits and meats were included in the FFQ to assess possible over- or under-reporting on individual food items. These questions took the form ‘How many servings of vegetables (excluding potatoes) do you usually eat a day?’ Response frequencies to these summary questions were then compared with the summed frequencies of consumption of individual items from the meat, fruit and vegetable groups from the FFQ.
Statistical analysis
The Shapiro–Wilks statistic, skewness and kurtosis values were initially used to assess the normality of the data. All analyses were performed on natural log-transformed data. Paired t tests were used to assess whether reported dietary intake differed between the original and repeat surveys for each of the variable pairs. All results were back-transformed to the original scale to aid interpretability.
To assess the reproducibility of the FFQ, weighted kappa and the intraclass correlation (ICC) statistics were calculated for each of the food groups and nutrients to determine the degree of similarity between FFQ1 and FFQ2. The ICC was calculated using a two-way mixed-model ANOVA with single measure reliability. This was performed using the SAS GLM procedure, with the ICC calculated as the ratio of the ‘between-subject mean square minus the error mean square’ and the total variance; 95 % confidence intervals were also estimated(Reference Snedecor and Cochran18).
We categorised the data into quartiles, with the cut-off points for the original survey variables also applied to the repeat variables. The weighted κ statistic was calculated using these categories. The Fleiss–Cohen(Reference Fleiss19) quadratic weighting scheme and 95 % confidence intervals were calculated with the SAS FREQ procedure. This weighting was chosen to produce values that are mathematically the most similar to the ICC measure of agreement. All statistical analyses were performed using the SAS statistical software package version 9·1 (SAS Institute Inc., Cary, NC, USA).
Results
Complete replicate FFQ were available for 100 participants (fifty men and fifty women) with a mean age of 60 years (range 22–79 years). The back-transformed mean intake of food groups and nutrients from FFQ1 and FFQ2 are presented in Tables 1 and 2, respectively. There was generally no difference in mean intake of food groups between FFQ1 and FFQ2. However, intake was significantly lower in FFQ1 than FFQ2 for red/yellow vegetables and other vegetables (Table 1). For nutrients, mean intake was significantly lower in FFQ2 than FFQ1 for total energy, carbohydrate, total fat, polyunsaturated fat, monounsaturated fat, riboflavin, thiamin, Ca and caffeine (Table 2).
Mean daily intake of various food groups, and weighted kappa statistics and intraclass correlation coefficients (ICC), comparing FFQ1 and FFQ2 administered one year apart in men and women (n 100) aged 22–79 years, Australia
*Analysed with ln(x + 1) transformation; back-transformed for presentation.
†Paired t test used to test differences in mean intakes of food groups between FFQ1 and FFQ2.
‡Weighted κ and ICC used to measure agreement between FFQ1 and FFQ2.
Mean daily intake of macro- and micronutrients, weighted kappa statistics and intraclass correlation coefficients (ICC) for the comparison between FFQ1 and FFQ2 administered one year apart in men and women (n 100) aged 22–79 years, Australia
*Analysed with ln(x + 1) transformation, back-transformed for presentation.
†Paired t test used to test for differences between FFQ1 and FFQ2.
‡Weighted κ and ICC used to measure agreement between FFQ1 and FFQ2.
The weighted κ measures of agreement for food groups between FFQ1 and FFQ2 ranged from 0·37 for breads and cereals to 0·71 for low-fat dairy and green leafy vegetables (Table 1). The ICC between the food groups ranged from 0·39 for ‘other vegetables’ to 0·73 for low-fat dairy and green leafy vegetables (Table 1). For the nutrients, weighted κ between FFQ1 and FFQ2 ranged from 0·44 for starch to 0·83 for alcohol (Table 2), while the ICC ranged from 0·51 for starch to 0·91 for alcohol (Table 2). Though the magnitude of the weighted κ corresponded well to the ICC, we calculated the latter to allow comparison with other studies. Stratified analysis by gender showed no difference in weighted κ or ICC between males and females (data not shown).
Mean intake estimated from the summary questions did not differ appreciably between FFQ1 and FFQ2, although mean meat intake was somewhat higher and mean fruit intake slightly lower on FFQ1 than FFQ2. Similarly, total intakes summed across the individual FFQ items did not differ greatly between FFQ1 and FFQ2. However, estimates from the summary questions were significantly lower than the estimates summed from the individual food items for meat, fruit and vegetables on both FFQ1 and FFQ2 (P < 0·02). Estimates based on the sum of individual FFQ items were generally 70–90 % higher than the estimates of the summary questions, suggesting an overestimation of meat, fruit and vegetable intake by the itemised FFQ method (Table 3).
Mean number of servings of fruit, vegetables and meat estimated from the summary questions and individual food items on FFQ1 and FFQ2 administered one year apart in men and women (n 100) aged 22–79 years, Australia
*Excluded potatoes.
Discussion
We assessed the reproducibility of an FFQ by administering the same FFQ to the same group of study participants on two occasions, one year apart, and comparing the food and nutrient estimates obtained. According to Landis and Koch(Reference Landis and Koch20), agreement for κ statistics of 0·40–0·59 is considered to be ‘moderate’, 0·60–0·79 is ‘substantial’ and 0·80–1·00 is ‘outstanding or almost perfect’. Reproducibility of the FFQ in our study population, as measured by the weighted κ statistic, was therefore ‘moderate’ to ‘substantial’ for all food groups and nutrients tested. Of the eighteen food groups studied, eleven (61 %) had weighted κ measures indicating ‘substantial’ agreement and six food groups (33 %) showed ‘moderate’ agreement. Of the twenty-eight nutrients studied, one (alcohol) had a weighted κ measure indicating ‘outstanding or almost perfect’ agreement, eight (29 %) showed ‘substantial’ agreement and nineteen (68 %) had ‘moderate’ agreement. This provides reassurance that the data obtained from this FFQ are reproducible. An earlier version of the FFQ was also shown to have good validity for measurement of diet(Reference Ashton, Marks, Battistutta and Green11–Reference McNaughton, Marks, Gaffney, Williams and Green14) and thus the FFQ can be used to rank subjects in an adult Australian population according to their dietary intake with some confidence.
Our results are comparable to those from studies of similar FFQ used in Western adult populations, where the ICC between two administrations of the FFQ ranged from 0·50 to 0·70(Reference Cade, Burley, Warm, Thompson and Margetts6) or from 0·40 to 0·95(Reference Johansson, Hallmans, Wikman, Biessy, Riboli and Kaaks21, Reference Ocke, Goddijn, Jansoen, Pols, van Staveren and Kromhout22). The ICC we obtained were closer to those from the Nurses’ Health Study(Reference Willett, Sampson, Stampfer, Rosner, Bain, Witschi, Hennekens and Speizer10) and the Health Professionals Follow-up Study(Reference Rimm, Giovannucci, Stampfer, Colditz, Litin and Willett23) than the Helsinki Diet Methodology Study(Reference Pietinen, Hartman, Haapa, Rasanen, Haapakoski, Palmgren, Albanes, Virtamo and Huttunen24) or the Stroke in the Elderly Study(Reference Lazarus, Wilson, Gliksman and Aiken25), which reported somewhat better repeatability perhaps due to the presence of older adults (55 years and older; see Table 4). Reproducibility of nutrient intake among the elderly has been reported to be particularly high(Reference Cade, Burley, Warm, Thompson and Margetts6, Reference Lazarus, Wilson, Gliksman and Aiken25) as older people may be more established in their dietary habits than younger groups(Reference Lazarus, Wilson, Gliksman and Aiken25). In our population, we observed for most foods and all nutrients that the weighted κ values were consistently slightly lower than the corresponding ICC values and this is also in agreement with previous studies(Reference Lazarus, Wilson, Gliksman and Aiken25, Reference Hankin, Yoshizawa and Kolonel26). It has been reported that weighted κ values are sensitive to the choice of cut-off points(Reference Hankin, Yoshizawa and Kolonel26); thus the ICC values are likely to be more robust.
Comparison of intraclass correlation coefficients for selected nutrients from various reproducibility studies
*Nutrients are unadjusted for energy.
†Australian Cancer Study/Australian Ovarian Cancer Study.
We have shown that there is reproducibility in the reporting of meat and vegetables using the summary questions, and in the reporting of meat and fruits using detailed items on an FFQ. We have also shown that intakes were higher when assessed by FFQ compared with summary questions. It is likely that this represents overestimation of intake by the FFQ items compared with the summary questions, as has been reported in several previous studies(Reference Amanatidis, Mackerras and Simpson27–Reference Calvert, Cade, Barrett and Woodhouse30). While this may cause problems with estimation of absolute levels of intake, it should not affect the ranking of individuals and thus is not a major problem for studies where this is the primary aim(Reference Mina, Fritschi and Knuiman29). Furthermore, this overestimation can potentially be corrected by weighting the itemised FFQ responses by a factor equal to the ratio of the reported frequency on the summary question divided by the estimated frequency from summing the individual items on the FFQ in the same food group(Reference Bogers, Dagnelie, Westerterp, Kester, van Klaveren, Bast and van den Brandt28, Reference Calvert, Cade, Barrett and Woodhouse30).
The estimates for nine nutrients were significantly lower on FFQ2 than FFQ1, but higher for two food groups. These changes could reflect a genuine change in diet over the 1-year period. Overall 11 % of participants reported that their diet had changed between FFQ1 and FFQ2; when this occurred they were asked to report their usual diet before the change. However it is possible that, in some individuals, differences between FFQ1 and FFQ2 reflect a genuine shift in intake over the year between the two questionnaires and, if this is the case, our data would tend to underestimate the reproducibility of our FFQ.
In summary, the 135-item semi-quantitative FFQ provided reproducible measurements for individuals over a period of 1 year, suggesting that it is reasonable to use for classifying individuals into groups based on their relative levels of intake.
Acknowledgements
Sources of funding: This study was supported by the National Health and Medical Research Council (NHMRC) of Australia (Programme No. 199600) and P.M.W. is a Senior Research Fellow of the Cancer Council Queensland. The funding bodies played no role in the design or conduct of the study; the collection, management, analysis or interpretation of the data; or preparation, review or approval of the manuscript. Conflict of interest declaration: None of the authors had a personal or financial conflict of interest. Authorship responsibilities: T.I.I. participated in cleaning diet data and was responsible for writing the manuscript; S.P. was responsible for collection of repeat FFQ data, preliminary data analysis and participated in writing the manuscript; K.M. performed statistical analysis and edited the manuscript; M.C.H. participated in cleaning diet data and performed nutrient data extraction; P.K.O. provided statistical expertise, interpretation of data and edited manuscript; P.M.W., principal investigator, conceptualized the study, performed field work planning and data collection, and edited the manuscript. Acknowledgements: The Australian Cancer Study (ACS) Investigators are Adèle C. Green, Peter G. Parsons, Nicholas K. Hayward, Penelope M. Webb and David C. Whiteman; the Australian Ovarian Cancer Study management group comprises David D.L. Bowtell, Adèle C. Green, Georgia Chenevix-Trench, A. deFazio, D. Gertig and Penelope M. Webb. We thank the ACS project staff and the study participants for their time and effort in completing the FFQ on two occasions.
Appendix
Food/nutrient groupings used in the reproducibility analysis
