Dietary modification is considered an important component in the prevention, management and treatment of chronic conditions such as hypertensionReference Moore, Conlin, Ard and Svetkey1, hypercholesterolaemiaReference Ammerman, Keyserling, Atwood, Hosking, Zayed and Krasny2, diabetesReference Steyn, Mann, Bennett, Temple, Zimmet and Tuomilehto3 and obesityReference Nicklas, Dennis, Berman, Sorkin, Ryan and Goldberg4. Recently, there have been a number of large-scale clinical trials that continue to test the effect of dietary modification and/or supplementation on primary and secondary prevention of a wide range of chronic and infectious diseasesReference Bazzano, Serdula and Liu5–Reference Fawzi, Msamanga, Spiegelman, Wei, Kapiga and Villamor8.
In large-scale community-based intervention trials, the more precise methods of dietary assessment such as interviewer-based dietary recalls, food records or lengthy diet histories can be prohibitively expensive and carry a respondent burden that contributes to a high drop-out rate – particularly in underserved communitiesReference Kristal, Feng, Coates, Oberman and George9. A practical and cost-effective alternative to diet recalls and diet histories is to assess dietary intake with a self-administered food-frequency questionnaire (FFQ). Numerous validation studies have shown that in a prospective cohort study, the FFQ can be a valid estimate of the food and nutrients measured by dietary recalls or diet historiesReference Willett10. To date, only a few studies have validated the use of the FFQ to assess baseline diet and dietary change (due to prescribed intervention) in an intervention trialReference Kristal, Beresford and Lazovich11–Reference Patterson, Kristal, Rodabough, Caan, Lillington and Mossavar-Rahmani15.
In the present study, we have conducted a validation analysis of data from a 6-month intervention trial that investigated the health effect of daily incorporation of walnuts into the habitual diet of free-living individuals. Dietary data were collected at baseline and during the follow-up by multiple 24-hour dietary recalls and by a self-administered 171-item semi-quantitative FFQ. The purposes of the present validation study were (1) to examine whether the FFQ provided a valid measure of nutrient intake among subjects enrolled in a clinical trial, and (2) to examine whether the FFQ can measure the effect of dietary intervention (walnut supplementation) on intake of specific nutrients. Findings from this study will contribute to the design of an FFQ that can be used in large-scale community intervention trials of diet and health outcomes.
Materials and methods
Study population
As previously describedReference Sabaté, Cordero-MacIntyre, Siapco, Torabian and Haddad16, the study population was recruited from 94 volunteers who responded to local advertisements from Loma Linda University about a feeding trial. Three of the 94 participants dropped out due to compliance difficulty, one did not fill out the FFQ and three completed fewer than six dietary recalls. Thus, the analytical population for this validation study consisted of 87 subjects who completed at least six dietary recalls and the self-administered FFQ. Among the recalls collected from the 87 subjects, five recalls were excluded due to their extreme total energy values. We defined ‘extreme’ as being outside the biologically plausible range of 800–4500 kcal among men and outside the biologically plausible range of 500–3500 kcal among women.
Study design
At baseline, subjects were randomly assigned to either the control (habitual or usual diet) or the walnut-supplemented group. The control group was instructed to eat their usual diet and refrain from eating walnuts ( ≤ 2 g of walnuts per day) and substantial amount of other nuts, including nut butters, during the study. The intervention group was provided with 25–56 g of pre-packaged raw walnuts to be eaten daily together with their usual diet in any way they preferred. This amount of walnuts accounted for ∼12% of the subject's total daily energy intake. Since the intervention was among free-living adults, we did not give any other dietary advice to the participants. During the 6-month follow-up, we collected, through telephone interviews, at least six 24-hour dietary recalls (mean = 6.9 recalls) at intervals of 2–5 weeks. The protocol for these recalls is given below. At the end of 6 months, the subjects completed a self-administered FFQ which assessed intake during the past 6 months.
FFQ
We modified a previously validated FFQ that was developed at our institutionReference Knutsen, Fraser, Beeson, Lindsted and Shavlik17, Reference Knutsen, Fraser, Linsted, Beeson and Shavlik18. The original questionnaire was designed to allow a measure of vegetarian diet patterns (i.e. meat analogues). As described previously, Reference Knutsen, Fraser, Beeson, Lindsted and Shavlik17, Reference Knutsen, Fraser, Linsted, Beeson and Shavlik18 for the intervention trial in a general population sample, we removed many of the items on specific meat analogues and added additional items on meat intake. The questionnaire was then pilot-tested for clarity, interpretation and improvement of format in a substudy of 10 individuals, who had similar demographic characteristics to the study group but were not participants in the study.
The FFQ is semi-quantitative and consists of 171 hard-coded food and 15 write-in items. The respondent is asked to report average consumption for the past 6 months. Food items are grouped under the following categories: breads, grains and starches; vegetables; legumes and nuts; eggs, dairy products, oils; fish and meats; fruits; beverages; sweets and baked goods; and condiments, dressings, miscellaneous. Portion sizes are based on average serving sizes using familiar measuring devices, e.g. cup, teaspoon, can and others. The frequency section consists of eight categories: never or rarely, 1–3 per month, 1 per week, 2–4 per week, 5–6 per week, 1 per day, 2–3 per day and 4+ per day. Except for four write-in items (cold breakfast cereals), lack of substantial respondent information precluded inclusion of 11 write-in items in the validation of this questionnaire.
Reference method: 24-hour dietary recalls
Research nutritionists collected 24-hour dietary recalls during telephone interviews with the participants using a protocol that has been previously describedReference Sabaté, Cordero-MacIntyre, Siapco, Torabian and Haddad16. The research nutritionists trained in the use of Nutrition Data Systems (NDS-R) for Research software package19. The NDS-R system enables collection of data on 18 000 foods, 8000 brand name products and many ethnic foods. For 24-hour recalls, the NDS software has built-in quality control with a multiple-pass approach interview methodology19. Default food description assignments on the software standardise data collection and therefore minimise inter-interviewer variation. A two-dimensional food portion visual was provided to help participants estimate food portion sizes.
The telephone interviews were unannounced, unscheduled, and included a standard script with instructions that were read by each research nutritionist before the NDS-R protocol was implemented. On average, recall interviews lasted 30 min. Each participant provided at least six dietary recalls (mean ± standard deviation (SD) for recalls = 6.9 ± 0.4). Day-to-day variation was accounted for by obtaining recalls on either (1) all days of the week or (2) at least one weekend day and five weekdays.
Data analyses
Validation
Collection and coding of dietary intakes on both the FFQ and dietary recalls were performed using NDS-R software. To determine average frequency of intake per day on the FFQ, we converted the frequency value of average consumption over the past 6 months into frequency of intake per day (e.g. frequency of 1–3 per month = 2/30 = 0.067, and so on). The factors were 0 for never or rarely, 0.067 for 1–3 per month, 0.143 for 1 per week, 0.429 for 2–4 per week, 0.786 for 5–6 per week, 1.0 for 1 per day, 2.5 for 2–3 per day, and 5 for 4+ per day. Nutrient amounts were relative to the fixed portion/serving sizes in the semi-quantitative FFQ. For multiple-food items, the nutrient compositions of the more commonly eaten varieties of the foods in the item were averaged, e.g. the nutrient composition computed for a multiple food-item that consists of roll, English muffin, bagel or soft pretzel was based on the average of the nutrient compositions of a medium size (white flour) roll, medium size English muffin, medium size (white flour) bagel and medium size soft pretzel. The average of raw and cooked nutrient components of vegetables known to be eaten either raw or cooked was used for vegetables such as carrots, broccoli and others. For single food items where several types exist, the NDS-R software computes nutrient composition for the most commonly eaten type – a determination made using nationally representative market research data19. Nutrient intake per day was then computed using the sum-product method, i.e.
where A = total intake of nutrient A per day; F i = frequency of food i intake per day; and a i = amount of nutrient A in food i.
To determine the relative validity of the FFQ's nutrient intake estimates at the group level, we compared its means and standard deviations with those of the reference method using paired samples t-test. For relative validity at the individual level, we performed correlation analyses. Since our intent was to determine the amount of variation explained by the test method on the reference method (i.e. R 2), the use of parametric (Pearson's) correlation analyses does not assume any particular underlying distribution of the variablesReference Singh, Fraser, Knutsen, Lindsted and Bennett20. To correct for within-person variation in multiple dietary intake measurements with the reference method (dietary recalls), the following formula was usedReference Willett10:
where r c = corrected/de-attenuated correlation coefficient; r0 = uncorrected/attenuated correlation between FFQ and multiple 24-hour recalls; 
= within-person variance of the multiple 24-hour recalls; = between-person estimate of variance in the reference method (24-hour recalls); and n = number of repeated measures of the dietary recalls.
De-attenuation of the correlation coefficients using the method described above creates conditions in which we could not assume normally distributed errors for r c. Thus, instead of the traditional asymptotic methods to determine confidence intervals about r c, we computed ‘distribution-free’, non-parametric 95% confidence intervals using the BCa bootstrap re-sampling methodReference Efron and Tibshirani21, with each confidence interval determined from the distribution of r cs from 2000 samples.
Nutrient variables were also adjusted for energy since amounts of walnuts allotted to the participants differed according to their energy intake. Nutrient intake distributions were normalised by logarithmic and square root transformations before using the residual method for energy adjustment. Energy-adjusted correlations were obtained using Pearson's correlations.
Intervention measure
We were interested in finding out if the FFQ would be able to assess the prescribed intervention measure in the two diet groups. The test food, walnut, is high in α-linolenic acid (ALA) (9.1 g/100 g walnuts) and linoleic acid (LA) (38.1 g/100 g walnuts). Walnut studies have shown increases in plasma levels of LA and ALA as a result of incorporation of walnuts into the dietReference Almario, Vonghavaravat, Wong and Kasim-Karakas22, Reference Chisholm, Mann, Skeaff, Frampton, Sutherland and Duncan23. Although the amount of LA in walnuts is much higher compared with that of ALA, LA is ubiquitous in the American diet and using it as a marker of intervention measure in this study may be confounded by other food sources of LA.
ALA is a more unique component of walnuts than LA. It is important to note that ALA is also found in items such as flaxseed, flaxseed oil, canola oil and soybean oil. We did not expect that walnut supplementation would substantially change intake of these items.
To assess further whether ALA would be the most appropriate marker of the intervention measure, we determined the percentage of energy accounted for by both LA and ALA, using 1.1% as the cut-off for ALA (i.e. the percentage of energy accounted for by ALA in 28 g of walnuts if the total energy requirement is 2000 kcal day− 1) and 4.8% as the cut-off for LA (i.e. the percentage of energy accounted for by LA in 28 g of walnuts eaten if the total energy requirement is 2000 kcal/day− 1). We then compared the calculated percentage of energy values for each diet group with this criterion. Reference method (dietary recall) values of mean percentage energy from ALA intake were 2.2% (SD = 0.4) for the walnut diet group and 0.6% (SD = 0.2) for the control diet group. For LA, however, reference method values of mean percentage energy from LA intake were 11.3% (SD = 1.9) for the walnut study group and 5.6% (SD = 1.6) for the control diet group. Determining a cut-off value for LA that will delineate intervention from non-intervention was more challenging than for ALA because LA is not unique to walnuts. Our use of ALA as the nutrient marker for the intervention is, therefore, justified.
We assumed that proportional increases in ALA intake would be a consequence of consuming 28–56 g of walnuts. As already shown earlier, for an individual whose daily energy requirement is 2000 kcal, 28 g of walnuts will result in an ALA intake that accounts for ∼1.1% of that individual's daily energy requirement. Utilising this logic, we defined the intervention measure as ALA intake ≥ 1.1% energy for the walnut diet group and ALA intake < 1.1% of energy for the control diet group.
Means of absolute estimates ( ± SD) of ALA by the FFQ and the 24-hour dietary recalls were computed for both diet groups. We used cross-classification to determine agreement between the two methods in assessing adherence to the prescribed intervention measure described above.
Graphical approaches were also used to determine agreement between the test and reference methods in the estimation of ALA – the nutrient most indicative of the intervention effect. Specifically, we plotted the FFQ and recall values of ALA in the control and walnut group (Fig. 1). We also constructed a Bland–Altman plot to assess agreement between the methods by plotting the difference between FFQ and recalls vs. the recall values.
Plot of α-linolenic acid (ALA) intake estimate values from the food-frequency questionnaire (FFQ) against dietary recall estimates of ALA intake
Results
Demographic characteristics of the study population of 87 subjects are given in Table 1 and are further stratified by intervention group. The study population was 74% Caucasian and was aged 30–72 years (mean = 54.7). We found no important demographic differences between the walnut-supplemented and control groups.
Demographic characteristics of study subjects
BMI – body mass index.
All values are reported as mean (standard deviation).
Mean height and mean weight are significantly different between males and females.
Validation of FFQ nutrient measures in all subjects
In Table 2 we have reported the mean values for 32 nutrients computed from the FFQ and the corresponding nutrients computed from the mean of the diet recalls. Mean nutrient intake on the FFQ was higher than dietary recalls for all nutrients, and these differences were statistically significant for all nutrients except for animal protein, ALA, cholesterol and alcohol. Total energy intake was overestimated by 160 kcal (8.4%), and for the remaining nutrients the overestimation ranged from 1.3% (retinol) to 87.5% (docosahexaenoic acid).
Nutrient intake estimates of the food-frequency questionnaire (FFQ) and 24-hour dietary recalls (DR) for all participants (n=87)
SD – standard deviation; CHO – carbohydrate; SFA – saturated fatty acid; MUFA – monounsaturated fatty acid; PUFA – polyunsaturated fatty acid; LA – linoleic acid; ALA – α-linolenic acid; EPA – eicosapentaenoic acid; DHA – docosahexaenoic acid.
* Significantly different from DR values at P < 0.0001.
Table 3 provides the de-attenuated (corrected for measurement error in the 24-hour diet recalls) correlations for 19 out of the 32 nutrients measured by the FFQ and the mean of at least six 24-hour dietary recalls. Statistically significant de-attenuated correlations were found for total energy (r = 0.39), total carbohydrate (r = 0.42), vegetable protein (r = 0.43), total fat (r = 0.51), polyunsaturated fat (r = 0.77), LA (r = 0.78), ALA (r = 0.79), and total fibre (r = 0.60). The de-attenuated correlation for saturated fat was high (r = 0.59), but did not attain significance. For the vitamins and minerals, a very high de-attenuated correlation was found for vitamin C (r = 0.96), and significant correlations in the range of 0.46–0.80 were found for calcium, phosphorus, magnesium, iron and potassium.
De-attenuated (corrected for measurement error in the 24-hour diet recall standard) correlation coefficients between the food-frequency questionnaire and 24-hour dietary recall (at least six recalls per subject) measures of 19 nutrients are presented with non-parametric confidence limits (CI)
CHO – carbohydrate; SFA – saturated fatty acid; MUFA – monounsaturated fatty acid; PUFA – polyunsaturated fatty acid; LA – linoleic acid; ALA – α-linolenic acid.
* The confidence interval could not be computed due to high between-person variance.
Table 4 provides uncorrected correlation coefficients for the remaining 13 nutrients that could not be corrected for measurement error due to very high between-person variance. Among these 13 nutrients, strong correlations (>0.40) were found for retinol, β-carotene, folate and alcohol. Further adjustment for total energy intake tended to strengthen the positive correlations for seven out of the 13 nutrients and, after energy adjustment, all attained statistical significance.
Correlation coefficients between the food-frequency questionnaire and the 24-hour dietary recall (at least six recalls per subject) measures of 13 nutrients are presented with and without adjustment for total energy intake
EPA – eicosapentaenoic acid; DHA – docosahexaenoic acid.
Correction for measurement error in the 24-hour recalls was not possible for these nutrients due to high between-person variation.
*P < 0.05, **P < 0.01, ***P < 0.0001.
Intervention measure
As shown in Table 3, ALA, a good nutrient marker for walnut supplementation, was measured well by the FFQ, with a correlation of 0.79 with the diet recalls. In the trial, we had used the following ALA-based criterion to distinguish between the two intervention groups: ALA was ≥ 1.1% of total energy for the walnut-supplemented group and < 1.1% of total energy for the controls. In Table 5 we present the percentage total energy from ALA for both groups that was measured by the FFQ and diet recalls. Both the FFQ and the diet recalls found a significantly higher percentage of total energy from ALA in the walnut-supplemented group (P < 0.05 for walnut-supplemented vs. control on both FFQ and diet recall). We also examined the sensitivity of the FFQ in classifying subjects into the intervention groups based on being ≥ 1.1% and < 1.1% total energy from ALA in their diet recalls. We found that in the controls, the sensitivity of the FFQ in detecting < 1.1% total energy from ALA was 80% and in the walnut-supplemented group the sensitivity of the FFQ in detecting ≥ 1.1% total energy from ALA was 98%.
Percentage energy from α-linolenic acid (ALA) for the walnut-supplemented diet and control that were measured by the food-frequency questionnaire (FFQ) and 24-hour dietary recalls (DR)
SD – standard deviation.
Figures 1 and 2 show that the FFQ estimates closely paralleled the recall values in their indication that ALA intake in the walnut-supplemented group was greater than that in the control group. The Bland–Altman plot (Fig. 2) indicates that there was good agreement between the two methods as evidenced by the symmetry of the data around the zero value (indicating a zero difference between the FFQ and recall measures of ALA). Taken together, these findings indicate that the FFQ was a good measure of the nutrient effect (i.e. increased ALA intake) of the walnut-supplemented intervention.
Difference between the α-linolenic acid (ALA) intakes estimated from the food-frequency questionnaire (FFQ) and the reference method (24-hour dietary recall) plotted against the mean of the two methods. The solid line represents the mean while the dotted line represents the limits of agreement (95% confidence interval)
Discussion
We sought to validate the measurement of 32 nutrients by a 171-item FFQ that was administered during a dietary intervention trial among free-living individuals. After correction for measurement error in the standard (multiple diet recalls during the follow-up), we found significant positive correlations between the FFQ and diet recalls for total energy (r = 0.39), total carbohydrate (r = 0.42), vegetable protein (r = 0.43), total fat (r = 0.51), polyunsaturated fat (r = 0.77), total fibre (r = 0.60), LA (r = 0.78) and ALA (r = 0.79) – the last nutrient being an excellent biomarker of the intervention with walnut supplementation. Among the nutrient correlations that could not be corrected for measurement error due to high between-person variance, we also found good validity (uncorrected correlations >0.40) for retinol, β-carotene, folate and alcohol. Energy adjustment tended to increase the magnitude and statistical significance of all uncorrected correlations.
Absolute intake estimates on the FFQ were significantly higher (up to 88%) for most of the nutrients compared with the reference method estimates. The length of the questionnaire, wherein a number of food items might not have been consumed or reported to have been consumed during any of the dietary recalls, could have contributed to this bias. While results vary, FFQs usually have higher estimates for most nutrients than the reference methodReference Potischman, Carroll, Iturria, Mittl, Curtin and Thompson24, Reference Subar, Thompson, Kipnis, Midthune, Hurwitz and McNutt25 particularly if the FFQ exceeds 100 itemsReference Ocke, Bueno-de-Mesquita, Pols, Smit, van Staveren and Kromhout26–Reference Yanek, Moy and Becker28. Overestimation in dietary assessment questionnaires may be due to measurement errors introduced by differences in conceptualisation of portion sizes, misinterpretation of specific itemsReference Willett10, and frequency and serving size differences between the test and reference methods. This is especially the case when there are multiple foods in an itemReference Flegal and Larkin29. Moreover, averaging amounts of intake over a long period of time, as with an FFQ, is subject to estimation misjudgement that may be different from estimation errors in reporting amounts of intake over the past few hours, as in dietary recalls.
The lower, albeit significant, correlation of 0.39 for total energy is noteworthy. This may be consistent with other studies indicating that subjects tend to under-report their total energy intake on diet recalls administered during intervention trialsReference Caan, Ballard-Barbash, Slattery, Pinsky, Iber and Mateski30. In our study, the FFQ values tended to be ∼160 kcal more than the diet recalls. Further investigation of under-reporting in the recalls would require validation of the standard to values from direct measurement such as the doubly labelled water technique. We examined under-reporting of total energy by comparing total energy values with change in body weight during the first 6 months of the trialReference Siapco, Burns-Whitmore, Haddad and Cordero-MacIntyre31 and found possible under-reporting of values. Using Goldberg's method, 38% of the participants were classified as under-reporters, and 40% of the participants who reported an energy intake lower than their estimated energy expenditure actually gained >0.50 kg body weight. We found that exclusion of under-reporters from the analysis did not substantially change the magnitude of the correlations.
Correcting for random errors associated with within-person variation almost always increases correlation valuesReference Block32, and our results for de-attenuated correlations follow this same trend. However, high within-to-between person variances resulted in very high correction factors for 13 of the 32 nutrients and therefore in Table 3 we reported only uncorrected correlations. Further studies with a larger sample size are needed to investigate corrected correlations for these 13 nutrients. Another noteworthy factor is the possibility of errors in the estimation of the within-person and between-person variancesReference Willett10.
The ability of the FFQ to detect the intervention measure (walnut supplementation) among the diet groups was clearly shown by the high correlations with ALA in Tables 3 and 5. Walnuts are unique as a nut because of their ALA (18:3n-3) content. Both diet groups were instructed at the beginning of the study to eat or to not eat walnuts according to their diet assignment. Thus, the possibility that a training effect and compliance bias may have contributed to the relatively high agreement between FFQ and dietary recall regarding ALA intake for both diet groups cannot be discounted. It is important to note that our study examined the correlation between diet recalls and FFQ values during the intervention trial. Further studies are needed to determine whether the change in nutrient intake between baseline and follow-up is detected by the FFQ.
Our findings indicate that the FFQ did demonstrate excellent relative validity (compared with recalls) in the estimation of specific fatty acids (i.e. ALA) associated with a single food intervention (walnut), but had less relative validity in the estimation of macronutrients (i.e. total protein and saturated fat). Further work is needed to determine whether the FFQ can estimate nutrient intake in trials where the intervention diet is complex (i.e. a diet pattern) and the treatment effect represents a change in macronutrient intake (i.e. change in protein).
Acknowledgements
We would like to acknowledge the research nutritionists who helped collect the 24-hour dietary recalls and the California Walnut Commission for the funding of this research.
