Collection of high-quality dietary data in large populations is a priority challenge in nutritional epidemiology. Usual methods for measuring food and nutrient intakes (food records, 24 h recalls, dietary history and FFQ) are based on self-administered questionnaires or interviews by trained dietitians. These methods are complex and costly, requiring substantial logistic resources when applied to large populations(Reference Thompson, Subar and Loria1). Over the past few decades, new technologies have been developed to enhance accuracy, increase speed and minimise the cost and inconvenience of assessing dietary intake. The use of information and communication technologies for dietary assessment has recently been reviewed(Reference Thompson, Subar and Loria1–Reference Ngo, Engelen and Molag3). Those reviews stated that new methods hold promise as novel, high-quality, cost-efficient tools. However, several limitations in previously published validation and comparison studies were identified: many were based on a small sample size (less than 100 subjects), focused on relatively young populations and examined a restricted range of nutrients (mainly macronutrients). Therefore, research into validating technology-based applications for dietary assessment in a variety of populations is needed(Reference Thompson, Subar and Loria1–Reference Ngo, Engelen and Molag3).
Previous studies compared a self-administered computerised (but not web-based) 24 h recall or record(Reference Baranowski, Islam and Baranowski4–Reference Zoellner, Anderson and Gould6) with a dietitian-conducted interview. They concluded that these new methods compare well with the usual methods. The National Cancer Institute (Frederick, MD, USA) is currently undertaking detailed evaluation studies to compare a new web-based 24 h dietary recall with traditional methods and biomarker data(Reference Schatzkin, Subar and Moore7–Reference Zimmerman, Hull and McNutt10). Although the use of web-based self-administered recalls or records has been reported in previous epidemiological studies(Reference Forbes, Storey and Fraser11–Reference Simunaniemi, Andersson and Nydahl13), only one study published thus far compared this method with a dietitian-based interview, and it focused only on children(Reference Hanning, Royall and Toews14). That study observed relatively good agreement between the two methods for energy and macronutrients.
The NutriNet-Santé study was launched in May 2009 in France to investigate the relationship between nutrition and health along with determinants of dietary behaviour (www.etude-nutrinet-sante.fr). This is the first web-based prospective nutritional epidemiology cohort study worldwide on such a broad scale; it is scheduled to include 500 000 volunteers (aged ≥ 18 years) over the next 5 years. Before this, we had conducted test studies to compare the web-based version of questionnaires with the corresponding usual methods. Results of the anthropometric questionnaire test study have recently been published(Reference Touvier, Mejean and Kesse-Guyot15).
In the NutriNet-Santé cohort study, dietary assessment includes three 24 h records and a semi-quantitative FFQ per year of follow-up. The objective of the present study was to compare, for 1 d of food intake, the NutriNet-Santé web-based self-administered 24 h dietary record with a 24 h recall conducted by a trained dietitian (corresponding to the same day and using the same computerised interface for data entry), by administering both methods to a sample of mature subjects. We sought to determine whether information filled out by participants via the Internet was comparable with information obtained by a trained professional. Our hypothesis was that there should be close agreement between the two methods for assessment of food consumption, energy and macro- and micronutrient intakes, along with substantial logistic and cost advantages associated with the web-based self-administered method.
Methods
Study population
In January 2009, a call for participation was launched among subjects of the ‘Supplémentation en VItamines Minéraux et AntioXydants’ cohort(Reference Hercberg, Preziosi and Briancon16). Persons interested in participating in the test studies were required to register their candidacy on a specific website and provide an up-to-date email address. A valid email address and potential access to the Internet were required for participation. Before the end of January, 1090 persons volunteered to participate in NutriNet-Santé validation studies. Among them, 170 were randomly selected to test the web-based 24 h dietary record. The present study was conducted according to the guidelines laid down in the Declaration of Helsinki, and all procedures were approved by the Ethical Committee for Studies with Human Subjects and the Commission Nationale Informatique et Libertés (CNIL no. 908450). Electronic informed consent was obtained from all subjects.
Methodology of the comparative study
Participants were asked to fill in the self-administered web-based 24 h record on a date agreed upon beforehand. An appointment was made with a dietitian for the following day in order to complete the 24 h recall by telephone.
The NutriNet-Santé self-administered web-based 24 h dietary record
The interactive record is designed for self-administration on the Internet. It is based on a secured user-friendly interface and includes detailed instructions in several forms (PDF user's guide, video, tips included within the questionnaire, etc.). It relies on a meal-based approach, recording all foods and beverages consumed (nature and quantity) for each eating occasion: breakfast, lunch, dinner and all other eating occasions. The time and place of each eating occasion is systematically recorded. First, the participant fills in the names of all food items eaten using three possible methods: (1) a food browser in which foods are grouped by category (vegetables, dairy products, etc.) into a classification tree in which the participant browses each branch until reaching the consumed food item; (2) a search engine that accepts spelling errors; (3) manual typing (in case the food was not found by the first two methods). For specific foods with potentially high nutrient variability, participants are asked to provide the trademark.
To avoid omissions, supervision is integrated at two levels: (1) for each food entered, the software proposes a list of other items usually associated with it (e.g. sugar in coffee); (2) at the end of the food entry step of each eating occasion, it reminds the user of usually consumed items, such as water, bread, sugar, salt, etc.
Next, the participant estimates portion sizes for each food and beverage previously listed using photographs directly included in the computerised interface. These photographs, from a validated picture booklet(Reference Le Moullec, Deheeger and Preziosi17), represent more than 250 foods (corresponding to 1000 generic foods) proposed in three different portion sizes. Along with the two intermediate and two extreme quantities, there are seven choices of amounts. Instead of using the photographs, the participant can directly enter the quantity consumed in g or volume, if known.
24 h Recall carried out by a dietitian
The self-administered record must be completed (without the possibility of modification or consultation by the subject or the dietitian) before the dietitian's interview. This phone interview (24 h recall) evaluated what was eaten on the same day as the self-administered record, and was conducted by a trained dietitian using the same computerised interface. To describe portion sizes during the interview, the validated picture booklet(Reference Le Moullec, Deheeger and Preziosi17) (paper version) was mailed to participants beforehand.
‘Satisfaction’ questionnaire
After completing the two dietary assessments, participants were requested to fill in a web-based ‘satisfaction’ questionnaire describing their attitude towards the web (self-perceived level of computer and web knowledge, type of connection), how long it took to fill out each evaluation, their opinion of each method and their preferred method.
Statistical analyses
Nutritional values of the diet were estimated using a published French food composition table(Reference Hercberg18), completed and updated. French recipes validated by food and nutrition professionals were used to assess the amounts consumed from composite dishes.
For foods, the intraclass correlation coefficient (ICC), estimated as a reliability coefficient(Reference Winer19), was used to measure the strength of agreement between the two methods. The ICC (3, 1) was used, as described by Shrout & Fleiss(Reference Shrout and Fleiss20), based on within-subject and residual variations provided by two-way ANOVA. For each food group, ICC were estimated in two ways: (1) among consumers only; (2) among all subjects (considering consistently null consumption of a food item in the two methods as being concordant per se). For nutrients, agreement was assessed by Pearson's correlation coefficients for non-adjusted and energy-adjusted nutrient intakes. Energy adjustment was performed using the residual method(Reference Willett21). To improve normality, nutrient intakes were logarithmically transformed before analysis. For ethanol, ICC were estimated due to a substantial proportion of non-consumers during the day of data collection. Analyses were performed for men and women separately, as previously recommended for dietary validation studies(Reference Gibson22). ICC and Pearson's correlations were also calculated stratified by age group ( < 60 years/ ≥ 60 years), educational level (university/less) and self-estimated computer knowledge (‘novice/inexperienced’/‘experienced/expert’). The Wilcoxon signed-rank sum test was used to compare mean ICC and mean Pearson's correlations between strata. Non-parametric Spearman's correlations were also assessed for foods and nutrients.
Answers to the ‘satisfaction’ questionnaire were compared between the two methods using Bowker's test of symmetry and McNemar's χ2 tests. Finally, we estimated the cost of each method.
For all analyses, the significance level was two-sided and set at 0·05. All statistical analyses were performed using Statistical Analysis Systems software (version 9.1, 2006; SAS Institute, Inc., Cary, NC, USA).
Results
Characteristics of the study population
We excluded twenty-three subjects from the 170 initially selected: three subjects could not be contacted by phone at all, three used a Macintosh computer (not compatible with our web tool), six had computer or connection problems related to their equipment (for example: full mailbox not able to receive new messages), seven stopped because they felt the web system was too complex for them, three completed only one of the two methods and one refused to be contacted by phone. Among the 147 individuals included in the analyses, 124 also filled in the satisfaction questionnaire. Table 1 presents sociodemographic and web characteristics of the study population. Participants were mature in age (mean 60·8 (sd 6·0) years; age range 48–75 years), 59·2 % were women and 22·6 % considered themselves to be ‘novices/inexperienced’ in computer use.
(Mean values and standard deviations, numbers and percentages)
* Five missing data for educational level and ten for occupational category. One hundred and twenty-four subjects returned the satisfaction questionnaire, thereby providing information on computer and web knowledge and type of connection.
† P value for the difference between men and women from Student's t test and Fisher's χ2 tests, where appropriate.
‡ Current occupation or most recent job if retired or unemployed.
Comparison between the web-based 24 h dietary record and the dietitian-conducted 24 h recall
Of the dietary records, 20 % corresponded to weekdays and 80 % to weekend days. With each method, only two subjects had energy intake below 3349 kJ (800 kcal) for the day of the study. One of them declared having intentionally reduced food intake that day in order to compensate for overeating during the previous days.
Food groups
Among consumers only, the median of ICC was 0·8 in men and 0·7 in women; range: 0·5 for fats/sauces (both sexes), breakfast cereals, cakes/biscuits/pastries and dairy food (women only) to 0·9 for fruits, pulses (both sexes), breakfast cereals, alcoholic drinks and meat (men only) (Table 2). When non-consumers were also included in the calculation of ICC, the median was 0·9 for both sexes (range 0·7–1·0) (data not tabulated). Mean ICC did not significantly vary across categories of age, education or computer knowledge (P>0·2) (data not shown). Spearman's correlations showed results similar to ICC (median for both sexes: 0·7 for consumers only and 0·8 for all subjects) (data not tabulated).
(Number of consumers, mean values and standard deviations, ICC and 95 % confidence intervals)
* ICC were calculated taking into consideration all subjects with non-null consumption for at least one of the two methods.
Only seven food items were entered manually using the interview method, compared with 107 food items using the web-based record. These 107 food items were reclassified (one dietitian work day); 80 % corresponded to pre-existing items in the classification and 20 % were new food items.
Nutrients
The median of non-adjusted and energy-adjusted Pearson's correlation coefficients was 0·8 in both sexes. The range for energy-adjusted Pearson's coefficients was 0·6 for polyunsaturated fat, retinol, vitamin E and Na (in women) to 0·9 for a wide range of nutrients (Table 3). The lowest coefficients were mainly related to rather small variations between the two methods in the quantity of foods containing high amounts of the nutrient (e.g. liver and camembert for retinol).
(Mean values and standard deviations, Pearson's correlations and 95 % confidence intervals)
RE, retinol equivalents; TE, tocopherol equivalents.
* The nutrient data were log-transformed to improve normality for the calculation of Pearson's correlations.
† Adjusted for energy intake (without alcohol) from the residual method.
‡ For ethanol, intraclass correlation coefficients (and not Pearson's coefficients) were calculated due to a substantial proportion of non-consumers.
The mean Pearson correlation was higher in subjects aged < 60 years (0·8 (sd 0·1) v. 0·7 (sd 0·2) in older subjects; P = 0·02) and in subjects who declared being ‘experienced/expert’ with computers (0·8 (sd 0·1) v. 0·7 (sd 0·3) in ‘novice/inexperienced’ subjects; P = 0·0003), but no statistical difference was observed across education categories (P = 0·12) (data not tabulated). Spearman's correlations showed results similar to Pearson's correlations (median for both sexes: 0·8) (data not tabulated).
Duration and satisfaction
Among the 124 subjects who completed the ‘satisfaction’ questionnaire, the mean completion time was similar for the two methods (web: 31 (sd 29) min and interview: 27 (sd 13) min, P = 0·2, median for both methods: 25 min). However, completion time was more heterogeneous in the web-based method (Table 4). The web-based method was preferred by 66·1 % of the subjects.
(Numbers and percentages)
* Among the 124 subjects who returned the satisfaction questionnaire.
† P value for the difference between the two methods from Bowker's test of symmetry for completion time and from McNemar's χ2 tests for other variables.
Web-based 24 h record
Up to 63·7 % of participants filled out the web-based 24 h record in one sitting. According to 92·7 % of the subjects, the web-based interface was user-friendly. About two-thirds of the subjects (63·7 %) consulted the ‘help’ section, and 88·6 % of them were satisfied. For 80·7 % of the subjects, the e-frame supervisor enabled them to correct and complete their record.
Dietitian-conducted interview
Nearly all subjects (96·8 %) appreciated the help provided by the dietitian during the recall. Two declared not being comfortable with describing their diet to a third party, five reported having problems remembering the previous day's diet and six said that being available for the date of the interview represented a constraint.
Cost assessment
Cost of development of the overall NutriNet-Santé study website (and not simply the 24 h record interface) amounted to 150 000€ (US $213 000; 9 months of development). It included the entire study process: secured inscription system; development and administration of baseline questionnaires; monitoring of the cohort (emails to the subjects, newsletter, etc.); equipment and hosting. These costs depend on technical skills and resources of the research teams involved and on standards of the product (user-friendliness, security level, quality of graphic design, etc.). There is no supplementary cost for an additional day of 24 h record.
Additional cost related to the dietitian's interview (saved in the web-based method) mainly involved the salary of the dietitians (€21·2/subject, calculated on the basis of the cost of one full-time equivalent: €2080/month, including charges), telephone expenses (€0·9/subject) and cost of printing and sending the picture booklet (€16·04/subject). This corresponded to a total of €38·14 (US $51·84) for one subject and €19 070 000 (US $25 921 851) for a cohort of 500 000 subjects (i.e. the targeted enrolment number in the NutriNet-Santé study).
Discussion
In the present comparative study, we observed that agreement between the two methods was high for food (median ICC: 0·8 in men and 0·7 in women) and nutrient intakes (median energy-adjusted Pearson's correlation: 0·8 in both sexes), although these results should be interpreted with caution due to potential overestimation of correlations, as discussed below. The web-based version was generally well liked, and permitted considerable logistic simplification and cost savings.
The nineteen million euro additional cost represented by an interview-based assessment would make that method hardly feasible for large-scale studies. Moreover, in a web-based study, no supplementary cost is required to increase the number of 24 h records, an important feature for accurately apprehending intra-individual variability(Reference Thompson, Subar and Loria1, Reference Margetts and Nelson23). In the NutriNet-Santé study, subjects complete three non-consecutive 24 h records at baseline for inclusion in the cohort and three records per year during follow-up. Although online data collection may not solve traditional problems of reaching the targeted sample population(Reference Lefever, Dal and Matthiasdottir24), several studies have argued in favour of high acceptability and feasibility of web-based questionnaires in aetiological studies(Reference Bexelius, Honeth and Ekman25–Reference Etter and Perneger28). As reported previously, Internet access is constantly increasing throughout developed countries and across most sociodemographic and age groups(Reference Touvier, Mejean and Kesse-Guyot15). Thus, the advantage of using web-based self-administered tools is irrefutable, with the key question being the quality of collected data compared with standard methods.
Consistent with our findings, self-administered 24 h records or recalls using computerised tools seem to compare well with standard methods(Reference Ngo, Engelen and Molag3). Three studies compared a self-administered computerised (but not web-based) 24 h recall with a recall conducted by a trained interviewer (on the same day). One focused on teenagers and observed good agreement between the two methods (Spearman's correlations: ≥ 0·86 for nutrients and ≥ 0·72 for foods)(Reference Vereecken, Covents and Sichert-Hellert5). Another study, carried out among children (mean age 9·6 years), concluded that the self-administered tool was valid, although somewhat less accurate than a dietitian's interview in terms of match, intrusion and omission rates (compared with direct observation)(Reference Baranowski, Islam and Baranowski4). A final study included adults (18–65 years) and reported a mean energy-unadjusted correlation coefficient of 0·6 (range 0·29 (folate)–0·99 (alcohol))(Reference Zoellner, Anderson and Gould6). One study compared a self-administered web-based 24 h recall with a dietitian-conducted recall (on the same day) among children(Reference Hanning, Royall and Toews14); energy and carbohydrate intakes were slightly underestimated by the web-based recall, but all ICC were ≥ 0·5 (for energy and macronutrients).
It has been suggested that the quality of computer-based assessment may decrease with age(Reference Klovning, Sandvik and Hunskaar29) and increase with educational level and computer knowledge(Reference Edwards, Slattery and Murtaugh30). However, a study on reliability testing of a web dietary questionnaire concluded that the tool was feasible for self-administration involving older rural women(Reference Boeckner, Pullen and Walker31). We did not observe any statistical relationship between data consistency and education in the present study, probably due to our homogeneously well-educated population sample. In the present study of mature subjects, seven participants stopped because they felt the web system was too complex for them. In addition, a lower Pearson coefficient for nutrient intake was observed in older subjects. Correlations between the two methods were slightly higher in men. This may be partly explained by a higher proportion of computer ‘novice/inexperienced’ women than men (30 v. 13 %), and lower nutrient correlations in subjects with less computer knowledge. The higher proportion of men having a more rapid Internet connection than women (41 v. 24 %) may also have contributed to this finding, since a lower connection speed could decrease the user-friendliness of the web-based tool. However, this difference in the speed of Internet connections was not statistically significant. In the present study, agreement between the two methods was relatively high, even for older subjects and/or less computer-literate persons. In addition, a previous study had reported that computerised assessments bore witness to a learning effect, indicating that users became more familiar with the website with repeated use(Reference Probst, Faraji and Batterham32).
Advantages and limitations of 24 h records have been reviewed previously(Reference Thompson, Subar, Coulston and Boushey33); thus, the present discussion focuses on the specificity of the web-based self-administered method compared with an interview. A dietitian-based interview is commonly used as a ‘gold standard’ to evaluate the validity of self-administered tools. Indeed, the interview confers several advantages. The dietitian can limit omission of foods by appropriate probes, especially for often forgotten items such as fats and sauces. Also, the dietitian directly confers with the subject when a mistake in the declared quantities is suspected, thus avoiding errors in potentially difficult portion size estimates. For example, in the present study, the lower coefficient observed for dairy products in women was partly related to overestimation of milk intake associated with coffee, tea and breakfast cereals during the web-based method. Indeed, several subjects declared having consumed a full bowl of milk, whereas they only drank a third of it. Following this pre-test study, warning and help messages were added to the software to limit this problem. Finally, manual typing of foods (and thus subsequent reclassification workload) is limited with the interview. In order to decrease the researcher's burden associated with manually typed items, the search engine, which had not been highly visible in the initial interface version, has now been highlighted. An algorithm is currently being developed to automate reclassification into existing food items, and each newly created food improves the proposed list for future users.
In contrast, the dietitian's interview has some disadvantages compared with the web. First, judgement bias could potentially increase the risk of conscious or unconscious food omisson and under-estimation of portion sizes for sweet/fatty items (and in contrast, overestimation of healthy foods). In the present study, among women, the declared intake of cakes/biscuits/pastries was higher using the web-based method. Moreover, several subjects were uncomfortable with the idea of describing their diet to the dietitian. It has been suggested that limited contact with the interviewer may encourage reporting of more food items(Reference Probst, Faraji and Batterham32). However, a web-based study observed differences in dietary under-reporting by sex and weight status similar to differences observed during a dietitian's interview(Reference Vance, Woodruff and McCargar34). Second, our web-based record provides greater flexibility and freedom, allowing the subject to complete the record at any time via a user-friendly interface. In contrast, several subjects declared that being available for the date of the interview represented a constraint, and some were excluded because of the impossibility or refusal of contact by phone. Third, our web tool enables step-by-step recording of food intake during the day, avoiding memory bias. In contrast, several subjects complained of memory problems during the interview. Finally, an interview is subject to data entry mistakes (as in self-reported methods), but also inaccurate interpretation or misunderstanding by the interviewer, as well as increased between-interview variability (despite the use of a standardised data entry interface). Therefore, the self-administered web tool has both advantages and disadvantages compared with the interview method.
One limitation to the present study, typical of comparative surveys, is that completion of the first method (i.e. the record) may have influenced completion of the second (i.e. the recall) by reducing memory bias during the interview. This could result in potential overestimation of agreement between the two methods. Moreover, the objective of the present study was to compare the method of administration of dietary assessment and not to validate the web-based method using the dietitian's interview as a ‘gold standard’. In validation studies, measurement errors of two compared methods should be independent(Reference Gibson22), which was not the case in the present study, where the two methods compared were similar in several aspects (same underlying food composition table, etc.) and had a similar measurement error structure. In a FFQ validation study, Kipnis et al. (Reference Kipnis, Midthune and Freedman35) observed that the use of dietary report methods (multiple recalls, records or diaries) as reference instruments produced overestimation of the FFQ correlation with true usual intake, assessed by biomarkers. Thus, correlation coefficients observed in the present study should not be interpreted as correlations between the web-based assessment and true intake, which are probably lower. Another limitation was that our participants belonged to middle/high education and occupational categories. Thus, caution is required when extrapolating the present results to the general population. One other limitation involved distribution of weekdays and weekend days (the latter were over-represented), which had been determined for convenience purposes in the present comparative study to facilitate organisation of the dietitian's time schedule and the subject's availability. It does not reflect the true distribution of weekdays nor the assessment algorithm in the NutriNet-Santé cohort. Although there is no reason to believe that better distribution of days of the week would substantially change the present results, this parameter should be taken into account and evaluated in future studies. Finally, although unannounced dietary assessments would avoid potential bias related to modification of the diet by the participants(Reference Thompson, Subar, Coulston and Boushey33), 24 h dietary records (and not recalls) are performed in the NutriNet-Santé study based on randomly selected dates that are announced to the subjects. Indeed, unannounced recalls would require the possibility of being constantly in contact with subjects, requiring them to check their emails daily, thereby introducing a strong constraint in participation and increasing the potential for selection bias by recruiting only intensive Internet users.
In conclusion, our objective was to compare, for a given day, dietary intake assessed by a web-based self-administered 24 h dietary record and by a traditional dietitian's interview. The present study showed that agreement between the two methods was good, and that the web-based tool had substantial logistic and cost advantages. In large-scale epidemiological studies, this web-based 24 h record should be used repeatedly (several records per year) in association with a food frequency or propensity questionnaire in order to obtain a complete overview of usual dietary intake. In this perspective, studies assessing the validity (notably v. biomarkers) and reproducibility of the entire process of usual dietary intake assessment will provide further information. Online dietary surveys are able to access wide and diversified populations and achieve quick returns. Their use should increase worldwide, with appropriate adaptation of web-based tools to country-specific cultural aspects(Reference Schatzkin, Subar and Moore7). This will advance our understanding of the relationship between nutrition and health outcome. The web also promises a new generation of tools for dietary improvement, including nutritional education and counselling(Reference Lewis and Burton-Freeman36, Reference Papadaki and Scott37).
Acknowledgements
The present study was supported by the Ministère de la Santé (DGS, French Ministry of Health), the Institut de Veille Sanitaire, the Institut National de Prévention et d'Education pour la Santé, the Fondation pour la Recherche Médicale, the Institut National de la Santé et de la Recherche Médicale, the Institut de Recherche en Santé Publique, the Institut National de la Recherche Agronomique, the Conservatoire National des Arts et Métiers and Paris 13 University. None of the authors had a conflict of interest. M. T. designed the study, supervised the analysis, interpreted the data and wrote the manuscript. E. K.-G., C. M. and K. C. contributed to designing the study, interpreting the results and drafting the manuscript. C. P. and A. M. coordinated data collection and contributed to interpreting the results and editing the manuscript. S. H. designed and coordinated the NutriNet-Santé cohort and supervised the study. All authors reviewed the manuscript. We thank the scientists, dietitians, technicians and assistants who helped to carry out NutriNet-Santé test studies, and all dedicated and conscientious volunteers. We especially thank Soizic Gueho (data manager) for performing the statistical analysis; Alexandra Lisi, Anahita Mohtadji, Dorothée Houet, Mathilde Lanotte, Emilie Tisseron, Florence Charpentier and Laura Houaslet (dietitians) for conducting telephone interviews and completing the food composition table, and Gwenael Monot (computer scientist), who is coordinating the computer aspects of the NutriNet-Santé study. We thank Voluntis (a healthcare software company) and MXS (a software company specialising in dietary assessment tools) for developing the NutriNet-Santé web-based interface according to our guidelines. We are grateful to Jerri Bram for English editing of the manuscript.
