In 1997, the Ministry of Agriculture, Fisheries and Food (the predecessor of the Food Standards Agency (FSA)) commissioned a critical review of dietary assessment methods for use in people living on low income, published as a ‘Scoping study’ in 1998(Reference Dowler, Draper, Nelson, Thomas and Dobson1). The report critically reviewed the applicability, feasibility and cost of seven different dietary assessment methods in terms of their appropriateness for a dietary survey of people living on low income in the UK. The dietary assessment methods reviewed were the weighed inventory, the household measures technique, the semi-weighed method(Reference Nelson and Nettleton2), the food checklist, the diet history, the 24 h recall and the FFQ.
Methodological strengths and weaknesses were considered and each method was rated using a simple ranking system. Each method was assigned a rank (+1, 0, −1) in relation to sixteen factors believed by the authors to be important for consideration in low-income households(Reference Dowler, Draper, Nelson, Thomas and Dobson1). The authors assigned ranks based on their own experience and reading of the literature. The ranks were assigned as follows: +1 if there was deemed to be a distinct positive advantage relating to the use of the method in low-income households; 0 if the method was deemed to have no distinct advantage or disadvantage; −1 if there was deemed to be a distinct disadvantage relating to the use of the method in low-income households. The individual ranks were then summed to create a score for each method (‘Total 1’), as published in the original Scoping study, shown in Table 1. The scoring was intended to capture a cumulative impression of the relative merits of each method for use in low-income households, rather than a strictly quantitative assessment.
Using the scoring system and an evaluation of the strengths and weaknesses of the methods, the choice of methods suitable for a comparative survey of diet methods in low-income households was narrowed down to three. The authors of the Scoping paper recommended to the FSA that the repeat 24 h recall (score +2), the semi-weighed method (score +1) and the food checklist (score +1) be tested against the reference method of a weighed inventory. The 24 h recall scored highly on issues relating to dietary variation, literacy, numeracy, language and respondent burden. Consideration was given to potential problems with this method such as difficulties making comparisons with previous National Diet and Nutrition Surveys (NDNS)(Reference Gregory, Collins, Davies, Hughes and Clarke3–Reference Finch, Doyle, Lowe, Bates, Prentice, Smithers and Clarke8) and possible ‘social desirability bias’.Footnote † The semi-weighed method was considered to be almost as precise as the weighed inventory, and would potentially facilitate more direct comparisons with previous NDNS surveys, although respondent burden would be high. The semi-weighed method also enabled all household members to record their food intake, addressing issues of within-household distribution. The food checklist was recommended by the authors as it was deemed to have a low respondent burden compared with other methods. The weighed inventory, although it did not score highly and showed distinct disadvantages with problems relating to language, literacy and numeracy, was recommended as a reference method to provide information on the relative validity of the test methods and for comparisons with previous NDNS.
These four methods were used in the subsequent method comparison study – the Low Income Diet Methods Study (LIDMS), the study design and initial findings of which have been reported elsewhere(Reference Nelson, Dick, Holmes, Thomas and Dowler9, Reference Holmes, Dick and Nelson10). Briefly, LIDMS was carried out in 2001 and involved a cross-sectional design, comparing three 4 d dietary survey methods (multiple-pass 24 h recall, food checklist and semi-weighed method) against reference measures in respondents living on a low income in London. Reference measures included a 4 d weighed inventory validated with four 24 h urine collections and comparisons of estimated energy intake with estimates of energy requirements based on BMR calculations and reported physical activity levels.
In most cases respondents completed three 24 h recalls face-to-face and one over the telephone. Analyses for the 24 h recall have been undertaken as one method, since only small differences were found between the face-to-face and telephone interviews(Reference Nelson, Dick, Holmes, Thomas and Dowler9). The study aimed to: (i) compare the validity and acceptability of three dietary survey methods against appropriate reference measures; and (ii) identify a method which was both valid and acceptable in low-income households. Low-income households were selected using a doorstep screening questionnaire. Six trained interviewers and two researchers (also working as interviewers) collected the data over a one-year period. Results were based on 384 respondents (159 males, 225 females) aged 2–90 years in 240 households. Forty-eight per cent of respondents described themselves as white, 31 % as black or black British and 9 % as Asian or Asian British. The remainder were of mixed ethnicity or from other ethnic groups.
The present discussion paper updates and extends the method comparisons carried out in the 1998 Scoping study review. The paper also sets out, for the first time, a comprehensive set of factors for systematic evaluation of the relative merits of approaches to dietary assessment in low-income households in developed economies.
Methods
A re-assessment of the original evaluation of the advantages and disadvantages of the four dietary methods used in LIDMS was undertaken by the authors. The key elements of the re-assessment that informed the authors’ discussions and formed a basis for their decision on the score for each factor included:
1. Data collection, analysis and comparison of methods, e.g. levels of under-reporting.
2. Systematic evaluations of the methods reported by the respondents, e.g. respondent burden and interference with lifestyle.
3. Systematic evaluations of the methods reported by interviewers and researchers, e.g. literacy, numeracy and language problems.
4. Regular structured discussions between researchers and interviewers throughout the fieldwork year and a formal interviewer debrief at the end of the year covering elements directly relating to fieldwork, e.g. the need to make repeated visits.
5. Regular structured discussions between the researchers and dietary data coders, e.g. quality of portion size information.
6. A re-assessment by the authors (including one of the two lead LIDMS researchers and the LIDMS research director) based on a structured review of the elements above plus a consideration of elements assessed across all methods, e.g. the quality and consistency of recording food eaten away from home.
The scoring system for the four methods was similar to that used in the Scoping study. Where ranks assigned were variable, this was indicated using the symbol ↔. The principles by which the ranks were adjusted were based on the elements as described above and the experience gained from LIDMS. Ranks were either increased or decreased (as appropriate) from the original score, usually by 1 point only. Some ranks remained unchanged where it was felt the original rank still reflected appropriate ratings for the method. The rationale for changes in ranks as a result of the present review is given in the Results section. Scores for each method were recalculated as the sum of the revised ranks.
In addition to the review of the original list of factors considered in the Scoping study, further methodological evaluative outcomes from the LIDMS study were introduced in the form of ten new factors not previously considered(Reference Holmes11). Methodological strengths and weaknesses were considered against these new factors and each method was ranked using the same system as that used in the Scoping study. The ranks for the new factors were allocated by the authors considering a number of elements from the LIDMS survey, including those used in the re-assessment listed above.
Results
Total 1 (shown in Tables 1 and 2) presents the ranks for the methods as rated in the Scoping study. Table 2 gives the updated ranks for each of the original factors including the direction of change. Total 2 in Table 2 gives the updated rank for each method. Reasons for changes in the ranks are given below. Overall, twenty-two updates were made to the ranks allocated in the Scoping study: 24 h recall (seven changes); food checklist (six changes); weighed inventory (five changes); semi-weighed method (four changes).
ScoresFootnote * allocated in the Scoping study for the comparison of dietary assessment methods specifically in relation to low-income households
NDNS, National Diet and Nutrition Survey.
* Scores: +1, distinct advantage for the method; 0, neutral, no advantage or disadvantage; −1, distinct disadvantage for the method.
† Source: adapted from reference 1. Order of factors and some headings appear slightly differently.
Direction of changeFootnote * and new scoresFootnote † for the comparison of dietary assessment methods specifically in relation to low-income households
NDNS, National Diet and Nutrition Survey.
* Direction of change: ↑ increase; ↓ decrease; – no change.
† New scores: +1, distinct advantage for the method; 0, neutral, no advantage or disadvantage; −1, distinct disadvantage for the method; ↔, variable.
‡Order of factors and some headings appear slightly differently to those presented in reference 1.
Update in ranks for the factors assessed in the Scoping study
24 h recall
Ranks for the 24 h recall were increased more often than for any other method (factors 5, 6, 11 and 16). Analyses showed that the method yielded the fewest number of low energy reporters across age and gender groups (15·6 % overall)(Reference Nelson, Dick, Holmes, Thomas and Dowler9).Footnote * The method was also rated as the least interfering with lifestyle compared with the other methods. The quality of the portion size information varied depending on the age and capabilities of the respondent.
Ranks for factors 1, 4 and 8 were decreased for the 24 h recall. Respondent’s poor memory was cited by interviewers to be the most common problem with the 24 h recall (44 % of problems), especially among older respondents (64 %), and for this reason the rank for reliable data for children and the elderly was decreased. Further decreases were made after interviewers reported that using the photographic food atlas(Reference Nelson, Atkinson and Meyer14) during the telephone 24 h recalls sometimes highlighted numeracy difficulties.Footnote † Additionally, the atlas contained only a limited selection of ethnic foods, therefore reducing the accuracy of the portion size estimation for these foods.
Food checklist
The greatest number of decreases in ranks occurred for the food checklist (factors 1, 3, 6, 8 and 15). Factor 14 – evenness of recording food eaten away from home – was the only factor for which the rank increased. Difficulties in reporting portion sizes and large amounts of reading were commonly reported problems, which resulted in less reliable data for children and the elderly. Feedback from interviewers indicated that although the checklist did not involve much writing, it required lots of reading (in order to find the item on the pre-printed list), which then highlighted problems with literacy.
Analysis showed that the food checklist yielded the greatest number of under-reporters (joint with the semi-weighed method, both 25 %)(Reference Nelson, Dick, Holmes, Thomas and Dowler9). This may have been linked with factors 6, quality of information on portion size, since only a standard portion was indicated on the list. Although the pre-printed list incorporated some ethnic foods, the range proved to be insufficient to reflect the diversity of eating habits in this population. Often this meant that respondents had to list ethnic foods at the end of the checklist in the space for ‘other foods’. Place of consumption was not recorded, so a separate analysis of food consumed away from home was not possible.
Semi-weighed method
No ranks were increased for the semi-weighed method with the exception of factor 16 (database needed to complete). Since the same database was used across all methods all ranks were brought to neutral. Ranks were decreased for three factors (6, 11 and 12). Portion size data were not as good as anticipated due to a lack of clarity regarding distribution of recipe-based foods to all household members. Experience from LIDMS also suggested that the method caused a much greater than anticipated interference with lifestyle. A particular advantage of the method is the efficiency of recording for the whole family if everyone is eating the same food at the same time. This was frequently the case in the 1970s when the method was originally conceived but this advantage is lost if members of the family eat different foods or eat the same foods but at different times, which was often the case in the present study. Comparability with the NDNS was lower than for the weighed inventory but greater than for the 24 h recall and food checklist and the rank was adjusted to reflect this.
Weighed inventory
Increases in rank for factors 5 and 16 were made to the weighed inventory while decreases were made to factors 6, 11 and 14. The level of under-reporting for the weighed inventory was lower than for the food checklist and semi-weighed methods but higher than that for the 24 h recall (21·6 %)(Reference Nelson, Dick, Holmes, Thomas and Dowler9, Reference Black12, Reference Goldberg, Black, Jebb, Cole, Murgatroyd, Coward and Prentice13). This is likely to have been impacted by factors 6 and 11: the quality of the portion size data collected which varied considerably depending on the age and capabilities of the respondent and the interference with lifestyle, which was very high with this method. The weighed inventory was the only method for which foods consumed away from the home were not recorded consistently (in the majority of cases household measures were used instead of weights).
New factors for the assessment of dietary assessment methods in low-income households
The ranks for ten new factors introduced into the method assessment are shown in Table 3. Reasons for their inclusion and the ranks assigned are given below.
New factors and allocated scoresFootnote * for the comparison of dietary assessment methods specifically in relation to low-income households
NDNS, National Diet and Nutrition Survey.
* Scores: +1, distinct advantage for the method; 0, neutral, no advantage or disadvantage; −1, distinct disadvantage for the method; ↔, variable.
† Based on average total time per household for a two-respondent household.
1. Problems with completion of the method
Each method included an interviewer feedback questionnaire on which interviewers specified if the respondent experienced any difficulties.Footnote * Interviewers identified the fewest number of difficulties with the 24 h recall (16 %), followed by the food checklist (30 %), the weighed inventory (51 %) and the semi-weighed method (59 %).
2. Level of detail collected
The level of detail collected for the food checklist was poor in comparison with the other methods, while detail varied for the semi-weighed method and weighed inventory depending on how well respondents kept the diaries. On some occasions the person who did the cooking was not a respondent and did not wish to contribute to the study, and this had an adverse effect on detail recorded. Additionally, for the semi-weighed method, the level of detail varied depending on the number of people in the household (typically, the more people in the household, the less detail). Since the interviewers collected the information for the 24 h recall, the detail was more likely to be at the level required for adequate coding of foodstuffs.
3. Difficulty in assessment of portion size for respondents
While the weighed inventory has the ability to obtain good-quality portion size information (see Table 2, factor 6), this is generally only the case if the method is completed well. Both respondents and interviewers identified using the scales or weighing food as the most common problem with the weighed inventory. The assessment of portion size using the photographic food atlas(Reference Nelson, Atkinson and Meyer14) in the 24 h recall was the least difficult for respondents.
4. Disruption caused by missed appointments
Low-income households suffer from higher levels of domestic chaos and stress compared with other households(Reference Dowler, Draper, Nelson, Thomas and Dobson1, Reference Nelson, Dick, Holmes, Thomas and Dowler9), and as a result many respondents failed to keep appointments. This caused little disruption for the 24 h recall where an appointment could easily be rescheduled. For the other methods, however, checking visits had been scheduled at specific stages of the data collection. The disruption was greatest for the semi-weighed method because missing the final visit rendered useless the value of weighing communally consumed foods,Footnote * since the weights no longer corresponded to the amounts recorded on the days in question(Reference Holmes, Dick and Nelson10).
5. Acceptance of interrogation
Although probing respondents was a common part of all of the methods, probing for foods missed off the food checklist proved difficult because time of consumption is not recorded and interviewers go through the items in the order they appear on the list. Respondents who felt they had completed the food checklist adequately disliked being probed for further information and, in some cases, this led to tension(Reference Nelson, Dick, Holmes, Thomas and Dowler9). For the semi-weighed method and weighed inventory, interrogation sometimes created tension when respondents felt the quality of their recording was being questioned, while interrogation and probing in the 24 h recall were accepted as part of the method.
6. Time taken for interviewer visits
In total, over the four days, the food checklist took the interviewers the least time to complete, followed by the 24 h recall, semi-weighed method and weighed inventory(Reference Nelson, Dick, Holmes, Thomas and Dowler9). The average amount of time per visit was marginally higher for the semi-weighed method than for the weighed inventory, but the total time per respondent was higher for the weighed inventory, reflecting the fact that four visits were made for the weighed inventory while only three were made for the semi-weighed method.
7. Interviewer opinion of method accuracy
At the end of the study, interviewers were asked a series of questions about the methods and their responses are reflected in factors 7 and 8. Three-quarters of interviewers rated the 24 h recall as the most accurate method, with one-quarter rating the weighed inventory as the most accurate method. All of the interviewers rated the semi-weighed method as the least accurate. These results are further supported by the assessments that interviewers made at the end of each day (for the 24 h recall) or the end of the method (food checklist, semi-weighed method and weighed inventory) on how good they felt the method was at reflecting what the respondent had eaten over the period of food recording. On balance, the 24 h recall was felt best to reflect respondents’ consumption (only 2·4 % of records were classified by interviewers as ‘poor’) compared with 6 % for the food checklist, 9 % for the weighed inventory and 17 % for the semi-weighed method.
8. Interviewer burden
The majority of the interviewers (75 %) found that the food checklist was the least burdensome of the four methods, due in part to the lower level of detail collected compared with the other methods. Eighty-eight per cent of interviewers rated the semi-weighed method as either the most burdensome method or the next most burdensome method. The level of interviewer burden was comparable for the 24 h recall and the weighed inventory, lower than for the semi-weighed method but greater than for the food checklist.
9. Respondent method preference
When asked at the end of the study which of the methods they had preferred, respondents showed a clear preference for the food checklist (46 %), followed by the 24 h recall (29 %), weighed inventory (21 %) and finally the semi-weighed method (1 %). Three per cent of respondents did not know which method they preferred or felt there was no difference(Reference Nelson, Dick, Holmes, Thomas and Dowler9, Reference Holmes, Dick and Nelson10). Respondents preferred the food checklist because it was easier than the other methods (49 %) and respondents did not have to do the recording themselves (as it was already written down for them; 26 %).
10. Household burden
The method that was the most interfering with the greatest burden for other people in the household (not those recruited into the study itself) was the semi-weighed method. This method required co-operation of all household members and visitors for the recording of food and drink consumed at home.Footnote † Often other household members refused to co-operate in the study and only limited information could be collected. The collection of information for other people in the household was, in some instances, a distraction for the two main respondents and also for the interviewers when they returned to check the diary.
In summary, Total 3 in Table 3 gives the score for the ten new factors included in the method assessment. Total 4 in Table 3 presents the final score for each method, taking into account the updated ranks from the factors assessed in the Scoping study (Total 2, Table 2) and the ten new factors (Total 3, Table 3).
Discussion
The Scoping study was originally designed to critically review different dietary assessment methods in relation to specific issues associated with the ability of respondents in low-income households to complete the methods. The results of the Scoping study informed decisions regarding the methodology to be used in LIDMS, which in turn informed decisions for the methodology of the national Low Income Diet and Nutrition Survey (LIDNS)(Reference Nelson, Erens, Bates, Church and Boshier15). The methodological evaluations made in the Scoping study in 1998 have now been updated based on more recent findings and systematic assessments from LIDMS and extensive structured discussions between investigators, interviewers and respondents. Of the sixteen original factors, only five remained unchanged in their ranks.
As a result of the re-assessment of the methods and the addition of the new factors, there has been a substantial change in the total scores for all four of the methods. Total 4 (Table 3), based on revised scores for the original factors plus scores for the new factors, showed that the final preferred order for use of the four methods in low-income households was 24 h recall, weighed inventory and food checklist (equal scores), and finally the semi-weighed method. The re-evaluation suggested more strongly than in the Scoping study that the 24 h recall is the method most suitable for assessing diet in low-income households.
The approach taken to the re-evaluation introduced a number of improvements on that taken in the Scoping study. First, questions were asked systematically of both interviewers and respondents regarding the factors that made the methods acceptable or unacceptable from their perspectives. They were also asked to provide purely subjective judgements regarding the extent to which they believed that the different methods reflected actual dietary behaviours. Second, quantitative information (e.g. on the time taken to carry out interviews, the numbers of missed appointments for different methods) provided a clear understanding of the ways in which the fieldwork protocol was applied and adhered to. Finally, regular structured discussions with the interviewers provided a solid basis for the comprehensive collection of information about the issues that arose during data collection.
The process of re-evaluation of the methods had a number of weaknesses. First, for each factor, the original ranking in the Scoping study (−1, 0, +1) provided only a crude placement of each of the methods in relation to one another. This was appropriate simply because there was no systematic overview of the relative strengths and weaknesses of the different methods being considered. Many of the judgements in the Scoping study had to be made by comparing reports from the literature based on very different types of comparisons between methods, and using the authors’ cumulative experience of fieldwork. Because the intention of the present discussion paper was to re-evaluate the original scoring from the Scoping study as well as to include additional factors that had not been considered before, it was necessary to retain the same approach. A systematic basis for a more detailed ranking (e.g. 1 ‘worst’ to 4 ‘best’ with clear definitions for each rank for each factor) could have been constructed, but the assignment of ranks would still, in many instances, have been based on subjective judgements. In the final analysis, the separation between the virtues of the 24 h recall and the weaknesses of the semi-weighed method was unequivocal, and it is unlikely that a more subtle evaluation would have yielded a different conclusion. It may, however, have yielded a better resolution of the balance of relative merits of the food checklist compared with the weighed inventory.
Second, the domains evaluated were to some extent arbitrary. They reflect the real life experience of interviewing and collecting dietary data. A more systematic approach to their definition might have yielded a different categorization of factors. Given the experience of the fieldworkers and study investigators, the many structured discussions that occurred regularly throughout the study and the extensive debriefing, however, it seems likely that the present findings captured the key aspects of the relative merits of the methods when used in low-income households. It is difficult to see how a different categorization would have yielded results materially different from those presented.
Third, all the factors were given equal weight when computing the scores since there seemed little justification or basis for assigning different weights. The purpose of the present discussion paper was to highlight the relative merits of different approaches to dietary assessment and to consider them in a comprehensive framework, rather than to produce a definitive score per se.
Finally, the work in LIDMS was carried out in London. The question then arises as to how generalizable the findings are to other low-income households in England, the UK or other developed economies. From a methodological perspective and from extensive fieldwork experience, it is likely that the types of problems encountered in low-income families in London were the same as those likely to be encountered in other parts of the country. Cultural differences in the ways in which families reported consumption, be it using the 24 h recall or the semi-weighed method, and issues of literacy and numeracy, may have been presented more commonly in the present sample because of the higher proportion of minority ethnic group households in London compared with other parts of the country. Overall, however, the balance of the problems reported and the relative merits of the methods did not differ appreciably between ethnic groups. Moreover, not all of the issues raised related exclusively to low-income households, but are inherent in relation to dietary assessment generally and were common to all groups within the study.
The present paper provides a comprehensive framework within which to evaluate the relative merits of different approaches to dietary assessment in different population groups. We are not aware of any similar evaluations in the literature. The approach taken here is clearly amenable to refinement, and it is hoped that future assessments of this type can usefully build on the framework and approaches described.
Acknowledgements
Source of funding: The Low Income Diet Methods Study (LIDMS) was funded by the Food Standards Agency. Conflict of interest declaration: The authors declare no conflicts of interests with respect to this research. Authorship responsibilities: B.H. carried out data collection, analysis and report writing for LIDMS and drafted the manuscript. M.N. was involved in the design, analysis, report writing for LIDMS and commented on the manuscript. Acknowledgements: The authors would like to acknowledge Katie Dick for her involvement in all aspects of the study; principal investigators Elizabeth Dowler and Roger Thomas for contributing to the progress of the study throughout; the interviewers – Rosa Bonini, Angela Costetsos, Edith Matovu, Paul McGee, Martina Philips and Caireen Roberts – who, together with B.H. and Katie Dick, gathered information from January to December 2001; Jo Francis for her administrative support; Sian Burr for her work as a reserve interviewer and food coder; the team who entered data and coded food records, in particular Oliver Redfern; Roy Sherwood for carrying out the urine analyses; members of the steering group for their advice and guidance; the FSA for funding the study; and the survey participants, without whose energy and commitment the study could not have taken place.
