Accurate and contemporary data on dietary intakes are essential for developing and evaluating evidence-based policies and programmes to improve nutrition, reduce obesity, address food insecurity and ensure food safety. In New Zealand (NZ), population diets have typically been monitored through national nutrition surveys that collect data on food and nutrient intake to assess the nutritional status of the population(1). However, a national nutrition survey has not been undertaken in NZ since 2002 for children (aged 5–14 years)(2) and 2008/2009 for adults (aged 15 years and over)(3). Thus, a national nutrition survey is warranted to collect up-to-date population-level dietary intake data to effectively address current nutritional challenges and guide future policy initiatives.
In 2021, two NZ government agencies (the Ministry of Health and the Ministry for Primary Industries) embarked on plans for a future national nutrition survey, including the design of suitable dietary assessment questionnaires(4). As NZ’s food environment and population have changed significantly since the previous national nutrition surveys, some existing survey questions were deemed likely to be outdated and potentially unsuitable for assessing contemporary dietary behaviours and household food security. For instance, NZ’s Dietary Habits Questionnaire was developed for the 2008/2009 NZ Adult Nutrition Survey and subsequently updated and cognitively tested for the NZ Health Survey in 2018/2019. The NZ Food Security Questionnaire, designed to capture a household’s access to nutritionally adequate, safe and culturally appropriate foods, was developed for the 1997 National Nutrition Survey and has not been updated since(Reference Parnell, Reid and Wilson5). In 2022, a thorough review of national and international dietary habits and food security questionnaires was undertaken to identify relevant questions to assess contemporary nutrition issues(Reference Ni Mhurchu, Te Morenga and Mackay6). This process led to the development of a draft dietary assessment questionnaire including new and amended questions, which had not yet been tested with the NZ population. Therefore, these questions required pretesting to explore how they were interpreted and whether further adjustments were needed.
To achieve this, cognitive testing methods were employed. Cognitive testing has been commonly used to design, evaluate and refine survey instruments(Reference Wolcott and Lobczowski7). It is used to gain insight into the cognitive processes used by respondents when answering survey questions with the goal of identifying problems with comprehension, recall and decision and answering processes(Reference Willis8). Cognitive interviews can help identify if such issues occur, the source of the error and how a question can be improved to achieve its purpose(Reference Collins9). This study used cognitive interviewing techniques to pretest dietary habits and food security questions in a sample of NZ children and adults from various ethnic and socio-economic backgrounds, with the aim of understanding if the questions were interpreted as intended. In addition, expert consultation was used subsequently to inform wording changes needed to improve participants’ comprehension of the questions.
Methods
Study design
Cognitive interviews were used to investigate respondents’ understanding of a selection of dietary habits and food security questions. Participants were asked to verbally express their thoughts and interpretations of questions that were either new, adapted or deemed unlikely to apply to today’s food environment. Findings were used to identify problematic aspects of the questions (e.g. difficult terminology, unhelpful examples or confusing instructions) to guide revision and subsequently improve the clarity and accuracy of those questions.
Selection of questions for cognitive testing
Work was undertaken to determine appropriate questions to include in a future NZ nutrition survey (further detailed elsewhere)(Reference Ni Mhurchu, Te Morenga and Mackay6). This selection process was guided by a Technical Advisory Group (consisting of experts in human nutrition, survey design and dietary assessment), a Māori Advisory Group (including experts in Māori health and nutrition, dietary assessment and food security) and representatives from government agencies (contract funders). Together, these experts ensured the questions were appropriate for a national nutrition survey and relevant to NZ’s indigenous population (Māori). Dietary habits and food security questions from previous NZ and international questionnaires were presented to these experts, who were asked to determine if the question should be included in the future nutrition survey (i.e. suit the purpose of a national survey), to suggest any revisions (e.g. wording, order, instructions) and to propose additional questions (e.g. to cover contemporary dietary issues and current priorities). Out of the total question item pool identified, fifty-four questions were selected for inclusion in a nutrition survey. A further twelve questions required cognitive testing as they were newly developed (n 3), amended from the original (n 5) or identified to no longer be reflective of the current food environment or concepts (n 4). The latter was particularly relevant to the food security questions, which were developed over 25 years ago. In recent times, the concept of food security has evolved and may therefore not be accurately captured by the previously used questions. Online supplementary material, Supplemental Table 1 summarises the original questionnaire items, the questions tested in the cognitive interview (if amended) and a rationale for their inclusion and/or amendments. An illustration of the question selection process and cognitive testing approach is presented in Fig. 1.
Nutrition survey question selection process and cognitive testing approach. TAG, Technical Advisory Group; MAG, Māori Advisory Group.
Study participants, recruitment and informed consent
Eligible study participants were (1) aged 11 years or older, (2) not working in or with a qualification in the field of nutrition and (3) willing to participate in a one-on-one or group interview. Although the nutrition survey will include participants aged two years and older, the recruitment age of 11 years was chosen to assess the cognitive abilities of participants who would complete the survey themselves. Parents of younger children (aged 2–10 years) complete the survey on behalf of their children, meaning that cognitive testing was not necessary for those aged under 11 years. Purposive sampling was used to recruit participants from various ethnic and educational backgrounds to ensure the questions were tested with a sample likely to include diverse perspectives and cognitive abilities. A minimum sample size of 56 was targeted in an effort to reflect NZ’s four major ethnic groups (Māori, Pacific, Asian and NZ European/Other (NZEO))(Reference Stats10) and three key age categories (11–17 years, 18–64 years and 65+ years). A minimum of four participants per age group within each ethnic group was targeted, resulting in a base sample of forty-eight participants. A higher recruitment target was set for the Asian group to account for the broader ethnic diversity within this category. This sampling strategy was designed to capture potential differences in the interpretation of words and concepts, cognitive processing of questions and cultural eating practices(Reference Wolfe, Frongillo and Acassano11–Reference Miller, Chepp and Willson13).
Participants were recruited using two methods, with different approaches used for specific ethnic groups. Drawing on the researchers’ long-standing networks and connections, Pacific and Asian participants were recruited through community organisations: The Fono and The Asian Network (TANI), respectively. Māori and NZEO participants were recruited through advertisements posted on local community social media groups (e.g. Facebook). People who expressed interest in the study were provided information in the form of either a child or adult participant information sheet, depending on their age. Once they agreed to participate, participants were scheduled for cognitive interviews. Adult participants were required to sign a consent form, while children provided their assent alongside parental or caregiver consent prior to taking part in the study.
Data collection
Options for different settings for the cognitive interviews (e.g. individual v. group; in-person v. online) were provided. When scheduling the cognitive interviews, participants were given the option of participating in either a one-on-one or group interview. The procedures for both formats were consistent, other than some minor differences detailed below. Those who opted for a one-on-one interview could complete the interview in person or using an online video-calling platform (Zoom), while group interviews were conducted exclusively in person. All in-person interviews were held in participants’ homes or a community hall. Participants recruited by the community organisation partners participated in interviews that were facilitated by a trained member of that organisation, with a member(s) of the research team present where appropriate. At the start of the interview and in the consent form, participants were made aware that confidentiality could not be fully guaranteed due to the group setting and possible facilitation of interviews by a local community organisation. Participants recruited directly through local community social media groups participated in interviews facilitated by a member of the research team.
At the start of the interviews, socio-demographic information was collected for each participant. Socio-demographic data included age group (11–17; 18–64; 65+ years old), gender (male; female; other), ethnicity (one or more of the following options: Māori; Samoan; Cook Island Māori; Tongan; Niuean; Chinese; Indian; NZ European; other), education level (none; primary/intermediate school; secondary school; diploma, certificate or trade; undergraduate degree; postgraduate degree; other) and financial security (not at all – I do not have enough money to meet my basic living costs; borderline – I am just getting by financially; secure – I have money left over at the end of the month)(Reference Gerritsen, Rosin and Te Morenga14).
Eight dietary habits questions and four food security questions were cognitively tested. These twelve questions covered fruit and bread intake, exclusion of specific food groups, salt use, school food programmes, food procurement, dietary supplement use, access to basic foods, reliance on different sources of food assistance and food preparation and storage facilities. The questions were available in English only, as the eventual NZ nutrition survey, similar to other national health surveys, will not be translated into other languages. Some questions were deemed only appropriate for respondents aged 18 years or older, and therefore, participants aged 11–17 years were asked six out of twelve questions. Questions not asked of younger participants included the four food security questions, the food procurement question and another on food preparation and storage facilities. These questions required participants to answer on behalf of their household, with adult respondents considered more appropriate for this task given their role within the household and greater involvement in food preparation or purchasing activities. Participants were presented with showcards(15), including the questions, answer options and images where appropriate (e.g. examples of the foods that the question referred to or typical serving sizes – see online supplementary material, Supplemental File 1). All questions and answer options tested are listed in online supplementary material, Supplemental Table 1. Although answers to questions were recorded, the focus of this study was participants’ interpretation of the questions and how they came to an answer. Participants’ answers were therefore not analysed.
The interview process was outlined to participants at the beginning of the session. A combination of the think-aloud and the verbal probing techniques was used to elicit responses and gain insight into participants’ cognitive processes(Reference Willis8,Reference Collins9) . While participants in individual interviews read aloud each question and then answered them, questions were read aloud by the interviewer in group interviews, where participants then took turns in answering if comfortable doing so. Participants were reminded to verbalise their thoughts as they worked through answering the question. Following each question, the participants were asked a series of additional probing questions (both pre-defined and spontaneous) if they were unable to adequately articulate their thought processes. Examples of probing questions used include ‘You picked [answer], how did you come up with that?’, ‘In your own words, what do you think the question is asking?’ and ‘How would you word this differently?’. In addition, participants were asked to provide feedback on the showcards (e.g. ‘What are your thoughts on this showcard?’).
Each session lasted 0·5–1·5 h and was audio-recorded. Participants received a gift voucher for their participation in the study. All interviews were conducted between September and December 2022.
Transcription, analysis and question refinement
Socio-demographic data were summarised using descriptive statistics including percentages for categorical variables and mean (sd) for continuous variables (Excel, version 2308, Microsoft Corporation). All audio recordings were transcribed verbatim by one of five members of the research team. One researcher (BF) reviewed all interview transcripts, and participants’ responses were grouped by test question. The grouped responses were summarised (BF) using short descriptive codes that captured key ideas or meanings. Further inductive analysis was conducted by two researchers (BF and CH) who reviewed, refined and grouped the initial codes(Reference Miller, Chepp and Willson13). Themes (i.e. issues that occurred repeatedly) were identified by comparison and grouping of the coded data for each question across all interviews. In addition, responses that were particularly insightful, regardless of their frequency (i.e. discoveries), were captured(Reference Willis16). Coding ambiguities and findings were reviewed and discussed in consultation with a third researcher (SM) to ensure consistency between coders. To illustrate and support each theme, relevant quotes from participants were selected. Findings were not compared by age or ethnic group because the numbers in each subgroup were small. Following the analysis, findings were presented to and discussed with government representatives, the Technical Advisory Group and the Māori Advisory Group to decide on improvements to questions.
Results
Study participants
Sixty-eight study participants took part in either group interviews (n 16 interviews, each involving between two and fifteen participants) or in one-on-one interviews (n 14). Socio-demographic characteristics are outlined in Table 1 for the total sample and by age group. The sample was predominantly female (n 48) with approximately equal representation by key ethnic groups, education level and financial security.
Socio-demographic characteristics of study participants
* Excluding participants aged < 18 years.
Although the questions tested were generally well understood, some aspects were identified that impacted participants’ interpretation of questions and subsequently their answers. These were (1) interpretation of ambiguous terms, (2) poor understanding of dietary or technical terms and (3) difficulty following complex or unclear instructions. Below, each theme is described more fully, and examples are provided. Table 2 details the questions presented for cognitive testing, the issues identified from the thematic analysis, the refinements made based on the findings and further discussion with expert advisors and the resulting final questions.
Summary of identified issues, modifications and revised questions
* Questions were refined based on cognitive interview findings and discussions with advisory group members and Ministry representatives.
† Questions were only asked to adult participants.
‡ Family unit: Everyone related to each other by blood, marriage or a marriage-like relationship including partners and foster children.
Interpretation of ambiguous terms
Several terms were interpreted inconsistently across participants, such as ‘basics’, ‘bread’, ‘household’, ‘others’, ‘adequate’ and ‘over the counter’. One food security question (Q8) asked participants to indicate whether they or their household ran out of basic foods due to a lack of money, with examples provided being ‘bread, potatoes, etc.’. When prompting participants to explain what type of basic foods they thought of, a wide variety of answers were given. Answers ranged from starchy types of foods, ‘bread, potatoes, rice, pasta’, to individual ingredients used for cooking or baking, such as ‘butter, sugar, eggs, flour, milk’, to the inclusion of almost all food categories within the dietary guidelines, like ‘fruit, vegetables, meat, fish, potatoes, bread’. Further discussions with the expert advisors revealed that it is important to capture cultural and generational differences in the interpretation of basic foods. It was therefore decided to keep the terminology open to interpretation and to remove the specific examples.
One dietary habits question asked about the consumption of bread (Q2). Examples of bread products were listed to prompt the participants to think of those and similar foods. However, when participants tried recalling the average number of servings of bread they consume per day, they included non-bread products. Several participants confused the term ‘bread’ with ‘carbohydrates’, asking for clarification, ‘So, I know bread as carbs. So is it, this question is directed to bread or carbs?’ (Female, 18–64 years). While most participants talked about slices of toast they would usually have, there were examples of participants including other grain foods such as breakfast cereals, rice and noodles.
The term ‘household’ was used in multiple questions (Q6, Q8–12), and the interpretation was perceived as challenging by some. With different living arrangements come different approaches to purchasing and preparing food. Those participants who lived in shared housing said that living with others did not necessarily mean sharing their food with their housemates. They were unsure whether to interpret the term ‘household’ as everyone they live with or only those with whom they share their food expenses:
‘Uh well other people in our house go fishing, diving, they have been given home kill, uh maybe fresh eggs as well, but we have not eaten them.’ and ‘Hmm and I guess for the household again, for example, it might be one of our flatmates [housemates] might have used it [food assistance] often, but we might never use it and so how would you know what to put?’ (Female, 18–64 years)
This issue is relevant to four food security questions, a question on food preparation and food storage facilities and a household food procurement question. After in-depth discussions with our expert advisors and considering typical household composition in NZ for different ethnic groups, it was decided that a household should be defined as the ‘family unit’ concept used in the US National Health and Nutrition Examination Survey (see Table 2)(17). An interviewer should clarify this concept before reading out the household questions to help participants better understand whom to consider in their answers.
Three of the selected food security questions asked about the reliance on a different source when the participant’s household ran out of food due to a lack of money (i.e. ‘others’, ‘food banks or food grants’, ‘churches, marae [Māori meeting house] or other community organisations’). For the first question, it was unclear who ‘others’ referred to, and participants interpreted it variably (Q10). As a consequence, participants’ responses included several sources that should have been included in subsequent questions. This sometimes led to confusion when reading the second and third questions, thinking they had already answered this, and they were repeated questions. For example, after reading the first question, a participant answered, ‘I think I may have only gotten a food grant like once or twice since the start the year’ (Male, 18–64 years), so selected ‘sometimes’ as the frequency answer, which was incorrect, given that the first question was intended to explore the use of personal sources of support like friends and family. Once prompted, they explained they did not consider friends and family in this question as they ‘never really relied on my whānau [family] to provide kai [food] if needed’ (Male, 18–64 years). As a result of these findings, the term ‘others’ was changed to ‘whānau/family, friends or neighbours’ to clarify what was being asked about.
Poor understanding of dietary or technical terms
Some participants had difficulty understanding dietary terms in questions or answer options, such as ‘serving size’, ‘gluten sources’, ‘dietary supplements’ or ‘foraging’. Two dietary habits questions asked participants to estimate the number of servings of fruit (Q1) and bread (Q2) they consumed on average over the past four weeks. The showcards included multiple visual examples of what constitutes a single serving (e.g. ‘1 medium piece’ showing an image of the palm of a hand with an apple). Often, participants noted examples of a serving that did not align with the instructions on serving sizes. While one serving of fruit could be made up of two smaller pieces (e.g. two plums) or one serving of bread could be half the size of how a product is typically consumed (e.g. half a bread roll), participants generally counted the times that they ate a food or the number of pieces they had, independent of the size of that item. One participant said, ‘I was counting one kiwifruit as one serving’ (Female, 65+ years), while another participant said, ‘Yeah, I would think most people would count one roll’ (Female, 18–64 years). The questions and showcards were subsequently amended to include more (and less seasonal-dependent) examples (and visuals) of a serving size, and gram weights were removed.
Another dietary habits question asked participants to select all the foods that they eat from a multiple options list (e.g. red meat, dairy products) (Q3). One of the answer options, ‘Gluten sources (e.g. wheat, barley)’, was poorly understood by some as they were unfamiliar with the term. To simplify, the question was reframed to ‘Do you completely exclude any of the following foods?’. The revised question should be easier to answer as only people who completely avoid gluten would answer yes. The examples of gluten sources were changed to bread and pasta to represent actual foods consumed rather than the grains, and ‘sources’ was removed.
Participants’ understanding of the term ‘foraging’ was variable (Q6). Some participants were able to provide a brief explanation of what they considered foraging to be, but others said that they did not know what the term was or thought it was similar to one of the other answer options. The examples listed for foraging (i.e. fruit, vegetables, nuts, herbs, mushrooms) were perceived as confusing as these foods could generally also be obtained through other sources such as a home or community garden. One participant was unsure, ‘Is foraging is that, is that what, is that just picking fruit from your backyard or is it that, what that is foraging?’ (Male, 18–64 years). The answer examples were subsequently changed to help respondents distinguish the difference between foraging and food usually obtained from a (community) garden by being more specific to foods sourced from the wild.
Difficulty following complex or unclear instructions
Participants had difficulty selecting the correct answer option in instances where instructions were more complex (e.g. averaging intake of fruit over four weeks) or unclear (e.g. types of foods included or excluded in specific food groups). For questions on the number of servings of fruit and bread (Q1–2), an example calculation was given (i.e. ‘1 apple + 2 small apricots = 2 servings’). Some participants had difficulty answering, stating that their intake of these foods varied significantly throughout the week. One younger participant selected ‘I don’t know’, ‘Because some days I just eat a random amount of fruit, so some days I have three servings and some days I have one’ (Male, 11–17 years). Additionally, calculating an average over the past four weeks was a complex task, which was likely amplified by difficulties related to estimating serving sizes as described earlier:
‘I am just confused about the serving, what we should put, because normally, out of thirty days of the month, we eat twenty days roti made at home. And they can be different in the size. Some people make very small, like I make very small, and thin. Some make big, and… so it is confusing to put the serving, how many servings should we put?’ (Female, 65+ years)
Participants tried to be precise in their estimation, resulting in cognitively difficult response processes. To overcome this issue, the term ‘on average’ was replaced with the term ‘usually’ in revised questions.
The food security questions, in general, were perceived as complex and wordy (Q8–11). This may explain why instructions were sometimes overlooked or unclear (i.e. the questions related to specific situations where people rely on food support due to the lack of money). For example, participants talked about situations where they had to rely on people or organisations but not due to the lack of money. The COVID-19 pandemic was a key reason for relying on food support, ‘Uh, I will say “sometimes”, but it was not due to money. It was due to not being able to go and get food because of COVID’ (Female, 18–64 years). The questionnaire structure for the food security questions was subsequently amended to simplify the instructions and ensure that participants answered affirmatively only when lack of money was an issue.
Two issues were identified for the question on salt added to food during preparation or cooking (Q4). A participant questioned whether salty sauces should be considered in answering the question on discretionary salt use:
‘Hmm this was not as easy as it should have been to answer. And the reason is, for myself, I never add salt to food. However, I cook Thai, so we use fish sauce, which is pretty much 50 % salt, is not it? So I have hidden salt. I do not get the white salt and add it.’ (Female, 65+ years)
Discussions with the expert advisors indicated it was important to capture the use of salty sauces, and better instructions were needed to ensure the question was interpreted as intended. The question was therefore rephrased to align with the WHO’s STEPS questions on salt use, which include salty sauces(18). While the salt question specifically asked about salt added in cooking or preparing foods, there was some confusion around whether salt added at the table should be included. Several participants noted their salt use after food had been prepared: ‘I would not always shake it on. You know, like sometimes I will just leave it and just eat what that person has prepared’ (Female, 65+ years). The full draft nutrition survey questionnaire includes two salt questions (i.e. added while preparing or cooking v. added at the table), the latter of which was not cognitively tested. To avoid confusion, both salt questions were displayed on one showcard for participants to be able to distinguish between the two different behaviours. No other formatting changes were made to the showcards that could potentially influence the interpretation of the questions.
Discussion
We aimed to cognitively test several questions selected for inclusion in a national nutrition survey to inform expert discussions, which together would guide the revisions needed to improve the questions. The cognitive testing undertaken with children and adults provided useful insights into respondents’ understanding of twelve questions assessing specific dietary habits and household food security. Challenges experienced by the participants were revealed using the think-aloud technique, while additional issues, sometimes not identified by participants, were identified through probing. The findings indicated where further refinements to questions and answer options were required to improve participant comprehension. Specifically, ambiguous terms were clarified, more technical dietary terms were simplified where possible, additional examples and descriptions of foods were added, more detailed explanation was provided in some question instructions and question order was adapted to increase questionnaire logic. The variable interpretations of key concepts and instructions in the questions, as originally worded, suggest that the validity of answers would likely have been reduced, highlighting the importance of cognitive testing as part of standard survey design to improve questions and maximise accuracy and validity. The process, from question selection through cognitive testing, expert consultations and subsequent amendments to questions, was iterative in nature. The discussion of test findings with experts and government representatives was a distinctive feature of this approach, ensuring any proposed changes would still adequately measure dietary habits and food security, align with the purpose of a national nutrition survey and maintain the ability to measure change over time using previous national survey data. Government representatives also considered how amendments would inform nutrition policy or food regulation. As these dietary habits and food security questions have been used in previous NZ national surveys, our findings are relevant to future nutrition surveys and to researchers considering using these or similar questions in other dietary studies. Even for questions that have been used frequently and over a prolonged period, it cannot be assumed that such questions remain appropriate or relevant without review. This underscores the need for regular cognitive testing to ensure that nutrition survey questions continue to accurately capture dietary habits and behaviours and reflect contemporary understandings and contexts.
Findings are consistent with similar studies in other populations, which also identified issues related to a lack of clarity in question instructions and wording(Reference Wolfe, Frongillo and Acassano11,Reference Drolet-Labelle, Bédard and Lemieux19,Reference Thompson, Adams and Vidgen20) . Various studies that cognitively tested either nutrition survey questionnaires or FFQ’s have reported that diet-related terms and concepts (e.g. fruits, beans, serving size, low fat, 100 % fruit juice) were misinterpreted or interpreted inconsistently(Reference Wolfe, Frongillo and Acassano11,Reference Thompson, Adams and Vidgen20–Reference Khadka, Sharma and Yuen-Esco24) . For example, in a study by Wolfe et al. (2001)(Reference Wolfe, Frongillo and Acassano11), participants found the term ‘serving’ difficult to define and included a wide range of examples that they considered to be a serving. Like these studies, we subsequently changed question wording or added more examples and instructions to address the cognitive challenges identified. Unfamiliarity with the concept of servings and difficulties in estimating the average servings consumed over a specific time period have commonly been reported in previous research(Reference Drolet-Labelle, Bédard and Lemieux19,Reference Subar, Thompson and Smith25) . Even in other disciplines, such as physical activity, questions that involve determining the frequency and duration of activities are difficult(Reference Heesch, van Uffelen and Hill26). A recent study by Drolet-Labelle et al. (2024)(Reference Drolet-Labelle, Bédard and Lemieux19) reported that older adults struggled to provide a response to an FFQ when having to calculate serving sizes and frequency of consumption. Changing question wording to ‘usually’ rather than ‘on average’ may help participants formulate a response, given that people tend to describe ‘typical’ or ‘normal’ patterns and behaviours when answering such questions(Reference Heesch, van Uffelen and Hill26). Furthermore, cognitive testing of questions frequently reveals that instructions on how to answer a question may be sub-optimal(Reference Wolfe, Frongillo and Acassano11,Reference Drolet-Labelle, Bédard and Lemieux19,Reference Carbone, Campbell and Honess-Morreale21,Reference Khadka, Sharma and Yuen-Esco24,Reference Heesch, van Uffelen and Hill26) .
With the food security questions we tested, several participants said that they found them complex and wordy and thus may have overlooked specific instructions referring to times that they ran out of money. Our findings indicate that people consider other factors beyond financial constraints to significantly impact their household’s food security, with the COVID-19 pandemic and its associated restrictions limiting access to food being mentioned most frequently. Furthermore, questions were deemed somewhat repetitive, and participants expressed uncertainty around which support services to include in their responses. Similar findings have been reported in studies exploring participants’ understanding of the US Household Food Security Module(Reference Nikolaus, Ellison and Nickols-Richardson12,Reference McClain, Johnson and DiRado-Owens27) . For instance, college students perceived the Household Food Security Module questions as repetitive and reported non-financial reasons for having to make less healthy dietary choices (e.g. time and transportation constraints) despite the question wording focusing on financial insufficiency(Reference Nikolaus, Ellison and Nickols-Richardson12). This may lead to misclassification of food security levels and, in turn, could have a major impact on policy implications. As such, several researchers have identified the need for more comprehensive food security questionnaires(Reference Calloway, Carpenter and Gargano28–Reference Coates30). Food security is multifaceted and context specific, with the FAO defining it as ‘All people, at all times, have physical, social and economic access to sufficient, safe and nutritious food which meet their dietary needs and food preferences for an active and healthy life’ (31). Recently, Calloway and colleagues developed food security questions addressing additional pillars of food security (i.e. availability, utilisation and stability), which can be used alongside the standard US Household Food Security Module questionnaire(Reference Calloway, Carpenter and Gargano32). In addition to the amendments we propose to the NZ food security questions, further research is recommended to identify new question domains relevant to the diverse NZ population to capture a broader experience of food insecurity.
This study had several strengths. It was conducted with a relatively large and diverse sample, capturing a wide range of possible perspectives, misinterpretations or understandings. Both think-aloud and probing techniques were used, followed by a detailed and rigorous analysis of the responses, to gain a comprehensive understanding of participants’ cognitive processes. No heterogeneity was observed in the findings of the group and individual interview modes, suggesting that offering both options could be a valuable approach for others. It allows for more flexible and participant-tailored methods of data collection. Expert advisors were involved in key steps of the process to consider the impact of the possible changes in question wording on the purpose and aims of a national nutrition survey. There were, however, some limitations. The generalisability of the findings of this study is limited, given the highly specific nature of the questions and the profile of the population they were tested with. The group interview format may have meant that some participants did not feel comfortable or able to respond to all questions. Despite a total sample of sixty-eight participants, when considered by age, gender and ethnic group, this resulted in small numbers in some subgroups, and detailed analysis was therefore only undertaken for the total sample. Additional cognitive issues relating to the food security questions may not have been captured, given that a relatively small proportion of study participants self-reported low financial security. Moreover, this research was conducted soon after the COVID-19 pandemic in NZ, with many participants referring to how being isolated had impacted their food security. Only a selection of questions (n 12) from the proposed nutrition survey were tested to minimise participant burden. Feedback showed that the omission of related questions may have led to some confusion (e.g. for the salt question). Cognitive testing typically follows an iterative approach, involving multiple rounds of testing with revisions made to the questions based on participant feedback within each round. However, we were unable to use such an approach due to time and budget constraints. Although the revisions made are likely to improve the accuracy of data collected, revised questions should be retested on a similar sample to ensure that revisions have improved respondents’ comprehension of the questions and have not introduced new areas of confusion.
Conclusion
The cognitive testing of twelve questions proposed for inclusion in a national nutrition survey provided valuable data, which led to the refinement and potential improvement of selected questions. A key feature of this study was engaging experts and government representatives in reviewing the findings to ensure proposed question wording changes remained aligned with the nutrition survey goals. Changes included simplified terminology, clearer instructions, improved examples and a more logical question order. These findings highlight that cognitive interviewing is a useful method when designing survey questionnaires to identify any problems in understanding questions as intended and thus develop more effective survey measures. Cognitive testing of questions used in national surveys should be conducted periodically to assess participants’ ongoing understanding of questions, though for studies where a time series is important, amendments need to be carefully considered.
Supplementary material
For supplementary material accompanying this paper, visit https://doi.org/10.1017/S1368980025101195.
Acknowledgements
Thank you to the community organisations for their support and the participants for their time. We acknowledge the hard work and commitment of our research assistants in transcribing the interview recordings. We would also like to thank our wider investigator group for their guidance, as well as the experts and government representatives who provided critical advice to help design the nutrition survey questions.
Financial support
The NZ Ministries of Health and Primary Industries funded this research. These funders played a role in developing the draft nutrition survey questionnaire, and they participated in group discussions on the cognitive testing findings and proposed revisions to the questions.
Competing interests
None declared.
Authorship
All authors contributed to the conceptualisation of the research design. B.F. led the data collection supported by J.G., M.M., C.H. and L.T.M. Together with our research assistants, B.F. and M.M. completed the transcriptions. Subsequent thematic analysis was performed by B.F. with help from C.H. and S.M. B.F. and C.H. wrote the initial draft of this manuscript, which was further edited by S.M., C.N.M. and L.T.M. All authors read and approved the final manuscript.
Ethics of human subject participation
This study was conducted according to the guidelines laid down in the Declaration of Helsinki, and all procedures involving research study participants were approved by the University of Auckland Human Participants Ethics Committee (UAHPEC24647, approved 5 August 2022). Written informed consent was obtained from all subjects.
Open access
