Hostname: page-component-8448b6f56d-tj2md Total loading time: 0 Render date: 2024-04-19T12:01:29.471Z Has data issue: false hasContentIssue false

A structured expert judgement elicitation approach: how can it inform sound intervention decision-making to support household food security?

Published online by Cambridge University Press:  05 February 2021

Sue Kleve*
Affiliation:
Department of Nutrition, Dietetics and Food, School of Clinical Sciences, Monash University, Melboure, VIC3168, Australia
Martine J Barons
Affiliation:
Department of Statistics, University of Warwick, Coventry, England, UK
*
*Corresponding author: Email suzanne.kleve@monash.edu
Rights & Permissions [Opens in a new window]

Abstract

Objective:

To examine structured expert judgement (SEJ) elicitation as a method to provide robust, defensible data for three determinants of household food security (food cost, household disposable income and physical access) for quantifying a proof-of-concept integrating decision support system for food security.

Design:

SEJ elicitation is a validated method for obtaining unavailable data, but its use in household food security in high-income countries is novel. Investigate Discuss Estimate Aggregate (IDEA) elicitation protocol was implemented, including quantitative and qualitative elements. Using specific questions related to three determinants, food security experts were encouraged to Investigate – estimate individual first-round responses to these questions, Discuss – with each other evidence on the reasoning and logic of their estimates, Estimate – second-round responses, following which these judgements were combined using mathematical Aggregation.

Setting:

Victoria, Australia.

Participants:

Five experts with a range of expertise in the area of household food insecurity participated in the SEJ elicitation process.

Results:

The experts’ ability to provide reliable estimates was tested and informed the aggregation of the collection of individual estimates into a single quantity of interest for use in decision support. The results of the quantitative elicitation show the impact of combinations of varying household income, food cost and physical access on household food security status and severity and is supported by the experts reasoning during elicitation.

Conclusion

This research provides insight to the application of SEJ where elicited data can inform and support intervention decision-making specific to household food security, especially where evidence is absent or of poor quality.

Type
Research paper
Copyright
© The Author(s), 2021. Published by Cambridge University Press on behalf of The Nutrition Society

Despite an abundance of food and relative wealth, food insecurity is a complex and serious public health challenge increasingly faced by high-income countries, affecting the health and wellbeing of those experiencing it(Reference Ramsey, Giskes and Turrell1Reference Barons and Aspinall6). Food insecurity is the limited or uncertain availability of households’ and individuals’ physical, economic, and social access to sufficient, safe, nutritious and culturally relevant food by socially acceptable means(7). Households may transition between episodic and chronic experiences of food insecurity, suffer anxiety and concern about adequate budget and/or supply of food, changes in diet quality, reduction in food intake and hunger(Reference Bickel, Nord and Price8).

The prevalence of food insecurity in high-income countries measured by different validated multi-item tools was found to be 12 % in Canada(Reference Tarasuk, Mitchell and Dachner9), 8 % in England, Wales and Northern Ireland(Reference Bates, Roberts and Lepps10), and 11·8 % in the USA(Reference Coleman-Jensen, Rabbitt and Gregory11). In Australia nationally, using a two-part measure, 4 % of people lived in a household that reported to be food-insecure(12). This measure assesses one aspect of the food security experience and potentially underestimates the prevalence of food insecurity in Australia by approximately 5 %(Reference McKechnie, Turrell and Giskes13). In Victoria, the second highest populated state of Australia, 4 % of Victorians were food-insecure(14).

Responses to address food insecurity in high-income countries vary from food relief initiatives, community food programmes to social protection policies and programmes. The dominant response is the provision of food relief through the charitable food sector; however, this does not focus on the underlying determinants of food insecurity(Reference Loopstra15). A particular challenge in food insecurity responses, including policy, is that the determinants are complex and interacting. Solutions neglecting to address these determinants are unlikely to have significant or widespread impact. The complexity derives from the interaction between the multidimensional determinants that include structural and socio-ecological factors, for example, low or unstable income, un- and/or under-employment, health status, stressful life events, household composition, low education, home ownership and geographic (urban v. rural or remote)(Reference Daly, Pollard and Kerr16Reference Temple18). In addition to these aforementioned factors, the four interacting dimensions underpin food security status that include

  • 1 Availability of sufficient nutritionally adequate food supply;

  • 2 Adequate financial and physical access to food;

  • 3 Resources, infrastructure and ability to utilise food;

  • 4 The stability of 1–3 over time(19). However, a primary predictor of food insecurity is inadequate income and other financial resources available for acquiring food.

Health promoters, public health practitioners and policymakers wishing to make evidence-based decisions to ameliorate household food insecurity require suitable information and evidence, to evaluate candidate policies. Using high-quality evidence in a relevant localised context is ideal, but key aspects of the system of determinants may not be amenable to investigation or their acquisition may be prohibitive based on cost, time, effort or ethical considerations. The incomplete nature of the supporting evidence provides a serious challenge and can leave policymakers using heuristics to fill the gaps.

Structured expert judgement (SEJ) elicitation is a robust and defensible method for producing evidence for policymakers. The use of expert advice and opinion to support policy decision-making is commonplace, but generally, the manner in which contributions are synthesised to inform the eventual decision is not transparent. Where informal heuristics and elicitation are employed, experts are subject to a number of well-documented biases: social biases, deferring to the member with the most compelling personality or who is seen as the most senior, bias towards the most readily available information and misunderstandings due to semantic differences(Reference Slovic20,Reference Kahneman and Tversky21) . The results are often not reproducible and can be unreliable and heavily biased. These difficulties can be significantly reduced using structured approaches designed to mitigate the most pervasive and debilitating psychological and contextual frailties of expert judgement(Reference Burgman22Reference O’Hagan, Buck and Daneshkhah25).

Structured elicitation of expert opinion in pursuit of decision support is an increasingly important technique across areas affecting health. Examples include, food and feed safety risk assessment(26), assessment of health risks(Reference Cooke, Wilson and Tuomisto27), and the quantification of uncertainty in the risks of herbicide-tolerant crops(Reference Krayer von Krauss, Casman and Small28). Validated protocols for SEJ fall into three broad categories based on how they aggregate the individual contributions of experts into a single estimate: behavioural aggregation (seeking consensus); mathematical aggregation (combining individual estimates using a formula) and mixed aggregation. There are several well-established methodologies for structured expert elicitation protocols, each with its own strengths and limitations(26). The recently developed Investigate Discuss Estimate Aggregate (IDEA) elicitation protocol(Reference Hanea, McBride and Burgman29) encourages experts to Investigate and estimate individual first-round responses, Discuss, Estimate second-round responses, following which judgements are combined using mathematical Aggregation. IDEA combines the strengths and ameliorates some limitations of older methods.

In this study, we used IDEA SEJ to provide evidence that was missing on food security determinants for Victoria, Australia, required to develop a food security integrated decision support system (IDSS) similar to the one developed for the United Kingdom(Reference Barons, Fonseca and Davis30). As part of the development of this, IDSS food security determinants for Victoria were mapped. However, there were no data to quantify households’ expected food security category for three of the key determinants of food insecurity: (1) food cost, (2) household income and (3) physical access.

This paper describes the application of a SEJ elicitation method using the IDEA protocol to derive the unknown quantities of cost of food, household disposable income and physical access. These quantities were required to support the development of a household food security proof-of-concept IDSS for Victoria, Australia, to predict the probability of household food security status to inform policy and intervention.

Methods

In the development of a household food security integrating decision support system, a determinants map was initially devised by the authors, informed by the pre-existing literature(Reference Kleve, Davidson and Gearon31) and localised knowledge. The IDEA protocol for SEJ was used to elicit information from local food security experts required to build a food security integrating decision support system(Reference Smith, Barons and Leonelli32). The IDEA protocol was previously validated in a forecasting tournament over a 12-month period. In summary, each month a new set of questions was released and experts had approximately 3 weeks to investigate the question, provide first-round individual estimates, view the estimates of other individuals in their group, discuss and provide second-round individual estimates(Reference Hanea, McBride and Burgman29). The IDEA protocol method is described according to its three stages of elicitation: before, during and after.

Before elicitation

Drafting elicitation and calibration questions

In the pre-elicitation stage, the risk of semantic or other misunderstandings of the areas to be elicited are minimised by careful and precise wording of the questions, which are used to elicit the missing data values, called questions of interest. This stage helps to identify suitable experts to participate in the elicitation itself. During the IDEA elicitation, in addition to the quantities of interest, experts are asked calibration questions. Calibration questions have answers that are unknown to the experts providing the missing quantities, but known to the analysis team. Calibration questions are used to assess individual experts’ ability to estimate probabilities, which is known to be a difficult task(26). The accuracy and informativeness of experts’ responses to the calibration questions are used to calculate performance measures and used as weights for the mathematical aggregation step of the protocol (Table 1). Formulae and details of the hypothesis test for calibration are given in Ref. 33. Calibration questions are close to the subject-matter of the questions of interest as possible because it is expertise in a specific subject area which is important for SEJ.

Table 1 Structured expert judgement elicitation and calibration questions

* Calibration questions.

Drawn from the determinants map the questions of interest pertained to the relation of the determinants of food cost, household disposable income and physical access prediction of household food insecurity (Fig. 1).

Fig. 1 Part of the integrating decision support model for household food security in Victoria, Australia

Expert identification

Sixteen academics and health promotion/public health professionals from Victoria identified through snowball sampling as experts in the areas of food insecurity determinants were invited to participate in the elicitation process via personal email sent by an independent researcher. These experts were specifically involved in knowledge and/or response generation related to specific determinants of food insecurity and/or population groups experiencing food insecurity across the health promotion continuum and sectors. Identified experts included epidemiologists, social service sector policymakers or health promotion workers (youth, housing disability and employment, energy sector, work and economic security), social planners, practitioners with specific portfolios in indigenous and asylum-seeker services, peak bodies of food and material aid relief. Since SEJ involves the combination of expert judgement, diversity of experts is more important than large numbers. Greater than fifteen experts does not significantly improve the findings, but fewer than five may reduce the chance of providing adequate diversity in opinion(34). Nine of these sixteen experts initially agreed to participate in the elicitation process including pre- and post-activity and 1-d face-to-face workshop. A list of supporting and briefing background materials including relevant food security literature identified through a literature search, an outline of the IDEA protocol and the quantities of interest refined to specific questions was provided in advance of the face-to-face session.

Due to unanticipated work commitments, four of the nine experts withdrew at short notice limiting the capacity for further expert recruitment. The final five participating experts were employed in university, local government, and not-for-profit sector with experience in areas of food security determinants, poverty alleviation and food security response planning, implementation and evaluation, and/or front-line service provision.

During elicitation

The 1-d face-to-face expert elicitation process was audio-recorded and transcribed to provide further insight to the processes, framing, discussions and negotiations that experts undertook through the elicitation. These discussions serve to improve the accuracy of experts’ quantitative estimates, including uncertainty bounds, improving the quality of the second-round estimates, which will be used as data in the IDSS model. Testing of the IDEA protocol showed that when this discussion exercise is undertaken on calibration questions, experts tend to move their estimates closer to the true values in this exercise.

Agreement on question meaning

For a successful elicitation, agreement upon clear definitions of the variables to be quantified was required. Initial proposed definitions provided by the researchers and informed by literature were discussed with the expert panel and collaboratively refined as required to reach consensus.

The validated and widely implemented United States Department of Agriculture Household Food Security Survey Module 18 item (USDA HFSSM) was used to determine food security status and severity level(Reference Bickel, Nord and Price8).The food security quantities of interest were the proportions of people expected to fall into the four food security severity categories defined in the USDA HFSSM, under each combination of the levels of the three determinants (income, food cost and physical access), making forty-eight questions of interest in all. Severity of household food security was defined according to the USDA HFSSM criteria of number of indications (affirmative responses) to survey items(Reference Bickel, Nord and Price8):

  • High food security: no reported indications of food access problems or limitations.

  • Marginal food security: one or two reported indications – typically of anxiety over food sufficiency or shortage of food in the house. Little or no indication of changes in diets or food intake.

  • Low food security: reports of reduced quality, variety or desirability of diet. Little or no indication of reduced food intake.

  • Very low food security: reports of multiple indications of disrupted eating patterns and reduced food intake.

To avoid over-burdening the experts participating in the structured elicitation, the number of categories in the income, food cost and physical access variables were restricted: equivalised disposable income was categorised as high, moderate and low; food access (cost of food) was categorised as good (low cost) or poor (high cost), and physical access was categorised as good (nearby) or poor (large distances).

Equivalised disposable income per household was defined as total household income divided by the sum of weightings for household members. The Australian Bureau of Statistics (ABS) uses the Organisation for Economic Co-operation Development (OECD)-modified equivalence scale which assigns a weighting value of 1 to the household head, 0·5 to each additional person 15 years or older, and 0·3 to each child under 15 years(35). The ABS mean weekly equivalised disposable household income quintile data for Victoria were used to inform income categories(35). Weekly equivalised disposable income was categorised as high greater than AUD$1152 (above 4th Quintile); moderate between AUD$859 and AUD$1152(3rd and 4th Quintile); and low below AUD$859 (below 3rd Quintile).

Food cost, influencing the availability of food, was determined using the validated Victorian Healthy Food Basket data collected across the state(Reference Palermo, McCartan and Kleve36). For a ‘typical family of 4’ (44-year-old male and female, 18-year-old female and 8-year-old male), the highest cost basket is AUD$561·08, moderate (median) cost basket is AUD$422·50 and lowest cost basket is AUD$359·48 per fortnight. Therefore, food availability was good if a healthy basket for a ‘typical family of 4’ costs AUD$422·20 (median) or less and poor if a healthy basket for a ‘typical family of 4’ costs more than AUD$422·20.

Physical access categories were informed by the 20-min neighbourhood project(37). Physical activity was categorised as good if the nearest supermarket or strip shopping was either: within 500 m walking distance, within 20 min on available private transport, within 500 m walking distance of access to public transport (bus stop, station) and shopping within 30 min journey time by shortest available route. Physical activity was categorised as poor if the nearest supermarket or strip shopping is further than these distances.

The calibration exercise

The calibration questions were derived from household food security survey data used to validate an Australian food security tool(Reference Kleve, Gallegos and Ashby38) and compare its performance with the USDA HFSSM questions(Reference Bickel, Nord and Price8). This data had not yet been made public. The calibration questions were treated in the same way as the questions of interest, and in this exercise, were mixed among the questions of interest in a single questionnaire. Using standard measures of performance according to Hanea (2017), the second-round answers to the calibration questions were analysed(Reference Hanea, McBride and Burgman29).

Expert estimates

Following agreement on question understanding for the questions of interest, the experts each provided private, individual first-round estimates for the lowest plausible, highest plausible and best estimate for the natural frequencies of how many people out of 100 would be in each food security category given various levels and combinations of income, food access and availability (Table 1). The first estimation was undertaken privately to encourage independent thinking and avoid anchoring on other experts’ estimates. The experts’ estimates to the scenario questions were plotted on graphs in anonymised form for the subsequent discussion phase.

Experts then discussed how they had arrived at their estimates and reasons for the width of the interval between their lowest and highest plausible estimates, using the anonymised graphs. In particular, it was important for the facilitators to understand whether a wide interval was indicative of the expert’s perceived uncertainty in the system or related to the question or a reflection of their own uncertainty. Following discussion, the experts each gave private, individual second-round responses in line with the IDEA protocol.

After elicitation

After the workshop, the experts’ first- and second-round estimates were compared. The first-round estimates form the basis of discussion, sharing information and challenging perspectives which improve estimates in the second round.

The second-round responses to the questions of interest were aggregated in line with the IDEA protocol: using a performance-weighted average, the aggregated lowest plausible, highest plausible and best estimates for the probability of categories of household food security status were calculated from the second-round estimates.

Those experts who gave the most accurate and informative answers on the calibration questions are given most weight in the aggregation of questions of interest.

Results

Estimates of proportions of households falling into each of four categories of household food security under a combination of key variables were obtained from five experts participating in the SEJ elicitation and aggregated based on performance on calibration questions. In each case, the lowest plausible, highest plausible and best estimate of the proportion were obtained and treated as 5th and 95th percentiles and median of a probability distribution, respectively.

The comparison of experts’ first- and second-round estimates across the questions indicated that whilst some responses were unchanged, others changed considerably, showing that the discussion brought new perspectives to the table. As example, Figure 2 describes the expert estimates with equal weight and performance weight combinations for comparison with two questions of interest. Question 7(Q7) ‘Out of 100 people with moderate equivalised disposable income and good physical access when food availability is good, how many will be food-secure?’ While Question 16(Q16) asks ‘Out of 100 people with low equivalised disposable income and good physical access when food availability is good, how many will be very low food-secure?’ The solid lines in Fig. 2 indicate the experts estimates of ranges of lowest plausible, highest plausible and best estimate (median). The broken lines depict each expert’s private second-round estimate after discussion. The top two lines are the equal-weighted and (top) performance-weighted combinations of all judgements. Expert estimates to all calibration questions and questions of interest are available in Supplemental Files 1 and 2.

Fig. 2 Expert estimates with equal weight and performance weight combinations for comparison

Of particular note, many of the experts reduced the interval between highest plausible and lowest plausible values in their second-round estimates, suggesting that they were more certain about the interval within which a good estimate should lie following the discussion.

Figure 2 illustrates that in Q7, there was a varied change in response between experts’ initial and second estimates. Expert 1 changed their estimate considerably following discussion on Q7 and increased their uncertainty. In contrast, Expert 4 reduced their uncertainty, and Experts 2, 3 and 5 had not changed their estimates following discussion. In Q16, the difference between round 1 and round 2 estimates was small, except in one case but despite this the plausible intervals overlapped. In Q16, Expert 5 had a very different estimate from the other four, and this supports a strongly different pooled estimate between the equal weight and performance weight combinations. The graphs for all the questions of interest and the calibration questions are included in Supplemental Files 1 and 2.

The discussions undertaken by the experts during the elicitation process provided opportunity for debate, clarification, agreement and reasoning of the questions of interest or calibration questions being considered and contributed to the outcome in the scenarios impacting on food security status. Careful inspection of the discussion showed the diversity of sector experience in these experts provided a range of viewpoints. As example of the expert discussion in the scenario Q7 good physical access, food availability (cost) and moderate equivalised income:

‘Expert 5: So we’re almost covering the whole scale

Expert 3: I used the same reasoning, where I put my majority on as moderate. So moderate income will be moderately FS, so I put that as my highest one out of 100

Expert 2: I would be very surprised if there were more than 50 % in this situation.

Expert 3: Perhaps I should increase my upper limit

Expert 1: This is interesting. Using the healthy food access basket, 25 % of your income goes on food with a moderate income of $859 …… that’s a fair whack of income going towards that healthy food basket. ….So maybe they’re spending less on healthy food and they were at that higher end of that income bracket. So I thought a lot of them would be FS, more than 50 % of them would be FS. But maybe even more.

Expert 2: But then isn’t income a really important determinant of FS? If that’s not high, it’s moderate?

Expert 3: For me, income is the biggest predictor. So if you’ve got moderate compared with high, then you have more of a chance at being moderately FS than being food-insecure with a moderate income…….

Facilitator: Because you think a significant influence on FS is income? The two other variables?

Expert 3: I always put income first. That’s what the evidence shows, isn’t it?

Expert 4: I do like your rationale (Expert 3). …. looking at the definition of moderate FS and how there’s actually no change to the diet or food intake but it can take into account that anxiety around stress, so someone could be eating a good diet and not change, but just have constant underlying worry…I won’t change mine’

This sample highlights the robust discussion that occurred across the experts, with diverse fields of expertise, when discussing the variations in the conditions in the questions of interest. It also illustrates the variety of perspectives within the expert panel.

The results of the calibration exercise found in Supplemental File 3: Supplemental Table 1 showed that performance-weighted aggregation was preferred to equal-weighted aggregation, although both are shown for comparison.

Table 2 highlights the sensitivity of household food security to all three factors. The probability that a household is food-secure, given good physical access to food, good availability of food (prices average) and a high equivalised income is predictably high, at median 88 % and between 80 and 96 %. Some food insecurity can occur even in these optimal circumstances because asset:debt ratio can be very high, meaning that the high income is spent on servicing debt. Similarly, given the worst possible scenario of poor physical access, poor availability (prices high) and low income, the most probable outcome is low food security (median 43 % and between 17 and 56 %). In this scenario, the probability that a household is food-secure is low (median 15% and between 7 and 25 %).

Table 2 Probability of food security status expressed as a percentage according to varied scenarios of physical access, food availability and equivalised income

* Medians were asked for, these did not necessarily sum to 100, so were normalised and the lowest and highest plausible values were normalised using the same factor.

In the scenario of low equivalised income, good physical access and food availability, Questions 13–16 (Supplemental table) the probability of high food security was 22·1 %, moderate 28·9 %, low 40·9 % and very low 8 % (Table 2). The probability of high food security varies dramatically between high and low income when the other two factors are the same, regardless of the levels of them. In contrast, food availability and physical access each have smaller effects on food security status for the same income levels.

Discussion

This paper reports the use of SEJ, specifically the IDEA protocol as a method to quantify unknown aspects of data when developing a proof-of-concept IDSS to describe food insecurity in Victoria, Australia. These data can be added to the model of complex, interacting determinants of food security to provide decision support to intervention (inclusive of policy) decision-makers. Using this approach, the probability of household food security status under various combinations of the determinants disposable household income, physical access and food availability (cost) could be predicted, overcoming the prohibitive difficulties of obtaining these by designed experiment or other data sources. SEJ provides a way to estimate these quantities in a transparent and defensible manner. In the elicitation of quantities from experts, the differences in expertise between acknowledged specialists can be properly and robustly addressed and reduced by the careful use of facilitated discussion, avoiding the severe problems associated with unstructured elicitation.

Consistent with being a primary determinant of food security status, the quantities provided by the experts supports that income, in this case disposable income, has a predominate effect on food security status. Hence, it is important for intervention decision-makers to understand the demands on household financial resources including cost-of-living pressures, housing, utilities, cost of food(Reference Kirkpatrick and Tarasuk39Reference Reeves, Loopstra and Stuckler41) and additional costs such as health conditions and medical expenses(Reference Tarasuk, Cheng and de Oliveira42). Subsequently, interventions targeting increasing disposable household income and determinants that impact on this are likely to be supportive of higher levels of food security(Reference Loopstra15). In addition, the information gathered through the elicitation allows for intervention decision-makers to determine likely effect of policies on food security under the effects of uncontrollable determinants, such as cost of food. For example, estimating the effect of economic or environmental policies on food availability (cost) can estimate the subsequent impact on household food security(Reference Smith, Barons and Leonelli32,Reference Smith, Rabbitt and Coleman- Jensen43) .

Eliciting best-estimate values for the probability of each food security outcome and also the lowest and highest plausible values (Table 2) provides further evidence of the varying impact of determinants. Whilst the effects of physical access and food availability on food security status are significant, household disposable income is by far the strongest determinant. This helps intervention decision-makers to anticipate the effect of different candidate responses at their disposal. For example, policies and or interventions to improve physical access, such as transportation and walkability alone, are unlikely to have a strong effect on household food security; however, they may have additional health-promoting benefits, for example, reducing social isolation. Similarly, improving access to food access through a range of interventions such as food bank/pantry, food literacy or community food programmes such as growing schemes will have limited improvement in ongoing household food security(Reference Loopstra15,Reference Pollard and Booth44) .

In order to quantify the likely effects of changes in the levels of determinants, perhaps via interventions, on household food security in a range of contexts health promotion, public health practitioners and policymakers may use the quantities derived through elicitation in combination with known data of other key determinants. This provides opportunity to test policy interventions alone and/or in combination for the likely impact on household food security status to inform policy choice or potential interventions.

A strength of this research is the use of an established and validated method to derive quantities of interest related to food security that may support health promotion and public health workforce in decision-making on how best to address this issue within a range of geographic and/or demographic contexts. The provision of the likely effect of combinations of determinants on household food security inclusive of those explored in this proof-of-concept decision support system is of importance to support policy design. In particular, these findings can be used to evaluate potential policies to support decision-makers in making evidence-based choices. The availability of calibration questions very close to the questions of interest is a strength of the study, providing high confidence that the weighting of the experts is appropriate. The experts who contributed to the elicitation process and provided estimates are experienced in a range of aspects food security, providing confidence that these estimates are likely to be reliable given the current state of knowledge.

The limitations of the study include the discretisation of the continuous variables. In order to undertake the elicitation in the available time, each of the three areas of interest for elicitation were reduced to two-three discrete categories. Ideally, a more nuanced categorisation would be preferred. However, more levels per variable lead to a rapid rise in the number of conditional probabilities to be elicited, hence in an increased elicitation burden. Finally, a number of experts who had committed to taking part in the elicitation process were unavailable at very short notice, reducing the number and potentially the breadth of expertise contributing to the elicitation. However, diversity of experts is more important than large numbers for SEJ. Greater than fifteen experts does not significantly improve the findings, but fewer than five may reduce the chance of providing adequate diversity in opinion(34). Nevertheless, a strength of the IDEA protocol is the discussion between experts allows verification that a broad range of views and considerations were represented when undertaking this elicitation. The example of this discussion highlights the complexity of food security and the importance of decision-makers to have an awareness and understanding of the range and interaction between determinants and their impact. The variety of opinions between experts (such as in Q16) is dealt with through the aggregation process, combining experts using their performance on the calibration questions. It is also possible to undertake sensitivity analysis, by including and excluding subsets of experts. Whilst a decision-maker may wish to re-run the elicitation with a larger group before using the full model for decision support, we are satisfied that the diversity in the experts’ backgrounds, experience and perspective is rich enough for a proof of concept.

Based on new methodology for coherent inference in networked systems(Reference Smith, Barons and Leonelli32), future work will include the incorporation of these values with other evidence on major determinants of household food security. This will provide a proof-of-concept IDSS, which would demonstrate how to support policymakers to evaluate the effect on household food security of plausible scenarios. Doubtless, were policymakers to adopt this methodology, they would wish to scrutinise the model itself and re-run the structured expert elicited judgement with the number and variety of experts that would be satisfactory to them.

Conclusion

This research provides insight to the application of a novel approach for food security studies in high-income countries, SEJ. This provides useful, transparent and defensible evidence, when evidence-based decision-making is required particularly in context of complex issues such as household food security. In order to address the issue of household food insecurity beyond current responses, interventions must consider the range of varied and interacting determinants. In the instance where evidence is absent or of poor quality, public intervention decision-makers may consider using structured expert elicitation as a method of evidence generation to inform intervention decisions.

Acknowledgements

Acknowledgements: The authors thank the experts who participated in the elicitation process and Dr Anca Hanea (AH) and Caitlin Kemp (CK) for supporting elicitation process. AH guided the authors in the preparatory stages of the IDEA elicitation process and facilitated the workshop and calibration discussions (aided by MJB, SK) and analysed the quantitative responses to produce the pooled estimates. CK recorded and documented the discussions. The authors thank for the support of The Monash Warwick Alliance – between Monash University and University of Warwick. Financial support: N/A. Conflict of interest: The authors confirm no support from any organisation for the submitted work except as listed above, no financial relationships with any organisations that might have an interest in the submitted work in the previous 3 years and no other relationships or activities that could appear to have influenced the submitted work. Authorship: MJB and SK conceived, designed and implemented the study. MJB and SK drafted the paper. Both authors critically revised the manuscript for important intellectual content and approved the final version. Ethics of human subject participation: This study was conducted according to the guidelines laid down in the Declaration of Helsinki and all procedures involving research study participants were approved by the by Monash University Human Ethics Committee – low risk 2017-11079-13745. Written informed consent was obtained from all subjects/patients.

Supplementary material

For supplementary material accompanying this paper visit https://doi.org/10.1017/S1368980021000525

Footnotes

Sue Kleve and Martine J Barons are first authors.

References

Ramsey, R, Giskes, K, Turrell, G et al. (2011) Food insecurity among Australian children: potential determinants, health and developmental consequences. J Child Health Care: Professionals Working with Child Hospital and Community 15, 401416.CrossRefGoogle ScholarPubMed
Ramsey, R, Giskes, K, Turrell, G et al. (2012) Food insecurity among adults residing in disadvantaged urban areas: potential health and dietary consequences. Public Health Nutr 15, 227237.CrossRefGoogle ScholarPubMed
Seligman, HK, Laraia, BA & Kushel, MB (2010) Food insecurity is associated with chronic disease among low-income NHANES Participants 1,2. J Nutr 140, 304310.CrossRefGoogle Scholar
Stuff, JE, Casey, PH, Szeto, KL et al. (2004) Household food insecurity is associated with adult health status. J Nutr 134, 23302335.CrossRefGoogle ScholarPubMed
Vozoris, N & Tarasuk, V (2003) Household food insufficiency is associated with poorer health. J Nutr 133, 120126.CrossRefGoogle ScholarPubMed
Barons, MJ & Aspinall, W (2020) Anticipated impacts of Brexit scenarios on UK food prices and implications for policies on poverty and health: a structured expert judgement approach. BMJ Open 10, e032376.CrossRefGoogle ScholarPubMed
Committee on World Food Security (2012) Coming to terms with terminology Rome (IT): Food and Agriculture Organisation Report No.: CFS 2012/39/4. http://www.fao.org/3/MD776E/MD776E.pdf (accessed February 2020).Google Scholar
Bickel, G, Nord, M, Price, C et al. (2013) Guide to Measuring Household Food Security, Revised 2000. Alexandria, VA: U.S. Department of Agriculture, Food and Nutrition Service.Google Scholar
Tarasuk, V, Mitchell, A & Dachner, N (2016) Household Food Insecurity in Canada 2014. Toronto: Research to identify policy options to reduce food insecurity (PROOF).Google Scholar
Bates, B, Roberts, C, Lepps, H et al. (2017) The Food and Your Survey Wave 4. Combined Report for England, Wales and Northern Ireland. UK: Food Standards Agency.Google Scholar
Coleman-Jensen, A, Rabbitt, M, Gregory, C et al. (2018) Household Food Security in the United States in 2017. Washington, DC: U.S. Department of Agriculture, Economic Research Service.Google Scholar
Australian Bureau of Statistics (2018) Australian Health Survey: Nutrition State and Territory results 2011–12 Cat no 4364.0.55.009. Canberra, AU: Australian Bureau of Statistics.Google Scholar
McKechnie, R, Turrell, G, Giskes, K et al. (2018) Single-item measure of food insecurity used in the National Health Survey may underestimate prevalence in Australia. Aust New Zealand J Public Health 42, 389395.CrossRefGoogle ScholarPubMed
Department of Health and Human Services Victorian Government (2017) Challenges to Healthy Eating: Food Insecurity in Victoria: Findings from the 2014 Victorian Population Health Survey. Melbourne, AU: Department of Health and Human Services, Victorian Government.Google Scholar
Loopstra, R (2018) Interventions to address household food insecurity in high-income countries. Proc Nutr Soc 77, 270281. https://www.cambridge.org/core/journals/proceedings-of-the-nutrition-society/article/interventions-to-address-household-food-insecurity-in-highincome-countries/F2D7D0B429C175D9098237B8F7CDDCDF (accessed February 2020).CrossRefGoogle ScholarPubMed
Daly, A, Pollard, CM, Kerr, DA et al. (2018) Using cross-sectional data to identify and quantify the relative importance of factors associated with and leading to food insecurity. Int J Environ Res Public Health 15, 2620.CrossRefGoogle ScholarPubMed
Gorton, D, Bullen, CR & Mhurchu, CN (2010) Environmental influences on food security in high-income countries. Nutr Rev 68, 129.CrossRefGoogle ScholarPubMed
Temple, J (2018) The association between stressful events and food insecurity: cross-sectional evidence from Australia. Int J Environ Res Public Health 15, 2333.CrossRefGoogle ScholarPubMed
Food and Agriculture Organisation (2018) An introduction to the basic concepts of food security. www.fao.org/docrep/013/al936e/al936e00.pdf (accessed February 2020).Google Scholar
Slovic, P (1999) Trust, emotion, sex, politics, and science: surveying the risk-assessment battlefield. Risk Anal: Offic Publ Soc Risk Anal 19, 689.CrossRefGoogle ScholarPubMed
Kahneman, D & Tversky, A (1984) Choices, values, and frames. Am Psychologist 39, 341350.CrossRefGoogle Scholar
Burgman, MA (2016) Trusting Judgements: How to Get the Best Out of Experts. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Aspinall, W (2010) A route to more tractable expert advice. Nature 463, 294.CrossRefGoogle ScholarPubMed
Cooke, R & Goossens, L (2008) TU Delft expert judgment data base. Reliab Engineering Sys Saf 93, 657674.CrossRefGoogle Scholar
O’Hagan, A, Buck, CE, Daneshkhah, A et al. (2006) Uncertain Judgements: Eliciting Experts’ Probabilities. New Jersey: John Wiley & Sons.CrossRefGoogle Scholar
European Food Safety Authority (2014) Guidance on expert knowledge elicitation in food and feed safety risk assessment. EFSA J 12, 3734.Google Scholar
Cooke, RM, Wilson, AM, Tuomisto, JT et al. (2007) A probabilistic characterization of the relationship between fine particulate matter and mortality: elicitation of European experts. Environ Sci Technol 41, 65986605.CrossRefGoogle ScholarPubMed
Krayer von Krauss, MP, Casman, EA & Small, MJ (2004) Elicitation of expert judgments of uncertainty in the risk assessment of herbicide-tolerant oilseed crops. Risk Anal: Int J 24, 15151527.CrossRefGoogle ScholarPubMed
Hanea, AM, McBride, MF, Burgman, MA et al. (2017) Investigate discuss estimate aggregate for structured expert judgement. Int J Forecast 33, 267279.CrossRefGoogle Scholar
Barons, M, Fonseca, T, Davis, A et al. A decision support system for addressing food security in the UK 2020. https://arxiv.org/abs/2004.06764 (accessed August 2020).Google Scholar
Kleve, S, Davidson, ZE, Gearon, E et al. (2017) Are low-to-middle-income households experiencing food insecurity in Victoria, Australia? an examination of the Victorian Population Health Survey, 2006–2009. Aust J Primary Health 23, 249256.CrossRefGoogle ScholarPubMed
Smith, J, Barons, MJ & Leonelli, M (2016) Coherent Frameworks for Statistical Inference serving Integrating Decision Support Systems. https://arxiv.org/abs/1507.07394 (accessed July 2016).Google Scholar
Cooke, R, Mendel, M & Thijs, W (1988) Calibration and information in expert resolution; a classical approach. Automatica 24, 8793.CrossRefGoogle Scholar
EFSA (2014) Guidance on expert knowledge elicitation in food and feed safety risk assessment. Eur Food Saf Authority (EFSA) J 12, 3734.Google Scholar
Australian Bureau of Statistics (2017) Household Income and Wealth Australia 2015–16 Cat no.65230.0. https://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/6523.02015-16?OpenDocument (accessed September 2017).Google Scholar
Palermo, C, McCartan, J, Kleve, S et al. (2016) A longitudinal study of the cost of food in Victoria influenced by geography and nutritional quality. Aust New Zealand J Public Health 40, 270273.CrossRefGoogle ScholarPubMed
State of Victoria Department of Environment L, Water and Planning (2017) 20 Minute Neighbourhoods. https://www.planmelbourne.vic.gov.au/current-projects/20-minute-neighbourhoods (accessed May 2020).Google Scholar
Kleve, S, Gallegos, D, Ashby, S et al. (2018) Preliminary validation and piloting of a comprehensive measure of household food security in Australia. Public Health Nutr 21, 526534.CrossRefGoogle ScholarPubMed
Kirkpatrick, S & Tarasuk, V (2011) Housing circumstances are associated with household food access among low-income urban families. J Urban Health 88, 284296.CrossRefGoogle ScholarPubMed
Emery, JCH, Bartoo, AC, Matheson, J et al. (2012) Evidence of the Association between Household Food Insecurity and Heating Cost Inflation in Canada, 1998–2001. Canadian Public Policy/Analyse de Politiques 38, 181215.CrossRefGoogle Scholar
Reeves, A, Loopstra, R & Stuckler, D (2017) The growing disconnect between food prices and wages in Europe: cross-national analysis of food deprivation and welfare regimes in twenty-one EU countries, 2004–2012. Public Health Nutr 20, 14141422.CrossRefGoogle ScholarPubMed
Tarasuk, V, Cheng, J, de Oliveira, C et al. (2015) Association between household food insecurity and annual health care costs. Can Med Assoc J 187, E429E436.CrossRefGoogle ScholarPubMed
Smith, MD, Rabbitt, MP & Coleman- Jensen, A (2017) Who are the world’s food insecure? new evidence from the Food and Agriculture Organization’s Food Insecurity Experience Scale. World Dev 93, 402412.CrossRefGoogle Scholar
Pollard, CM & Booth, S (2019) Food insecurity and hunger in rich countries—it is time for action against inequality. Int J Environ Res Public Health 16, 1804.CrossRefGoogle ScholarPubMed
Figure 0

Table 1 Structured expert judgement elicitation and calibration questions

Figure 1

Fig. 1 Part of the integrating decision support model for household food security in Victoria, Australia

Figure 2

Fig. 2 Expert estimates with equal weight and performance weight combinations for comparison

Figure 3

Table 2 Probability of food security status expressed as a percentage according to varied scenarios of physical access, food availability and equivalised income

Supplementary material: File

Kleve and Barons supplementary material

Kleve and Barons supplementary material 1

Download Kleve and Barons supplementary material(File)
File 13.2 KB
Supplementary material: PDF

Kleve and Barons supplementary material

Kleve and Barons supplementary material 2

Download Kleve and Barons supplementary material(PDF)
PDF 1.2 MB
Supplementary material: PDF

Kleve and Barons supplementary material

Kleve and Barons supplementary material 3

Download Kleve and Barons supplementary material(PDF)
PDF 1.3 MB