Scientific Advisory Committee on Nutrition’s evidence evaluations

Since its inception in 2001, SACN has produced a range of publications. The different approaches and nomenclature have changed over time, reflecting the need for flexibility in the types of evaluations undertaken by SACN. The following approaches, with clear definitions and processes, were agreed for consistency in future evidence evaluations: reports (full risk assessments); rapid reviews; position statements; updates to reports/rapid reviews/position statements and joint reports/rapid reviews/position statements (to cover assessments jointly undertaken with other scientific committees).

It was agreed that the appropriate approach should be chosen at the outset of an evaluation, with the rationale for the selection included in the methods section. The choice of approach would depend on consideration of issues such as the research question(s), the nature of the available evidence and the urgency and timeframe for completion.

Interpretation of statistical methods and data

In previous SACN reports, findings with a P value < 0·05 were considered ‘statistically significant’ and providing evidence of an effect (from randomised controlled trials) or association (from observational studies), while those with a P value ≥ 0·05 were considered as ‘not statistically significant’ and providing insufficient evidence of an effect or association. During the preparation of more recent reports, SACN members had raised concerns about using the P< 0·05 criterion alone for interpretation of results. Studies with P values just below 0·05 might also be at greater risk of publication bias because those with ‘statistically significant’ results are more likely to be published^{(Reference Hopewell, Loudon and Clarke2)}.

The subgroup agreed that P values should not be considered in isolation and that it would be more informative to also consider effect size and confidence intervals. Clinical or biological significance and public health relevance should also be considered since findings might be ‘statistically significant’ but may not be clinically or biologically important. It was also noted that a small effect size may have little value at an individual level but could be important at a population level.

For interpretation of study results, it would be important to define outcomes and the effect sizes considered beneficial for public health at the outset of an evidence evaluation. Interpretation of P values should be informed by consideration of study size and power. It would also be essential to consider the methodological quality of studies because, irrespective of the P value, findings from poor quality studies or studies with a high risk of confounding may not be reliable. The subgroup agreed that future evidence evaluations should not describe results as ‘significant’ or ‘non-significant’ but report the exact P value, estimated effect size and confidence intervals where available, alongside the direction of any effect or association and consistency of findings.

Assessing the quality of evidence

Assessing systematic reviews

Since the majority of SACN’s assessments are based on evidence from systematic reviews, two established quality assessment tools for systematic reviews and meta-analyses were considered: AMSTAR 2 (a measurement tool to assess systematic reviews)^{(Reference Shea, Reeves and Wells3)} and ROBIS (risk of bias in systematic reviews)^{(Reference Whiting, Savovic and Higgins4)}. The usability in practice of AMSTAR 2 and ROBIS was compared by applying both to assess the quality of two systematic reviews with meta-analyses (Korsmo-Haugen et al.^{(Reference Korsmo-Haugen, Brurberg and Mann5)}, Sainsbury et al.^{(Reference Sainsbury, Kizirian and Partridge6)}) that were previously reviewed in the SACN report on Lower carbohydrate diets for adults with type 2 diabetes⁽⁷⁾.

The AMSTAR 2 tool assesses methodological quality through a checklist of sixteen questions. Seven of the questions are considered ‘critical’ to the validity and conclusions of the systematic review (although appraisers may add or substitute other critical domains). Responses for eleven questions are dichotomous (yes/no), while responses for five questions include an additional response (partial yes). An overall judgement of confidence (high, moderate, low or very low) in the results of a systematic review is based on the assessment of the critical and non-critical items. The AMSTAR 2 assessments of the systematic reviews with meta-analyses by Korsmo-Haugen et al.^{(Reference Korsmo-Haugen, Brurberg and Mann5)} and Sainsbury et al.^{(Reference Sainsbury, Kizirian and Partridge6)} are summarised in online Supplementary Table 1.

The ROBIS tool assesses risk of bias in three phases. The 1st phase assesses relevance of the systematic review to the research question of interest by comparing both in terms of participants, interventions, comparisons and outcomes. The 2nd phase identifies concerns within the systematic review process and comprises twenty-one questions within four domains (study eligibility criteria, identification and selection of studies; data collection and study appraisal; synthesis and findings). There are five possible responses to the questions (yes, probably yes, probably no, no or no information). The 3rd phase comprises three questions and considers if the systematic review, as a whole, is at risk of bias. A judgement is then made on the overall risk of bias (low, high or unclear).

In the 1st phase of the ROBIS assessments (relevance to research question) both Korsmo-Haugen et al.^{(Reference Korsmo-Haugen, Brurberg and Mann5)} and Sainsbury et al.^{(Reference Sainsbury, Kizirian and Partridge6)} were considered relevant to the research question but Sainsbury et al.^{(Reference Sainsbury, Kizirian and Partridge6)} was judged to be a partial match since it addressed only two of the three outcomes of interest. The concerns identified within the systematic review process (phase 2) are summarised in online Supplementary Table 2, and judgement on the overall risk of bias (phase 3) is provided in online Supplementary Table 3.

The overall judgements on the quality of the 2 systematic reviews using AMSTAR 2 and ROBIS were in agreement. For the systematic review by Korsmo-Haugen et al.^{(Reference Korsmo-Haugen, Brurberg and Mann5)}, overall confidence in the results was ‘high’ using AMSTAR 2, and risk of bias was ‘low’ using ROBIS; for the systematic review by Sainsbury et al.^{(Reference Sainsbury, Kizirian and Partridge6)}, overall confidence in results was ‘low’ using AMSTAR 2, and risk of bias was ‘high’ using ROBIS.

The subgroup agreed that both AMSTAR 2 and ROBIS provided a structured approach to assess the quality of the systematic reviews. Overall, AMSTAR 2 was simpler and easier to use. ROBIS provided a more rigorous assessment of risk of bias and consequently took much more time to complete. The longer completion time was identified as potentially problematic if the quality of several systematic reviews needed to be assessed for a SACN evidence evaluation. The main advantages and disadvantages of the AMSTAR 2 and ROBIS tools, experienced during the comparison exercise, are summarised in Table 1.

Table 1.

Advantages and disadvantages of AMSTAR 2^{(Reference Shea, Reeves and Wells3)} and ROBIS^{(Reference Whiting, Savovic and Higgins4)}

The subgroup agreed to recommend use of the AMSTAR 2 tool for quality assessment of evidence from systematic reviews. It was agreed, however, to further explore the practicality of using ROBIS in the future.

Assessing the certainty of evidence

The rationale for assessing the certainty of a body of evidence is to inform and guide recommendations. Previous versions of the SACN Framework did not include guidance for assessing evidence certainty. However, evidence was graded in 4 SACN reports using an approach devised by SACN. The approach was conceived initially for use in the SACN report on Carbohydrates and health⁽¹¹⁾. It was developed for use in the SACN report on Saturated fats and health⁽¹²⁾ and then further developed for the SACN reports on Lower carbohydrate diets for adults with type 2 diabetes⁽⁷⁾ and Feeding young children aged 1–5 years⁽¹³⁾.

The subgroup agreed that a consistent and standardised approach, with wide international recognition and comparability, was required for assessing the certainty of evidence in future SACN evidence evaluations. Four approaches were considered: Grading of recommendations, assessment, development and evaluation (GRADE); Nutrigrade^{(Reference Schwingshackl, Knuppel and Schwedhelm14)}; United States Department of Agriculture (USDA) Dietary Guidelines Advisory Committee (DGAC)⁽¹⁵⁾ and Hierarchies of Evidence Applied to Lifestyle Medicine (HEALM)^{(Reference Katz, Karlsen and Chung16)}. Nutrigrade and HEALM required detailed evaluation of individual studies. Since SACN usually considers evidence from published systematic reviews, these approaches were not considered further. It was agreed to focus on the GRADE approach because it is the most recognised and widely used tool for assessing evidence certainty. It specifies 4 levels of certainty (high, moderate, low and very low) that can be assigned to a body of evidence per outcome (see Table 2).

Table 2.

GRADE certainty ratings

Evidence from randomised trials starts with a ‘high’ certainty rating. This can then be downgraded after considering 5 criteria: risk of bias; imprecision; inconsistency; indirectness; and publication bias. Evidence from observational studies usually starts with a ‘low’ certainty rating because of potential bias due to lack of randomisation and because confounding is always a concern in even the most rigorously conducted observational studies. The certainty rating from observational evidence can be upgraded if any of 3 criteria are met: (1) large magnitude of effect; (2) clear dose-response gradient; (3) residual confounding is likely to decrease rather than increase the magnitude of effect.

The subgroup agreed that the advantages of the GRADE approach included consistency, transparency and comparability with other guidelines. It also allowed flexibility to exercise judgements in the 5 key domains. However, GRADE presented specific challenges for assessing nutritional evidence. These included: (1) likelihood of ‘low’ certainty being assigned to macronutrient and whole-diet intervention trials, where blinding is impossible or unrealistic; (2) potentially undervaluing evidence from prospective cohort studies (which provide evidence of long-term effects in real life conditions) because the starting point for all observational evidence (regardless of type) is ‘low’ certainty; and (3) limited flexibility to upgrade evidence from observational studies.

Implementation of the GRADE approach in practice was explored by conducting a grading exercise, comparing it with the USDA/DGAC approach. Both approaches were applied to grade the evidence for the effect of lower compared to higher carbohydrate diets on an outcome considered in the SACN report on ‘Lower carbohydrate diets for adults with type 2 diabetes’⁽⁷⁾. The primary studies in all the systematic reviews considered in this report were randomised controlled trials. The outcome considered in the grading exercise was glycated haemoglobin (HbA1c) concentration in the longer term (≥12 months), in a systematic review with meta-analysis by Sainsbury et al.^{(Reference Sainsbury, Kizirian and Partridge6)}.

Using GRADE, the evidence for HbA1c (≥ 12 months) was graded as: moderate certainty of no difference in effect. The process for reaching this grade is summarised in online Supplementary Table 4.

With the USDA/DGAC approach, a grade (strong, moderate, limited, grade not assignable) is assigned for each of 5 elements: risk of bias, consistency, directness, precision, and generalisability. The final grade reflects consideration of all the grading criteria. Using this approach, the evidence for HbA1c (≥ 12 months) was also graded as: moderate certainty of no difference in effect. The process for reaching this grade is summarised in online Supplementary Table 5.

In practice, the GRADE approach was considered to be more straightforward to use than the USDA/DGAC approach. The stepwise process for reaching a final grade was transparent and the reasons for downgrading were clear. With the USDA/DGAC approach, each of 5 assessment domains were assigned a grade but no guidance was provided on weighting the separate domains to make a judgement on the overall grade. Another concern was that the grades could inflate confidence in the evidence because there was no ‘weak’ or ‘low’ category.

Although the grading exercise generally favoured the GRADE approach, there were still concerns about the criteria for upgrading observational evidence. GRADE stipulates that observational evidence can be upgraded if there is a large magnitude of effect (risk ratio > 2 or < 0·5); however, such large effect sizes are rarely observed in nutrition evidence and a smaller effect size could be important in terms of public health. It was suggested that at the outset of an evidence evaluation, SACN working groups should decide and then specify the magnitude of effect that would be considered ‘large’ for each outcome under consideration. The agreed threshold could then be used as the basis to make a judgement about upgrading evidence from prospective cohort studies.

The secretariat subsequently met with two representatives of the GRADE Public Health Group to discuss interpretation of GRADE in relation to public health and nutrition evidence. The GRADE representatives agreed it would be appropriate for SACN working groups to set the threshold for a ‘large’ effect size (with justification provided) for a particular outcome. They also advised that any suggestions on making GRADE more usable for public health would be considered by the GRADE developers.

A remaining concern about adopting GRADE was its appropriateness in a nutrition context, where there is a paucity of evidence from large long-term randomised controlled trials or where allocation of intervention is masked from the participants. Instead, evidence for longer term hard endpoints is largely drawn from observational studies where grading starts at ‘low’ certainty.

The previous grading exercise, comparing the GRADE and USDA/DGAC approaches, was extended to a meta-analysis of prospective cohort studies^{(Reference Te Morenga, Mallard and Mann17)} that considered the association between sugar-sweetened beverage consumption in children aged 1–5 years and body weight in later childhood (5 prospective cohort studies/7 comparisons, n 7255). Comparison of higher v. lower sugar-sweetened beverage intakes suggested a higher risk of being overweight associated with higher sugar-sweetened beverage intakes (odds ratio 1·55, 95 % confidence interval 1·32 to 1·82, P< 0·001). The certainty of the evidence was assessed as ‘low’ using the GRADE approach and as ‘limited’ using the USDA/DGAC approach.

In common with the previous grading exercise (comparing both approaches to grade a meta-analysis of randomised controlled trials), applying the stepwise approach of GRADE was found to be more straightforward. The USDA/DGAC approach involved making separate judgements on each domain and seemed more subjective. It was noted that both approaches had reached a similar grade but that the process was more transparent with GRADE.

The subgroup agreed that, in general, SACN’s public health recommendations should be based on evidence assessed as ‘high’ or ‘moderate’ certainty. In some cases, however, expert judgement could also be used to make recommendations based on ‘low’ certainty providing that a clear explanation of the rationale for such a decision was included.

Overall, it was agreed that advantages of GRADE included its transparency, broader international recognition and its comparability. In addition, the GRADE developers were open to evolving the methods in response to feedback. Although the subgroup still had some reservations about applying GRADE to nutritional evidence, it was agreed to recommend its use in future SACN evidence evaluations. Since the updated Framework was intended to be a ‘live’ document, the approach to grade the certainty of evidence could be changed if GRADE was found to be unsuitable in practice.

The subgroup agreed that it would be important for the Framework to clearly describe the two-step process of (1) assessing evidence quality (applied at the systematic review level) and (2) grading evidence certainty (applied at the outcome level).

The updated Framework was published on the SACN website in January 2023. It was further updated in October 2024 to include more information on how SACN determines its work programme; recommendation to use the ROBINS-E^{(Reference Higgins, Morgan and Rooney18)} (risk of bias in non-randomised studies – of exposures) tool to assess the quality of observational cohort studies; addition of text stating that (exceptionally) recommendations could be based on ‘very low’ certainty of evidence and a flow diagram illustrating the process for selecting systematic review/meta-analysis to grade evidence for an exposure–outcome relationship. The changes to the previous version were chronicled in an Annex to the Framework.

Next steps

Since publication of the updated Framework, the subgroup has provided support and guidance to the SACN Nutrition and maternal health working group on applying GRADE to assess the certainty of evidence for the draft SACN report on nutrition and maternal weight outcomes. The subgroup has also considered and made recommendations to SACN for improving the consistency and clarity of terminology to express energy intakes and recommendations for fat and carbohydrate intakes. A number of topics have been identified for future consideration.

As a ‘living’ document, the Framework will be subject to regular review by the subgroup and continue to evolve in line with best practice. The subgroup will respond to any issues experienced by SACN, the subgroup on maternal and child nutrition or its working groups in applying the Framework to ongoing evidence evaluations.

The subgroup would welcome and consider any comments or feedback from the scientific community on the SACN Framework.