Whatever the economic setting, health systems are struggling with healthcare resources allocation and the implementation of universal health coverage (UHC) (Reference Wilkinson, Sculpher and Claxton1). The World Health Assembly held in 2014 marked health technology assessment (HTA) as a crucial tool for UHC (Reference Wilkinson, Sculpher and Claxton1). HTA supports policymakers in dealing with their budgetary constraints by facilitating an equitable, efficient, and high-quality health system (Reference Wilkinson, Sculpher and Claxton1;Reference O’Rourke, Oortwijn and Schuller2). Economic evaluation (EE) is an essential component of HTA, requested in many countries to support reimbursement decisions by providing information on the cost-effectiveness of health technologies (Reference Dilokthornsakul, Thomas, Brown and Chaiyakunapruk3;Reference Weatherly, Cookson and Drummond4). For harmonization and comparability purposes, economic evaluation guidelines (EEG) have been developed and applied in many countries, particularly European and North American countries, where HTA is well established (Reference Zhao, Ming and Qu5–Reference Dacccahe, Rizk and Dahham7). Along with other health systems guidance (HSG), EEGs provide critical input and support for decision-making for effective health system programs; accordingly, a well-described process is essential to ensure the transparency and objectivity of these guidelines (Reference Bosch-Capblanch8;Reference Lewin, Bosch-Capblanch and Oliver9).
Recently, the interest in and use of HTA has increased in low- and middle-income countries (LMICs), leading to the development of EEGs in these countries, such as Brazil, Cuba, Egypt, and Indonesia (Reference Dacccahe, Rizk and Dahham7). Although EEGs are emerging in LMICs, several countries do not yet have an established EEG (Reference Dacccahe, Rizk and Dahham7). Developing national EEGs is crucial to providing researchers with practical steps for conducting their EEs (Reference Treweek, Oxman and Alderson10). International organizations have demonstrated interest through their contribution to this quality domain (Reference Bosch-Capblanch8;11;12). The World Health Organization (WHO), in collaboration with the Swiss Center for International Health, developed a handbook for supporting the development of HSG (Reference Bosch-Capblanch8). Similarly, in 2015 the International Network of Agencies for HTA (INAHTA) created the LMICs working group, Guidelines International Network (GIN), to explore methods for promoting guideline development, adaptation, dissemination, implementation, and research within developing countries (12). Furthermore, the Appraisal of Guidelines for Research and Evaluation (AGREE) enterprise developed a new appraisal tool for assessing the quality of HSG, the Appraisal of Guidelines for Research and Evaluation-Health Systems (AGREE-HS) (11). At the 2020 international HTA global policy forum, researchers proposed the adoption of transparency, inclusivity, and impartiality as starting principles in developing a deliberative process in HTA (Reference Bosch-Capblanch, Lavis and Lewin13;Reference Bond, Stiffell and Ollendorf14), bringing attention to the process of developing guidelines.
Despite the importance and the value of the development process of EEGs, there is a lack of published comprehensive literature on this subject. Understanding how previous EEGs have been developed could provide relevant insights for the next countries to develop EEGs in their context. Accordingly, this systematic review aimed to review the process and sources of evidence in developing guidelines for EEGs in LMICs, with the ultimate goal of assisting and supporting developers from LMICs, in addition to identifying gaps in research, and providing recommendations about each significant step in the development of guidelines for LMICs aiming to design their own EEGs.
Methods
A systematic review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA-S) guidelines (Reference Moher, Liberati, Tetzlaff and Altman15). The protocol of this review was registered with the Open Science framework platform (Registration DOI: https://doi.org/10.17605/OSF.IO/DHRYF).
Systematic review
Literature Search Strategies
We searched MEDLINE (Ovid), PubMed, EconLit, Embase (Ovid), and the Cochrane Library from the inception date to 9 February 2020, and performed a literature search update on March 2021.
In our search strategy, we combined three key concepts: “economic evaluation” AND “guidelines,” AND “low and middle-income countries.” For the last concept, we used the Cochrane filter 2012 (16), which we adapted to the 2020–21 World Bank classification. For the three concepts, we mapped controlled vocabulary, such as Medical Subject Headings (MeSH) terms and keywords. A medical information specialist validated the strategy. The full search strategies for MEDLINE, PubMed, and Embase are provided in Supplementary Material 1.
Moreover, the gray literature was searched, including the Web sites of the World Bank, WHO, the Professional Society for Health Economics and Outcomes Research (ISPOR), pharmacoeconomics guidelines around the world, the international Decision Support Initiative (iDSI), the Guide to Health Economic Analysis and Research (GEAR), as sources of eligible documents, and Epistemonikos as a source of systematic reviews. In addition, the Web sites of country-specific HTA agencies or Ministries of Health were reviewed (provided in Supplementary Material 2). A backward citation tracking for relevant systematic reviews and included guidelines was conducted. The search was neither limited to a specific language nor to a publication date.
We performed these literature searches in preparation for conducting two systematic reviews; the first was to identify and review the key features of EEGs in LMICs, and the second one to understand and to review the development process of these EEGs (Reference Dacccahe, Rizk and Dahham7).
Eligibility Criteria
The official versions of EEGs, including pharmacoeconomics guidelines (PEGs) and drugs guidelines, developed by the national agencies of relevant LMICs, and publications related to their development process were included.
EEGs from high-income countries, EEs studies and reports regarding diseases, interventions, and other topics not relevant to guidelines and their development process were excluded. Nonoriginal research documents and studies or expert opinions regarding EEGs, systematic reviews, and unofficial published guidelines were also excluded. An article is considered original research if it is the report of a study written by the researchers who actually did the study, if the researchers describe their hypothesis or research question and the purpose of the study, if the researchers detail their research methods, if the results of the research are reported, and if the researchers interpret their results and discuss possible implications (17).
Study Identification and Screening
All records identified by the search were retrieved. To determine eligibility, titles and abstracts were screened by one reviewer (C.D.) and full texts of selected references were assessed by two pairs of reviewers (C.D./R.K. and C.D./R.R.). A calibration exercise was conducted prior to proceeding. Documents in languages other than English, French, and Arabic were translated via the online.doc.translator Web site. Discrepancies were resolved through discussions between reviewers.
Data Extraction
Data related to the methods and steps used in the EEG development process, the responsible party engaged in setting the guidelines, and the composition of the development team if reported, were extracted. A calibration exercise was conducted before data extraction. Two pairs of authors (C.D./R.K. and C.D./R.R.) extracted relevant data from the included studies independently and in duplicate. Disagreements between reviewers were solved through discussions to reach a consensus.
Data Analysis
A narrative synthesis of collected data was done. Then, a comparative summary table was presented.
Quality Assessment
We conducted a quality assessment of the development process of included EEGs using the AGREE-HS tool, which was adapted to the scope of our study.
While the AGREE-HS tool represents five domains (topic, participants, methods, recommendations, and implementability), we selected two quality domains that answer our research questions in identifying the steps in developing EEGs in LMICs and the composition of the development team. Accordingly, we adopted the term process to define participants and methods domains. “Participants” is the item that addresses the composition of the HSG development team. “Methods” addresses the use of systematic methods and transparency in reporting, along with use of the best available and up-to-date evidence (11). We developed a quality assessment document for each country, where appraisers reported their comments and scoring (Supplementary Material 3). Each item was assessed on whether its criteria have been met, whereby appraisers took into consideration whether the item-related content was well reported, easy to find, and easy to understand. For consensus, we performed a practical appraiser’s exercise (AGREE-HS appraisal of an eligible EEG).
Appraisers rated each item using a 7-point response scale. A score of 1 (lowest quality) was given if there was no information relevant to the AGREE-HS item, if the criteria were very poorly reported in the HSG document, or if the authors explicitly stated that it was not done; whereas a score of 7 (highest quality) was given if the information related to the AGREE-HS item was exceptionally well reported, all criteria related to the item were considered during the development of the guidance, and the information related to the item was applicable in its context. Scores between 2 and 6 were assigned when not all criteria of the AGREE-HS item were met, depending on the completeness and quality of reporting. Scores increased as more criteria were met (11). Two appraisers (C.D./R.K. and C.D./R.R.) independently assessed the process adopted by each country in developing its EEGs, using the accompanying eligible studies related to the development process. The mean AGREE-HS item scores were calculated and presented for each document.
In the absence of any empirical basis for deriving thresholds and defining high- and low-quality HSG, appraisers assessed the process based on a relative comparison between LMICs of each quality domain scores.
Results
The flowchart of study selection is detailed in Figure 1. In February 2020, twenty-one records were included; of these, thirteen were EEGs and eight were discussing and describing the development process. In addition, with the literature update in March 2021, one guideline reporting on the first edition of the Philippine HTA methods guide published in September 2020, new Chinese guidelines developed in 2020 to replace the previous ones, one article published in 2021, entitled: “Current Development and Practice of Pharmacoeconomic Evaluation Guidelines for Universal Health Coverage in China,” and one Indian guideline entitled: “Health Technology Assessment in India: A manual” were included.
PRISMA diagram of study selection. BIA, budget impact analysis; EE, economic evaluation; EEG, economic evaluation guidelines; GEAR, guide to health economic analysis and research; HICs, high-income countries; ISPOR, Professional Society for Health Economics and Outcomes Research; LMICs, low- and middle-income countries.
EEGs in LMICs
Based on the 2020–21 World Bank classification, fourteen LMICs (10.4 percent) had EEGs, including none of the twenty-nine low-income countries (LICs), five out of fifty lower-middle-income countries (Bhutan, Egypt, India, Indonesia, and the Philippines) and nine out of fifty-six upper-middle-income countries (Brazil, China, Columbia, Cuba, Malaysia, Mexico, the Russian Federation, South Africa, and Thailand). Moreover, Mercosur, a union of countries that includes Argentina, Brazil, Paraguay, Venezuela (upper-middle-income countries), Bolivia (a lower-middle-income country), and Uruguay (a high-income country, which is not covered by this review), has developed a common guideline (18–37).
Table 1 includes the country’s name, its guidelines, and articles describing the development process, responsible party, publication year, and type of document.
Summary of Countries’ Eligible Documents
EMTD, Essential Medicines and Technology Division; HTA, health technology assessment; HTAP, health technology assessment panel; IETS, Instituto de Evaluación Tecnológica en Salud; InaHTAC, Indonesian Health Technology Assessment Committee; NA, not applicable.
Responsible Party and Team Composition
The Ministry of Health was the responsible party handling the development of three-fourths of the identified guidelines (n = 10) (Bhutan, Brazil, Cuba, Egypt, India, Indonesia, Malaysia, the Philippines, the Russian Federation, and South Africa), while nonprofit independent institutes were commissioned to develop 28.6 percent (n = 4) (China, Colombia, Mexico, and Thailand) of the identified guidelines.
The majority of LMICs who have developed EEGs did not explicitly report the discipline of those who were involved in the development process. Elsisi et al. and the Indonesian Health Technology Assessment Committee (InaHTAC) clearly reported the make-up of the teams involved in developing guidelines in Egypt and Indonesia (20;Reference Elsisi, Kaló and Eldessouki37), respectively, while others specified the working position (19–21;28;32) or the sector (22;25;34;35) or did not report it at all (18;23;24;26;27). Heterogeneity in extracted data was observed; the team composition results categorized members mostly as academicians, workers in the private or public sectors, and program managers. We assessed the team composition in the second part of this paper.
Development Methods
Table 2 summarizes the development steps of EEGs in LMICs. Each country carried out a specific process for developing its guidelines; many steps have been reported and identified. Some countries (n = 6) (Colombia, Cuba, Egypt, Malaysia, the Philippines, and Thailand) began by reviewing other national guidelines. In addition, nine countries (Bhutan, Brazil, China, Cuba, Egypt, Malaysia, Mexico, the Philippines, and Thailand) arranged workshops and consultation meetings for experts and stakeholders to propose recommendations and updates.
Summary of the Development Steps of EEGs in LMICs
EEGs, economic evaluation guidelines; LMICs, low- and middle-income countries.
Five out of fourteen countries (Bhutan, Colombia, Cuba, India, and Indonesia) developed their EEGs with the support of international experts. Furthermore, in four of the fourteen countries (Brazil, Cuba, Indonesia, and the Philippines), International organizations such as WHO, the Pan American Health Organization (PAHO), the European Union, iDSI, the Health intervention and technology assessment program (HiTAP) of Thailand, and the National Institute for Health and Care Excellence (NICE), contributed in the development of EEGs.
Three countries (India, the Russian Federation, and South Africa) did not report the steps followed to develop their EEGs.
Quality Assessment Results
Table 3 summarizes the quality assessment results of the two domains: participants and methods. The Cuban guidelines had the highest final score for participants, while the Egyptian and Philippines guidelines had the highest final score for methods. In detail, Cuba clearly reported in its guidelines that their development team for HSG includes members who have an interest or stake in the recommendations. Moreover, the guidance development team was multidisciplinary and multisectoral. In the two documents describing their guidelines development process, Egypt reported that systematic and transparent methods were used to identify and review the evidence, and to reach a consensus regarding the final recommendations. In addition, Egypt is considered the best available and most contextually relevant evidence. Similarly, in their guidelines document, the Philippines addressed the use of systematic methods and transparency in reporting, as well as the use of the best available and up-to-date evidence.
Quality Assessment Results
* Each item was assessed on whether its criteria have been met. A score of 1 was assigned for lowest quality, a score of 7 was assigned for the highest quality, and scores ranging between 2 and 6 were assigned when some of the criteria of the AGREE-HS item have been met, with higher scores indicating higher quality.
Discussion
A transparent approach to reporting the development process of EEGs is fundamental to assessing this process and generating information that can be beneficial for guidelines developers (Reference Bosch-Capblanch, Lavis and Lewin13;Reference Gagnon38). This is especially true for LMICs, where experience in developing EEGs is limited and this process is uniquely challenging (Reference Dacccahe, Rizk and Dahham7). In this paper, we systematically reviewed the development process of fourteen EEGs in LMICs, that is, the participants and the steps or the methods followed, and we assessed the quality of this process.
We identified fourteen EEGs in LMICs, compared with the thirteen reported by Daccache et al. in the systematic review conducted in June 2020 (Reference Dacccahe, Rizk and Dahham7). This highlights that guidelines development is an ongoing process in LMICs, which are slowly but steadily adopting EEGs as a support tool for advice about the allocation of their scarce resources.
Our findings indicate that the responsible party or the development agency in the majority of included EEGs was the Ministry of Public Health, while in high-income countries (HICs) such as Canada, the United Kingdom, and Australia, independent bodies manage the development process (39–41). The responsibilities of Ministries of Health in system guidance in LMICs, where regulations and standards are de jure governance roles, are emphasized in the literature (Reference Sheikh, Sriram and Rouffy42–44). Outside the government, this process can be improved by consultation with other major stakeholders, including the health professions, the private sector, civil society, philanthropic organizations, and academia, and by encouraging feedback by publishing discussion papers that set out draft proposals (44). Developers should bear in mind that during consultation, industry groups might seek to weaken regulation and to shift the goals that the government is seeking to achieve (44). The ministries’ involvement may ensure the implementability of these guidelines in developing countries.
Interestingly, we found reporting to be poor about participants involved in the development process: sector, interest, and discipline were not explicitly disclosed in the vast majority of included guidelines. Due to the heterogeneity of the extracted data, we failed to coherently present the team composition results. Inconsistencies regarding this issue are found even in HICs. For instance, Australia’s Pharmaceutical Benefits Advisory Committee (PBAC) did not report the disciplines of the guidelines developers, while Canada’s Canadian Agency for Drugs and Technologies in Health (CADTH) named the members of the guidelines working group, and the UK’s NICE briefly described the major roles of their guideline development team (39;41;45). A review of guideline organizations in Australia, Canada, Europe, New Zealand, and the U.S. found that guideline development groups (GDG) are composed of 10–20 members, and the number of disciplines is often three to five per group (Reference Fretheim, Schünemann and Oxman46). In addition, guideline development was endorsed by the support of methodological experts (epidemiologists, information specialists, and others) and patient representatives (Reference Fretheim, Schünemann and Oxman46). Adopting a multisectoral approach and engaging a multidisciplinary team is crucial for ensuring well-structured and high-quality guidelines (Reference Bosch-Capblanch8;Reference Sheikh, Sriram and Rouffy42). The GDG should include stakeholders such as consumers, health professionals who work within the relevant area, managers or policymakers, and individuals with the necessary technical skills, including information retrieval, systematic reviewing, group facilitation, project management, writing, editing, and expertise in health economics (Reference Fretheim, Schünemann and Oxman46). The poor reporting on the development team prevented us from drawing conclusions about participants, and from identifying potential competing interests of development team members. This highlights the importance of good reporting and identification of the roles of members of the working group involved in the process, as well as clear conflict of interest statements.
A transparent and systematic process is fundamental for EEGs developers to accomplish their objective. We identified four main steps followed by EEGs developers in LMICs: conducting a review on national guidelines, organizing workshops, and getting support from either international experts or from organizations. Nevertheless, the steps followed were heterogeneous across guidelines. Only one country, Cuba, followed the four steps. According to the WHO, the optimal process for supporting the development of HSG is through finding evidence and translating evidence into guidance (Reference Bosch-Capblanch8). That is, finding evidence using systematic reviews and translating evidence using a deliberative process, which may include informal or formal consensus involving several stakeholders with varied backgrounds, on development methods (Reference Bosch-Capblanch8). Only six of the fourteen LMICs (Colombia, Cuba, Egypt, Malaysia, the Philippines, and Thailand) conducted reviews to generate evidence in developing their guidelines, and only five followed up this step by the deliberative process (Cuba, Egypt, Malaysia, the Philippines, and Thailand) (22;25;Reference Chaikledkaew and Kittrongsiri29;34;35;Reference Elsisi, Kaló and Eldessouki37). Other countries followed incomplete steps, that is, either a workshop, or international support, or a review (Bhutan, Brazil, China, Colombia, India, Indonesia, and Mexico) (18–21;Reference Reinharz, Ra and Asche31;Reference Yue, Li, Wu and Guo33;34). Furthermore, while the AGREE enterprise provides a structured, systematic, and standardized methodological framework for developing and reporting HSG (11), we could not identify any country, except Cuba, following it. This is also the case in developed countries. For instance, Australia did not describe their guidelines development process, nor did the UK, which developed a manual for NICE guidelines, with the exception of some guidance including the appraisal of technology (47). In contrast, Canada, in their guidelines for the EE of pharmaceuticals, described the process for setting their guidelines (a workshop with Canadian stakeholders and international experts); this process was explicitly detailed by Towse in 1997 and Torrance in 1996 (Reference Glennie, Torrance and Baladi48–50). Moreover, in 2018, Canada briefly described this process in the fourth edition of the guidelines (39). Standardizing the guideline development process would facilitate the production and adaptation of guidelines to different contexts and reduce spending resources on guideline development; this is of utmost importance for LMICs (Reference Schünemann, Fretheim and Oxman51).
Finally, quality assessment scores were highest (nine out of fourteen) for the “methods” domain (means above 4), while lower scores were observed for the “participants” domain (seven out of fourteen with means below 4). Specifically, we noted that one-third of the identified EEGs failed to report or to engage multisectoral or multidisciplinary developers, and approximately 14 percent did not follow or did not report any recommended step as part of the development process. These results are in line with those reported by the systematic review of EEGs in LMICs, whereby countries with the lowest quality scores for the two domains had the most methodological weaknesses and gaps observed (Reference Dacccahe, Rizk and Dahham7). Unfortunately, to our knowledge, the quality of the development process of EEGs in developed countries has not been previously assessed. Hence, a comparison with our results cannot be performed.
To our knowledge, this systematic review was the first to explore the development process of EEGs. Providing such information is crucial for guidelines developers, especially in the context of LMICs, where EEGs are increasingly being developed, and international organizations are supporting these countries in the development of guidelines. One major strength is that we followed both a predefined protocol and the recommended methods for reporting search strategies and systematic reviews (the PRISMA 2020 Statement). The search was broad and sensitive, without restrictions for language or publication date. Full-text screening and data extraction were done following best practices (in duplicate and using predefined forms). Two reviewers assessed the quality independently, following an in-depth overview of the AGREE-HS user manual and related documents.
One potential limitation to this work is that it could not include the EEGs published by Iran. A citation of these guidelines was identified on the ISPOR Web site; however, the full guidelines were inaccessible, even after contacting the ISPOR Iranian Chapter. Moreover, as for all systematic reviews, it is always possible that our search could have missed some guidelines. In addition, our quality assessment concerned what was presented in the guidelines and their related documents, and the development may be slightly different from what was reported. Finally, one limitation pertains to the absence of a threshold for differentiating between the high, moderate, and low quality of included studies. This limitation is inherent in the AGREE-HS tool, given the lack of an empirical basis for defining thresholds (11).
Conclusion
The current work fills a gap in the research pertaining to the development of guidelines in LMICs (Reference Dacccahe, Rizk and Dahham7;Reference Fretheim, Schünemann and Oxman46;Reference Schünemann, Fretheim and Oxman51). We highlight the scarcity of published information related to this process and the suboptimal quality of included studies. This paper pinpoints the importance of including a multisectoral capacity and the relevance of a transparent, explicit, and systematic approach in developing and reporting on the development process of EEGs in LMICs. This work provides relevant material that assists international organizations—for example, WHO, GIN, AGREE enterprise—and supports guidelines developers in LMICs to identify the best practice for developing EEGs that fit their national context, and recommends a transparent approach in designing and reporting methods for the development of guidelines. Future studies should also investigate the barriers as well as facilitators that developers of guidelines faced while developing and setting their guidelines.
Supplementary Material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462322000186.
Conflicts of Interest
The authors have no conflict of interest.
Author Contributions
All authors were involved in the concept and design, reviewed and edited the manuscript, and approved the final version of the manuscript. C.D. performed the searches and drafted the manuscript. C.D. and R.R. conducted the title and abstract screening. C.D., R.K., and R.R. conducted the full text screening, performed the data extraction, and conducted the quality assessment. The content of this manuscript has not been published nor is it being considered for publication elsewhere.
Data Availability Statement
The authors declare that all the data supporting the findings of this study (i.e., search strategy and the information extracted from the studies included in this review) are available within the article and the Electronic Supplementary Material.
Open access
