Health technology assessment (HTA) is a multidisciplinary process that uses explicit methods to determine the value of a health technology at different points in its lifecycle. The purpose is to inform decision making in order to promote an equitable, efficient, and high-quality health system. This process includes governance and structure, scoping, assessment, appraisal, and implementation and monitoring (Reference Børlum Kristensen, Husereau, Huić, Drummond, Berger and Bond1). To guide HTA agencies in a way that gives legitimacy to their processes and ultimate decisions, evidence-informed deliberative processes (EDPs) were developed (Reference Baltussen, Jansen, Bijlmakers, Grutters, Kluytmans and Reuzel2).
EDPs emerged from two conceptual frameworks: multi-criteria decision analysis, which requires to explicitly identify the decision-making criteria and how they are going to be considered; and the Accountability for Reasonableness framework which provides conditions for fair and legitimate decision making. The main underlying premises of EDPs are that stakeholder participation and transparency contribute to legitimizing the institution, the process and the decisions. To support the use of EDPs, we developed a practical guide targeting HTA agencies (Reference Oortwijn, Jansen and Baltussen3). The guide is centered around the EDP framework that consists of five steps: installation of an appraisal committee, selecting technologies and criteria, assessment, appraisal, communication and appeal, and contextual factors for HTA development.
The operationalization of these steps for implementation in a specific country or region depends on the context and the guide offers specific advice depending on the level of HTA development (nascent or mature HTA system). As such it is not meant as a blueprint.
To identify the level of use of EDPs around the globe, we surveyed members of the International Network of Agencies for Health Technology Assessment (INAHTA) (4). The INAHTA members are mainly located in high-income countries. A key finding from the survey study was that EDPs can support the decision-making process of HTA-agencies but may also be relevant for countries that have not (yet) established an agency (Reference Oortwijn, Jansen and Baltussen5). The study results also indicated that there is potential for use of EDPs in low- and middle-income countries (LMIC). As there is interest in assessing the global utility of EDPs (Reference Culyer6), we aimed to establish the feasibility of EDPs application in LMIC. We therefore decided to collect views and experiences from HTA organizations and/or experts in LMIC regarding to what extent they applied the steps and elements stipulated in the EDP framework, identify their need for guidance regarding elements of the framework, and to learn about what they perceive as best practices. In this article, we present the main findings from the survey among experts from LMIC and provide some strategies to support the successful implementation of EDPs in the context of LMIC.
Methods
We adapted the original survey (Reference Oortwijn, Jansen and Baltussen5) that targeted INAHTA members to specifically address LMIC context. The process was that a senior researcher (LK) with ample experience in and originally from a LIC independently reviewed the original survey. She was not involved in the development of the original survey, and was asked to provide feedback in order to make the survey applicable to LMIC without a formal HTA agency. Her feedback was discussed with the study team, and consequently included. The changes included the provision of additional information about HTA in the introduction of the survey, and the addition of seven questions. This concerned two general questions about the respondent's familiarity with HTA and the involvement of their organization in HTA. Also, we specifically asked about the perceived legitimacy of the appraisal committee (two questions) and three questions were added about the availability of specific documents in the public domain. The questionnaire for the semi-structured online survey was developed based on the EDP framework, consisting of elements that reflect each step of the framework and the contextual factors for HTA development. We asked respondents (a) about the extent to which elements of the EDP framework were present and whether there was a need for further guidance, (b) to list elements that they felt were missing and (c) to list best practices regarding each part of the framework. The questionnaire is provided as Supplementary Material.
We used different strategies for recruiting potential survey respondents. First, we contacted the Health Technology Assessment International (HTAi) Interest Group on Developing Countries (DCIG). The steering committee of DCIG provided consent to invite their members to participate in the survey. Second, Web sites of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) were reviewed to collect information about the committee members of the regional chapters. Third, we listed focal point contacts and contacts of relevant national HTA agencies provided by the 2015 World Health Organization (WHO) HTA survey report on assessing the status of HTA in Member States (7). Finally, we identified potential respondents through our own HTA networks in LMIC. In total, we sent invitations to 416 potential respondents of which most were located in India, China, and South Africa (respectively 38, 33, and 23 individuals).
Respondents that did not respond to the first email were sent two reminders. The first reminder was sent 2 weeks after the initial invitation and a second reminder 1 week after the first reminder. The data were collected, using an online tool CheckMarket, between 21 May and 1 September 2019.
Two survey respondents explicitly wished not to disclose their affiliation. Therefore, we provide the results anonymously. We used basic descriptive statistics (frequencies, presented as percentage), derived from the CheckMarket tool, and performed qualitative analyses of the open questions to summarize the findings.
Results
Response Rate
From CheckMarket, we retrieved that 259 persons opened the survey and received seventy-one responses (response rate of 27 percent). We received fifty fully completed survey forms and twenty-one only partly completed the survey. Survey respondents represented thirty-four countries, of which thirty-two were LMIC. Five survey forms (two completed and three partly completed) were excluded from the analysis; of these, three respondents were from (two) high-income countries, one person responded twice (complete and partly), and the other had already participated in the study among INAHTA members. Hence, this article is based on the analysis of forty-eight respondents who completed the survey and eighteen respondents who provided meaningful responses to some of the questions (Supplementary Table 1). Therefore, the total number of responses reported in this article will vary according to the number of complete responses we received for a particular question. The top three responding countries included India (n=5), South Africa (n=5), and Ghana (n=4), while we received three responses from experts located in Bulgaria, Burkina Faso, Colombia, Ecuador, Romania, or Ukraine.
Familiarity with HTA
We asked respondents about the extent to which they are familiar with HTA, and specifically with HTA in their country. Of the fifty-nine completed responses to this question, 85 percent (n = 50) reported that they were either very familiar or familiar with HTA in their country. Of the sixty responses to the question regarding the involvement of their organization in HTA, thirty-eight experts (63 percent) indicated that they or their organizations were involved in HTA. Most of them indicated being involved in HTA through conducting HTA research and/or capacity strengthening (training and education).
Contextual Factors
Respondents were asked about the presence of factors that are supportive of HTA development (i.e., factors reflecting the linkage between HTA policy and practice; the level of institutionalization of HTA; and the ability to networking and capacity building), and whether these factors needed guidance. Overall, the respondents (n = 55) mentioned that the contextual factors were not present or present to some extent in their countries/regions. Consequently, between 60 and 75 percent of the respondents felt that guidance was needed for specific factors (Supplementary Table 2). Thirty-six respondents (55 percent) answered the question about which HTA practice serves as best practice. Of these respondents, 19 percent mentioned HTA methodology of other countries, such as the EUnetHTA Core Model and those from France, Germany, and the United Kingdom (UK).
Installation of an Appraisal Committee/Stakeholder Panel
With regard to the existence of an appraisal committee/stakeholder panel and related guidance, it became apparent that 61 percent of the fifty-four respondents mentioned to have such a committee/panel installed in their country. Additionally, 54 percent of the respondents indicated the (appraisal) committee to be legitimate. Of the respondents, twenty-three (43 percent) provided reasons for the committee not being legitimate, including the absence of an HTA committee (n = 5) or a formal HTA mechanism (n = 3). Absence of multidisciplinary stakeholder engagement and participation was also mentioned by some respondents (n = 4).
According to approximately one-third of the fifty-four respondents, a guideline or document that describes the roles and responsibilities of the committee/panel, and its procedures appears to be present (respectively, 30 and 28 percent). A document that describes the composition, terms, and selection of members, as well as the roles and responsibilities of stakeholders involved in the process was present according to, respectively, 28 and 20 percent of the respondents. The respondents expressed the need for guidance with respect to various elements of an appraisal committee/stakeholder panel, ranging from 63 to 70 percent for the different elements (Table 1). Six respondents (11 percent) also explicitly mentioned that participation of multiple stakeholders, with emphasis on patient involvement, should be enhanced.
Views regarding the presence of elements related to an appraisal committee/stakeholder panel, and the need for guidance (n = 54)
Three respondents (6 percent) indicated that the guidelines/document describing the composition, terms, and selection of members of the HTA (appraisal) committee were publicly available. Thirty-one respondents (47 percent) completed the question about the respondents’ awareness of any HTA practice that could serve as best practice. The National Institute for Health and Care Excellence (NICE) in the United Kingdom was mentioned most often as such an exemplar HTA practice (13 percent).
Selection of Health Technologies in Need for Assessment
The majority of the fifty-two respondents (81 percent) indicated that a horizon scanning system did not exist in their country or region. Furthermore, the elements in relation to selecting technologies and criteria (i.e., existence of an early warning system/horizon scanning, and existence of a scoping procedure) were most often not present according to the majority of the respondents. Subsequently, the majority of the respondents indicated that guidance was needed for all specific elements (ranging from 79 to 85 percent) (Table 2).
Views regarding the presence of elements related to selecting technologies and criteria, and the need for guidance (n = 52)
Respondents did not indicate whether any of the guidelines/document was publicly available. Twenty-six respondents (39 percent) answered the question regarding best practices related to horizon scanning and scoping, but only one respondent indicated an existing best practice (EUnetHTA); most were not aware of best practices in this field.
Assessment and Appraisal
The findings show that guidelines/documents on how to undertake the HTA in terms of data collection and analysis and a template to report the HTA results were most often indicated to be present to some extent (44 percent; n = 50). The existence of an approach for stakeholder consultation to review evidence reports was often mentioned not to be present (52 percent). In line with this, the respondents overall felt that there was a need for guidance, ranging from 72 to 82 percent for the different elements (Table 3). One respondent (2 percent) indicated that a tool/template for reporting and summarizing the (quality of the) evidence per relevant aspect as part of HTA (assessment) is publicly available in his/her country. Of those who answered the question on best practices (n = 29; 44 percent), three respondents (10 percent) mentioned international HTA practices, such as NICE and the Canadian Agency for Drugs and Technologies in Health (CADTH).
Views regarding the presence of elements related to conducting and reporting assessments as well as to appraisal, and the need for guidance (n = 50)
Of the fifty respondents that completed the questions related to appraisal, 58 percent indicated that there was a formal framework for appraisal/HTA decision making in their countries/regions. However, less than 25 percent indicated that there was a guideline/document describing the elements related to appraisal (Table 3). Consequently, more than 78 percent of the respondents identified a need for guidance with 14 percent explicitly mentioning the need for guidance on how to apply appraisal processes, especially multi-criteria decision analysis and how to explicitly involve stakeholders in a transparent way. One respondent (2 percent) indicated that guidelines/document describing the process, roles and responsibilities of stakeholders, or methods were publicly available. Only, 39 percent of the respondents (n = 26) answered the question on best practices but most indicated not to be aware or did not mention a specific practice.
Communication and Appeal
About half (54 percent) of the fifty respondents indicated that the decisions and the underlying reasons were made public. Of the respondents, 42 percent specified modes through which decisions were publicized sources. These included webpages of Ministry of Health, television, social media, or in print. However, the respondents also indicated that guidelines or documents describing relevant aspects of communication and appeal were less present. Not surprisingly, over 80 percent of the respondents indicated a need for guidance regarding these aspects (Table 4). Notably, only five respondents (10 percent) considered stakeholder participation and involvement in the HTA process as important in this step. One respondent (2 percent) indicated that guidelines/document describing the mechanism(s) for appeal, how to propose revisions, and to receive a reasoned response was available. With regard to best practices, twenty-six participants responded to this question of whom two respondents (8 percent) indicated that communication and interaction with stakeholders were of utmost importance.
Views regarding the presence of elements related to communication and appeal, and the need for guidance (n = 50)
Discussion
This study collected views and experiences from experts in LMIC about the extent to which elements of the EDP framework were present and whether there was a need for further guidance. There are some limitations to this study. While the intention was to get input from many LMIC, the response rate (27 percent) suggests that some selection bias might be present. We are also aware that certain steps and/or elements can be more important than others, depending on the contextual factors/country context. For example, the mandate of an organization might influence the remit and scope of conducting HTAs. Therefore, the findings should be mainly viewed as indicative for the extent to which elements of the EDP framework were present in LMIC.
The finding that contextual factors to support HTA development are overall not present or present to some extent is in line with the findings of other HTA surveys in LMIC. For example, in 2015, the WHO mapped the existence, capacities and requirements for HTA in LMIC (8). They concluded that there is often limited awareness of HTA methods, scarce human resource capacity, a lack of HTA institutionalization, lack of clear mandate from legal and policy authority, and limited political support.
However, it is interesting to note that the majority of the respondents mentioned to have an appraisal committee/stakeholder panel in place. This might be due to the fact that several of LMIC have existing structures through which stakeholders can participate in health system decision making (Reference Panda and Thakur9). For example, many LMIC have established public health committees, health unit management committees, or National Immunization Technical Advisory Groups.
Furthermore, respondents indicated that specific guidance was needed for all elements related to selecting technologies and criteria, including scoping, assessment, appraisal, and communication and appeal. With the exception of guidance for the assessment phase, this was also found in our survey among INAHTA members (Reference Oortwijn, Jansen and Baltussen5).
The lesson that we learned from this survey is that EDPs have the potential to provide the necessary steps for legitimate decision making using HTA in LMIC. As we have recently argued in a commentary on the potential use of EDPs for low-income countries, successful implementation of EDPs in HTA would be augmented by considering how to deal with the lack of transparency and participation culture in some contexts, and by providing additional guidance on how to set up HTA organizations, especially for countries that lack designated HTA organizations (Reference Kapiriri, Baltussen and Oortwijn10). For this reason, monitoring and evaluation of implementing EDPs in different contexts are of utmost importance. Implementation projects, such as currently underway in Kazakhstan and Pakistan, will provide empirical information about the utility and feasibility of applying EDPs and would inform further refinement of guidance for EDPs in different contexts around the globe. In this vein, the WHO refers to our practical EDP guide (Reference Oortwijn, Jansen and Baltussen3) in relation to universal health coverage and priority benefits package (11).
Conclusions
This survey among experts in LMIC intended to further develop the guide on EDPs and to explore to what extent the countries were applying the steps and elements stipulated in the EDP framework. The results provide useful and additional information for further developing practical guidance on the implementation of EDPs around the globe. We found that the respondents indicated that their countries are in the early phases of applying (elements of) EDPs. It became clear that contextual factors to support HTA development are overall not present or present to some extent. Interestingly, the majority of the respondents indicated that an appraisal committee/stakeholder panel is in place. However, the majority of the respondents felt that guidance is needed for all steps. It is promising that respondents clearly noted that guidance regarding transparent processes and stakeholder involvement is needed; these are the underlying values and principles of EDPs. The results can serve as a baseline measurement for future monitoring and evaluation of the level of EDP application in LMIC. This, as well as reference cases from LMIC, could support the countries in improving their processes and enhancing legitimate decision making when using HTA.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462320000549.
Acknowledgements
The work described in the manuscript is conducted as part of a research program that rethinks the theory and tools to value interventions for priority setting in health care and is supported by a grant from the Netherlands Organisation for Scientific Research (in Dutch: NWO). WO and SO were supported by this grant. LK was supported by a visiting scholar fellowship of the Netherlands Organization for Scientific Research (NWO). The funding organization did not have any influence on the design, data collection, analysis, and reporting of the study.
Open access
