Introduction
Health technology assessment (HTA) is defined as a multidisciplinary process that uses explicit methods to determine the value of a health technology at different points in its lifecycle. The purpose is to inform decision-making in order to promote an equitable, efficient, and high-quality health system (Reference O’Rourke, Oortwijn and Schuller1).
In Ukraine, HTA is enshrined in a legislative framework, and the main decision criteria used for determining the value of health technology include comparative effectiveness, safety, cost-effectiveness, and budget impact. In line with this framework, the HTA department of the State Expert Centre (SEC) of the Ministry of Health (MOH) of Ukraine conducts HTAs upon the request of the MOH or industry for inclusion in the national essential medicine list or the nomenclature, conducts dossier reviews for inclusion in the state formulary, and develops methodological frameworks (e.g., HTA guidelines, budget impact analysis) that are submitted to the MOH for approval (Reference Topachevskyi, Leleka, Oleshchuk, Zaliska, Kovtun and Piniazhko2).
Because HTA as a formal instrument is relatively new in Ukraine, capacity building among the main actors was needed and supported by the Management Sciences for Health, Inc. (MSH), which is implementing the international technical assistance project “Safe, Affordable, and Effective Medicines for Ukrainians” (SAFEMed) in Ukraine with the financial support of the United States Agency for International Development (USAID). HTA capacity building can be defined as “the process by which individuals and organizations develop or strengthen abilities related to understanding, providing input to, conducting, or utilizing HTA for health policy and decision-making, as well as, developing awareness and support in the environment within which HTA is being used” (Reference Pichler, Oortwijn, Ruether and Trowman3). Actors involved in capacity building activities in Ukraine include staff of the HTA department, experts in the MOH, National Health Service of Ukraine (NHSU) and other governmental bodies, and other stakeholders, such as patients, clinicians, hospital managers, academics, and industry. Although earlier activities had strengthened capacity and awareness of HTA in Ukraine, after full legalization of the HTA function in 2020 key stakeholders expressed interest in a broader training program based on needs of HTA doers, users, and trainers. The cadre of trainers is important to optimize sustainability and institutionalize HTA capacity building within Ukraine. SAFEMed subcontracted Radboud university medical center (Radboudumc) to co-lead the design and delivery of the training program. Work commenced in July 2022 for a planned duration of one year. This paper describes the approach taken to develop and implement the HTA capacity building program and presents the results of the program in terms of knowledge gained by the participants. This description can serve as inspiration and guidance for those who want to develop HTA capacity building activities in countries that are starting to institutionalize HTA.
Approach
Because there is a lack of literature regarding practical approaches to implement HTA capacity-building activities (Reference Abraham, Kvamme and Magrin Sammut4), our approach was based on our practical experience with providing formal and informal HTA training around the globe. It comprised four elements: assessment of training needs, developing and providing the training program, evaluating whether participants gained knowledge and skills, and developing and executing a train-the-trainers program.
Training needs
For the identification of training needs, we developed an online survey based on the validated HTA skills assessment survey of Bidonde et al. (Reference Bidonde, Meneses-Echavez, Asare, Chola, Gad and Heupink5) and a survey tool we used to assess HTA knowledge and skills in Moldova and Tanzania (Reference Surgey, Mori and Baltussen6). The survey captured biographical information, level of experience with HTA (number of times conducted or used), level of confidence in and training needs (open question) related to specific value dimensions of HTA (e.g., clinical effectiveness, cost-effectiveness, patient-relevant outcomes (Reference Staniszewska and Soderholm Werko7)), and related methods (e.g., budget impact analysis, ethical analysis), as well as five (true/false) statements to test specific HTA knowledge. The English version of the survey is provided as Supplementary File 1. The SAFEMed team in Ukraine provided the translation and logistics of the training program, reviewed the survey, translated it into Ukrainian, and identified and invited potential participants of the training program. The translated survey was transferred to Google Forms and sent out via email. The survey invitees had 10 days to respond, and multiple reminders were sent to optimize the response rate. We used basic descriptive statistics (frequencies, presented as percentage) to summarize the findings per target group (doers, users, and trainers).
Training program
The 160 hours (20 days) training program consisted of lectures, self-study assignments (reading articles or literature and watching videos), and practical exercises to develop skills (see Supplementary File 2). The five training components (modules) were based on the HTA competency framework developed by Mueller et al. (Reference Mueller, Gutierrez-Ibarluzea, Chiumente and Oortwijn8), the specific framework core competencies for ethics experts in HTA (Reference Refolo, Bond and Bloemen9), and the training needs identified via the survey (Supplementary File 1). The number of days per module was based on the requirements of using HTA in Ukraine and the decision-making criteria that apply, giving certain modules (comparative effectiveness and cost-effectiveness/budget impact) more weight than others (introduction to HTA, qualitative evidence synthesis, and ethical and other issues). The specific content of each module included state-of-the art HTA theory, methods, and tools and was developed using available materials from compulsory courses of the HTA track as part of the Biomedical Science Program at Radboud University, recent literature, international HTA reference guidance, and the HTA Core Model® and Cochrane Online Handbooks (10). Participants were provided with a course manual (translated into Ukrainian) prior to the start of the training program comprising an overview of the training, details on the training modules’ content (learning goals, background material, study time, and learning activities for each training day), and the course coordinators/lecturers (WO, WK) and other contributors (an information specialist and six senior lecturers from Radboudumc). The instructional design of the training program was based on active and participatory learning, which is relevant for professionals already working in the field. For this purpose, the examples and exercises in the training were made context-specific whenever possible. Due to the war in Ukraine and resulting limitations on the ability to meet in-person, the training was provided online via Zoom.
Evaluation of participant’s gained knowledge and skills
We conducted an evaluation of each participant’s gained knowledge and skills per training module. For this purpose, participants were asked to rate each learning goal of a specific session of the module. Before the training, we asked each participant to indicate their level of confidence regarding each learning goal using the following categories:
-
• No knowledge on this topic
-
• I have heard of the topic, but do not feel confident to do it
-
• I feel slightly confident to do it
-
• I master this topic
At the end of the module, participants were asked to rate each formulated learning goal again. This was done through polling in Zoom during the training sessions of three modules and via an online survey for two modules (clinical effectiveness and cost-effectiveness/budget impact) due to the substantial number of learning goals of these modules. Respondents could fill out the survey in their own time and received follow-up communications in case of non-response. We used basic descriptive statistics (frequencies, presented as percentages) to summarize the findings.
Train-the-trainer program
Train-the-trainer is a framework for training potential instructors or subject matter experts to enable them to train other people. This means that we were looking for participants with ample expertise and experience in HTA (i.e., they should feel confident to do it/master the topic). Potential trainers for future HTA training in Ukraine were identified via the training needs assessment survey (Supplementary File 1), in which we asked the respondents to identify themselves as a doer, user, or trainer. In addition, SAFEMed experts identified additional trainers based on knowledge of capable doers within the SEC and MOH. The train-the-trainer program was tailored to the trainers’ personal learning goals to better deliver training in HTA. Prior to the start of the train-the-trainers program, we asked the identified trainers to what extent they felt confident (not at all; to some extent; completely) in delivering the content of each module and in their training delivery skills, as well as about their training needs and personal learning goals (open question). We used a written questionnaire (see Supplementary File 3) that was translated and distributed by e-mail to identified trainers. The training needs that emerged from the responses served as input for the design of the train-the-trainers program.
Results
Training needs
The survey was sent out to forty-eight identified HTA professionals of which forty completed the survey, resulting in an overall response rate of 83 percent. Table 1 presents the general descriptive statistics of the survey participants, which shows that the majority of respondents were female, between 25 and 60 years old with up to 15 years of work experience, and most did not serve on a governmental or ministerial committee. The doers, users, and trainers represented a variety of stakeholders from different governmental, regional, and health care organizations, industry, and academics. The highest degrees or levels of education completed were in medicine, pharmacy, epidemiology, and economics.
Table 1. General characteristics of survey participants (n = 40)
In Table 2, we present the experience of survey participants with conducting and/or using HTA and related research. Overall, more than 50 percent of the twenty-one users indicated not to have conducted HTA (57 percent), systematic reviews (90 percent), economic evaluations (71 percent), health-related quality of life studies (86 percent), studies focusing on ethical, social, cultural, and/or legal issues (100 percent), organizational and/or environmental aspects (90 percent), or those targeting relevant outcomes for patients, relatives, carers, and/or the general public (76 percent). Similarly, more than half of the twenty-one users mentioned not to have used the results of these kinds of research, except for 62 percent who had used economic evaluations.
Table 2. Experience of survey participants with doing or using HTA (n = 40)
For the seventeen doers we see a different pattern. More than 59 percent of doers have conducted HTAs or economic evaluations, but they have less experience conducting systematic reviews (29 percent), health-related quality of life studies (6 percent), studies focusing on ethical issues (0 percent), social, cultural, and/or legal issues (6 percent), organizational and/or environmental aspects (12 percent), and those targeting relevant outcomes for patients, relatives, carers and/or the general public (6 percent). Regarding use, almost two-third (65 percent) of the respondents indicated to have used HTA reports, and even a higher percentage used systematic reviews (76 percent), economic evaluation (71 percent), or health-related quality of life studies (76 percent). Concerning studies focusing on ethical issues, social, cultural, and/or legal issues, organizational and/or environmental aspects, and those targeting relevant outcomes for patients, relatives, carers, and/or the general public, the percentage of doers that indicated their use is much lower; 24 percent, 12 percent, 29 percent, and 29 percent, respectively.
The two trainers indicated to have experience with both conducting and using HTA and related research, except for the conduct of health-related quality of life studies, studies that assess ethical issues, and studies assessing relevant outcomes for patients, relatives, carers, and/or the general public.
The results can be explained by the extent to which the respondents indicated to be familiar with HTA; 62 percent of the twenty-one users, 71 percent of the seventeen doers and both trainers mentioned to be familiar with HTA. As about 30 to 40 percent of the doers and users appear not familiar with HTA, there was a need for a general introduction to HTA.
The survey results also enabled us to identify specific knowledge and skill gaps, which were in line with the experience described above. A high percentage of participants reported “No knowledge on this topic” or “I have heard of the topic, but do not feel confident to do it” with regard to the following topics: (1) searching for studies and synthesis of evidence; (2) ethics; (3) patient and public involvement; and (4) health economics. These gaps have been translated into specific learning goals of the different training modules and are described below.
Training program
The online training program consisted of five modules: Introduction to HTA, Comparative effectiveness, Cost-effectiveness/budget impact, Qualitative evidence synthesis, and Ethical issues and other relevant HTA aspects (see Supplementary File 2). The training was provided between January 24 and May 17, 2023, with a study load of about 1.5 days per week. Forty-eight participants were initially selected to participate in the entire training program, while additional stakeholders were invited to attend the first module “Introduction to HTA,” which was anticipated to be beneficial for them.
During the entire period of the training, participants were encouraged to ask questions regarding any issue they might face when implementing HTA in practice. The queries that were received were addressed by Radboudumc lecturers who coached the participants through email and online communication. Finally, 2 months after the training program, two online question and answer sessions with Radboudumc lecturers were organized for all participants. On July 10, 2023, the focus was on questions related to the comparative effectiveness and cost-effectiveness modules, whereas on July 18, 2023, questions related to the other three modules were addressed.
Evaluation of training program
In total, seventy-five persons participated in the training program with varying levels of attendance per training session (between thirty and seventy-five participants).
The following sections include module-specific results regarding how confident participants felt about the specified learning goals before and after the training.
Module 1: Introduction to HTA.
Fifty-two respondents rated their confidence in the three learning goals of module 1 before the training, while forty-three repeated their ratings at the end of the training.
Table 3 shows a learning effect of the training module: the percentage of participants that had no knowledge or were not confident to do it overall decreased after the training, while the percentage that felt slightly confident to do it or master the topic increased. The only exception is with regard to the third learning goal (Identify the various steps in the HTA process, and the requirements in terms of quantitative and deliberative analysis [appraisal], where the percentage that master the topic decreased (from 6 to 2 percent). This could be explained by the lower number and potentially different respondents (forty-three after the training compared to fifty-two before), as well as the broader audience that attended (in addition to the forty-eight identified users and doers of HTA). The fact that not all respondents may have attended all training sessions could also explain why a small number still reported having no knowledge of the learning goals after the training (2–5 percent).
Table 3. Responses and number of respondents per learning goal of the training modules Introduction to HTA and comparative effectiveness, before and after the training
Module 2: Comparative effectiveness.
In Table 3, we also present the responses of the participants with regard to the fourteen learning goals of this module before and after the training. The number of respondents varied between twenty-one and thirty-one for the initial rating (at the start of the training), while the number of respondents after the training was fifty for all learning goals. The latter number might be higher because different subsets of people participated in the training sessions and the polling at the start of each session, while after the training participants could fill out the survey in their own time and we sent reminders to submit their ratings.
The results presented in Table 3 indicate a learning effect of the training for all learning goals of the module on comparative effectiveness: the percentage of participants that had no knowledge or were not confident to do it overall decreased after the training, while the percentage that felt slightly confident to do it or master the topic increased. The only exception is with regard to learning goal seven (Define what data to extract and create data extraction forms) where a small percentage (2 percent) still felt that they had no knowledge with regard to the learning goal after the training.
Module 3: Cost-effectiveness/budget impact.
In Table 4, we present the responses of the participants with regard to the eighteen learning goals of the module on cost-effectiveness/budget impact before and after the training. The number of respondents varied between sixteen and thirty-four for the initial rating (at the start of the training), while the number of respondents after the training was thirty-seven for all learning goals.
Table 4. Responses and number of respondents per learning goal of the training module Cost-effectiveness/budget impact analysis, before and after the training
As with the first two modules, a learning effect of the training for all learning goals of the module on cost-effectiveness/budget impact can be observed from Table 4. The percentage of participants that had no knowledge or were not confident to do it overall decreased after the training, while the percentage that felt slightly confident to do it or master the topic increased. Also, no participant indicated that they lack complete knowledge on this topic after the training.
Module 4: Qualitative evidence synthesis.
In Table 5, we present the responses of the participants with regard to the five learning goals of the module on qualitative evidence synthesis before and after the training. The number of respondents varied between fifteen and seventeen for the initial rating (at the start of the training), while the number of respondents after the training varied between nineteen and twenty-two.
Table 5. Responses and number of respondents per learning goal of the training modules Qualitative evidence synthesis and Ethical issues and other relevant HTA aspects, before and after the training
Again, we see a learning effect of the training. The percentage of participants that had no knowledge or were not confident to do it overall decreased after the training. No participant indicated that they lack complete knowledge on this topic. Notably, the participants that felt slightly confident to do it increased remarkably, while the percentage that indicated mastery of the topic increased for two out of three learning goals (explain the different approaches to patient-related evidence and explain the use of patient-based evidence in HTA) after the training.
Module 5: Ethical issues and other relevant HTA aspects.
In Table 5, we also present the responses of the participants with regard to the nine learning goals of the module on ethical issues and other relevant HTA aspects before and after the training. The number of respondents varied between sixteen and twenty-one for the initial rating (at the start of the training), while the number of respondents after the training varied between eleven and sixteen.
In this module of the training program we also see a learning effect. The percentage of participants that had no knowledge or were not confident to do it overall decreased after the training. It is also notable that some participants (6 percent to 14 percent) indicated that they (still) lack complete knowledge on this topic after the training. Notably, the participants that felt slightly confident to do it increased remarkably, while the percentage that indicated mastery of the topic increased for four out of nine learning goals after the training with no change for the other five goals.
Train-the-trainer program
With regard to the train-the-trainers program, we invited twelve HTA professionals all of whom participated in the above-described training program. Half of the trainers were academics and the other half were employees of the HTA department of the SEC. Eleven out of twelve trainers (92 percent) responded to (parts of) the written questionnaire (Supplementary File 3) to evaluate their confidence in providing the modules. The results showed that respondents felt confident to some extent or completely to provide one or more modules (see Supplementary File 4).
Specifically, a majority (seven) of the trainers felt confident to provide an introductory course module on HTA, while for the other modules only one or two trainers felt completely confident to teach the content.
The identified training needs were related to gaining more practical experience with some of the topics of each module (content-related) and learn more skills related to giving presentations, guiding group work and discussion, effectively involving participants, preparing self-studies, and designing assignments.
For this purpose, we organized six online sessions of 3 hours each between May 30 and July 15, 2023 for the eleven trainers (with ten participants on average attending each session). General topics such as defining learning goals and related actions, storytelling, and interactive learning were discussed. Additionally, trainers addressed and worked on their personal learning goals to (better) deliver training in HTA. The program also included 6 hours of self-study.
Discussion
In this paper, we provide an overview of the approach taken to develop and implement a tailored HTA training program in Ukraine, and we present the results in terms of knowledge and skills gained by the participants. Overall, the results demonstrate an acquisition of confidence with the subject matter among participants for all training modules. As such, there seems to be a rationale for continuing the training program, which consists of lectures, self-study assignments (reading articles or literature and watching videos), and practical exercises to develop skills.
As stated by Pichler et al., (Reference Topachevskyi, Leleka, Oleshchuk, Zaliska, Kovtun and Piniazhko2) capacity building should address three levels: the individual level (i.e., training to conduct HTA), organizational level (i.e., development of processes to make use of HTA results more effectively), and the environmental level (i.e., awareness of the added value of HTA for decision-making). The provided training primarily focused on the individual level (HTA doers), although it also addressed capacity building at the organizational level (by involving users and trainers) and the environmental level (e.g., by inviting a broader audience to the introductory module on HTA). However, we realize that the provided in depth knowledge about the different methodologies used in HTA is not necessary for all user groups. In addition, we noticed that the competency levels (knowledge and skills) between user and doers in Ukraine varied (see Table 2). Therefore, it could be beneficial to offer separate training (modules) for current and future users on a basic knowledge level and for doers on an advanced level. This is increasingly important because the overall ecosystem of HTA continues to expand in Ukraine while the international HTA knowledge base and the required related competencies are also expanding (e.g., on the use of artificial intelligence in conducting HTA).
It is encouraging that the vast majority of the trainers indicated feeling confident to provide an introductory course module on HTA. It could be valuable for offering such training broadly within Ukraine in order to continuously raise awareness of HTA. As noted by other authors (Reference Abraham, Kvamme and Magrin Sammut4), while awareness can be increased through reports and presentations, decision-makers must take action to ensure the successful implementation of the HTA process. Ongoing development of knowledge and skills at the individual level remains essential, including training for decision-makers that rely on HTA evidence. For the other modules, trainers need to gain more experience before delivering the training in practice. In particular, repeated application would be necessary to build the confidence required for teaching. It should be noted, however, that it is common in multidisciplinary fields such as HTA to compose teams of researchers and lecturers with different backgrounds and expertise covering all domains considered important in HTA. This would mean that it is not necessary that a potential trainer feel confident in teaching all modules.
The fact that the training sessions were held twice a week (4 hours each) during business hours may have caused overlap or conflicts with other meetings of participants, including academic trainers. Shorter sessions – ideally no longer than 2.5 hours – would be more beneficial for all participants in order to stay focused. Furthermore, the context of the war in Ukraine and inability of Radboudumc trainers or MSH international experts to be in Ukraine may have limited its effectiveness. Offering the training program face to face whenever feasible would help participants and trainers get to know each other from the outset, fostering engagement and peer learning throughout the program.
To provide sustainable capacity building, it would be beneficial to build on the existing HTA expertise and infrastructure in Ukraine. It is recommended to explore whether one or more licensed universities in Ukraine, which already offer related study programs such as pharmaco-economics, are willing to provide (parts of) the training program as (part of) a Master program or as advanced (postgraduate) course(s). In the national public register of licensed educational institutes in Ukraine (i.e., Unified State Electronic Database on Education) the following universities would qualify: Danylo Halytsk Lviv National Medical University; Horbachevsky Ternopil National Medical University; and Bogomolets National Medical University. Furthermore, collaboration with the HTA department or other key stakeholders (NHSU, other universities/identified trainers, industry, MOH) would be beneficial to ensure that students could work on internship projects/topics of interest to stakeholders or learn from practice during their training program. Finally, it is recommended – given Ukraine’s European Union (EU) candidate status - to consider the implications of the EU HTA Regulation (HTAR). Per 1 January 2025 joint scientific consultations and joint clinical assessment are conducted at EU level and these activities require specific competencies. For this purpose, the European Commission is funding a specific EU HTAR capacity building program via the EU4Health Program. In order to comply with the key principles of the HTAR and to enhance capacity building in this area, enrolment in this program, as well as funding from international donors which support HTA institutionalization, such as the World Bank could be explored.
Conclusion
The SAFEMed experience of training HTA users, doers, and trainers provides a solid foundation and insights that will be useful for planning and delivering future HTA capacity development efforts in Ukraine. It may also serve as inspiration for other countries aiming to increase HTA capacity. Continued institutional and financial support for capacity building is recommended as the sustainable development of knowledge and skills takes time.
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/S0266462325103243.
Acknowledgments
We thank all SAFEMed colleagues, trainers from Radboudumc, staff of the HTA Department and trainees for their participation in this study.
Funding statement
This research was funded by Management Sciences for Health through the United States Agency for International development (USAID), reference number RFP-2021-11-18. USAID did not have any influence on the development of the data collection methods, analysis, and reporting.
Competing interests
The capacity-building grant from MSH for the conduct of the study was paid to Radboud university medical center.
Open access
