Introduction
The coronavirus disease 2019 (COVID-19) pandemic has affected all aspects of health systems and the field of Health Technology Assessment (HTA) is no exception (Reference Lorgelly and Adler1). Nonetheless, HTA in Asia has played a significant role in policymaking during the pandemic, either by assessing the value of interventions (Reference Utami, Rendrayani and Khoiry2) or by adapting their research capabilities to answer various policy questions (Reference Ananthakrishnan, Luz and Kc3‒Reference Cai, Lim and Huynh6). A robust and dynamic HTA system is therefore desirable during both peacetime and public health emergencies (Reference Lorgelly and Adler1; Reference Ananthakrishnan, Luz and Kc3; 7‒Reference Miglietta, de Waure and Chronaki9). As we exit COVID-19 and prepare to strengthen our response system for future pandemics, it is crucial to understand the impact of COVID-19 on organizations that conduct HTA-related research in Asia as a priority.
Commitment to Universal Health Coverage (UHC) has increased political buy-in from countries and allowed HTA to flourish across Asia (Reference Teerawattananon, Rattanavipapong and Lin10; 11). It continues to shape national health policies in many countries; however, the field is far from reaching its full potential in Asia (Reference Teerawattananon, Rattanavipapong and Lin10). HTAsiaLink, a network of non-for-profit organizations that conduct HTA-related research, plays an important role in the growth of HTA in the region (Reference Teerawattananon, Luz and Yothasmutra12). Established in 2011 with three founding members, HTAsiaLink now hosts over fifty organizations from twenty member countries. It has three main objectives (i) strengthen individual and institutional capacity in HTA research and integration of HTA evidence into policymaking; (ii) promote collaboration and reduce research waste; and (iii) share best practices among members (Reference Teerawattananon, Luz and Yothasmutra12).
Several studies have studied the impact of COVID-19 on HTA including the speed of regulatory decisions (Reference Mundy and Kearney13; Reference Thomas, Craddy and Foxon14; 18), changes in engagement with the industry, patients, and caregivers, the use of real-world data, and emerging elements of value (Reference Barham and Lovell19), as well as the process of conducting HTA studies (Reference Boss, Bending and Rawson20). A study from Brazil (Reference MMdA, Thabane and LGMd15) explored the impact of the COVID-19 pandemic on the HTA process of the National Commission for the Incorporation of Technologies into the Brazilian health system. The National Centre for Pharmacoeconomics Review Group (Reference Leahy, Hickey and McConnell16) and a global survey (Reference Elvidge and Dawoud17) studied the methodological challenges presented by COVID-19. One commentary highlighted how HTA organizations in low-and middle-income countries (LMICs) adapted to the pandemic needs by leveraging existing research skills and networks (Reference Ananthakrishnan, Luz and Kc3). Although existing literature sheds light on the impact of COVID-19 on HTA process, methods, and market access, there is sparse information on how COVID-19 has affected HTAOs at the organizational level, specifically those in the Asia-Pacific region.
This study seeks to understand how COVID-19 has impacted organizations that conduct HTA-related research in the Asia-Pacific region, specifically the members of HTAsiaLink.
Methodology
Online Survey and Virtual Consultation
An online cross-sectional survey was administered among HTAsiaLink members (forty-four organizations across twenty countries in the Asia-Pacific) between December 2022 and January 2023 to understand the impact they experienced from COVID-19. The survey was first disseminated at the HTAsiaLink Member meeting during the tenth HTAsiaLink Annual Conference held in Pattaya, Thailand, on 2 December 2022, and subsequently circulated to all member organizations via email. The questionnaire was piloted among four HITAP staff to improve clarity and comprehensibility. The survey was developed and administered on Survey Sparrow, a web-and mobile-based platform. Members were asked to submit only one response per organization. In addition to the survey, a virtual consultation was held on 29 May 2023 to validate and supplement the study results. Informed consent was sought from all respondents and participation in the online survey and the virtual consultation was voluntary.
Questionnaire Development
Impact on “something” may be inferred by the performance it ensues (Reference Puga, List, Anheier and Toepler21). Hence, the questionnaire was developed using an existing framework (Reference Lafortune, Farand, Mondou, Sicotte and Battista22) which provides guidance on measuring the performance of HTAOs. The framework presents four functions of HTAOs as indicators to their performance (i) adaptation; (ii) production; (iii) culture and values; (iv) goal attainment. Given the difficulty in delineating these functions, we further classified them under the three key domains of the logical framework (23), namely (i) inputs, (ii) process, and (iii) outputs, to assess impact.
The questionnaire had eight sections and a total of thirty questions. No personal data were obtained unless they chose to be a collaborator in the data analysis and manuscript writing process.
The survey questions were based on the three key domains mentioned above: (i) inputs (changes in human and financial resources), (ii) process (changes in the HTA process and ability to independently design, conduct, and disseminate research), (iii) outputs (number of projects, HTA related publications, and public accessibility to publications). The full questionnaire can be found in Supplementary Tables A1 and A2. Organizations were categorized as either an HTA agency or a research agency. An HTA agency was defined as a dedicated national HTA agency for the country while a research agency was defined as an organization that conducts research, including HTA, but not a dedicated national HTA agency. Projects were referred to as any research projects, including COVID-19 related, while publications were limited to HTA publications (e.g., economic evaluation and HTA guidelines). As COVID-19 hit countries at different times in 2020, the years 2019 and 2021 were selected as a period before and during COVID-19, respectively. Considering the time needed for publications and human resourcing, 2022 was chosen as a reference point for these two sub-domains.
Data Analysis
Responses from Survey Sparrow were imported into Excel for data cleaning and analysis. The virtual consultation was recorded and transcribed into a summary. Upon validating and supplementing the survey data with the summary from the virtual consultation, descriptive results were presented either in count and percentages for quantitative or in narrative form for qualitative data. Results were presented to reflect the impact of COVID-19 on inputs, processes, and outputs of HTAOs as they relate to conducting research activities. No statistical analysis was performed on the quantitative data.
Results
Out of the forty-four HTAsiaLink members, twelve (27 percent) organizations representing ten countries in the Asia-Pacific region responded to the survey. Among the respondents, the oldest organization was the Malaysian Health Technology Assessment Section (MaHTAS), established in 1995, and the most recently established in 2020, was the HTA Resource Centre within the Jawaharlal Institute of Postgraduate Medical Education and Research (JIPMER). Table A1 summarizes the respondents’ profiles and responses.
Changes in number of staff and funding during COVID-19.
Impact on Inputs (Human and Financial Resources)
Some organizations e.g. CDE, HITAP, and ACE, reported high staff turnover during COVID-19. Nonetheless, most organizations (8/11, 73 percent) also managed to hire more staff during this period. Only HITAP and ACE were able to hire or retain international staff during COVID-19. MaHTAS and MCH did not hire additional staff during COVID-19. MaHTAS was unable to do so as they would need approval from the Ministry. To mitigate this, MaHTAS adapted their ways of working to accommodate the increased demand for research. No notable difference was observed between HTA and research agencies (Figure 1).
Most of the organizations (8/11, 73 percent) managed to secure similar or higher levels of funding during COVID-19 compared to the pre-COVID-19 period. Only CNHDRC, MCH, and UHS received less funding during this period. MCH, MaHTAS, ACE, and INAHTAC received a fixed budget from the government or hospital, while the rest received project-based funding. CNHDRC reported difficulties securing adequate funding during COVID-19 from the private sector, consequently, there was a reduction in funding. Despite receiving less funding and having fewer staff, CNHDRC conducted more projects during this period to meet the high demand for evidence from the government. CNHDRC mitigated this issue by collaborating with other research agencies to conduct research studies.
Impact on Processes
All organizations indicated that there were slight adjustments to the HTA process during the COVID-19 pandemic. ASERNIPS, MaHTAS, and MCH reported that the HTA process was conducted in a shorter timeframe to provide timely recommendations. All organizations reported that COVID-19-related research was prioritized in their respective countries, and therefore, all organizations, except CDE, undertook COVID-19-related research during this period. HITAP reported having to adapt their research design to meet policymakers need for evidence on topics such as protocol for rationing the use of critical resources (Reference Marshall, Archer and Witthayapipopsakul24), early HTA for COVID-19 vaccines (Reference Wang, Luangasanatip and Pan-ngum25), and bilateral travel policies using vaccination certificates (Reference Cai, Lim and Huynh6).
Most of the organizations (10/12, 83 percent) reported that COVID-19 had affected their regular data collection process (Table A2). These organizations therefore resorted to other means including online data collection, retrieving available real-world data, literature review, or expert consultation. CDE reported that COVID-19 did not impact their data collection process and cited their reliance on secondary data since the pre-COVID-19 period to conduct research studies.
HITAP, INAHTAC, MaHTAS, and MCH, reported that COVID-19 had impacted their ability to independently design research methods, draw conclusions, and publish results design (Table A2). CNHDRC, due to regulations on data, were unable to publish their research in peer-reviewed journals. As a result, they resorted to communicating their research findings through policy briefs that were catered to decision-makers. ASERNIPS, MaHTAS, and MCH specifically reported facing challenges in completing their research and providing policy recommendations in a timely manner due to short deadlines from policymakers. Organizations mitigated this issue by conducting rapid reviews or rapid HTA when appropriate.
Impact on Outputs (Projects, Publications, and Accessibility)
Most of the organizations (8/11, 73 percent) saw an increase in the number of research projects undertaken during COVID-19 compared to the pre-COVID period (Figure 2). Only UNPAD, UHS, and MCH had a reduction during this period. The type of new projects included by the respondents include rapid review, horizon scanning, HTA assessments and appraisals, clinical and implementation guidelines on health technologies, and health policy evaluation. As reported earlier, all organizations conducted at least one research on COVID-19 to inform decision-making.
COVID-19 impact toward research projects.
Only ACE, INAHTAC, MaHTAS, and MCH reported an increase in HTA-related publications during COVID-19 compared to the pre-COVID period. ACE, INAHTAC, MaHTAS, HITAP, and MCH reported increased accessibility to HTA-related publications during COVID-19, while CDE, ASERNIPS, CNHDRC, and CMHTA reported it being similar to the pre-COVID-19 period. UHS and UNPAD reported being worse. For organizations that reported increased accessibility of their publications, one of the potential reasons cited was the increased use of policy briefs and digital platforms for communication during COVID-19. In addition, COVID-19 raised public health awareness among the public which may have induced evidence-seeking behavior.
Discussion
This study surveyed HTAOs in the Asia-Pacific region proxied by HTAsiaLink members to understand the impact of COVID-19 on their inputs, processes, and outputs as they relate to conducting research activities.
We find that COVID-19 has had a relatively low impact on HTAO’s ability to secure funding. In fact, the majority of HTAOs attracted similar or more funding during this period which may be explained by increased demand for evidence from policymakers (in light of the pandemic), resulting in increased availability of funds for research at national and global levels and fast-tracked funding and ethical approvals. While no studies can confirm this in the region, this was found to be true in the U.K. (Reference Chinnery, Pearce, Kinsey, Jenkinson, Wells and Watt26). Furthermore, funding mechanisms (either grant-based and/or routine government budget line) may have helped HTAOs maintain adequate funding during COVID-19.
Although our study found that more staff were hired during the pandemic, staff retention remained a challenge for almost all HTAOs, presumably due to the high workload and better opportunities. While other studies did not report changes in staff levels, observations of high staff turnover and burnout were evident (Reference Elvidge and Dawoud17) and this is not limited to the field of health research (Reference Poon, Lin, Griffiths, Yong, Seah and Liaw27; Reference Sungbun, Naknoi, Somboon and Thosingha28). As staff levels affect HTAO’s research productivity and, ultimately, their ability to inform decision-making, organizations should prioritize managing burnout and increasing resilience, especially while working during emergencies. While not-for-profit HTAOs may not be able to offer attractive remuneration like the private sector may (Reference O’Rourke, Werkö, Merlin, Huang and Schuller29), they could find innovative ways (for example, flexibility in working hours and location, scholarships, staff exchange programs, career progression, leadership roles, etc.) to attract and retain staff. HTAsiaLink could commission a study on factors associated with staff retention in not-for-profit HTAOs, which may provide insights and appropriate strategies. A more sustainable solution is to routinely build the capacity of early career researchers in the region, a key mandate of HTAsiaLink (Reference Teerawattananon, Luz and Yothasmutra12) and an increasing priority for other regional networks (Reference Mundy and Maddern30).
Our study points to a significant impact on HTAO’s process of conducting research. Traditional HTA research for UHC was deprioritized by an influx of COVID-19-related research according to our respondents. Similar observations were reported by other studies (Reference Elvidge and Dawoud17; 18; Reference Boss, Bending and Rawson20). Consequently, this appears to have reduced the number of traditional HTA publications (UHC-related) from these HTAOs. However, this provided HTAOs with opportunities to expand their scope of work and adapt research methods and capabilities to address policy needs of the hour (Reference Ananthakrishnan, Luz and Kc3; Reference Cai, Lim and Huynh6; Reference Wang, Luangasanatip and Pan-ngum25). As we enter the post-COVID-19 era, HTAOs should continue to diversify their research portfolio and build capabilities beyond traditional HTA to remain relevant to policymakers and attract diverse talents and funding.
In terms of assessment, COVID-19 induced lockdowns appear to have affected HTAO’s ability to collect primary data and conduct in-person stakeholder consultations, an observation reported by others (Reference Barham and Lovell19; Reference Boss, Bending and Rawson20). However, these challenges were overcome by using secondary data and leveraging digital platforms, a trend that was prevalent globally (Reference Sharon Wolf, Brown, Tsinigo and Edro31; Reference Sarah Coates32). While our study did not examine the impact on HTA research methodology, studies have discussed the challenges and potential increase in uncertainty and reduction in quality of HTA studies resulting from such disruptions (Reference Leahy, Hickey and McConnell16; Reference Elvidge and Dawoud17). Despite the risk of reduced quality of research, HTAOs in our and other studies reported the pressure of presenting evidence to policymakers in a short timeframe (Reference Mundy and Kearney13; Reference Elvidge and Dawoud17‒Reference Barham and Lovell19). As Chris Whitty rightly says, “An 80 percent right paper before a policy decision is made is worth ten 95 percent right papers afterward, provided the methodological limitations imposed by doing it fast are made clear” (Reference Whitty33), speed is indeed more valuable over quality in emergency settings. However, research ethics, standards, and independence should be maintained to the extent possible to avoid the misuse and misinterpretation of low-quality evidence to forward agendas (Reference Elvidge and Dawoud17). Fortunately, some guidance is now available on applying HTA during pandemics (Reference Leahy, Hickey and McConnell16) and maintaining standards while ensuring timeliness (Reference Teerawattananon, Kc and Chi34). Future studies could apply such guidance to their research, comment on their usefulness, and build on as appropriate such that timely yet reliable evidence can be produced during future pandemics.
To combat some of the challenges discussed above, including lack of data and balancing quality and timeliness, the HTAOs in this study recommend expanding collaboration and strengthening partnerships within and across regions. For example, during the pandemic, ten organizations from HTAsiaLink representing ten countries collaborated in an initiative called the COVID-19 Vaccination Policy Research and Decision-Support Initiative in Asia (CORESIA) (Reference Kc, Faradiba and Sittimart35), to fast-track the exchange of reliable cross-country information, data, and evidence on issues related to COVID-19 policies. The Ministry of Public Health Intelligence Unit (MIU) in Thailand, through HITAP, routinely leveraged this collaboration to gather evidence from the region to inform national policies (in the absence of local data). In the post-pandemic era, HTAsiaLink is working to (i) harmonize clinical data required to conduct HTA studies and (ii) establish a comprehensive registry for ongoing HTA-related studies in the region. These efforts are aimed at filling the data, technical capacity, and evidence gap that many LMICs face while rewarding existing studies by increasing visibility and reducing duplication of efforts.
Our study has several limitations. Only 27 percent of the HTAsiaLink members participated in the survey, therefore, study results may not be representative of the impact across all HTAOs in the region. However, response from twelve organizations representing ten countries may be considered high compared to other studies. Only a few performance indicators from the LaFortune et al.’s framework (Reference Lafortune, Farand, Mondou, Sicotte and Battista22) were chosen to study impact on HTAOs, hence, other important dimensions such as impact on research coordination, learning, approval decisions, timing, methodology, etc., were not considered. In this study, we only presented descriptive results and did not perform any statistical analysis due to the limited sample size. Therefore, no causal inferences can be made from our results. Furthermore, data from the survey were only estimates and may be subject to inaccuracies. Our study did not factor in contextual differences such as baseline capacity, capabilities, link to policymakers, etc., of HTAOs which could provide a more nuanced understanding and nature of impacts from the pandemic.
Conclusion
COVID-19 has impacted several dimensions of HTAOs in the Asia-Pacific region. Nonetheless, HTAOs have shown resilience and adapted well to the pandemic and were able to consistently support policymakers in their countries. With commitment to efficient and equitable UHC, there is growing reliance from countries to adopt HTA. Therefore, adequate financial support from the governments and funders to undertake research, continuous capacity building efforts with adequate support system for staff, innovation in HTA methods to incorporate dynamic policy questions, and pragmatic yet robust evidence to policy process are some key takeaways as we prepare for the next pandemic.
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/S0266462324000357.
Acknowledgments
The authors express their deepest appreciation to the members of the HTAsiaLink network who participated in this study and the members who reviewed the manuscript.
Funding statement
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.
Competing interest
Dr Piyameth Dilokthornsakul, an author of this manuscript, received research grants from Norvartis (Thailand), Pfizer (Thailand), and Boehringer Ingelheim (Thailand) for other projects at the Center for Medical and Health Technology Assessment (CM-HTA), Chiang Mai University. However, these funding sources are unrelated to the research presented in this manuscript. The other authors declare no conflict of interests.
Appendix
Respondents profile and responses
COVID-19 impact on the research process
Open access
