A review of implementation frameworks to operationalize health technology assessment recommendations for medical technologies in the Singapore setting

Abstract Background Methodologies of health technology assessment (HTA) for medical technologies are well established; yet, operational frameworks that enable appropriate uptake of HTA recommendations into routine clinical practice are lacking. This review aims to identify the key themes needed to guide the planning and implementation of HTA subsidy decisions for medical technologies such as diagnostics, medical devices, and services and to monitor their impact on a complex multipayer healthcare system like Singapore. Methods A literature search of implementation frameworks was conducted up to 20 December 2020 and was documented in a flow diagram. A thematic analysis of the evidence base was performed using the Braun and Clark approach to identify key themes, from which an implementation framework suitable for Singapore's healthcare system could be developed. Results The searches yielded forty-four articles for review, from which twenty themes were identified. The top ten themes constituted the key themes of implementation essential for local adaptation and were categorized into five domains: implementation strategy, organizational support, stakeholder engagement, information dissemination, and implementation outcomes and evaluation. These domains provide operational guidance to methodically identify gaps to facilitate sustainable implementation of HTA-informed medical technology subsidy decisions. Conclusion A robust and adaptable implementation of HTA-informed subsidy decisions is crucial to optimize its intended impact of improving patient outcomes per dollar spent. The key themes of an implementation framework should capture the important aspects of organizational feasibility to ensure successful adoption in a complex multipayer healthcare system like Singapore.


Background
Universal health care in Singapore is provided by a combination of government subsidies, compulsory individual healthcare savings accounts (Medisave), risk-pooling via both voluntary private and mandatory government health insurance plans (MediShield Life), out-of-pocket contributions from patients, and a government endowment fund that acts as a safety net for the needy (Medifund). Although Singapore has a successful healthcare system that is well known for its efficiency and quality, it is facing rising healthcare costs driven by a rapidly aging population, an increasing burden of chronic disease, and growing demands from citizens for expanded medical services (1).
To overcome such challenges, the Agency for Care Effectiveness (ACE) was established in August 2015 within the Ministry of Health (MOH) as the national health technology assessment (HTA) agency to initially focus on evaluating drugs for subsidy considerations. HTA is used as a tool to review health technologies and provide evidence of the value that these technologies can deliver to patients and their families, health system stakeholders, and to society more broadly. It is a multidisciplinary process that uses explicit methods to determine the value of a health technology at different points in its life cycle, with the purpose of informing decision making in order to promote an equitable, efficient, and high-quality health system (2).
In 2017, the ACE expanded its capacity to assessing other health technologies beyond drugs and supporting the MOH Medical Technology Advisory Committee (MTAC) to make evidence-based subsidy recommendations for medical technologies, including diagnostics, medical devices, and services, but excluding models of care, information technology system, and telemedicine (3). To increase transparency in decision making and facilitate implementation, the ACE publishes guidance on the rationale of subsidy recommendations, which include a summary of the comparative safety, effectiveness, and cost-effectiveness evidence along with the subsidy criteria for eligible populations.
Although established structures and processes exist in the Singapore's public healthcare system for implementing drug subsidy decisions, these are lacking for medical technologies, given that their adoption is highly dependent on the organizational feasibility of healthcare institutions. Adopting new medical technologies has a far-reaching impact beyond generating additional costs to the healthcare system, which may include changes in the organization of care and the modification of facilities, manpower resources, training, and credentialing requirements.
Establishing a robust and adaptive implementation framework for medical technology subsidy is important in Singapore. This ensures that funding decisions are translated into sustainable and effective clinical practice changes, leading to improvements in patient health outcomes. We aim to identify actionable steps to implement HTA recommendations for medical technologies in the Singapore setting.

Methods
For this review, the appropriate steps required in Singapore to operationalize HTA implementation beyond the subsidy decision itself were defined as the key themes of HTA implementation.

Search Strategy
We conducted a literature search adopting three complementary searches-electronic databases, a manual search of reference lists, and a review of specific HTA agency Web sites-to identify articles on implementation frameworks and related strategies. EMBASE, PUBMED, and CRD databases were searched. The search strategy employed three concepts: (1) Implementation OR Adoption OR Diffusion OR Dissemination OR Spread. The search concepts were deliberately broad to comprehensively capture relevant articles because HTA implementation was poorly reported. The first concept captured the aspects of implementation, whereas the second concept captured the aspects of HTA of medical technology. The third concept was a combination of concepts one and two. The search period was from 2008 to 2020, pragmatically chosen to simultaneously cover more recent developments in implementation science and seminal HTA articles published in 2008, such as by Drummond et al. (4). The electronic search was supplemented by a manual search of the reference lists of articles identified through electronic databases. There was no period restriction imposed on the manual search, as the intent was to identify widely cited landmark articles in implementation science that may have preceded 2008. Articles were restricted to the English language and included primary research, reviews, and grey literature such as conference posters and abstracts. Articles were selected if they described an implementation framework, process model, or theory, including case studies of prior implementation efforts. Our evidence hierarchy prioritized articles that contributed to implementation frameworks with direct relevance to our research question. Articles focused on the implementation of drugs, e-health interventions, digital health services, clinical practice guidelines, or patient safety guidelines were excluded because they did not meet the local definition of "medical technology", and HTA implementation for medical technologies was more challenging in Singapore (3;4).
We also reviewed the online resources of specific HTA agencies including the Canadian Agency for Drugs and Technology in Health (CADTH), Alberta Health, U.K. National Institute for Health and Care Excellence (NICE), Australian Medical Services Advisory Committee (MSAC), U.S. Agency for Healthcare Research and Quality (AHRQ), and the International Network of Agencies for Health Technology Assessments (INAHTA). The INAHTA Secretariat was contacted in November 2020, seeking responses from member agencies for existing policies, procedures, or frameworks that support the implementation of HTA-informed subsidy decisions for medical technologies.

Data Abstraction and Analysis
Two independent reviewers evaluated full-text articles to minimize bias. Each article was reviewed with the aim of eliciting whether an implementation model was proposed and what themes were proposed in those models. The specific themes from each model were synthesized. A thematic analysis was conducted using the Braun and Clark (5) framework as a guide to identify key themes using a coding frame (Supplementary File 1).

Thematic Analysis
An inductive method was applied to the evidence synthesis and a semantic approach was used to code the emerged themes. We used open coding, with numeric codes assigned to describe the content in sentences extracted from the articles. We then collated the assigned codes into broader themes, which were refined through iterative discussions to remove overlaps and contradictions across the different models that were analyzed. The final set of themes, including names and definitions, was shared with all coauthors and was accepted to be coherent and distinctive.
We counted how often themes appeared in each article, and the top ten themes were identified. We assumed that a high occurrence, especially in identified landmark articles, corresponded to high internal validity. Unique counts were assigned to the same theme, which may have occurred more than once in an article. Landmark articles on implementation frameworks were internally defined as frameworks that were widely cited and applied in qualitative or quantitative studies in our evidence base. We were cognizant that, although frequency of occurrence was chosen to present the data, it might not necessarily imply significance in answering our research question. For the purposes of internal validity, a matrix analysis mapped the top ten themes occurring in the implementation frameworks that were widely cited in our evidence base.

Results
The literature searches were initially conducted on 15 October 2019 and were updated on 20 December 2020. Our searches yielded 9,854 hits that comprised 9,828 retrievals from electronic databases and 22 citation reviews and 4 articles from HTA agencies. After removing duplicates and screening titles and abstracts, 121 articles remained. Of these, seventy-seven were excluded. Of the remaining forty-four articles, eighteen articles were retrieved from the databases, twenty-two were retrieved from citation reviews, and four were found on HTA agency Web sites. A flow diagram of our search results is shown in Figure 1.
INAHTA member agencies who responded to the survey were NICE UK, CADTH Canada, CDE Taiwan, SBU Sweden, CONITEC Brazil, and NIPH Norway. They do not have formal frameworks in place to support the implementation of subsidy decisions for medical technologies.

Prominent Implementation Frameworks
There were eleven prominent implementation frameworks widely cited in the retrieved literature. The top ten themes that emerged from the evidence synthesis were mapped back to the implementation frameworks and are reported in Table 1.
The most often cited framework was CFIR (7) with 4,097 citations, but it only covered seven themes and missed the "Information Dissemination" domain. The most often cited theme was "conducting an organizational needs assessment". The framework that included eight out of ten themes and covered all domains was the QIF (12), but it did not include "conducting a pre-implementation context analysis of various domains", and "employing diverse and intelligent strategies to disseminate information, for example, knowledge translation tools".

Helpful Findings for Developing Singapore's Specific Implementation Framework
The top ten recurrent themes and examples of the coding frame are shown in Table 2. Supplementary File 2 captures the five domains and respective themes and details their descriptions.
The top ten themes holistically covered key concepts in the prominent implementation frameworks and were considered the most relevant findings for developing a specific implementation framework for Singapore. These were shared with all coauthors and were further synthesized into five distinct domains. Broad acceptance was obtained that these themes were representative of good practice standards for HTA implementation.

Five-Domain HTA Implementation Framework
A five-domain model for HTA implementation was developed: (1) Implementation Strategy,

Implementation Strategy
A prevalent assumption of an implementation strategy was "clinical serendipity"-a natural and smooth adoption of new innovations or interventions by clinicians (24). In contrast, successful implementation required various, if not all, parts of the implementation strategy (e.g., dissemination, training, and support) to be carefully designed (7). This included outlining specific tasks and timelines (7;12) and ensuring that the implementation strategy was adaptable to address unique requirements in any particular context (25).

Organizational Support
Organizational readiness assessments were referred to explicitly, with specific tools created to assess readiness (14) based on organizational factors such as communication channels (7), leadership (26), culture and morale (8), and resource allocations (26); and implicitly by inferring the need to develop an understanding of the local operating environment or context (6;7;13). Different medical technologies require different aspects of organizational readiness to be surveyed, taking context into account.

Stakeholder Engagement
The theme of stakeholder engagement was widely reported. Implementation efforts should be consultative, allowing for ample stakeholder participation. Early engagement of stakeholders helped address and manage any concerns upstream (4) and by doing so, ensured support for the implementation strategy (12;13). Stakeholders representative of a range of different   International Journal of Technology Assessment in Health Care backgrounds catered for adaptability in the implementation strategy (27;28).

Information Dissemination
Dissemination was defined as an active process of knowledge transfer, one that was vertical, planned, formal, and centralized (29). The more extensive and multipronged the mode of information dissemination was, the greater the success in implementation (20;30). Specific to HTA, the INAHTA's guidance document highlighted that mail-outs and journal presentations might be ineffective as standalone modes of communication, and instead, recommended a mixed approach. This involved face-to-face consultations, in tandem with the distribution of written materials (15). Having a localized unit (12;19) within the organization to orchestrate implementation efforts was shown to improve the success rate of implementation. Such a unit introduced structure and clarity of roles, key deliverables, and timelines for seamless information flow (12). It was suggested that these units possessed  the necessary technical, communication, financial, and project management skills required to carry out the implementation effort (24).

Implementation Outcomes and Evaluation
Evaluation of implementation outcomes was explicitly described as evaluating fidelity, adoption, appropriateness, costs, feasibility, penetration, and sustainability (31) and was implicitly described as incorporating a component for audit (9) or feedback (26) in any implementation strategy. Regardless of the method chosen, ensuring that some form of feedback mechanism was hardwired into the evaluation enhanced the robustness of an implementation strategy. Evaluation outcomes created in partnership (22) with stakeholders ensured that the outcomes were relevant and specific to their context, instilling greater ownership among stakeholders.

Discussion
Based on our literature review, we identified key themes for implementation and conceptualized them into an HTA implementation framework with domains most relevant to Singapore. The framework is a visual representation of the key themes that could be considered in sequence, when implementing HTA recommendations of medical technologies. In Singapore, the provision of services involving medical technologies is highly influenced by institutional practices and user preferences at healthcare institutions. In recognition of the fact that a "one-size-fits-all", "top-down" implementation strategy is not adaptable nor practical, we attempted to cater implementation efforts to unique operating environments. As the national HTA agency, the ACE developed an Organizational Readiness Assessment (ORA) tool (Supplementary File 3) to highlight backend processes at the healthcare institutions that might be affected when a medical technology is recommended for subsidy. This self-assessment allows hospital institutes to identify potential implementation barriers early and make changes before subsidy comes into effect.
Clinical stakeholders such as doctors often assumed the role of clinical champions to identify and solve implementation issues unique to their institutions. However, a proper implementation team at the healthcare institutional level, comprising people with the necessary skillsets and authority to drive multifaceted changes, should be established to solve organizational problems. Establishing such teams will complement existing multipronged modes of information dissemination that the ACE employs, including face-to-face consultations, frequent email exchanges, issuing formal circulars, and publishing guidance documents on our Web site. The ACE has yet to explore mobile applications as a mode of information dissemination but acknowledges that it is worth exploring to cater to increasing demand from information-seeking behaviors.
Implementation success in Singapore is currently measured by the timeliness of subsidy implementation. Such a measure is not reflective of the true benefits to patients and the healthcare system. As such, implementation outcomes should be codeveloped with relevant stakeholders to ensure the collection of meaningful data that measures changes in clinical practice and improvements in patient outcomes in a sustainable manner. We acknowledge that the key themes of the five-domain model were considered most relevant to HTA implementation in the Singapore setting. Other HTA agencies may consider its local relevance and applicability. The ten key themes of implementation need to be regularly reviewed to keep pace with the evolving needs of the healthcare landscape and be adapted to address any potential barriers to framework compliance.

Conclusion
Policy decision making in Singapore is increasingly reliant on HTA when assessing the value of medical technologies. A robust and adaptable implementation of HTA-informed subsidy decisions for medical technologies is crucial to optimize patient outcomes commensurate with costs. Although there is a plethora of implementation science literature to guide the implementation of novel or complex health interventions, much of the existing literature is about conceptual developments. The top ten themes were identified to emphasize the important aspects of organizational feasibility, which should be considered when implementing HTA recommendations for medical technologies in the Singapore setting. We will continue reviewing the applicability of key themes when implementing HTA-informed subsidy decisions as we gain more experience in the future.
Supplementary material. The supplementary material for this article can be found at https://doi.org/10.1017/S0266462321000222.
Ethics approval and consent to participate. Not applicable.

Consent for publication. Not applicable.
Availability of data and materials. The dataset(s) supporting the conclusions of this article are included within the article and its Supplementary files.