Introduction
Procedures of health technology assessment (HTA) traditionally include the clinical benefit and, for some, a measure of economic impacts (Reference Oortwijn, Sampietro-Colom and Habens1;Reference Tanios, Wagner and Tony2), whatever the type of technology considered (drugs, medical devices [MDs], professional practices). Despite its undisputed value, this process has been the subject of increasing criticism for many years, particularly with respect to the assessment of value, which can be seen as restrictive (Reference Busse, Orvain and Velasco3–Reference Goldman, Lakdawalla, Philipson and Yin6). Various researchers suggest extending HTA to other dimensions of value. For example, articles suggest considering the impact on public health, disease severity, unmet needs (Reference Angelis, Lange and Kanavos7), indirect or unintended outcomes, ethical and legal considerations, the conditions of use of the technology and, more broadly, contextual considerations regarding the implementation of the the technology (Reference Oortwijn, Sampietro-Colom and Habens1–Reference Busse, Orvain and Velasco3;Reference Angelis, Lange and Kanavos7–Reference Henshall and Schuller10).
This is particularly true for the assessment of MDs that are associated with complex interventions and the need to adapt the assessment of their clinical effectiveness (as compared to drugs) (Reference Drummond, Griffin and Tarricone11–Reference Schnell-Inderst, Iglesias and Arvandi16), although all health technologies may also be affected. In terms of economic evaluation, MDs are specific due to the learning curve, incremental innovation, and also organizational impacts (OIs) that need to be modeled adequately (Reference Tarricone, Torbica and Drummond14). In particular, adoption of MDs in clinical practice frequently requires substantial organizational investments, multidisciplinary teams, need for supervision (Reference Tarricone, Callea, Ogorevc and Prevolnik Rupel12), change in the relationships between different organizations, or training for patients and professionals (17). Nevertheless, these organizational changes seem imprecisely defined. Integrating them into HTA procedures with a pragmatic but rigorous approach is a challenge.
In France, the topic of OIs emerged in 2016 at the request of the Strategic Council of Health Industries and many healthcare professionals, who advocated for their inclusion in HTAs of health technologies (17) and in the criteria for price negotiations. The Haute Autorité de Santé (HAS), the French HTA agency, convinced of the need to move forward, initiated research in 2018. It was carried out in several stages until November 2019.
In terms of clinical and economic evaluation, the HAS already had, for each type of health technology, both methodological guidelines and a doctrine (based on jurisprudential data built from the experience and deliberations of its commissions). In the clinical evaluation of these technologies, OIs could already be integrated, via a criterion for evaluating the benefit of the health technology for the community, that of “public health interest” (PHI), which includes organizational elements. These mainly refer to the consumption of resources linked to patient care: for example, the foreseeable effect of the technology on the organization of care and expenditure (number of consultations, hospitalizations, procedures, etc.). Similarly, economic evaluation methods (medico-economic or budgetary impact) have also integrated PHI through criteria related to resource consumption at the community level.
These practices have raised a number of recurring questions for the HAS: (i) are all the macro-OIs identified (is the approach exhaustive?); (ii) Should the OIs other than those at the macro level, such as impacts at the care team or patient level – meso and micro levels – be considered (is it relevant to have only a macro approach?); (iii) If OIs are added to clinical assessments and economic analyses, is there a risk of counting the same things twice (especially between inputs and outputs)?; (iv) How to take into account the impossibility of subjecting certain technologies to an HTA procedure because they do not claim collective clinical or economic impacts, while they clearly generate major impacts on other aspects (would assessing OIs independently of the clinical and economic evaluation be the right solution?).
These questions led the HAS to launch a research project. Specifically, the question was whether existing researches allowed for a description of the OIs of a health technology, in the form of a generic comprehensive checklist to be adapted to different purposes, but also allowing for the integration of different levels of micro/meso/macro analysis. In other words, and using the taxonomy of Pfadenhauer (Reference Pfadenhauer, Gerhardus and Mozygemba18), the objective was to identify the existence of an OI assessment framework. This article describes (part 1) what methods were used to carry out this research, (part 2) what the results are, and (part 3) the questions raised by these results. We conclude (part 4) with possible avenues for reflections that emerged from this analysis.
Methods
The study was carried out by a project team comprising three academics, two of whom being specialized in health economics and health services management, and four experts belonging to the HAS’s department in charge of the medical, economic, and public health evaluation of drugs, professional procedures, and devices. A methodological guidance group was also established starting June 2019 to ensure methodological support and to validate the findings. It consisted of ten experts on HTA, with research profiles in health engineering, public health, health services management, health economics, and patients’ rights.
Several investigations were completed between May 2018 and November 2019, primarily due to the lack of results matching our expectations. A triangulation process, using multiple research methods to achieve our goal, was implemented to help strengthen the validity and credibility of the research results. Three types of investigation were carried out following iterative and parallel processes:
-
– Between May 2018 and January 2019, an analysis of documents published by a sample of selected HTA organizations (agencies, bodies, institutes, and networks) known for their work on HTA;
-
– Between May 2019 and November 2019, a rapid review (19;20) by mixing several successive keyword searches on a database and a search by the pearl method (;22);
-
– In a parallel process, between July and September 2019, a questionnaire survey to the International Network of Agencies for Health Technology Assessment (INAHTA) experts.
First of all, the work of HTA organizations from ten countries and two European expert networks in HTA was analyzed. The HTA organizations were selected on the basis of two key studies of HTA organizations around the world (Reference Ciani, Wilcher and Blankart23;Reference Fuchs, Olberg, Panteli and Busse24). Fuchs et al. (Reference Fuchs, Olberg, Panteli and Busse24) identified three European Union HTA national organizations which explicitly mention OIs in their official documents: the former Danish Centre for Health Technology Assessment (DACEHTA) (Denmark), the Ludwig Boltzmann Institute (LBI) (Austria), and the National Institute for Health and Care Excellence (NICE) (United Kingdom). In Ciani et al. (Reference Ciani, Wilcher and Blankart23), the scope was HTA national organizations outside of the European Union. They identified the Brazilian HTA agency, the Department of Science and Technology – Brazilian Health Technology Assessment General Coordination (DECIT-CGATS) as the only agency having “developed scientific methodological guidance for the HTA of MD.” Thus, the Ministry of Heath of Brazil -DECIT was included in our sample. Furthermore, the main HTA organizations of respectively Sweden, Australia, USA, and Canada were also taken into account in this study, as they are often referred to in other national and international documents when it comes to in-depth HTA procedures. Germany and Italy were also included, in order to allow the comparison with two large countries bordering France. Originally the New Zealand agency (National Health Committee) was also to be analyzed, but it was eventually discarded; its documents could not be identified, probably due to its merger with the Ministry of Health in 2016. In addition to the ten selected organizations, the work of the European Network for Health Technology Assessment (EUnetHTA) and the European research program Adopting Hospital Based Technology Assessment (AdhopHTA), which aims to improve HTA practice in hospitals, were also considered.
The homepage of each website was “hand-searched” for potentially relevant documents, using web site navigation, web site search engines, and sitemaps. Firstly, we searched for an explicit mention of OIs. When the term was missing, we searched for any guide, manual, or report concerning general HTA methods (and not regarding the evaluation of a specific technology). When a choice was to be made, it was the most complete documentation available and concerning the most general level that was retained. Overall, sixteen relevant documents were identified (Table 1). Each document was analyzed according to the following criteria: explicit mention of organization as an aspect, level, or dimension of the assessment; explicit definition of OI; explicit formulation of a list of OIs; mention of technology-induced structural changes (e.g., number of hospital beds, space allocation), process changes (e.g., care pathway, patient flows), and cultural changes (e.g., patient and professional acceptance/reluctance of the new technology) as assessment criteria.
HTA organizations website and selected documents
Note: This table describes the HTA organizations whose websites were analyzed and the reference documents that were selected.
After the previous researches, a rapid review of the academic literature was carried out. The choice of a rapid review was guided by the objectives and constraints of the mission entrusted by the HAS. Firstly, the HAS’ objectives were not to have a structured, extensive and critical analysis of the literature on OIs, but to know whether or not there was already a framework for evaluating OIs that could be used in the context of rigorous evaluation procedures in France. Furthermore, following the work on the documents of the sample of HTA organizations, it was appropriate to verify that no alternative work existed from the scientific community. Finally, time and human resources were limited and did not allow for more structured research.
Concerning the selection of articles, first, twenty-nine relevant articles were identified and processed based on the pearl method. Second, in addition, a search by keywords in Medline over the period from 2009 to 2019 (i.e., the 10-year period preceding the launch of the project) was also carried out. This process was carried out in three iterative steps and through nine queries (Supplementary Table S1). Regarding the keywords used, the Merriam Webster Dictionary defines an impact as “the force of impression of one thing on another: a significant or major effect.” As this definition is very broad and there is no universal definition of OI, a wide range of synonyms were used: consequence, change, modification, innovation, side-effects, effects, and outcome. These terms were combined with keywords referring to medical equipment, supplies, technologies, and devices on the one hand, to the organization, administration, and production of care on the other. The objective was to find articles which propose OI evaluation framework complementary to what had previously been identified.
A total of 928 records were identified from the literature search. Twenty-two duplicate records were removed and 149 others without abstracts; 801 records were reviewed by title and abstract by two researchers who searched for the same criteria as used for the hand-searching websites’ homepages. Seven hundred and sixty-seven records were screened out. In the end, thirty-four full-text articles were relevant. The rapid review was carried out on these thirty-four articles, to which were added the twenty-nine identified by the pearl method, minus seven articles that were duplicates. Finally, fifty-six articles were read and analyzed in detail (Figure 1) (Table 2).
PRISMA flow diagram. The PRISMA diagram details our search and selection process applied during the rapid review.
List of publications included in the review
Note: This table lists the publications which were included for review.
In parallel, a questionnaire was also submitted to experts from the forty-nine HTA organizations that were (at the time of the survey) members of the INAHTA network to obtain a broader picture of their practices. The questionnaire was sent to the email addresses of HAS correspondents. It included four questions asking whether (i) an OI, in relation to the introduction of a new technology, is included in their evaluation criteria, (ii) if so, what were the types of health technology involved, (iii) what were the specific criteria used for the evaluation of OIs, and (iv) the reference to related guidelines. Ten organizations responded to the questionnaire, representing a response rate of 20.4 percent of the total (Supplementary Table S2).
Results
Three ways to approach OI assessment procedures as conducted by HTA organizations were identified. Ranging from the most to the least detailed explanation (advanced, intermediate, low), they are as follows (Table 1):
1) The HTA organization explicitly mentions the OI and provides a more or less detailed categorization of the OI elements: DACEHTA – closed in 2012 (Denmark) and EUnetHTA CORE MODEL;
2) The HTA organization explicitly mentions OIs but does not provide a detailed categorization (at best, examples of OIs not classified into operational categories): AdHopHTA, the Institute for Quality and Efficiency in Health Care (IQWiG) (Germany), and the Italian National Agency for Regional Healthcare Services (AGENAS) (Italy);
3) HTA organization does not mention OIs but only elements that may relate to OIs and that appear under another name (e.g., economic, budgetary, social aspects): NICE (United Kingdom), LBI-HTA (Austria), the Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU) (Sweden), the Medical Services Advisory Committee (MSAC) (Australia), and the MS-DECIT (Brazil); the Institute for Clinical and Economic Review (ICER) (United States), the Canadian Agency for Drugs and Technologies in Health (CADHT)
(Canada).The analysis highlights that, despite the vagueness of the definition of OIs and their imprecise delineation from other dimensions of HTA assessments, recurring elements emerge from the OI assessment. These can be classified into three broad general categories, namely processes, structure, and culture, to echo the themes mentioned by DACEHTA and EUnetHTA.
Process-related elements typically include the impact on care pathways and procedures, patient flow and length of stay, as well as workforce size and employee training. Also considered are the evaluations and follow-ups of the new technology once it is implemented, as well as the procedures for selecting patients eligible for the new treatment. The structural aspect addresses the impact on the number of beds available, the location or accessibility of the new technology, premises, facilities, and equipment. The cultural aspect concerns the perceptions of the new technology by patients and professionals.
These classifications do not appear to be based on a systematic analysis and therefore cannot constitute a taxonomy. Since they are not based on a theoretical definition of the concept of OI, they do not constitute a typology either (Reference Bailey25). They therefore appear as a list of examples according to experts, which cannot suffice to construct an extensional or intentional definition of the concept (Reference Cook26). The authors of the basic EUnetHTA model (17) themselves state that their own evaluation criteria are very general and invite using detailed methodology to collect and analyze additional data.
The rapid review provides clarification regarding the definition of OIs, even though it does not presently appear to be a unanimous definition in the scientific community. As mentioned by Allen et al. (Reference Allen, Towne and Maxwell27), there was considerable variability in the conceptual and operational definitions of organizational constructs. For example, Gurtner (Reference Gurtner28) defines OIs as “the sum of all necessary changes in infrastructure, human resources, training, and organizational procedures.” Roussel et al. (Reference Roussel, Carbonneil and Audry29) state that “OIs are the consequences (upstream and downstream) of the introduction of a MD in terms of resources, production processes, availability, and information/training.”
Another more specific approach is that of Ottardi et al. (Reference Ottardi, Damonti and Porazzi30), who have carried out a comprehensive evaluation of an innovative MD. OIs were assessed quantitatively and qualitatively. At the quantitative level, the additional investments required for the proper application of the technology were taken into account, as was the impact of the latter on hospital processes. At the qualitative level, the impact was considered from the perspective of professionals: staffing needs, training courses, sessions and communication, learning curve, and acquisition or renewal of equipment/materials (Reference Ottardi, Damonti and Porazzi30).
A more detailed definition is provided by Fattore et al. (Reference Fattore, Maniadakis, Mantovani and Boriani31). Here, the perspective is reversed: the authors do not ask which organizational changes the new technology will bring about, but how the organization can adapt to achieve the best results from the use of the new technology. This normative perspective differs from the aim of this project, but it nevertheless has the advantage of identifying organizational parameters that are related to the introduction of a technology: training, procedures, definitions of tasks and roles, coordination mechanisms, and complementary technologies. Similarly, Nielsen et al. (Reference Palmhøj Nielsen, Funch and Kristensen32) point out that to properly assess health technologies, both the consequences and the prerequisites for their introduction, such as adapting organizational structures, work processes, and culture, must be determined.
The literature reveals dimensions and elements already identified in the official documentation of HTA organizations, namely a structural dimension (investments, infrastructure, equipment, materials, and human resources) and a process dimension (care and other procedures, training, tasks and roles, communication, and coordination).
Despite these attempts, scholars and HTA organizations do not propose a complete and systematic evaluation framework of OIs. To our knowledge, the only attempt in this direction is that of Roussel et al. (Reference Roussel, Carbonneil and Audry29), which defines twelve categories of OIs of MDs and applies them to sixteen types of MDs considered by a group of experts as having a significant impact on the organizations. However, the twelve categories used contain heterogeneous elements and are not always mutually exclusive. For example, it is difficult to distinguish between the categories “Work or care production process,” “Care pathway,” and “Patient flow.” The same is true for the categories “Accessibility” and “Logistical Circuit.” The category “Type and level of patient/caregiver involvement” groups two distinct impacts, namely the transfer of certain acts from professionals to patients or caregivers on the one hand, and the acceptability of the new technology by patients on the other.
Other classifications/typologies of organizational aspects are found in the literature (33;34), often linked to multi-criteria procedures, for decision-makers. Others are not necessarily related to the specific field of HTA. For example, Piña et al. (Reference Piña, Cohen and Larson35) listed, from a broader perspective, the characteristics of “care delivery organizations and systems.” These characteristics were grouped into six domains. In the field of integrated care, Evans et al. (Reference Evans, Grudniewicz, Baker and Wodchis36) have developed a conceptual framework for organizational capabilities. The authors identified eighteen organizational factors classified into three categories. Here too, we found some commonalities with the dimensions discussed above (i.e., “Structure,” “Process,” and “Culture”).
The responses to the questionnaire submitted to the INAHTA network confirm the elements already identified in the websites of the HTA agencies, institutions or organizations, and the rapid review. Most of them reported including OIs in their evaluation criteria. But none of them have developed specific guides or guidelines for evaluating OIs. When they do include OIs, they rely on the EUnetHTA CORE MODEL or INAHTA HTA recommendations or AdHopHTA Handbook (Reference Sampietro Colom, Lach and Haro37) for all or only some technologies (Supplementary Table S3).
Discussion and conclusion
The objective of this research was to determine whether HTA organizations or researchers had developed an OI evaluation framework that could be used by the HAS. The results show that this issue remains largely unresolved. There is no explicit conceptual definition of OIs or a generic comprehensive checklist to be adapted to different purposes.
However, there are some limitations to this study. First, practical constraints may have limited our review of HTA organizations websites. The list of selected HTA organizations was based on two key articles and was broadened but others, not included on these lists, may have provided additional modeling guidance. In addition, HTA organizations guidance not featured on a website was not included nor has guidance in a national language other than English or French. The analysis could also undoubtedly have been supplemented by a questionnaire to a broader panel of experts than the representatives of the forty-nine agencies in the INAHTA network. Similarly, the choice of a rapid literature review naturally induces risks of bias, in particular: the fact of having used only one database, over a limited period of 10 years, in English or French, and of having excluded grey literature. In addition, the analysis of the selected articles was carried out in the form of a rapid narrative synthesis, aiming first of all to respond to the request of the HAS, that is, “whether or not there is an analysis framework for OIs that could be used by the HAS.” This initial objective was broadened in view of the disappointing results, to allow a better understanding of the state of the subject and its limits, in the international scientific community. In addition, the question of the scope of the research arises. Indeed, the analysis of HTA practices was limited to a review of the literature relating to the health sector as a whole. We could probably have made more use of the work on evaluation in hospitals, particularly that carried out by the AdHopHTA network, which stresses the importance of taking into account the context, the stage of development of the technology under consideration, and the levels of impact (meso and micro). However, the perspective of the assessment proposed by AdHopHTA is at a less global level than that of a health technology agency such as the HAS, which raises the problem of the exhaustiveness of the criteria for capturing macro-level effects. Indeed, as the perspective of the evaluation proposed by AdHopHTA is at the hospital level, the criteria are more restricted than those required by the HAS, which works from the perspective of compulsory health insurance and society as a whole.
Despite these limitations, several works and publications published outside the period of this research show that the issue of defining and integrating OIs by HTA organizations is still relevant and timely (Reference O’Rourke, Oortwijn and Schuller38). HTA organizations and researchers seem to agree on the need to broaden the HTA criteria, and in particular to include OIs, especially when it comes to MDs. However, defining OIs are a prerequisite to this issue. The construction of this definition can take a deductive or inductive form. From a deductive point of view, the current lack of a conceptual definition of OIs prevents the construction of a typology. Conversely, from an inductive point of view, the additional OI criteria proposed in some publications should then be examined with appropriate methods to constitute true taxonomies.
In addition, the organizational dimension faces two major challenges that need to be addressed to move forward.
The first issue concerns the definition of the unit of analysis, in other words, the definition of the organization studied. As organizational theory has shown, the concept of organization cannot be limited to the existence of a legal structure. In the health domain, organizations and their interactions are so complex and numerous that it is difficult to represent them in a single model. This issue has recently been reiterated by several authors (18;39–41). One way to reduce this complexity is to conduct the analysis at three different levels, as proposed by EUnetHTA Core Model (17) and Pfadenhauer (Reference Pfadenhauer, Gerhardus and Mozygemba18): the micro-organizational level, corresponding to the intra-organizational level (e.g., structure and process of care in a hospital or a ward); the meso-organizational level, corresponding to the inter-organizational perspective (e.g., clinical pathways); and the macro-organizational level, corresponding to the health care system level. Each of these levels involves different stakeholders. As a result, aspects of OIs are specific to each level. For example, an MD can impact the care provided by nurses (intra-organizational level). The MD can then impact the work of the doctor in the hospital department in which they work (intra-organizational level), as well as relations with another follow-up care hospital for example (inter-organizational level, or even macro-organizational level). Moreover, depending on the point of view adopted (that of the hospital or society for example), it is a different aspect of the organizational dimension that is considered, as is the case in economic evaluations. This topic is clearly identified by the revised HTA definition which highlights “the intended and unintended consequences of using a health technology” and underlines that “the overall value may vary depending on the perspective taken, the stakeholders involved, and the decision context” (Reference O’Rourke, Werko, Merlin, Huang and Schuller42).
The second issue concerns the delimitation of the boundary of the evaluation of OIs with the economic domain. To the extent that resources are considered an aspect of the organization, the modification of a resource can simultaneously be considered as an OI and an economic impact. There is, therefore, a risk of monetary double counting when the HTA is carried out in the perspective of a request for financing or reimbursement. This risk is particularly high when technology introduces changes in intangible assets such as skills, organizational structures and processes, and culture, the economic effects of which are difficult to demonstrate (Reference Brynjolfsson, Hitt and Yang43). Thus, if a new technology involves improving the skills of health professionals, the need for new skills could be listed, on the one hand, in the OIs, and, on the other hand, valued in the economic impacts. Although these effects do not occur at the same time, the promoter of this new technology would thus risk highlighting the same impact twice, in different categories but also valuing it twice.
In conclusion, this research could also interest other economic sectors. Indeed, any technological innovation has an impact on the organizations that develop or use it, and even more broadly on society as a whole. It therefore seems useful to go beyond this boundary.
Abbreviations
- AdHopHTA
-
Adopting Hospital Based Health Technology Assessment
- AGENAS
-
Agenzia Nazionale per I servizi sanitari regional
- CADTH
-
Canadian Agency for Drugs and Technologies in Health
- DACEHTA
-
Danish Centre for Health Technology Assessment
- EUnetHTA
-
European Network for Health Technology Assessment
- ICER
-
Institute for Clinical and Economic Review
- IQWiG
-
Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen
- LBI-HTA
-
Ludwig Boltzmann Institute for Health Technology Assessment
- MS-DECIT
-
Ministry of Health of Brazil, Secretariat of Science, Technology and Strategic Inputs, Department of Science and Technology
- MSAC
-
Medical Services Advisory Committee
- NICE
-
National Institute for Health and Care Excellence
- SBU
-
Swedish Agency for Health Technology Assessment and Assessment of Social Services
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/S0266462322003221.
Acknowledgments
The authors thank the members of the methodological support group of this project: Vincent Augusto, École des Mines, Saint-Étienne; Valérie Buthion, University Lyon 1; Dominique Costagliola, University Pierre and Marie Curie Paris 6; Benoit Dervaux, University of Lille; Carine Franc, Centre d’Epidemiologie et de Santé des Populations, Villejuif; Thierry Garaix, École des Mines, Saint-Étienne; Bernard Guillot, University of Montpellier; Julie Le Cardinal, Centrale-Supélec, Paris; Jean-Claude Moisdon, École Nationale Supérieure des Mines Paris; Pascal Sellier, member of an association of patients and users of the health system; as well as the services of the High Authority of Health involved Chantal Andriamanga, Emmanuelle Blondet, Juliette Chazareng, Thierno Diatta, Marie Georget, and Estelle Piotto. The authors would also like to extend special thanks to the anonymous reviewers of the article whose comments significantly improved the original manuscript.
Funding statement
This work was supported by the Haute Autorité de santé (French HTA agency) which has allowed us to use the results of this research.
Conflicts of interest
The authors declare none.
Open access
