Hospitals are often the main entry level for new health technologies (HT), and they invest a considerable volume of resources in implementation of new HT. There is growing interest in hospital-based health technology assessment (HB-HTA) as a tool to facilitate hospital decision making that is based on a systematic and multidisciplinary assessment of the evidence for new HTs (Reference Gagnon1). The background knowledge and available scientific evidence that underpins HB-HTA is the same as that used for national and regional HTA. However, the information required for making decisions about the introduction of a new HT, the time frame and the relative importance given to the different types of information may differ according to the organizational level of the healthcare sector at which decisions are made (Reference Sampietro-Colom, Morilla-Bachs, Gutierrez-Moreno and Gallo2).
HB-HTA links evidence-based clinical data with the unique organizational and economic implications of a new HT at the local hospital level, thus providing shorter and more timely HTA reports to hospital decision makers (Reference Sampietro-Colom, Morilla-Bachs, Gutierrez-Moreno and Gallo2). The decision-making process at hospitals, including the informational needs of hospital and clinical managers when deciding whether or not to invest in health technologies, is generally not well described, however (Reference Greenberg, Peterburg, Vekstein and Pliskin3).
One of the most widely used set of guidelines on how to perform and report HTAs is the Core Model developed by EUnetHTA (the European network for Health Technology Assessment), from a collaboration of primarily national HTA institutions. The Core Model describes a large number of potential elements for assessment (topics and items) that are divided into nine domains (Reference Lampe, Mäkelä and Garrido4). HB-HTA products should be aligned with the needs of the final decision makers at hospital level, but the level of agreement between the Core Model and the informational needs of hospital decision makers is currently unknown.
The primary objective of this study was to review empirical studies that analyze the information hospital decision makers want to have at their disposal before making decisions on HT investments. Furthermore, we wished to rank the importance of the requested information based on the number and frequency of mentions in the identified literature.
This systematic review is part of the European research project AdHopHTA (Adopting Hospital Based Health Technology Assessment in EU; http://www.adhophta.eu/) which aims to strengthen the use and impact of HTA in hospital settings. The results of this literature review will be used to develop qualitative interviews and a questionnaire survey among hospital decision makers in Europe.
METHODS
Identification of Information Types
A systematic review of empirical studies was carried out to identify the informational needs of hospital and clinical managers when deciding about HT investments. We searched for empirical studies published in English or Danish from 2000 to 2012 (November) in the PubMed, Embase, Cochrane Library, and Web of Science databases. The subject of the literature search had three topics (decision maker [who], informational need [what], hospital setting [where]) and we limited the search to specific study designs. For each of the three topics, we defined the various queries (with similar search terms using both Medical Subject Headings [MeSH] and free text) and combined them into a final query. The search strategy was reviewed and refined by a senior research librarian. Supplementary Table 1 provides the full search histories.
After exclusion of 517 duplicates between the databases, the 2,689 articles that were identified as being potentially relevant were reviewed by two authors (K.K. and A.M.O.), working independently of each other. In case of disagreement, a third opinion (M.B.O.) was sought and in- or exclusion was resolved by discussion among the authors. Assessment of the literature was carried out in two phases—first by examining relevance by reading only the title and abstract, and then by reading the full text of those articles still deemed relevant. A kappa coefficient was calculated to measure the correlation between assessments made by the two independent reviewers. This coefficient takes into account the fact that part of the observed correlation between two assessments is due to chance (Reference Landis and Koch5).
When reviewing the literature, we looked specifically for evidence of which information that hospital decision makers needed when deciding on HT investments. This information was retrieved from empirical studies of hospital managers’ attitudes and of decision-making processes in hospitals. Inclusion criteria were: (i) articles reporting on informational needs in a decision-making situation, (ii) in a hospital context, and (iii) based on an empirical study (not commentaries, letters, opinions, etc.). Systematic reviews of empirical studies were also accepted. The included articles were reviewed and the main types of information needed by hospital decision makers were listed.
Categorization of Information Types
The different types of information requested by hospital decision makers were categorized according to the nine domains of EUnetHTA's Core Model. The types of information were discussed and categorized by the authors jointly after reviewing the Core Model. The level of agreement between the identified types of information and the topics included under each domain in the Core Model was taken into consideration when categorizing the information.
The different types of information were seldom clearly defined in the literature, making it difficult to assess whether informational needs with very similar wording had the same meaning. Therefore, even when the identified types of information appeared to be very similar, the types were not merged unless their wording was exactly the same. The lack of clear definitions also meant that some types of information could be interpreted in different ways and could be categorized under two different domains.
Ranking of Importance of Information Types
The relative importance of the identified types of information was determined according to (i) the number of different information types within each domain, and (ii) the frequency with which the information types within each domain were mentioned in the literature. This second approach based on the frequency of mentions in the literature has been used in previous studies (Reference Galani and Rutten6;Reference Guindo, Wagner and Baltussen7). The results were compared with the results of studies in which hospital decision makers were explicitly asked to assess the relative importance of different types of information.
RESULTS
Identification of Information Types
The combined search strategies identified 2,689 articles, from which 2,664 were excluded after review of the title and abstract. The remaining twenty-five full-text articles were reviewed and fourteen empirical studies or reviews of empirical studies were considered relevant and included in the analysis (Figure 1). Relevant characteristics of the included literature are presented in Table 1.
Flow-chart of the systematic literature review, including reasons for exclusion of articles.
Characteristics of the Studies Included in the Systematic Review and Their Findings Regarding Decision-Makers’ Need for Information.
Reasons for exclusion were that (i) the aim of the study was not to identify informational needs of decision makers (wrong aim), (ii) the study did not deal with decision making by hospital or clinical managers (wrong population), (iii) the study did not take place in a hospital setting (wrong context), (iv) the article was not based on an empirical study or was based on an empirical study of low quality (wrong/poor study design), (v) the article was not in English or Danish (wrong language), or (vi) the article was not retrievable (n = 2). The kappa coefficient was 0.54, indicating moderate agreement between reviewers in the initial assessment of articles.
Half of the fourteen included studies were conducted in Canada, Spain, and Israel, and the rest were conducted in other European countries, United States, and Australia. The study methodologies varied, but typically included semi-structured interviews, questionnaire surveys and literature reviews. From the fourteen articles, we identified seventy-four different types of information requested by hospital and clinical managers.
Categorization of Informational Types
Most of the seventy-four types of information could be categorized within the nine domains of EUnetHTA's Core Model (Table 2). Due to ambiguity in wording, fifteen (20.3 percent) of the seventy-four information types were categorized under a maximum of two domains.
Categorization of information Types According to Ten Domains, and the Frequency with Which These Types Were Mentioned in the Fourteen Articles Included in the Systematic Review
Note. For example, ten of the information types that were identified in the literature (Table 1) were categorized within Domain 1 (number), and they were mentioned a total of thirteen times (frequency). The shadowed cell indicates a new domain that is not covered by EUnetHTA's Core Model.
Ten (13.5 percent) of the seventy-four information types did not fit easily into any of the domains of the Core Model and were categorized under a new, tenth domain entitled “political and strategic aspects” (Table 2). Six of the fifteen (40.0 percent) information types that were not clearly defined were categorized under both the new, tenth domain and one of the other domains. Thus, a total of sixteen (21.6 percent) information types were categorized under the new domain covering “political and strategic aspects” of HT investments (Table 2). Political aspects referred to, for example, the alignment between the decision to invest in a given technology and the local political climate or values. Strategic aspects referred to, for example, the fit between a given technology and the hospital's research strategy, as well as prestige and competition between hospitals in relation to a specific technology or health problem.
The third domain (D3) dealing with “clinical effectiveness,” included information about clinical outcomes (e.g., quality of life) and effect sizes (e.g., patient impact) as well as characteristics of the evidence (e.g., quality of evidence).
The fifth domain (D5), dealing with “costs and economic evaluation,” included information from traditional health economic evaluations with a broad societal perspective (e.g., cost-effectiveness analysis) as well as narrower budget impact analyses with a hospital perspective (e.g., business cases).
The domains dealing with ethical (D6), safety (D4), and social (D8) aspects had the fewest number of information types (Table 2). Information on the patient perspective appeared only once in the literature (Domain 8 in Table 2).
Ranking of Importance of Information Types
Regardless of the approach used to rank the domains in terms of importance (i.e. number of different information types vs. frequency of mentions in the literature), the same five domains were ranked as the most important, albeit in differing order (Table 2). These domains were those dealing with information about political and strategic aspects (D10), clinical aspects (D3), economic aspects (D5), organizational aspects (D7), and the health problem and current use of the technology (D1). Mainly information about organizational aspects changes from being the second most important type of information when looking at the number of different information types within each domain (column 2 in Table 2) to being the fourth most important type of information when looking at the frequency of mentions in the literature (column 4 in Table 2).
Three of the fourteen reviewed articles described studies in which hospital decision makers were directly asked to rank different types of information in order of their relative importance—one study from Spain (Reference Sampietro-Colom, Morilla-Bachs, Gutierrez-Moreno and Gallo2) and two from Israel (Reference Greenberg, Peterburg, Vekstein and Pliskin3;Reference Greenberg, Pliskin and Peterburg14). These studies used Likert scales, rank-order, or both to assign importance.
Most of the information types that emerged from these studies as being important came under the same five domains that we identified as being important based on frequency of mentions in the literature. However, information about political and strategic aspects (D10) was rarely mentioned as important in the three direct measurement studies. Furthermore, a description of the technology and its technical characteristics (D2) and information about safety (D4) and legal aspects (D9) were ranked as important aspects in the direct measurement studies, but were not identified as important using our approaches. Of note, the social (D8) and ethical (D6) aspects of a new technology were rarely considered as important information for decision making.
DISCUSSION
This systematic review identified fourteen relevant empirical studies that contained seventy-four different types of information requested by hospital decision makers when deciding on HT investments. The EUnetHTA guidelines for performing HTA, that is, the nine domains of the Core Model (Reference Lampe, Mäkelä and Garrido4), included most of these different types of information (Reference Lampe, Mäkelä and Garrido4). However, we identified types of information that related to a new, tenth domain covering “political and strategic aspects.”
The additional domain dealing with the strategic and political aspects of investments in new health technologies is in line with McGregor's (Reference McGregor19) assertion that investment decisions are dependent on political and social pressures and the opinion and values of the hospital decision makers. According to Gray (Reference Gray20) healthcare decisions are based on a combination of three factors: evidence, values, and resources. At present, some decisions may be driven principally by values and resources—a process Gray (Reference Gray20) describes as opinion-based decision making. The new domain dealing with “political and strategic aspects” (D10) cover information that by definition is something else than evidence. These types of information are, however, requested by hospital decision makers and, therefore, it may be worth considering including information on these more value-based aspects of HT investments as part of a basis for decision making in hospitals.
We found that information about political and strategic aspects of new technology (D10), together with clinical (D3), economic (D5), and organizational (D7) aspects were mentioned most frequently in the literature and were also those domains with the highest number of different information types. The importance of information about clinical effectiveness and economic aspects was confirmed in the three direct measurement studies, but not information about political and strategic aspects (D10). The reason may be that the hospital decision makers were not directly asked to consider these (new) aspects of HT investments in these studies. A recent systematic review of decision criteria for resource allocation and healthcare decision making showed that the most frequently cited category of criteria was “Overall context” including (among others) political aspects and stakeholders interests and pressures (Reference Guindo, Wagner and Baltussen7). It does seem then, that consideration of the political and strategic aspects are important in hospital decision making.
Our results suggested that hospital decision makers rarely focus on information about ethical (D6), safety (D4), social (D8), and legal (D9) aspects of new HT. This may be due to less familiarity with this type of information or with the way in which the information (especially about legal and ethical aspects) is collected. It may be that safety information was assumed to be included in the domain on clinical outcomes and effectiveness (D3). Furthermore, it was somewhat surprising that information on patient satisfaction and patient preferences was rarely directly requested, given that the patient perspective is one of four main categories in the Danish template for mini-HTA (21).
Future studies on the relative importance of different types of information to hospital decision makers should distinguish between information on clinical outcomes and effect sizes on the one hand, and information on the characteristics and quality of the evidence on the other. Both of these types of information were categorized here within the domain of clinical effectiveness (D3), but it is possible that hospital decision makers weight these types of information differently. Quality of evidence is a key determinant for the strength of recommendations for or against the adoption of a given HT (Reference Tanios, Wagner and Tony22) and this type of information was considered the most important in one of the three direct measurement studies (Reference Sampietro-Colom, Morilla-Bachs, Gutierrez-Moreno and Gallo2).
It may also be relevant to examine more closely the relative importance of the broad societal perspective and the narrower hospital perspective in economic analyses of HT. Most of the information types identified from the literature related to the local hospital perspective, but investigating the relative importance of different perspectives when directly asking hospital decision makers to prioritize between information in future studies will be interesting. McGregor (Reference McGregor19) noted that cost-effectiveness of a given technology does not determine investment decisions, but budget impact does. Similarly, Gallego et al. (Reference Gallego, Taylor and Brien15) found that information on the local budget impact of a new technology was more often requested by hospital decision makers than conventional economic evaluations with a broad societal perspective.
Likert scales and rank ordering exercises were used in the three direct measurement studies (Table 1). The Spanish study used only a Likert scale, and the results showed that the decision makers mostly agreed that all the types of information were important to some extent (Reference Sampietro-Colom, Morilla-Bachs, Gutierrez-Moreno and Gallo2). The use of Likert scales alone does not force decision makers to prioritize between different types of information. A combination of Likert scales and ranking (Reference Greenberg, Pliskin and Peterburg14) is, therefore, recommended in future studies.
A recent international questionnaire survey invited healthcare decision makers to report which criteria they consider when making decisions on healthcare interventions (Reference Tanios, Wagner and Tony22). Respondents were asked to indicate whether each decision criterion was “currently considered” or “should be considered” and its relative weights. The most relevant criteria were found to be: (i) clinical efficacy/effectiveness, (ii) safety, (iii) quality of evidence, (iv) disease severity, and (v) impact on healthcare costs. Organizational and skill requirements were frequently considered, but had relatively low weights, suggesting that their impact on the final decision might be fairly small. These results are largely consistent with the results of the three direct measurement studies in the current review.
We found that EUnetHTA's Core Model did not include all the types of information considered in hospital decision making. What are the implications of this for HB-HTA? Should guidelines for HB-HTA be adjusted and thereby differ from those for full HTA, for example, EUnetHTA's Core Model?
One possibility is that HB-HTA should focus exclusively on the clinical, economic, safety, organizational, strategic, and political issues associated with the introduction and use of a specific HT. McGregor (Reference McGregor19) suggests that there is often a disconnection between the rigorous and careful collection of evidence as part of the HTA process, and the failure of this process to influence policy decisions. This is partly due to the HTA often being delivered too late for inclusion in the final basis for decisions. Results from a Polish study suggested that healthcare managers favor speed over accuracy of information in evidence-based decision making (Reference Niedzwiedzka13). A more focused and targeted approach to assessing HT, leaving out, for example, ethical and social aspects not valued highly by hospital decision makers, might allow a faster and less resource-consuming assessment at hospital level.
Further research is needed before concluding anything definitive about the practical implications for HB-HTA. We need more knowledge about what hospital decision makers understand by strategic and political aspects and what importance they place on the quality of clinical evidence and the different perspectives used in economic analyses of HT. These issues will be among those investigated in the interview study and the questionnaire survey among hospital decision makers in Europe that are part of the next steps in the AdHopHTA project.
Methodological Considerations
Several methodological considerations need to be taken into account when interpreting our results. First, the literature search was restricted to articles in English and Danish, and relevant literature in other languages could have been missed. In fact, no relevant literature in Danish was identified, so this language choice had no impact on the final results. In addition, we could not retrieve two potentially relevant articles.
Second, although all the reviewed studies included hospital managers and/or clinical managers, it was not always possible to isolate their results from those of other decision makers included in the study. Some studies were conducted at hospital level (micro), others at regional (meso) and national (macro) levels of decision making. Thus, we cannot be sure that our results are based solely on the informational needs of hospital and clinical managers.
Third, the included articles had very different purposes and research questions. Some articles investigated the relevance of a single criterion or element (e.g., the patients’ perspectives) or particular product (e.g., mini-HTA or economic evaluation), while others focused on decision making in relation to a specific type of technology (e.g., expensive pharmaceuticals).The included literature also involved different study methodologies (typically systematic reviews, semi-structured interviews, and questionnaire surveys) and very different sample sizes, which is consistent with the methods used in the AdHopHTA project. This might, however, have influenced the importance ranking of the ten domains based on the number of information types or the frequency of their mentions in the literature, and these results should, therefore, be interpreted with caution. Notice, that the three direct measurement studies were among the fourteen relevant articles included in this review (Study no. 5, 8, and 14 in Table 1).
Fourth, the literature used a variety of concepts to describe the information that decision makers need when deciding whether or not to invest in HT, including, for example, “decision support,” “information,” and “decision criteria.” We have not distinguished between these concepts, and this study thus concerns the need for “information” among hospital decision makers, which is a wider concept than the specific and measurable “criteria”.
Fifth, the different types of information were discussed and categorized jointly by the authors after a thorough review of the Core Model. It would have been preferable for the authors to categorize the information independently before having a joint discussion. However, the result of the categorization was subsequently discussed with and validated by a group of HTA experts in the AdHopHTA project, which enhances the quality of the analysis (please see http://www.adhophta.eu/ for further details about the AdHopHTA project).
Because the different types of information were seldom clearly defined in the literature, we found fifteen of them to be sufficiently ambiguous that they had to be categorized under two domains. For example, information on the level of maturity of the technology (“Technology is still experimental”) could be placed either in domain 1 (D1: Health problem and current use of the technology”) or domain 2 (D2: Description and technical characteristics of the technology). Similarly, information about the effect of a specific technology on the exploitation of the hospital (“Influence on hospital's exploitation”) could be categorized under domain 5 (D5: Costs and economic evaluation) or domain 7 (D7: Organizational aspects). Even when we forced each of these fifteen ambiguous types of information into one domain only, the results remained largely unchanged. Thus, the most important types of information were still within the same five domains. The only difference was that information on organizational aspects (D5) went from being the third most important to the second most important information together with information on clinical aspects (D3) when using the number of information types as a measure of the relative importance.
Finally, the included literature was based on research conducted in ten different countries. Results cannot be transferred uncritically from one context to another because of national differences in healthcare systems, decision-making processes, and attitudes toward the use of HTA.
CONCLUSION
The results of this systematic review suggest that hospital decision makers are able to describe their informational needs when deciding on HT investments. The domains of EUnetHTA's Core Model appeared to cover most of the informational needs of hospital and clinical managers, but full agreement was not observed. In addition to the domains of the Core Model, hospital decision makers also seek information on strategic and political aspects not covered by the model. Furthermore, the domains are not of equal importance to hospital decision makers. Clinical, economic, and strategic/political aspects are mentioned most frequently in the literature. The importance of clinical and economic aspects is confirmed in studies of relative importance of different types of information among hospital decision makers. Finally, this literature review also shows that the relative importance that hospital decision makers assign to different types of information has seldom been examined.
The results of this systematic review provide further knowledge about the types of information that hospital decision makers consider relevant when they decide on HT investments. This information will be useful for directing future empirical studies on this subject, including the interview study and questionnaire survey that will be conducted in the next phase of the AdHopHTA project.
SUPPLEMENTARY MATERIAL
Supplementary Table 1 http://dx.doi.org/10.1017/S0266462315000665
CONFLICTS OF INTEREST
Dr. Ølholm reports grants from EC Seventh Framework Programme, during the conduct of the study. Drs. Kidholm, Birk-Olsen, and Buck Christensen have nothing to disclose.
Open access
