Work Package 4 (WP4) of the EUnetHTA Project (Reference Kristensen, Mäkelä and Allgurin Neikter6) developed a standardized structure for health technology assessments, that is, the HTA Core Model (4;5). Its novel elementary structure allows rigorous production and transparent presentation of health technology assessment (HTA) information (Reference Lampe, Mäkelä and Velasco-Garrido7). The HTA Core Model consists of 9 domains (Table 1). Each domain is divided into more specific topics, and further into issues in the form of generic questions. The combination of domain, topic, and issue defines an assessment element, the basic unit in the Model. The elements are divided into core and noncore elements based on their importance and transferability (Reference Lampe, Mäkelä and Velasco-Garrido7). The Model guides the HTA doers first to consider the relevance of each assessment element for the technology. For each relevant element, the generic question is translated into a specific question concerning the technology. A Core HTA is the compilation of the questions and answers of relevant core elements for a specific technology, and a summary chapter.
Domains in the HTA Core Model
HTA, health technology assessment.
The EUnetHTA Project piloted the HTA Core Model with two topics: a therapeutic intervention and a diagnostic technology. These pilot Core HTAs (2;3) tested the concept and structure of the Model and the adequacy of the assessment elements. Because piloting primarily aimed at experimenting the Model and its use, not all issues were assessed with full scientific thoroughness. Therefore, the pilot Core HTAs should not be considered as stand-alone HTAs or used in actual decisions in health care.
The Finnish Office for Health Technology Assessment (FINOHTA) guided the production of the Core Model and the pilot HTAs. Practical guidance for future work is compiled in a Handbook (Reference Lampe and Pasternack8).
AIMS AND OBJECTIVES
This article analyzes and describes the processes and outcomes of the two pilot Core HTAs that were used to test the HTA Core Model. The focus is on the essential points for using the Model: scoping and framing, assessing relevance of generic issues, translating issues into research questions, and reporting answers in the structured format. We also discuss some relevant problems encountered during the two pilot Core HTA projects, their solutions, and the applicability of the HTA Core Model.
STRUCTURE AND METHODS
The pilot Core HTAs on drug eluting stents (DES) and multislice computed tomography (MSCT) in coronary angiography were based on draft versions of the HTA Core Model (4;5). Description of the Model and its domain specific guidance are presented elsewhere in this journal (Reference Lampe, Mäkelä and Velasco-Garrido7). Supplementary Table 1, which is available at www.journals.cambridge.org/thc2009002, presents the number of investigators and reviewers in each of the domain specific working teams. The task of the teams was to select relevant questions in the Model and produce evidence based answers to them. Internal and external feedback was sought during a validation process.
Data for this article were gathered from the first public drafts of the two pilot Core HTAs (2;3), their validation feedback, a questionnaire sent to investigators in domain teams (10 of 18 replied), minutes of project and team meetings, emails, and direct discussions among EUnetHTA coordinators and primary investigators at FINOHTA.
RESULTS
Topic Selection
Following a two-step, Web-based voting procedure, organizations participating in WP4 selected DES (intervention) and MSCT (diagnostic) as pilot topics out of twenty proposals for interventions and fourteen proposals for diagnostic technologies. Five criteria were considered in the selection: European scope, transferability, relevance for several domains, feasibility of carrying out the assessment, and voter's preference (Supplementary Table 2, which is available at www.journals.cambridge.org/thc2009002).
Scoping and Framing
Comparing drug eluting to bare metal stents was a meaningful scope for the DES pilot. This framing was used in three domains, that is, Description and technical characteristics, Effectiveness, and Costs (Table 1: Domains 2, 4, and 5). Other domains framed at least some of their questions more broadly: they considered stents in general, cardiac interventions (including coronary bypass surgery), or even medical devices at large in their answers. For instance, in the Social domain, the research question about postintervention work ability referred to studies comparing stents with angioplasty. In the Organizational aspects domain, the description of information flow in integrated patient pathways was not specific to DES patients but applicable to any patient undergoing surgery.
Scoping for MSCT coronary angiography had different problems. This noninvasive imaging modality could replace some invasive coronary angiographies (ICA) in diagnosing coronary artery disease (CAD). ICA is the gold standard for CAD diagnoses, but for MSCT, ICA is not the clinically most relevant comparison test. Unlike MSCT, ICA can also be extended to a therapeutic intervention: the obstructed vessels can be opened during the procedure. The investigators therefore identified the following scope for their assessment: comparing a diagnostic pathway with and without MSCT in patients with low to moderate risk of CAD. This scope was strictly followed in the Effectiveness domain, but less so in other domains.
Assessing the Relevance of the Assessment Elements
The draft Intervention Model contained 163 assessment elements; on average 18 (range, 6–29) per domain. Ninety of them were labeled as core elements. The investigators in the DES pilot considered 77 percent of all elements and 79 percent of the core elements to be relevant in the context of DES. The percentages were smaller than in MSCT, partly because elements in the Safety domain were not specifically assessed in DES.
The draft Diagnostic Model contained 158 assessment elements, 87 of them belonging to the core. Investigators in the MSCT pilot considered 90 percent of all elements and 89 percent of the core elements relevant in their context. All Effectiveness and Organizational elements were considered relevant in both pilot HTAs.
From Issues to Research Questions
Typically, one generic issue in an assessment element was translated into one specific research question. Among the 117 generic issues answered in the DES pilot, 6 (5 percent) were translated into several research questions. In the MSCT pilot, 11 of 128 answered issues (9 percent) were each translated into two to four research questions.
We asked the investigators how easy it was to translate the generic questions in the Model into technology-specific research questions. The task was simple according to eight of ten respondents. Investigators of the Ethical analysis domain noticed that formulating the question is a crucial step requiring careful thought, or even an analysis of the stakeholders’ values attributed to the technology and its use. Some believed that the categorization or naming of generic issues hindered translation, which led to proposed changes in the Model.
Finding Information for Answers
A systematic literature review was the standard method used to find information to answer questions. An information specialist at FINOHTA designed and performed the basic searches, with search terms covering the disease and the technology. The teams were urged to supplement these with more specific searches. In some instances, a systematic review was clearly infeasible, or even unnecessary. For instance, answers to the questions about incidence or the current management practices of CAD were based on information derived from recent HTAs or guidelines, instead of systematically reviewing original research. Some investigators had recently performed traditional HTAs, or systematic reviews on DES or MSCT, and could transfer relevant pieces of information directly into the pilot Core HTAs. Earlier work was used in domains 3, 4, and 5 of the DES pilot and in domains 1, 2, 3, and 5 of the MSCT pilot.
Due to the lack of published studies, the Organizational and Social domain teams did their own research, mainly using semistructured or structured interviews of experts, key officials, clinicians, or patients. The Costs and economic evaluation domain used modeling. Compiling information from national registries, databases, manufacturers’ information sheets, and standard protocols provided answers especially to domains 1, 2, and 9. Answers in the Ethical aspects domain were based on published data of stakeholder views, but the final output usually required additional primary inquiries.
The MSCT pilot could use results created earlier for DES. Five structured pieces of information from the DES Core HTA, about characterization, symptoms, incidence, and mortality of CAD, were exploited as such, or in a slightly modified form, in the MSCT Core HTA.
Reporting the Results
Half of the Domain teams managed to answer all relevant issues. In the DES pilot, 10 of 126 relevant issues were left unanswered. Six of them belonged to the core elements (8 percent of all relevant core elements). In the MSCT pilot, answers to 17 of the 143 relevant issues were missing, 6 of these being core elements (8 percent of all relevant core elements). The typical reason given was time pressure. Lack of capacity or motivation could also explain this, as some Domain teams had few substance or methods experts. The Safety domain teams did not produce any text for DES, considering it too artificial to discuss safety and effectiveness issues separately in this particular context.
Preparing the Summary
No summary was written for the DES pilot. Afterward, discussions in WP4 and the validation feedback underlined the need for a summary. The General Design team of the MSCT pilot decided that brief domain summaries were compiled by the editor into a two-page summary for the whole Core HTA document. No conclusions or recommendations were included.
Validation
The first public draft of the Core HTA on DES was published in June 2007 and on MSCT in July 2008. These documents were submitted for validation to EUnetHTA participants and INAHTA members by email. Public feedback was sought with a separate questionnaire on the EUnetHTA Web site and personal reminders to organizations. The validation questionnaires covered the feasibility of the structure from the readers’ point of view, adequacy of the research questions, and the usefulness of the answers in informing policy. There were twenty-three EUnetHTA/INAHTA respondents on DES and seventeen on MSCT, and (including collated responses from ISPOR) eight public feedback respondents on DES and two on MSCT. The responses were mainly positive, and the work was considered to be promising (Table 2). Supplementary Table 3, which is available at www.journals.cambridge.org/thc2009002, gives an overview of the problems detected in the validation, and Supplementary Table 4, which is available at www.journals.cambridge.org/thc2009002, presents domain-specific results.
Validation Results
Note. Percentage of respondents that agreed with the statements in the validation of the pilot core health technology assessments (HTAs) on drug eluting stents (DES) and multislice computed tomography (MSCT) in coronary angiography.
DISCUSSION
Scoping and Framing the Topic
The scope of the assessment, in terms of explicit definition of the patients, intervention, comparison, and outcomes should be identical throughout a Core HTA. Otherwise, the information provided in various domains might not be commensurate. In the DES pilot, the scope was drug eluting versus bare metal stents in CAD, and Domain teams kept to this. In the MSCT pilot, the scope was determined to be a comparison of a management pathway with, versus without MSCT, in patients with low risk of CAD. The scope was chosen because it was considered important to assess the value of MSCT in ruling out CAD and reducing the need for ICA in low-risk patients. The selected scope, however, was not observed by all Domain teams. The Costs domain found that there was insufficient evidence to assess the cost-effectiveness of MSCT compared with not using MSCT in the target population, which would in practice mean a blank chapter for this domain. However, for the purpose of testing the HTA Core Model, it re-defined its objectives to verify the basis for the commonly asserted statement that MSCT is cost-effective compared with ICA in a specific patient population.
Scoping study questions in the pilot projects thus had various rationales, clinical relevance being the most obvious. At times, a scoping is selected because systematic reviews are available. The problem with this kind of opportunistic scoping is that it may introduce an unbalanced HTA where problems important for implementation are left out.
When the scope of the assessment is clear, it is possible to let the Domain teams frame their research questions differently. Different framing means that research questions can be expanded to include a broader category, still retaining the scope. For example, drug eluting stents represent stents in general. Stenting is a coronary intervention like angioplasty and coronary bypass; they all belong to cardiac interventions. Often a technology shares problems of the group it belongs to. Drug eluting stents are devices that release a drug; their drug licensing issues can be shared with drugs in general. MSCT shares safety issues of computer tomography technologies in general, or any imaging modality using radiation.
Variation in framing across the Domains of a Core HTA or across single question answer-pairs is not a problem if explicitly reported. The Social domain of the DES pilot broadened their frame from the strict drug eluting versus bare metal stent comparison to allow assessment of the social aspects of stents in general, as the team assumed that the impact on a patient would be largely independent of stent type. In the absence of research in an area, presenting evidence from a differently framed, but well researched, angle may be useful, although the information might not be directly applicable.
Overlaps Across Domains
Half of the generic questions in the Model are obviously interrelated with each other. Questions similar at first glance usually should be answered from the viewpoint of each domain. For example, the element of autonomy and informed consent in the Ethical analysis domain reflects the views of patients themselves; in the Legal domain, it is a formal prerequisite to invasive procedures.
Obvious linkages between elements call for an organized HTA process: some assessment elements use information from elements in other domains. Knowledge about disease incidence is required when translating generic questions into actual research questions in Effectiveness Domain, and effectiveness and safety data are needed for Costs and Economic Evaluation domain answers. Linkages between elements are a problem only if real overlapping leads into duplication of work. Future Core HTAs must start by identifying elements requiring input from several domains to agree on timing and division of labor.
Core Elements
Each element in the HTA Core Model has been assigned a category for importance and transferability. In DES and MSCT pilots, some domain teams made a topic-specific reassessment of these categories, although it is, at the moment, not considered as a standard feature of Core HTAs. They encountered the same difficulties in the assessment of importance and transferability as the Model doers. For example, in the Ethical analysis domain, the authors believed that almost every element could be defined as important from some stakeholder's point of view. Transferability was considered not the property of the question alone while it depends very much on how extensive and detailed the answer will be. Detailed incidence data per country may be more transferable than aggregate figures for Europe. An assessment element describing variation in the use of a technology in several countries, as occurred in DES, may importantly hint at over- or underuse of the technology locally. In the Costs domain of the MSCT pilot, results were obviously not transferable to other contexts, but investigators considered the issues to belong in the core; unit costs are rarely transferable as such, but they provide a useful perspective to local data. It is the HTA user who eventually determines the transferability of the information presented in a Core HTA. An Italian decision maker finds Italian cost data transferable, whereas his/her Swedish colleague does not. When the Swedish reader gets cost information from seven European countries, he might consider these partially transferable or useful for putting his own costs into perspective, even without Swedish data. Transferability assessment is further discussed by Turner et al. in this journal (Reference Turner, Chase and Milne9).
Coordination
The shared opinion of the pilot HTA investigators was that, in the future too, it is possible, in principle, to divide the work between HTA agencies. One or two domains per agency would be a natural way to share Core HTAs. Contributions could also be less; only some topics or issues from one domain. Strong coordination of the work by professional project leaders was considered essential if agencies are to share the work. This requires monitoring to ensure that the Domain teams provide answers to all relevant questions, with the same scope, in a reasonable order and without duplication of work. Editors need to guarantee congruence and organize quality control in terms of methodological validity, comprehensiveness, and timeliness. English language checks are important in this multilingual environment with few native English speakers.
Communication remains a major issue when performing network HTAs. Conference calls or e-meetings were planned for pilot HTAs, but proved impractical at an early stage and were seldom used. However, with increasing experience, the performance and utility of firmly organized e-meetings was improving. Working only through email was believed to be tedious. The new communication platform proposed by the Information Management System of the EUnetHTA Collaboration (Reference Allgurin Neikter, Rehnqvist, Rosen and Dahlgren1) might provide a solution for the future.
Reporting Core HTAs. In the feedback for both pilot Core HTAs, several respondents pointed out poor reporting. They called for justification of methods used and information on search strategies, inclusion criteria, and study quality. The common methodology chapter for the whole domain was considered inadequate. Readers generally preferred having the methods described separately for each research question. Some of the introductory texts on domains in the pilot HTAs were deficient: definitions or focus were missing, or the introduction already presented a summary of the answers.
The pilot Core HTAs reported results as question-answer pairs. This was generally considered to improve consistency and clarity. On the other hand, the authors of the Ethical analysis domain did not find the splitting of results a good solution. A balanced discussion, written in HTA prose, was preferred. An ethical analysis was considered more complex than a set of answers to specific technical research questions. Additionally, the answers usually required some background information for the reader to understand the message. This means we would need to link identical pieces of text containing background information, to several answers. In traditional paper format, this redundancy is not ideal. As Core HTAs are ultimately electronic collections of assessment elements (with a summary text), in the form of question-answer pairs, such redundancy is acceptable and even necessary. Because each element should be understandable alone, cross-references to text, for example, “see above,” are not encouraged.
Readers’ View. There were varying opinions in the validation feedback on the required extensiveness and level of detail of the Core HTAs. This probably reflected the two reader groups; researchers and decision makers. Researchers, mainly HTA producers, preferred the elementary structure and detailed information on sources and quality of evidence. Decision makers found the current 200-page format too extensive. One respondent even asked for separating issues relevant for decision makers and providing details in an appendix. People had different views about the role of the summary chapter in a Core HTA. Suggestions ranged from a collection of domain results to a single conclusion or a recommendation with pragmatic instructions for action.
Future Use of the Core Model
The domain-specific lists of generic questions were considered useful checklists, even when starting to prepare a traditional HTA report. Flexible selection of issues, including noncore issues, is necessary. Instead of doing a whole Core HTA, several people who had participated in the pilot Core HTAs considered it possible to provide answers, at least to some of the assessment elements in English when making a local HTA, and sharing them through the EUnetHTA Collaboration. This would gradually build toward a joint European pool of structured HTA information. Using assessment elements in the pool, as such or in a slightly modified form, is an apparent benefit that might save resources.
The HTA Core Model is evolving; new issues, and even domains, can be brought into the Model, and others combined or removed. The decision in the DES pilot to exclude all issues in the Safety domain was motivated by the strong linkage between safety issues and effectiveness. Therefore it was asked, whether a separate Safety domain was necessary. After considerations, it was determined that safety issues deserve their own domain, separate from the Effectiveness domain in the HTA Core Model. This was apparent already in the MSCT pilot. Safety is not simply the inherent, unwanted effects of the intervention to the patient, but also includes preventable, performance-related issues, and possible risks caused to practitioners and public health.
The HTA Core Model represents a major shift in the content and work processes of a traditional HTA. Not all HTA agencies envisage sharing work with other HTA units yet. Some agencies already foresee true European collaboration, provided that the future online system of the HTA Core Model truly facilitates group work.
The role of Core HTAs in decision making needs further pilot testing and discussions. The overall opinion of readers and investigators was that a Core HTA should be an easily shared and updated package of HTA information, with sufficiently detailed and extensive presentation, and it should provide information, not instruction, for decision making in health care.
CONCLUSION AND RECOMMENDATION
The HTA Core Model was generally welcomed as a means to facilitate European HTAs and improve their quality. Pilot Core HTAs were created for the EUnetHTA Project by a transnational group working under considerable time pressure, which caused procedural and quality problems. The feedback will be used for further development of the HTA Core Model and work processes within the EUnetHTA Collaboration.
SUPPLEMENTARY MATERIALS
Supplementary Table 1
Supplementary Table 2
Supplementary Table 3
Supplementary Table 4
CONTACT INFORMATION
Iris Pasternack, MD (iris.pasternack@thl.fi), Research Officer, Heidi Anttila, PT, PhD (heidi.anttila@thl.fi), Special Researcher, Marjukka Mäkelä, MD, PhD, MSc (marjukka.makela@thl.fi), Director; Professor, Department Of General Practice, University of Copenhagen, P.O. Box 2099, 1014 Copenhagen K, Denmark, Tuija S. Ikonen, MD, PhD (tuija.ikonen@thl.fi), Senior Medical Officer; Adjunct Professor, Department of Surgery, Turku University Central Hospital, Kiinamyllynkatu 4–8, 20520 Turku, Pirjo Räsänen, PhD (pirjo.rasanen@thl.fi), Senior Researcher, Kristian Lampe, MD (kristian.lampe@thl.fi), Senior Medical Officer, Finnish Office for Health Technology Assessment, National Institute for Health and Welfare, P.O. Box 30, 00271 Helsinki, Finland
Hans van Brabandt, MD (hans.vanbrabandt@kce.fgov.be), Medical Expert, Irina Cleemput, PhD (irina.cleemput@kce.fgov.be), Senior Health Economist, Chris De Laet, MD, PhD (chris.delaet@kce.fgov.be), Senior Medical Expert, Belgian Health Care Knowledge Centre, AC Kruidtuin, Doorbuilding (10th floor), Kruidtuinlaan 55, 1000 Brussels, Belgium
Marco Marchetti, MD (mmarchetti@rm.unicatt.it), Chief, Health Technology Assessment Unit, “A. Gemelli” University Hospital, 8 Largo A. Gemelli, Rome, Italy 00168
Pietro Refolo, PhD (pietro.refolo@rm.unicatt.it) “A. Gemelli” University Hospital, 8 Largo A. Gemelli, Rome, Italy 00168; Institute Of Bioethics, “A. Gemelli” School of Medicine, Università Cattolica del Sacro Cuore, 1 Largo F. Vito, Rome, Italy 00168
Dario Sacchini, MD, PhD (dsacchini@rm.unicatt.it) “A. Gemelli” University Hospital, 8 Largo A. Gemelli, Rome, Italy 00168; Assistant Professor, Institute of Bioethics, “A. Gemelli” School of Medicine, Università Cattolica del Sacro Cuore, 1 Largo F. Vito, Rome, Italy 00168
Regina Kunz (rkunz@uhbs.ch), Associate Professor, Alain Nordmann, MD, MSc (nordmanna@uhbs.ch), Senior Researcher, Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, Hebelstrasse 10, 4031 Basel, Switzerland
Dagmar Lühmann, MD (dagmar.luehmann@uk-sh.de), Senior Researcher, Institute for Social Medicine, University of Lübeck and University Hospital Schleswig-Holstein, Campus Luebeck, Beckergrube 43/47, 23552 Lübeck, Germany
Bo Freyschuss, MD, PhD (freyschuss@sbu.se), Project Director, Swedish Council for Technology Assessment in Health Care, Box 5656 114 86 Stockholm, Sweden; Department of Internal Medicine M52, Karolinksa Institutet, Karolinska University Hospital Huddinge, 141 86 Stockholm, Sweden
