The systematic evaluation of health related technologies within a health technology assessment (HTA) is a multistage process including not only the assessment of clinical effectiveness (Reference Busse, Orvain and Valesco1–Reference Drummond, Schwartz and Jonsson4). The applicable jurisdiction and decision process determine the conduct of a HTA. Typically, it starts with the priority setting of assessment topics. Next is the scoping, where a specific research question is selected and explicated on the basis of predefined criteria. The assessment in a narrower sense, the creating of the assessment report, starts after these steps and ideally considers clinical benefits and harms, ethical, social, legal, and organizational aspects, and, if applicable, cost-effectiveness of the procedure using the best available evidence. After evaluating the scientific evidence, the results are appraised. Finally, the results are disseminated and periodically updated. Sometimes implementation of results is supported.
Each step in the HTA process uses different sources of information and involves different groups of persons. For the assessment step, the best available scientific evidence should be regarded. Randomized controlled trials (RCT)—or even better, synthesis of several trials—provide the most reliable evidence for estimates of the clinical effectiveness of therapies (5); however, they are not always available and do not serve all research questions. Expert opinion, on the other hand, is situated on the lowest level of the hierarchy of evidence. Nevertheless, technical experts are consulted when empirical data are not available or for example for questions of practice patterns, preferences, procurement processes, or economic data (Reference Fleischner, Mittermayer and Wild6;Reference Siebert, Sroczynski and Aidelsburger7).
Sullivan and Payne (Reference Sullivan and Payne8) diagnosed an increasing need for using expert opinion in economic models due to the growing number of assessments of interventions without data from randomized controlled trials, namely diagnostics. Despite the wide application of expert data in modeling studies, it has been argued that expert opinion was applied inconsistently and its use reported insufficiently (Reference Evans and Crawford9).
When information of a low evidence level is used in HTA, it is even more important to systematically plan and document the data collection and analysis and to report it in a transparent way. The aim of the present study was to review the current state of recommendations from general HTA guidelines and handbooks regarding the involvement of experts and expert opinion in different steps of the HTA process.
METHODS
Terminology
We refer to “experts” as external persons with a special knowledge or expertise in a relevant field who are not routine members of the HTA team. With the term “expert opinion,” we summarize all contributions experts can make to the HTA process. This can be quantitative and qualitative data, experience, or value judgments.
Literature Search
We performed a Web search for guidelines and methods papers considering expert involvement in HTA on the Internet pages of seventy-seven international HTA agencies and organizations. We selected all agencies participating in the European network for HTA (EUnetHTA) during Joint Action 1 and all members of the International Network of Agencies for HTA (INAHTA) and searched their Web sites by means of search fields or hand search in the sections “tools,” “methods,” “resources,” “publications,” etc.. All screened institutions are listed in the Supplementary Table 1. As an additional source of publications, we screened the bibliography of the recent EUnetHTA guideline on methods for health economic evaluations, which is based on a review of methodological guidelines developed by the members of EUnetHTA (10).
Finally, we consulted the database “Pharmacoeconomic Guidelines Around The World” on the ISPOR Web site (www.ispor.org/PEguidelines/index.asp). The initial search from 2011 was updated in March 2015. Documents were retrieved and read in full-text if they were available in English or German language, contained the search terms “expert, specialist, professional,” and if they had a general methodological focus. They were excluded from further analysis if they did not address our research question regarding expert involvement in the HTA process. We complemented our search by an electronic database search in Medline and Embase for further guidelines in German and English language through August 2011. The syntax is provided in the Supplementary Tables 2 and 3. The Internet search and download of documents was performed by one author (T.H.) and included documents reviewed by a second author (N.S. or P.S-I.). Results from the database search were screened in a two-step process (first title and abstract, then full text) by one reviewer (T.H.).
Information Synthesis
For the document analysis, relevant information was extracted according to a predefined scheme by one author (T.H) and reviewed by a second independent author (N.S. or P.S-I.).
We defined five steps of the HTA process and applied this structure to all evidence tables. First, we identified at what HTA step each document mentioned expert involvement and how detailed this information was. We also extracted available information about who is considered an expert and how they are identified and selected by the HTA organizations. The nature and content of experts’ contributions was summarized and illustrated by statements from the documents. We listed the various elicitation methods by HTA step.
RESULTS
Existing Guidelines
From the HTA agency Web sites, the electronic database search, the bibliography screening and the ISPOR guideline database, we retrieved 120 documents for full-text screening (see Supplementary Figure 1). We finally included forty-two documents from twenty-two institutions that address the involvement of expert opinion in the HTA process (Table 1). None of the included documents stemmed from the electronic database search.
Consideration of Expert Involvement at Different HTA Steps According to HTA Handbooks
Note. The table displays the five main steps in HTA and indicates for what step handbooks mention expert involvement. +, experts only mentioned; ++, advice how to involve experts; +++, detailed description of the expert involvement.
AAZ, Agency for Quality and Accreditation in Health Care and Social Welfare; AHTAPol, Agency for Health Technology Assessment in Poland; ASERNIP-S, Australian Safety and Efficacy Register of New Interventional Procedures –Surgical; CADTH, Canadian Agency dor Druds and Technologies in Health; CRD, Center for Reviews and dissemination; CVZ, College voor Zorgverzekeringen; DACEHTA, Danish Centre for Health Technology Assessment; DAHTA@DIMDI, German Agency for HTA at the German Institute for Medical Documentation and Information; EUnetHTA, European Network for Health Technology Assessment; GÖG, Gesundheit Österreich GmbH; HAS: Haute Autorité de Santé; HIQA, Health Information and Quality Authority; INAHTA, International Network of Agencies for Health Technology Assessment; IQWiG, Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen; KCE, Belgian Federal Health Care Knowledge Centre; LBI-HTA, Ludwig Boltzmann Institut for Health Technology Assessment; MoH, Ministry of Health; NICE, National Institute for Health and Clinical Excellence; OHTAC, The Ontario Health Technology Advisory Committee; PHARMAC, Pharmaceutical Management Agency.
The twenty-two organizations are very heterogeneous in the manner they integrate experts in HTA and in the extent they report about it in their handbooks. The National Institute for Health and Care Excellence (NICE), for example, gives a detailed description of its process of expert involvement (11–14). Several organizations have more than one guidance document and expert involvement sometimes differs with respect to the area which is covered by the guideline, for example, economic evaluation or systematic literature review.
Selection of Experts
The guideline documents generally define experts as persons who are not routine members of the HTA team (11;15–17). However, there are different conceptions about who is regarded an expert that should be involved in the HTA (Table 2). First, there are content experts who have particular knowledge of the technology under investigation. Content experts can come from different areas of expertise (18), primarily healthcare professions (19), but also ethicists are recruited as content experts (20). Second, there are methods experts, whose expertise are analytical techniques for evidence evaluation or synthesis (15;21). Methods experts are also mentioned for quality assurance (3;22;23). A third group that contributes to HTA are patients, patient representatives, dependents, and caregivers, because they provide expert information about the perspective of the persons concerned (11;13;14). Beyond that, some institutions refer to stakeholders (e.g., providers, manufacturers, payers) as separate expert group (24).
Different Kinds of Experts Are Considered at Different Steps in the HTA Process
For the selection of content experts, NICE for example, contacts professional bodies, follows the recommendations of individuals who have formerly served as advisors, and performs literature searches to identify experts (11;25). Usually, two clinical experts, two patient experts and two commissioning experts are selected for NICE's Technology Appraisal Program (14). A list of advisors is published on the agency's Web site. Patient commentators are recruited by means of hospitals. The German Institute for Quality and Efficiency in Health Care (Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen, IQWIG) publishes a call for proposals containing qualification requirements on its Web site (16).
Only eight institutions give details of the expert recruitment process (3;11;16;20;21;Reference Cleemput, Neyt, Van de Sande and Thiry26–28). Most of them state that experts are selected according to criteria reflecting expertise in methods and/or content and that they must declare potential conflicts of interests. The Dutch Health Care Insurance Board explicitly states that experts that were consulted by the registration holder of the technology under investigation will not be involved (29). We found reference for the inclusion of either individual experts or panels with varying numbers of members. Some agencies recommend the inclusion of more than one expert to get a range of views (27;Reference Fröschl, Bornschein and Brunner-Ziegler30). The size of expert panels is between 5 and 20 individuals in the Irish guideline for stakeholder engagement (31).
Expert's Contribution
Table 1 provides an overview of all included documents and indicates the HTA step, where they mention involvement of experts. We found that all but two guidelines (n = 40; 95 percent) involve experts in the assessment of the technology in one way or another. Fifteen reports (36 percent) involve experts in the scoping process, that is, twenty of twenty-two institutions. Nearly thirty percent (n = 12) of the reports consider experts for the dissemination of the HTA report which also includes quality assurance and review of the final HTA report. Another twelve reports mention experts for appraisal of evidence and six (14 percent) note their role in the prioritization of assessment topics.
In general, we distinguish between experts giving advice regarding the conduct of an HTA and providing background information on the technology under study, and those who represent an additional data source for the analysis of clinical and cost-effectiveness, thus substituting empirical data. Table 3 provides a summary of the information from handbooks about the nature of input institutions seek from experts at each HTA step.
Nature of expert contribution to HTA
Note. Statements from HTA handbooks about the kind of contribution that experts provide are listed for each HTA step. The assessment step is separated into data collection on the one hand and analysis and synthesis on the other.
To prioritize possible topics for HTA reports, experts and stakeholders are asked to identify and weight relevant topics for assessment. During scoping, experts usually help framing the research question (3;11;23), denote issues where evidence is missing (22;29), and indicate points to be considered in the assessment. Particularly, they give background information about the technology under investigation and affected disorders. NICE, for example, involves clinical and patient experts in the committee's entire upfront discussions about the technology that is assessed (11).
In the preparation of the assessment report, a major contribution of external advisors is the support of the literature search. They are frequently consulted to locate relevant studies (3;15;23;Reference Fröschl, Bornschein and Brunner-Ziegler30;Reference Pertl, Fröschl, Rosian-Schikuta and Schtürzlinger32) and provide information about ongoing and unpublished research (12;27). After literature has been collected, experts discuss results of study inclusion and appraise the quality of included studies (3;15;23;Reference Fröschl, Bornschein and Brunner-Ziegler30). The Centre for Reviews and Dissemination (CRD) (15) recommends involving different kinds of experts in every step of the literature search.
Another crucial area for expert opinion during the assessment step is the evidence synthesis and analysis. In our review, HTA guidelines agree that expert data is low-grade evidence, but many suggest expert advice, for example, for health-economic and decision-analytic modeling, if other evidence is deficient (16;Reference Cleemput, Neyt, Van de Sande and Thiry26;33–Reference Swartenbroekx, Obyn, Guillaume, Lona and Cleemput35). The majority of recommendations for data analysis refer to decision-analytic and health economic modeling. In this context, most method papers emphasize to state in the report which assumptions or data which populate the model are based on expert opinion. For model parameters based on expert opinion, the Canadian Agency for Drugs and Technologies in Health (CADTH) recommends sensitivity analysis (36;Reference Mittmann, Evans and Rocchi37).
The expert contribution to the model structure includes, among others, assessing the process of patient management (16), selecting comparators (36;38), determining usual or recommended care as well as actual dosing (36) and resource consumption (15;16;22;Reference Swartenbroekx, Obyn, Guillaume, Lona and Cleemput35;36;39), judging the transferability of resource data between countries, or general validation of the model (3;16;34;40) (Table 3). To parameterize and populate the model with data, experts are asked to assess the variability of input parameters (34;36;41), value quality of life data (36;Reference Alves da Silva, Gouveia Pinto, António Pereira, Drummond and Trindade42), determine weights for age and health states (Reference Alves da Silva, Gouveia Pinto, António Pereira, Drummond and Trindade42), determine discount rates, and ascertain the required precision of cost assessment (16). Regarding the analysis techniques, CRD explicitly recommends seeking additional input from statisticians with special expertise for Bayesian methods, complex indirect treatment comparisons, mixed data scales, and meta-analysis of individual patient data (15). Particularly for Bayesian methods, experts are asked to provide prior information for the analyses (43).
Experts’ contribution is also of matter in the field of ethical and social aspects of a health technology (20;44;45). More general questions to experts during the assessment step focus on the substantiation of existing knowledge (11;Reference Cleemput, Neyt, Thiry, DeLaet and Leys46) and the implementation into clinical practice (45). Finally, we found recommendations to seek expert review to assure the quality of the HTA report (3;16;22;23;Reference Pertl, Fröschl, Rosian-Schikuta and Schtürzlinger32;Reference Alves da Silva, Gouveia Pinto, António Pereira, Drummond and Trindade42). Some guidelines request to report all external experts involved in the assessment (13;14;25), a justification and the method of elicitation (20;36;41).
Expert opinion for critical appraisal is recommended in HTA manuals but only limited details are provided. In summary, the experts are asked to evaluate and complement the information gathered from research and to contextualize the scientific evidence within the clinical and social context of the use of a technology (11;16;41;45).
The contribution to the dissemination step ranges from a general review of the HTA report (13;21;27) to the detailed answering of questions regarding “readability, the proposals made, and any future action to be taken regarding the development or distribution of the technology” by means of checklists (47). The National Authority of Medicines and Health Products (INFARMED, I.P.) guideline (Reference Alves da Silva, Gouveia Pinto, António Pereira, Drummond and Trindade42) explicitly calls on experts to disseminate the HTA results and stimulate the inter-peer discussion, while the German Agency for Health Technology Assessment (DAHTA) at the German Institute of Medical Documentation and Information (DIMDI) (48) only mentions general support of experts in stimulating information exchange with other agencies.
Elicitation Methods
Most documents give the advice that, if expert opinion is a source of information, it should be reported how it was obtained, but the majority do not specify this any further. HIQA states that “where expert opinion is used, it should be sought in a manner which minimizes bias and the process should be documented in sufficient detail” (49). The advice for expert elicitation (Table 4) is mainly restricted to vague instructions: “seek input” (36), “consult” (15), “the methods should be described” (15). NICE and HAS's ethical aspects guideline describe the methods used for information retrieval from experts in more detail (11;13;14;20). Also the Belgian Health Care Knowledge Centre KCE (Reference Cleemput, Neyt, Van de Sande and Thiry26) lists specific reporting items for the expert elicitation. They include potential blinding of participants to the study's purpose, remunerations, background information provided, the medium used to collect the opinions, questions asked, use of iteration in collation, approach used to present variability in opinions, and approach used to finalize the estimates from different experts. Altogether, established methods of social research are recommended (16), namely focus groups, interviews, written surveys (16;Reference Fröschl, Bornschein and Brunner-Ziegler30), or consensus panels (Reference Alves da Silva, Gouveia Pinto, António Pereira, Drummond and Trindade42) (e.g., Delphi method) (Reference Fröschl, Bornschein and Brunner-Ziegler30) and also triangulation of methods (Reference Fröschl, Bornschein and Brunner-Ziegler30). The Ontario Health Technology Advisory Committee states in its handbook that systematic methods for elicitation of societal and ethical values are being developed (45).
Different Methods of Expert Elicitation Reported in HTA Handbooks
DISCUSSION
While many public HTA institutions consider experts in the HTA process, we found little advice on how they do this. Most of the screened methods guidelines at least mention expert involvement. In the core step of an HTA, the assessment of benefits and risks of a technology, experts may represent a data source if empirical studies are not available. For this essential task, some fundamental advice is given in the agency handbooks, but the recommendations usually remain vague or just generally demand a description of the expert elicitation.
To our knowledge, this is the first systematic overview of the consideration of expert opinion in HTA guidance documents. Our research is limited by the document search on organizations’ Web sites. This way of information retrieval is not completely systematic, and we did not apply quality criteria to the included documents. The organizations we screened might have additional internal standards which we did not include. Even the organizations that did not publish any guidelines on their Web sites might still have practical guidance documents for the preparation of HTA reports. It may have been beneficial to contact the agencies and ask for such internal handbooks, but this was not possible in the time frame of the study. We limited our analysis to documents issued by HTA agencies and similar institutions, although there are other producers of HTA as well.
The reason for this was that we want to describe the methods of institutions that have some kind of authority, for example, because they are government appointed, and their guidelines have a binding character, also for third parties. This may possibly bias the methods we found to more conventional and conservative approaches. More experimental methods may be used by other producers of HTA, for example, academic research groups or manufacturers doing internal assessments or early HTA. Certainly, further research should also consider academic institutions. Another limitation it that even though we screened international organizations we included only reports in English and German language. Particularly when considering national HTA agencies, there are often no translations of their documents available. The results from the electronic database search were screened by only one reviewer, but as this search was only meant to complement the Web search, we believe this to be sufficiently comprehensive.
Our analysis reflects the methodological standards of organizations that perform or commission HTA, but we have not validated their theoretic guidelines against how they are actually implemented in HTA studies. It is possible, that in practice, the approaches to expert data are different. For example, HTA doers might follow published methods for expert elicitation coming from different sources and research areas.
When clear guidelines on selection of experts, elicitation methods, and implementation of expert data in an HTA are lacking, this might distort the findings based on this study. The formalized elicitation of numerical data (point estimates or distributions of parameters) should be judged differently from using expert opinion, for example, to obtain background information on the technology use. A recent methodology review of expert judgement on distributions in model-based HTA concludes that “reports of expert elicitation are insufficient in detail and this impacts on the perceived usability of expert-elicitated probability distributions” (Reference Grigore, Peters, Hyde and Stein50).
Where expert data replaces empirical data, it should be treated with the same rigor as data from trials, which usually undergo quality appraisal before they are included in an HTA analysis. Before deciding to use expert opinion, authors should become clear about what the risks of using expert data are and how to avoid them. Crucial points for the use of expert data in HTA are to make the process transparent, and to minimize, quantify, and describe uncertainties around expert data (Reference Sullivan and Payne8).
Existing research about expert elicitation in HTA and related fields comes mainly from decision-analytic modeling and clinical guideline development. The former combines various methods, while the latter mainly applies consensus panels as expert opinion. Guideline development also deals with methods to formalize the integration of subjective judgments (e.g., GRADE) (Reference Onate-Ocana and Ochoa-Carrillo51); however, the scope and procedure of guideline development differs from that of HTA and transferability of its methods is limited. Many established methods come from social research and are also applied in other fields than healthcare. Looking at the nonmedical literature, we see that for example ecology, environmental research, operations research or risk analysis, are confronted with similar problems in their specific form of technology assessment, mainly regarding elicitation of quantitative data. Bayesian methods, for example, are used in different disciplines and allow for incorporation of expert opinion regarding prior distributions of parameter estimates (Reference Ades, Sculpher and Sutton52).
The Delphi method is probably the most common technique for elicitation from groups of experts (Reference Brown53). It is characterized by anonymous answers with feedback over one or more rounds. However, it is sometimes used as a catch-all term for different kinds of consensus groups (Reference Sullivan and Payne8). Sullivan and Payne (Reference Sullivan and Payne8) suggests to better elicit single expert data for parameter values and aggregate them mathematically instead of applying consensus methods.
Evans provides a hierarchy of expert data, reflecting for which categories experts are more likely to give credible and correct estimates (Reference Evans and Crawford9). Model validation and information on treatment patterns were judged more reliable than costs and probability estimates. A survey among twenty-four HTA agencies found that 75 percent used expert opinion in the majority or all of their HTAs to amend and interpret trial data (Douw 2008, HTAi, conference abstract). The agencies mainly obtained qualitative data (70 percent) through informal conversation (61 percent). This indicates that expert knowledge data like cost and probability estimates are not the primary goal of expert consultation. However, the proportion of quantitative estimates might increase due to the rising number of modeling studies.
CONCLUSION
The issuers of publicly funded HTA reports consider expert opinion as an information source, but prevailingly do not give sufficient guidance for elicitation in their handbooks. HTA doers should be aware of the uncertainty surrounding expert opinion and the influence of the elicitation design. Further research should focus on how expert elicitation is actually performed in HTA and on how to transfer established methods from other fields into HTA.
Policy Implications
HTA Agencies should integrate existing knowledge on expert elicitation methods from all related fields into their guiding documents to formalize the use of expert data and make the process more transparent, systematic, and reliable. Particularly when experts provide quantitative data, explicit recommendations should be given.
SUPPLEMENTARY MATERIAL
Supplementary Table 1: http://dx.doi.org/10.1017/S0266462316000209
Supplementary Table 2: http://dx.doi.org/10.1017/S0266462316000209
Supplementary Table 3: http://dx.doi.org/10.1017/S0266462316000209
Supplementary Figure 1: http://dx.doi.org/10.1017/S0266462316000209
CONFLICTS OF INTEREST
T.H. received grants from the Tiroler Wissenschaftsfonds (Tyrolian Science Fund), grant number GZ: UNI - 4040/1009. The other authors have nothing to disclose.
