ETHICS AND HTA
The aim of health technology assessment (HTA) is to provide decision makers with a sound evidence base for health policy decisions, including those on public reimbursement of—or disinvestment from—particular health technologies, screening programs, and changes in clinical guidelines (1–Reference Walley7). Toward that aim, HTA studies the “medical, economic, social, and ethical implications of the development, diffusion, and use of health technologies”—such as pharmaceuticals, medical devices, and procedures—used for the purposes of prevention, screening, diagnosis, palliation, and treatment of illness (1;2;8).
The ethics dimension is thus, at least in definition, a part of HTA analysis. That the ethics dimension is a crucial part of HTA is also becoming increasingly recognized in the literature (Reference Autti-Ramo and Makela9–Reference Van der Wilt, Reuzel and Banta19). Arguments for inclusion of ethics considerations in HTA have been advanced on a variety of grounds, including that health technologies have normative implications; are morally challenging and value-laden; involve the values of patients, communities, professionals, and producers; and that the HTA process itself is likewise far from neutral, involving multiple value judgments on the parts of the health technology assessors (Reference Hofmann, Cleemput and Bond16). Nevertheless, despite its importance for HTA, the methodology in ethics for HTA is generally perceived as relatively under-developed compared with other areas of HTA (Reference Saarni, Hofmann and Lampe4;Reference Autti-Ramo and Makela9;Reference Duthie and Bond12;Reference Van der Wilt, Reuzel and Banta19–Reference Tantivess22).
This perception is not entirely correct, as in the last decade, much work occurred in this space and numerous methodological advances are now evident (Reference Assasi, Schwartz, Tarride, Campbell and Goeree23). These advances include, for example, the proliferation of methods for integrating ethical issues into HTA. EUnetHTA's Core Model identifies multiple approaches to addressing ethics issues in HTA, including casuistry, coherence analysis, participatory HTA approach (iHTA), principlism, social shaping of technology, wide reflective equilibrium, as well as several local approaches used in Quebec, Finland, Norway, and Italy; EUnetHTA has also developed its own framework for ethics analysis (13). Methodology for searching for literature on ethics issues in HTA has also been developed (Reference Droste, Dintsios and Gerber11). And several agencies, including IQWiG, OSTEBA, and SBU, are currently trialing instruments for identifying when an in-depth assessment of ethical issues is required as part of an HTA (Reference Stoklosa and Bond24).
On the other hand, the perception of methodological under-development in ethics for HTA appears to be accurate with respect to methodology for quality assessment of ethics analyses conducted for HTA purposes (Reference McDougall25–Reference Strech, Synofzik and Marckmann29). We note, for example, that in a recent systematic review of methodological guidance documents for evaluation of ethical issues in HTA (Reference Assasi, Schwartz, Tarride, Campbell and Goeree23), there is no discussion of quality assessment tools. This is in contrast to other types of evidence evaluated as part of the HTA processes (e.g., clinical studies, economic models), where quality assessment methodology is well developed, a multitude of quality assessment instruments tailored by study type exist, and quality assessment is accepted as an integral part of the process. In ethics analyses, as in systematic reviews more generally, a smaller number of high quality articles may be preferable to a large number of poor quality ones. This, however, requires a means of differentiating high quality from low quality articles.
Our aim in this study is, therefore, to advance the methodology in ethics for HTA, by exploring and articulating the range of issues involved in quality assessment in this domain. More specifically we aim: (i) to verify existing tools for quality assessment of ethics analyses for HTA, (ii) to consider some arguments for and against the need for quality assessment tools for ethics analyses for HTA, and (iii) to propose a preliminary set of criteria that could be used for assessing the quality of ethics analyses for HTA.
TAXONOMY OF ETHICS ANALYSES CONDUCTED FOR HTA PURPOSES
Before considering the issues around quality assessment of ethics analyses conducted for HTA purposes, however, the notion of “ethics analyses for HTA purposes” first requires clarification. Two distinctions can be drawn here: a type distinction and a level distinction.
Type Distinction
Ethics analyses can be broadly differentiated into two types: descriptive and normative. In descriptive ethics analyses, empirical methods such as interviews and surveys are used to identify issues, attitudes, views, or practices (Reference McCullough, Coverdale and Chervenak30). These would include, for example, a survey of healthcare providers’ views toward moral permissibility of active voluntary euthanasia. In descriptive ethics, the conclusions are “is statements” rather than “ought statements.”
Descriptive analyses can be contrasted with normative analyses, where arguments are marshalled to establish moral permissibility or impermissibility of a decision or action (Reference McCullough, Coverdale and Chervenak30). Unlike in descriptive analyses, the conclusions are “ought statements,” for example, that human reproductive cloning technologies are morally objectionable or that a specific type of hormone replacement therapy should be provided to a given patient group.
Level Distinction
The second distinction that can be drawn in the context of ethics analyses for HTA purposes is a level distinction. At the first level of analysis, we can place a set of arguments for or against a technology, a listing of several arguments in favor of or against adopting a particular technology that is not necessarily comprehensive. This often takes on the form of an article arguing for moral permissibility or impermissibility of adopting a particular health technology, and can be thought of as ethics analogue of an individual trial of clinical effectiveness. In the HTA context, this type of analysis could take the form of an analysis of permissibility of a single health technology that is limited to consideration of the commonly cited four principles of Principlism: beneficence, nonmaleficence, autonomy, and justice.
At the second level is a systematic review of arguments for or against a particular technology, that is, an ethics analogue to a systematic review of clinical effectiveness studies. These types of reviews may (Reference McCullough, Coverdale and Chervenak31) or may not (Reference Sofaer and Strech26) include a weighting and balancing of the arguments to reach a normative conclusion about moral permissibility or impermissibility of a health technology, much like systematic reviews of clinical evidence may or may not include a recommendation, depending on the remit of the HTA agency by which they are being performed. (Even more fine-grained distinctions are possible here) (Reference Sandman and Heintz32).
Combining the Type Distinction and Level Distinction
Combining the type distinction with the level distinction yields a four-way taxonomy of ethics analyses that may be conducted as part of an HTA: (i) a set of arguments that is descriptive (i.e., a description of arguments), (ii) a set of arguments that is normative (arguments amounting to a recommendation), (iii) a systematic review of arguments that is descriptive, and (iv) a systematic review of arguments that is normative.
Thus, in context of quality assessment of ethics analyses for HTA purposes, we need to consider that a variety of analyses are captured here, and consequently, that a variety of quality assessment instruments may exist for this purpose.
METHODS
To establish whether this is indeed the case, we adopted a three-pronged approach. First, we examined all of the guideline documents identified in a recent systematic review of existing guidelines for integrating ethics into HTA, conducted by Assasi et al. (Reference Assasi, Schwartz, Tarride, Campbell and Goeree23). We did so, as methodological guidelines for integrating ethics analyses into HTA would be a likely place to contain instructions or suggestions for how to assess the quality of such analyses. Second, to identify any additional guidelines published between Assasi et al.’s searches and our own, we updated their search, replicating their strategy (kindly provided by Assasi et al.) and examined additional guidelines identified through the updated search. Third and finally, to identify additional, unpublished or in-progress quality instrument tools targeting the quality of ethics analyses for HTA, we convened two workshops gathering experts in methodology in ethics for HTA, in Edmonton and Cologne in October 2013 (Reference Stoklosa and Bond24), as they would be the most likely to know of existing quality assessment instruments or be personally engaged in developing them.
Assasi et al. (Reference Assasi, Schwartz, Tarride, Campbell and Goeree23) Systematic Review
Assasi et al's search strategy consisted of: a database search (including: Medline, EMBASE, PsycINFO, PubMed, Wiley's Cochrane Library and the CRD HTA Database), a search of Web sites of HTA bodies, and review of article bibliographies as well as contacting experts. The searches were unrestricted by language or start date, with a search end date of October 1, 2013 (Reference Assasi, Schwartz, Tarride, Campbell and Goeree23). Their search identified forty-three documents: twenty-one methodological articles and twenty-two HTA guideline documents. We read in full all of documents included by Assasi et al. to identify any tools or instruments or suggestions for conducting quality assessments of ethics analyses conducted for HTA purposes. Of the twenty-one methodological articles included by Assasi et al., two (Reference Hofmann14;Reference Grunwald33) recognized the importance of ensuring the quality of ethics inquiry, but neither proposed a formal instrument for this purpose. Of the twenty-two guidelines included by Assasi et al., two likewise acknowledged the import of ensuring quality, but again, offered no formal instrument for this purpose (Reference Busse, Orvain and Velasco34;Reference Droste, Gerhardus and Kollek35).
Updated Search
We replicated and updated the database and HTA Web site searches conducted by Assasi et al. (Reference Assasi, Schwartz, Tarride, Campbell and Goeree23) to determine whether any new methodological guidance articles or guidance had been released since the last date of Assasi's search (October 1, 2013). The database search consisted of replicating the strategy outlined in Assasi et al. (Reference Assasi, Schwartz, Tarride, Campbell and Goeree23) and kindly provided by to the present authors in full, and updating it to 1 December 2014. Websites of fifty-seven international and national HTA producers (see the Supplementary Appendix 1) were searched between February and April 2015 for documents in the English language that provided any guidance on identifying or addressing ethical issues when conducting HTA.
Whereas Assasi et al. (Reference Assasi, Schwartz, Tarride, Campbell and Goeree23) conducted a Google search for documents produced by fifty-seven HTA agencies or organizations, we used the same list of agencies to search the agencies’ Web sites themselves. In some cases, Assasi had searched for agencies or organizations that are no longer in existence (e.g., AETMIS and DACEHTA) and did not search for documents archived or produced by successor agencies or organizations. For this reason, our updated search, which did search sites of successor committees, agencies, and organizations, as well as searches of countries with established HTA programs omitted by Assasi et al. (e.g., Ireland), differs slightly in the number of agencies searched from that conducted by Assasi et al. In addition, this updated search provides a more current and thorough picture of the availability of current guidance for assessing the quality of ethics analysis in HTA and not just guidances on how to conduct ethical analysis in HTA as it was the primary purpose of Assasi's paper.
The updated database and Web site searches together yielded 420 additional references: 413 from database searches, seven from searches of HTA bodies’ Web sites (see Figure 1). On removal of duplicates, and application of exclusion criteria, forty-two documents remained: forty identified through database searches, and two identified through HTA agency Web site searches. These were reviewed in full. Six documents met the inclusion criteria: four articles from database searches (Reference Damian, Necula, Sandu, Iliescu and Ioan36–Reference Thebaut39), and two from a review of HTA agencies’ Web sites (40;41).
Results of the update of Assasi et al.’s original search, updated to December 1, 2014.
All four articles identified through database searches offered methodological guidance on ethics for HTA and underscored the importance of integrating ethics analysis into HTA. Hofmann et al. (Reference Hofmann, Droste, Oortwijn, Cleemput and Sacchini37) additionally advised to assess ethics issues and arguments identified in literature searches for: logic and coherence, reliability, validity, and actuality. No instrument for this purpose was offered, however. Similarly, while both included documents that were identified through HTA agency Web site searches (40;41) recognized the importance of integrating ethics into HTA and the need for quality of evidence in this area, however, neither contained an instrument for doing so.
Soliciting the Views of Experts in Methodology in Ethics for HTA
In light of the gaps in the literature, we, therefore, solicited the views of the experts in methodology in ethics for HTA. Experts in this area convened over a 2-day period in Edmonton (October 18–19, 2013) and in Cologne (October 25–26, 2013), to discuss issues around quality assessment of ethics analyses for HTA (Reference Stoklosa and Bond24). The workshops included thirty-two participants from ten countries: Australia, Canada, Germany, Norway, Spain, Sweden, United Kingdom, Italy, France, and The Netherlands. Participants represented a variety of professional backgrounds, including academics, HTA producers, clinical ethicists, members of government bodies, decision makers, and members from research funding agencies. Many of those participating are active members of the HTA Ethics Interest Group, and jointly work with INAHTA. In those, and subsequent discussions, the known approaches to assessment of the quality of ethics analyses for HTA were identified, examined, and discussed with the participants (see Table 1). A detailed report of the workshop aims, participants, and discussion is publicly available (Reference Stoklosa and Bond24).
Existing Approaches to Assessing Quality of Ethical Analysis for HTA Purposes
QA, quality assessment.
It is evident from Table 1 that the existing approaches to assessment of quality of ethics analyses focus primarily on assessing ethics analyses at the lower level, that is, the article level (Reference McDougall25;Reference Sofaer and Strech26;Reference Strech, Synofzik and Marckmann29;Reference McCullough, Coverdale and Chervenak30). Even at that level, however, it is apparent that quality assessment is in the early stages of development. Moreover, the identified approaches emphasize assessment of the content of the ethics analysis, and the validity of the arguments or the clarity of the resulting analysis. What is absent is the assessment of the process of the analysis, for example, the search methodology used to identify content, how the ethics analysis was related to the technology (e.g., with regards to complexity), to the HTA process, where it fits into the structure of decision making, which stakeholders were involved, etc.). While this is partly explained by the focus on quality assessment at the article rather than systematic review level, even at that lower level, some type of literature search is conducted and some type of process for including arguments and positions is deployed and it is impossible to perform a valuable quality assessment if one does not consider these broader issues.
Thus, although inchoate, approaches to assessing the quality of ethics analyses at the article level do exist. In contrast, it is noteworthy that neither the literature searches, nor the advice of the experts in the field, identified any tools or instruments, even rudimentary ones, for assessing the quality of ethics analyses at the systematic review level. It is worth emphasizing here that although the approaches to assessment of quality at article level could offer a starting point for this purpose, they would likely require considerable redevelopment and tailoring, analogous to the way in which different instruments were developed for assessing the quality at individual clinical study level (e.g., Jadad scale for randomized clinical trials) and systematic review of clinical studies level (e.g. AMSTAR for systematic reviews).
SHOULD ETHICS ANALYSES BE ASSESSED FOR QUALITY?
Unlike in other areas of methodology in ethics for HTA, then, there appears to be a considerable room for development in the area of quality assessment, particularly at the systematic review level. However, whether this is a genuine methodological gap, depends on whether systematic reviews of ethics issues for HTA purposes indeed do require quality assessment tools. To establish if this is indeed the case, we need to first step back and consider the aim of quality assessment in HTA more generally.
The recognition of the importance of quality assessment in HTA more generally, both of the process of identification of evidence and of the resulting content, is well documented. For example, this is demonstrated by the existence of INAHTA's special working group on quality assurance, which has resulted in a series of checklists (Reference Hailey42). It is also demonstrated by a multitude of other quality assessment initiatives focused on specific products or domains of the HTA process (Reference Kidholm, Ehlers, Korsbek, Kjaerby and Beck43;Reference Niederstadt and Droste44), which have resulted in instruments such as AGREE for Clinical Practice Guidelines (45), AMSTAR for systematic reviews (Reference Shea, Grimshaw and Wells46), QUADAS for diagnostic tests (Reference Whiting, Rutjes and Westwood47), etc. Generating quality assessment instruments for ethics analyses for HTA would therefore align ethics analysis with other aspects of HTA.
Beyond that, quality assessment instruments would have the advantage of increasing the transparency and readability of the ethics analyses, by providing a set of criteria on which these analyses could be assessed. This would, furthermore, assist with international transferability of the analyses, which is especially salient in the area of ethics for HTA, because although some HTA agencies do conduct ethics analyses as part of their HTAs, many do not yet have the capacity or resources to undertake such analyses. A tool for assessing the quality of these analyses would help these agencies to decide whether or not to include and adapt other jurisdictions’ ethics analyses, as is currently done for adapting other jurisdictions’ health technology assessments.
Finally, quality assessment tools in this space would also offer a benefit to the wider HTA community, by providing a common language and a means of structuring the dialogue between the various participants in the HTA process (clinicians, epidemiologists, ethicists, patients, caregivers, etc.).
The many potential benefits of these tools notwithstanding, several caveats, both more general and more specific, ought also to be acknowledged. At the most general level, the use of quality assessment tools could lead to an undesirable oversimplification of ethics analyses. For example, if a quality assessment tool were to stipulate the assessment of elements A, B, and C, ethics analyses could potentially narrow to considering only those elements. The potential for this to happen will depend on who is the user of the quality assessment tool, the potential is probably lower for this occurring in the hands of a skilled ethicist than a nonethicist. However, whether this would eventuate is an empirical question, which is resolvable, at least in part, by evaluating whether such oversimplification has happened in other areas of HTA where quality assessment tools are currently in wide and accepted use.
Another concern here is the potential for the ethics analysis to become a box-ticking exercise against a potential quality assessment instrument. How significant a problem this would be, will depend on the content of the actual quality instrument proposed and how it is used. However, it is worth noting that several elements of an appropriately robust ethics analysis may be amenable to assessment through a box-ticking exercise, this is particularly the case for the assessment of the process of the analysis (e.g., the searches conducted to identify relevant content), but even some content elements are amenable to this. For example, whether the concepts deployed in the analysis have been adequately defined and clarified, whether the arguments put forward in support or against acceptability of a health technology have been related to more basic values and norms, whether possible sources of bias have been noted and considered, could all potentially be assessed in this manner. And, although it is difficult to find universal agreement on what constitutes good ethics analysis (Reference Gillon and Higgs48), given the fairly specific nature of ethics for HTA, such agreement may not be impossible, particularly with regard to general formal criteria, such as comprehensiveness, consistency, transparency, and so on.
Finally, it is possible that quality assessment instruments, in particular in the form of checklists, may fail to include subtle or less common issues pertinent to specific health technologies or particular jurisdictional contexts. However, this suggests that what is required to be built into any quality assessment instrument is a flexibility to amend it to suit health technology-specific and jurisdictionally-specific needs. Such flexibility is built into quality assessment instruments in other dimensions of HTA, for example, the QUADAS instrument (Reference Whiting, Rutjes and Westwood47), so there is no reason that these issues would be insurmountable in the case of quality assessment tools for ethics analyses in HTA.
PROPOSED APPROACH TO ASSESSING THE QUALITY OF ETHICS ANALYSES
In light of the existing gap, as well as the identified need, we therefore propose here a preliminary instrument for quality assessment of ethics analyses (see Table 2). This instrument arises out of the work undertaken at the aforementioned workshops (Reference Stoklosa and Bond24) and encompasses two major categories of assessment domains: internal category and external category.
Proposed Approach to Assessing the Quality of Ethics Analyses
Internal Category
The internal category focuses on assessing the internal quality/strength of the ethics analysis, and can be thought of as analogous to evaluating the internal quality of an argument, that is, its validity. The domains to be assessed for quality within this category include: perspective, assumptions, premises, conclusions, premises/conclusions relationship, and objections. Perspective domain focuses on whether the position(s) adopted for the ethics analysis identify from whose perspective they are put forth—for example, is something morally acceptable from the point of view of a patient, the health system, healthcare professionals, etc. Assumptions pertain to the principles taken for granted in the analysis—for example, that a publicly funded health system is a public good. Premises are statements or reasons offered in support of conclusion, for example, appeal to an ethical principle (equality, say) in support of a conclusion that a technology ought to be funded. Conclusion is what the analysis is trying to show, for example, that a particular technology ought to be publicly funded for all, or only for a specific group of patients. The relationship between premises and conclusions centers on whether the reasons offered in support of a conclusion, indeed do support that conclusion, for example, does the principle of fairness truly support public funding of a life-saving drug for a small number of patients with a rare disease. Finally, it is common in ethics analyses to consider objections, that is, dissenting points of view presented in a way that its proponents would accept, and address them; an appropriately conducted ethics analysis should include this domain.
The quality assessment of these domain focuses on the clarity with which the domain is identified in the analysis—clearly, unclearly, or not at all—in the case of the following domains: perspective, assumptions, premises, conclusions, objections. Reasonability with which the domain is approached applies to the following domains: assumptions, premises, conclusions. Strength of the relationship is the quality element assessed for the domain of premises/conclusion relationship, and finally, objections domain is assessed on whether objections are identified, and whether they are addressed.
External Category
The external category focuses on evaluating domains that can be thought of as analogous to evaluating the external quality of an argument or its soundness. The domains to be assessed for quality within this category, include: transferability, implications, completeness, and bias. The transferability domain focuses on whether the argument (or analysis) is transferable to the context of assessment, for example, in a casuistry-style analysis, is an extrapolation of an argument about the moral duty to fund cochlear implants, say, indeed transferable to the assessment of the ethics issues around funding of prosthetics. The quality assessment here focuses on whether the argument is indeed transferable (fully or in part) or not at all. Implications center on whether the policy implications of the ethics argument are identified, with the clarity with which this is done being the subject of the quality assessment, as well as whether the different implications for the various stakeholder groups are differentiated (patients, healthcare professionals, health system, etc.). Completeness centers on the thoroughness with which the ethics analysis was conducted, was the literature searched, and were the gaps identified and acknowledged. Finally, in assessing the bias domain, the focus is on whether potential sources of bias have been identified, with the quality assessment centering on the clarity with which this was done, and whether any steps have been taken to address potential sources of bias.
NEXT STEPS
Although legitimate concerns exist about the use of quality assessment instruments for any ethics analyses, and especially in HTA due to the consequences of adopting its recommendations, they remain largely empirical issues and issues that are not insurmountable. Moreover, the multiple benefits of developing the quality assessment methodology cannot be ignored. As our searches show, this area is presently underdeveloped in HTA both at an article level and systematic review level of analysis. The need for methodological development is perhaps greater at the latter level, as an increasing number of systematic reviews of ethics issues around various health technologies and healthcare delivery are being published, including those on ethics issues around autologous stem cell transplantation (Reference Droste, Hermann-Frank, Scheibler and Krones49), access to drugs post-trial (Reference Sofaer and Strech26), management of psychiatric disorders with concealed medications (Reference McCullough, Coverdale and Chervenak31), and overriding parents’ medical decisions for their children (Reference McDougall and Notini50).
The framework proposed above outlines the key elements for quality assessment of ethics analyses for HTA. It is a preliminary one, it requires further testing, refinement, contextual adaptation, and elaboration. This, however, cannot occur in a small workshop setting, it requires an open and transparent professional engagement and debate. The tool proposed here is intended to spur precisely this kind of engagement.
SUPPLEMENTARY MATERIAL
Supplementary Appendix 1: https://doi.org/10.1017/S0266462316000556
CONFLICTS OF INTEREST
The authors declare no conflict of interest.
