Designing and Implementing Deliberative Processes for Health Technology Assessment: A Good Practices Report of a Joint HTAi/ISPOR Task Force

Objectives Deliberative processes for health technology assessment (HTA) are intended to facilitate participatory decision making, using discussion and open dialogue between stake-holders. Increasing attention is being given to deliberative processes, but guidance is lacking for those who wish to design or use them. Health Technology Assessment International (HTAi) and ISPOR—The Professional Society for Health Economics and Outcomes Research initiated a joint Task Force to address this gap. Methods The joint Task Force consisted of fifteen members with different backgrounds, perspectives, and expertise relevant to the field. It developed guidance and a checklist for deliberative processes for HTA. The guidance builds upon the few, existing initiatives in the field, as well as input from the HTA community following an established consultation plan. In addition, the guidance was subject to two rounds of peer review. Results A deliberative process for HTA consists of procedures, activities, and events that support the informed and critical examination of an issue and the weighing of arguments and evidence to guide a subsequent decision. Guidance and an accompanying checklist are provided for (i) developing the governance and structure of an HTA program and (ii) informing how the various stages of an HTA process might be managed using deliberation. Conclusions The guidance and the checklist contain a series of questions, grouped by six phases of a model deliberative process. They are offered as practical tools for those wishing to establish or improve deliberative processes for HTA that are fit for local contexts. The tools can also be used for independent scrutiny of deliberative processes.

researching the quality or completeness of a deliberative process or seeking to compare these processes across HTA bodies.

Definitions and Scope
As there is no widely recognized definition of a deliberative process for HTA, the first goal of the Task Force was to create a consensus definition. Existing definitions of "deliberation," developed for health policy or other related activities (e.g., (9-10)) did not appear satisfactory for our purposes. They were too focused on a particular application of the concept for one form of HTA, for example, pharmaceuticals. They also excluded some items suitable for deliberation, such as agendas for meetings, which may be appropriate in specific contexts but ought not to be ruled out by definition. They also missed the importance of exploring the political, economic, and social context in which HTA is to be applied. The Task Force, therefore, developed the following definitions (Box 2).
Those designing and supporting deliberative processes for HTA should not underestimate the importance of selecting and clearly stating the desired goals and outcomes of deliberation, as these will determine all other design choices (11). Deliberation can, for example, be used to provide an opinion (e.g., advice or recommendation), but can also be used to understand (diverging) views regarding a specific issue. Deliberation can create opportunities for exposing possibly conflicting values and perspectives and reaching resolutions that will be useful to ultimate decision makers (including resolutions that are not unanimous).
These values and perspectives are typically provided by a variety of stakeholders (patient[s], public/citizens, providers of care, payers, producers and innovators of health technology, principal investigators in research, and policy makers) and experts (e.g., experts in medicine, law, ethics, economics, epidemiology, bioengineering, and patient-based evidence). Stakeholders and experts can have financial, professional, or reputational interests in the outcomes of deliberative processes, which points to the need for their roles to be clearly defined. An HTA body must then decide how these perspectives will be represented, how participant views are exchanged, how participants are identified and selected, how partiality and bias are to be minimized, and whether and how the identities of participants are to be publicly disclosed.
Good deliberation includes the consideration and examination of general factors, such as the identities of participants, their roles, key decision criteria and possible conflicts between them, the scope of costs and benefits and their measures, the measurement and significance to be attached to factors, such as equity and public acceptability, and more specific factors related to a specific health technology, such as the evidence for its best use, its precise and accurate interpretation, and-when this is the goal-the dissemination and implementation of the output(s) of deliberation. These and other design and implementation considerations are part of the checklist for deliberative processes for HTA (Table 1) and elaborated upon after introducing the checklist and how to use it.

The Deliberative Processes for HTA Checklist and How to Use It
The checklist (Table 1) is intended to be used prior to designing an individual deliberative process or when reviewing an existing one. It is intended to be completed for each new process, whether for developing the governance and structure of an HTA program or managing one or more different HTA-related stages.
The checklist contains a series of questions, grouped by six phases of a model deliberative process. Users of the checklist should simply check all answers that apply to each question posed. The answers represent a minimum set of considerations and design features that are required in order to qualify a process as deliberative. The checklist is not intended as a scoring tool, but allows users to identify items that are not checked, and more readily scrutinize whether improvements can be made.
Some questions relate to the key features of a deliberative process that should be stated in its terms of reference. These questions are labeled with an asterisk "*" in the checklist. Asterisked items may also be considered a minimum set of features for any process to be considered and can be used for comparisons across processes.

Guidance for Designing and Implementing a Deliberative Process Phase I. Determining the Need for a Deliberative Process
The need for a deliberative process depends on its goal(s), the desired outcome(s), and scope.
Why Deliberate?-Deliberation allows participants to exchange their views and perspectives. Participants may learn about other views and perspectives as a result of deliberation, leading to predeliberative opinions being updated or revised. Participation through deliberation also gives some degree of influence and ownership of the final output to those involved. Deliberation can lead to an expansion of what counts as evidence in the process. Well-designed and well-executed deliberative processes also have the potential to provide stakeholders with reasons to perceive the HTA process as (un)fair, based on its procedural characteristics (e.g., inclusivity of all relevant stakeholders), even if they disagree with the outcomes of the process (2;10; [13][14]. It may yield outputs that are more inclusive, better framed, more balanced, and more feasible. This will increase trust in institutions and their decisions (15).
What Are the Desired Outcomes of the Deliberative Process?-HTA bodies might consider several desired outcomes related to the goals of deliberation. These include (i) the sharing and potential shifting of different viewpoints among participants, (ii) reaching common ground that underpins a collective decision; (iii) an opportunity to provide reasons for and against outputs, and a possible revision of them following deliberation; and (iv) strengthening mutual respect and understanding among participants. Although agreement can also lead to a collective opinion (e.g., advice, recommendations, or decisions), this is not always the required output of a deliberative process (2;10;14).
What Is the Scope of Deliberation?-Deliberation can inform a range of processes within an HTA body (Box 1), from formal discussions of an HTA organizational structure (deliberation about processes) to topics for assessment and the revisiting of an earlier decision on a specific health technology in the light of new evidence (deliberations within processes).
Deliberation about the HTA Process-Deliberation about the HTA process will be shaped by historical, sociocultural, political, economic, and institutional context for healthcare decisions, and these will (need to) be reflected in the autonomy, mandate, capacity, and governance of an HTA body (16). Therefore, whether (and at what stage) to deliberate will likely be driven by judgments about its (potential) use or effectiveness given institutional attitudes toward independence, transparency, and inclusiveness (17). Decisions like these may be informed by specific actors, such as regional administrators or ministries of health. In Canada, for example, Abelson et al. (18) have developed a public and patient involvement framework for an HTA body's process that addresses decision making occurring at a provincial level. The depth and complexity of actions (to be) undertaken by an HTA body may also dictate what degree of deliberation is required (19).
Deliberation within the HTA Process-Deliberation is needed or more desirable when there are anticipated and potentially controversial differences in what facts are relevant to the deliberation, how empirical information ought to be interpreted and valued, as well as what values are relevant to the decision and what these imply regarding the collection of information to inform deliberation; this may apply to the HTA process, as described above, as well as at several HTA stages (15;20). Examples of deliberation at different HTA stages are described in Box 3.
The benefits of deliberation must be balanced against the possible costs, the capacity to manage it, and timelines for decision making. The benefits of achieving a certain degree of rigor in deliberation may be costly, and it may be unnecessary or make a particular model infeasible. In addition, contextual factors (e.g., who is accountable for the process overall and to whom, available HTA content knowledge, and perceptions of conflict) may facilitate or hinder the feasibility of implementing deliberative processes and limit their contribution to legitimate decision making. For these and related reasons, what might be the most important aspects of a deliberative process to invest in can be determined by using the checklist (Table 1).

Phase II. Preparing for a Deliberative Process
Contextual factors, guiding principles, as well as what should be documented and communicated to the public are important considerations when preparing for a deliberative process.
of participants, the knowledge and skills of applying HTA methods, and the capacity or financial resources to conduct HTA and communicate HTA decisions (27). External factors include the extent to which countries have established mechanisms to incorporate evidence into decision making, the perceived role of HTA in decision making, and the mandate given by legal or policy authority (27).
Another important external contextual factor is influence by interest groups, including politicians, ministers of finance, industry, patients, and care providers. This has led to a wider range of stakeholders being involved in HTA processes and pressure to consider a broader range of criteria in decision making (28). Poorly implemented deliberative processes that do not manage these power dynamics can lead to distractions that dilute fruitful discussion (29). This creates ethical challenges for patients (30) as well as risks of delay or poor decision making (see Box 4; (31)).
There are also contextual factors specific to a decision about use of a health technology that is established during the deliberation about the technology itself (see Box 5).
Having an HTA infrastructure that supports transparency, participation of relevant stakeholders, and accountability of decision makers will be an important minimum requirement for enabling deliberation in any setting (27). We cannot assume that the implementation of deliberative processes automatically improves decision making across all jurisdictions equally. Variation in internal and external factors will shape and constrain implementation. Identifying factors that may hamper the effectiveness of a deliberative process enables planning to mitigate or overcome these barriers.
Ways of mitigating these barriers include policy statements on the willingness to use HTA in policy or practice, and development of transparent procedures for deliberations even if the outputs need to remain confidential, such as the communication of who is involved, their conflict of interest declarations, and how the process is undertaken.
There is an opportunity among countries with less-developed deliberative processes to start building them with features that take account of decision-making needs and environmental and societal constraints. Most low-and middle-income countries are in the early stages of developing HTA systems and can potentially incorporate learnings from the experiences of others (6). In contrast, most high-income countries have already established institutions and processes that may be more difficult to change (33-34). This means that these countries will require approaches to evaluate their current deliberative processes and to identify where improvements might be made.
What Are the Guiding Principles?-Principles are intended to guide the design and conduct of deliberative processes. They can be substantive or procedural (35). Efficiency in HTA, for example, is an important principle where resources for HTA are lacking. Walton et al. (36) reported, for example, that the National Institute for Health and Care Excellence (NICE) judged the resources dedicated to discussing individual topics to be not efficient and unsustainable.
To be useful to decision making, principles should be consistent with the overarching goals of the HTA body or healthcare system that the deliberative processes are intended to support (e.g., equal access to care). Such goals are often multidimensional, complex, and value-laden. Values such as transparency, impartiality, and inclusivity were recently identified as important procedural principles related to deliberations about drug coverage decisions (5). Other values may include timeliness, consistency, and verifiability (e.g., making recommendations that are timely and understandable) (14). Values can be differently applied to the various stages of an HTA process and may conflict. An example of the potential conflict between values of transparency and impartiality in developing drug funding recommendations is shown in Box 6.
The selection and application of principles and their related values may be affected by local contexts as different decision makers may have different goals, work in different sectors, and hold different attitudes toward (the need for) deliberation.
In some countries, technology producers implicitly determine the topics for assessment by submitting reimbursement decision dossiers to HTA bodies throughout the year, whereas in other topics are nominated and selected with the involvement of patients, payers, ministry of health officials, healthcare managers, and clinical experts (37).
What Should Be Documented and Communicated to the Public?-The need for a deliberative process, the guiding principles, and key elements for conducting and supporting a deliberative process as reflected in terms of reference, its outputs, and how they are used, as well as the monitoring and evaluation thereof (see sections below), need to be documented and communicated to the public. This documentation will also aid in sharing and studying what design features work best, and assist in promoting standards for HTA bodies in years to come.

Phase III. Conducting a Deliberative Process
When conducting a deliberative process, it is important to consider several design features, including the selection and identities of participants (including the chair[s] or facilitator[s]), openness, type, length, and the rules of deliberation.
Who Deliberates?-Participants in a deliberative process may include stakeholders, experts, or both. Some participants may be asked to play more than one role. For example, an industry representative is both a stake-holder affected by a decision, but may also have expertise in trial design and interpretation of evidence within a therapeutic context. The mix of participants and approaches to recruiting them should be carefully considered and will depend heavily on the tasks to be completed. Participation could take the form of membership of a standing committee where patient representatives have the task to represent their personal experiences or represent the wider experience of a group of patients. There might be ad hoc deliberative processes on selected topics in which a wider range of stakeholders (e.g., industry groups, healthcare payers, clinical experts, and patients) participate.
Other models of representative deliberation exist (e.g., consensus conference, citizen forum, and platform for representative deliberative democracy [G1000] (38)), and may be more feasible or better for meeting the goals of the HTA process. In the Netherlands, a deliberative citizen forum has been used by the Zorginstituut Nederland (ZIN; the national HTA body) to obtain insight into the criteria-informed citizens would propose for public reimbursement (39). Even within the same political context, participants involved in deliberation may vary according to what is being deliberated, and the goals and intended outcomes of deliberation (see Box 7).
The number of participants should be dictated by the goals and principles of a deliberative process. Although ten to fifteen participants have been suggested as an effective number for multidisciplinary advisory committees (40), others have stated, "there is no magic number that constitutes the ideal-sized group." Trade-offs between quality and speed must be made (41). In the case of controversial HTA questions (e.g., when major ethical issues are raised by the health technology that is being investigated), a larger group-including all relevant perspectives-that takes more time to deliberate (and may have higher quality deliberation) may be more desirable for engendering trust and legitimacy.
In practice, HTA bodies operate with standing deliberative committees consisting of nine (ZIN in the Netherlands) to twenty-nine members (Haute Autorité de Santé [HAS] in France). HTA bodies should balance the need for representativeness with administrative burden and group effectiveness. The form and composition of the group and (if applicable) who has voting rights will also vary according to its mandate or legislation. Publicly stating who is participating, their interests and how conflicts will be reported will enhance the perceived legitimacy of the process. A common approach to support effective participation in deliberative processes of HTA bodies is asking all participants to declare their interest(s) prior to each meeting (42).
What Membership Arrangements Enable Effective Deliberation?-Although an ad hoc group may be suitable for the purposes of deliberating about HTA processes, deliberation within processes typically involves one (e.g., in Brazil and the Netherlands) or more (e.g., in Australia, France, and Thailand) standing committees. A hybrid form can also be used. For example, in Germany, patients and representatives appointed by the Conference of Health Ministers of the German States have discussion and petition rights on all agenda items of the Federal Joint Committee (standing committee) meetings. In addition, several associations have representation at meetings with participation rights on specific topics.

How Will Participants Be Selected?-Identifying and selecting participants (stakeholders and experts) can be accomplished through an open process and interviewing
(as done in the Netherlands) or by targeting groups with informative perspectives (as done in Brazil). HTA bodies may use advertising or invitations to accomplish either approach.
How Are Perspectives to Be Represented?-Participants may represent the views of others or only express their individual views (43). Depending on the topic(s) under discussion and the stage at which it is being deliberated, an individual stake-holder or expert could also be considered an active participant in deliberation, or alternatively, simply as a provider of information to support deliberation by others (i.e., through consultation). HTA bodies need to decide how to include harder-to-obtain or often underrepresented perspectives (e.g., from minority groups) and how these views inform the output(s) of deliberation (i.e., whose votes count and with what weight regarding the recommendation or decision).
Participation of patients as well as public/citizens in the HTA process is also increasingly being considered. Despite the differing perspectives offered by individuals in these roles (44), there has been a tendency to consider patient and public/citizen roles together (45). There is also considerable uncertainty regarding how best to capture both perspectives, and efforts continue to establish best practices in this area (13;18;45). Challenges with patient involvement have been identified, and include ethical issues related to coercion and selection bias when only the most strongly motivated patient representatives participate. For example, a recent analysis of patient representation at the Canadian Agency for Drugs and Technologies in Health (CADTH) highlighted concerns about a lack of attention to preventing some patient voices from "dominating the process" over others (46).
Public involvement may be an even more challenging perspective to represent. Some HTA bodies have used separate citizens' juries or panels to inform expert discussion (47), whereas others have attempted to have the public perspective represented on advisory committees. For public involvement, factors identified as being critical ingredients to effective deliberation include (i) credibility, that is, an ability to contribute knowledge that is considered valid and relevant and that will result in mutual learning and generation of new solutions; (ii) legitimacy, that is, an ability to speak on behalf of others affected by healthcare decisions; and (iii) power, that is, an ability to influence healthcare choices (48). An exploration of public expectations in Australia revealed the public wanted HTA processes to "include a diversity of individuals, be independent and transparent, involve individuals early in the process, and ensure that public input is meaningful and useful to the process" (44).
How Will Participants' Identities Be Disclosed?-HTA bodies must decide if and how participants' identities will be disclosed. Disclosing who is involved, their interests, and how conflicts will be decided and settled will enhance the perceived legitimacy of the process. HTA bodies should also consider that full transparency on these issues could lead to privacy concerns for some participants, including undesired media attention or pressure from interest groups (49). . Although an open process will result in more transparency, HTA bodies should also consider that full transparency could lead to strategic behavior of some participants.

How Open Should the Deliberation
What Type and Length of Deliberation Is Needed?-The mode of deliberative processes may be virtual (e.g., in Brazil, Scotland, England, and the Netherlands), face-toface, or both. Participants may exchange views at a single meeting, at several meetings, at the same time, or at different times. The value of face-to-face deliberation may improve communication among participants, although better communication must be balanced against the potential for undue influence by dominant participants to minimize poor decision making by groups (50). These can include nominal group techniques, consensus-building approaches, and expert elicitation techniques. However, these approaches are seldom used by HTA bodies (51). HTA bodies should decide what level of rigor for deliberation is required and what is feasible given the available resources.
What Are the Rules of Deliberation?-Establishing rules of deliberation requires consideration of the goals of the process, who is participating, and how they will be engaged. Rules of deliberation can encompass both rules regarding how participants interact and share views, over what time frame, as well as rules for what specific information or topics can be discussed. The rules of deliberation are far more influential on affecting the outputs of deliberation than the participants involved or the information considered (52). The rules of deliberation may also be of particular importance when there are multiple deliberative forums and a perceived need for consistency in approaches.
For those deliberative processes intended to provide an opinion (e.g., advice or recommendation), rules are important regarding what procedure will be used to lead to the conclusion of deliberation. For example, for drug coverage decision making, consensus building (e.g., in the Netherlands), majority voting (e.g., in France and Scotland), or both (e.g., in Australia, Brazil, and England) are used by HTA bodies. Deliberative processes intended to result in an opinion may need to consider how disagreement is managed, which could include voting rules. In contrast, processes related to understanding (diverging) views, establishing expectations, creating narratives, and empowering patients and communities will more strongly emphasize participatory dialogue approaches.

Phase IV. Supporting a Deliberative Process
Supports for deliberation include (i) organizing interaction to create participant perceptions of meaningful interaction (53), (ii) effective communication of information to facilitate deliberation (54), and (iii) providing comprehensive terms of reference. These supports not only serve to make deliberation effective, but are also intended to empower participants. A significant concern is that participants may undergo social pressure to conform to the wishes of the group and leave the process unsatisfied (52). They may also be susceptible to cognitive or confirmation biases, that may be further subject to social influences (56). As decisions may be swayed by allowing one member of a group to dominate, the importance of rigorous and balanced facilitation or approach to exchange cannot be underestimated (57).
What Information Will Be Made Available?-HTA bodies need to consider the amount and type of information made available to participants, when it will be made available, as well as how it is gathered, interpreted, and communicated. The information provided will depend on the HTA stage.
For example, for topic selection and prioritization at the NICE (England), all participants receive similar written information, and they are backed up by clear process documents or standard operating procedures (in addition to Terms of Reference). When contextualizing HTA, some countries provide evidence synthesis using structured guidance (e.g., the Grading of Recommendations Assessment, Development, and Evaluation (short GRADE), as used in Germany; https://www.iqwig.de/methoden/general-methods_version-6-0.pdf), whereas others provide a draft HTA report (e.g., Scotland and England), or provide a draft HTA report with the reactions and perceptions of stakeholders on the draft report clearly identified (e.g., in the Netherlands).
To ensure an appropriate amount of scrutiny, consideration should be given to presenting a complete package of information to all participants in a deliberation, rather than presenting redacted or interpreted summaries, which are subject to error and may reduce perceptions of legitimacy. This information could include original clinical studies in addition to a synthesis report (as done at the CADTH in Canada), an original economic model in addition to a summary report (as done at the NICE in England), or other relevant information (e.g., patient-based evidence gathered through interviews, clinical reviews, and assessment checklist results).
If specific criteria are used to guide a decision or recommendation, an HTA body may additionally provide standardized summaries or lists that address these. Checklists, similar to standards used in published research (58) but tailored to a specific jurisdictional context, may also be useful. To further reduce errors when interpreting or synthesizing information for participants, the use of standard analytic judgments (e.g., consensus guidance for analysis, such as in Australia) could be considered. Participants will also benefit from an overview of what information is missing as well as the limitations of interpreting available information.

What Information Will Be Considered by Participants and How Will
Information Be Reviewed and Revised?-Just as allowing dominant viewpoints can be a threat to meaningful participation, so can the introduction of information that has not been equally considered by all participants. This can be remedied by creating enforcing rules regarding what information is allowable (i.e., not allowing participants to introduce new or modified, and potentially out-of-scope, information during deliberation).
How Will Information Be Presented?-Meaningful participation requires a clear and common understanding of available information and its interpretation (55). Effective participant understanding of presented information, including scientific evidence, is facilitated by the use of appropriate media (written, verbal, or both), reporting structure(s), clarity and level of language used, and ensuring a mutual understanding of how the information is identified and synthesized.
In presenting complex, technical information, HTA bodies will need to consider levels of health literacy among participants and introduce appropriate supports (e.g., training or briefing sessions) to ensure exchange that is not hindered by avoidable misunderstanding. This could also include standardized approaches to producing lay summaries and allowing participants sufficient time to clarify misunderstandings prior to deliberation. In England, NICE's Public Involvement Programme provides direct support and training to patient and carer consultee organizations, their representatives, and individual lay persons, patients, or carers. This enables them to contribute to the discussions during deliberative meetings (https://www.nice.org.uk/about/nice-communities/nice-and-the-public/ public-involvement/public-involvement-programme).
What Specific Supports for Participants Are Available?-HTA bodies will need to consider if and what remuneration they might provide to participants in deliberation as this may lead to more willingness to participate and better engagement. They can follow governmental policies on financial compensation to serve at committees of public agencies, as done in Canada, Poland, the Netherlands, and England (42). Better engagement can also be facilitated through training and education to orient participants to the deliberative process as done by the National Committee Health Technology Incorporation in Brazil and the HITAP in Thailand.
Tools and checklists may aid effective deliberation. A checklist for HTA bodies to design their efforts for meaningful stakeholder participation has been developed (12). Wale et al. (59) also provide good practices more narrowly focused on patients and public perspectives and expertise in HTA deliberative processes. Examples include the participation of patient experts to answer questions during the contextualization process of the Scottish Medicines Consortium in Scotland. HTAi has also developed relevant tools and guidance to support patient organizations in completing a patient group submission template (https://htai.org/ interest-groups/pcig/). In Australia, the International Association for Public Participation (60) and the Patient Voice Initiative have similarly developed standards and online tools for patient groups and communities when making submissions to the Pharmaceutical Benefits Advisory Committee's (PBAC; https://www.patientvoiceinitiative.org/resource-library/). In the United States, the National Health Council has a range of resources to support patient input to "value assessment" including basic HTA training (https://nationalhealthcouncil.org/ education/value-classroom/).
Are Comprehensive Terms of Reference in Place?-To enhance public trust and be transparent about the key aspects of (conducting) a deliberative process, comprehensive terms of reference should be created and be publicly available. A term of reference defines the purpose, scope, outcomes, and structures of a deliberative process. We recommend that questions labeled with an asterisk "*" in the checklist (Table 1) are addressed in such a document and reviewed periodically (e.g., every 3-5 years).

Phase V. Development and Communication of the Outputs of Deliberation
Reporting and communicating the process of deliberation, its output, as well as mechanisms for reconsideration of the outputs are important design features to consider.

How Will the Process and Output(s) of a Deliberation Be Reported and
Communicated?-Documenting the process and output(s) of deliberation is the responsibility of the organization that is establishing or managing HTA processes and can be facilitated through video (livestreaming or published recording) and by taking meeting minutes. Meetings can be reported back to all relevant stakeholders in entirety or as a summary, for example, by using a standardized format that includes an explicit description of the issue, the information used, and the views and argumentation that have been put forward. A rapid communication of the output(s) (e.g., recommendation) to an organization submitting a health technology dossier is an important part of processes where appeals or resubmissions are an option (see below). On the other hand, rapid, abbreviated reporting can lead to information loss of specific considerations (48). Providing a complete record can facilitate a better understanding of situation-specific considerations that may arise within deliberation for all stakeholders not involved in the deliberation.

How Will the Outcomes of Deliberation Be Reconsidered?-Possible mechanisms
for reconsideration include appeal, revision, and additional deliberation. The scope and process related to any of these mechanisms may vary depending on legislative or regulatory frameworks. Some HTA bodies, such as the Health Insurance Review and Assessment Service [HIRA] in Korea (61), the NICE in England, and the PBAC in Australia have established mechanisms or guidance for lodging an appeal (see Box 8 for the process at the NICE).
Appeal refers to a mechanism that gives stakeholders the possibility to apply for a revision of a recommendation, involving arguments offered by the individual or group making an appeal, and leading to a reasoned response (62). In designing appeal mechanisms, it is important to be explicit about what constitutes grounds for appeal and/or revision, the characteristics of the body in charge of the appeal (e.g., whether it is independent and the members of the appeal committee) (5), who can or cannot appeal, how the appeal will be considered and decided, and the timelines involved.
For deliberative processes that do not result in a decision, a review and revision of viewpoints, including additional deliberation, may be warranted, particularly if new information from external experts and stakeholders is developed and presented.
The responsible organization should make efforts to ensure that output(s) of deliberation are well communicated and target all relevant stakeholders, using a variety of channels and approaches. These may include the use of official documents (e.g., an official journal), Web sites of relevant organizations, policy briefs, newsletters, and news items on social media to address the broader public.
Consideration may also need to be given to the kind of information and different levels of detail for different audiences. For example, a health technology developer may expect a greater level of detail related to the assessment of evidence and uncertainty, in order to plan for any resubmission or appeal, whereas a patient may be more interested in the implications of the decision for access and possible next steps in that direction.

Phase VI. Monitoring and Evaluation
Monitoring and evaluation are necessary to determine if a deliberative process is achieving its intended aims, to identify aspects that are being done well, and ways in which the process might be improved.

Is the Desired Change(s) from Implementing a Deliberative Process Established and Does It Align with the Health System Values or Principles
Established and/or with Those of the HTA Body?-HTA bodies should start with a comprehensive description of how a desired change is expected to happen from a deliberative process in their context, including the resources (inputs) needed along with the rationale for such change. HTA bodies may assess whether the health system values and guiding principles of deliberation are upheld. The outcomes of a deliberative process could be assessed to evaluate the extent to which the desired change has been established.

How Will the Desired Change(s) from Implementing a Deliberative Process Be
Measured?-Questionnaires, interviews (63), document reviews, or live meetings held with participants are some mechanisms to evaluate deliberative processes. The HTA body can ask for feedback on the process of how the evidence report has been developed, and how experts and stakeholders perceived the process in terms of fairness, transparency, and timeliness, as well as the level and impact of participation (64).
HTA bodies may also consider assessing the outcomes of deliberation that are tied to the broader aims of HTA, such as distributional justice in health care or overall improved health (5). It is important to note that certain outcomes may not be achieved immediately, not because the processes are ineffective but rather due to a time lag for measuring the impact after implementation. It must also be recognized that both process (inputs and activities) and outcome measures are interrelated, with the inputs and activities mainly informing short-term success, whereas the outcomes reflect longer-term impact.

How Will the Desired Change(s) from Implementing a Deliberative Process
Be Assessed?-Monitoring and evaluation require considerations of who does the assessment, how it can be done, and what tools can be used. In addition to involving participants in deliberative processes, there are advantages to seeking an external party, such as an independent evaluator, who may provide an unbiased perspective to the evaluation. Who to involve largely depends on how monitoring and evaluation are done, whether monitoring is done on an ad hoc basis, or more routinely and embedded within a broader framework for evaluating other components of HTA. The monitoring and evaluation approach can be integrated into an existing monitoring and evaluation framework of the HTA body. The HTA body may consider developing or adapting established tools, such as checklists or questionnaires for eliciting feedback. Alternatively, using qualitative methods, such as focus groups and in-depth interviews with key stakeholders either directly or indirectly involved in deliberative processes may be considered for ad hoc assessments. Table 2, we suggest indicators, stakeholders to be involved, and methods for monitoring and evaluation of deliberative processes. These are by no means comprehensive. It is recommended that those leading the monitoring and evaluation adapt them and develop context-specific measures for each, which may be qualitative or quantitative. The process by which these are defined may also be done through deliberation.

What Indicators Will Be Used to Monitor and Evaluate the Deliberative Process?-In
Improving Deliberative Processes for HTA-The guidance and accompanying checklist are offered as practical tools for those wishing to establish or improve deliberative processes for HTA that are fit for local contexts. They can also be used for independent scrutiny of deliberative processes.
As deliberative processes for HTA are rarely seen in actual operation, we encourage those that (intend to) use the checklist and guidance or have recently implemented a deliberative process to share their experiences with the corresponding author of this report. We will use this information to further optimize the guidance and accompanying checklist and to inform future activities of the Task Force.

1.
Governance and structure of an HTA program (deliberation about processes)

Box 2 Definitions
Deliberation in HTA is the informed and critical examination of an issue and the weighing of arguments and evidence to guide a subsequent decision.
A deliberative process for HTA consists of procedures, activities, and events that support deliberation in HTA.

Europe PMC Funders Author Manuscripts
Box 3

Identification and prioritization of relevant topics for HTA
In Thailand, twice a year, the HTA body invites representatives of relevant stakeholders, such as policy makers, health professionals, academics, patient associations, industry, civil society, and the public to suggest topics for assessment. This process involves a multistage process of deliberation (in the form of a working group comprised of stakeholders who have their own stakeholder group deliberations prior to nominating the topics to the working group) informed by evidence and the use of explicit criteria, such as size of the affected population, severity of disease, effectiveness of interventions, variation in practice, economic impact on household expenditure, and ethical and social implications (21-22).

Providing scientific advice to technology developers
The National Institute for Health and Care Excellence (NICE) in England provides scientific advice to developers of drugs, medical devices, biotechnology, digital health technologies, and diagnostics regarding their evidence-generating plans needed for HTA. A formal process is in place in which NICE employees work together with independent clinical and health economic experts, as well as carers and/or patient experts. The participation of patients and/or carers provides an opportunity to understand what is important to patients about their condition and their treatment. All experts equally participate in developing the advice, and support mechanisms are available. Due to the confidential nature of the service, the meeting with the technology developer is not open to the public, not recorded, filmed, or minuted. After the meeting, experts are asked to confirm statements attributed to them. For more information: https://www.nice.org.uk/ about/what-we-do/life-sciences/scientific-advice.

Scoping, assessment, and synthesis of information
Brereton et al. (23) explored the role of different stakeholders and experts in scoping an HTA on palliative home care in seven European countries (England, Germany, Italy, Lithuania, the Netherlands, Norway, and Poland). They included 132 stakeholders and experts, of which eighty were commissioners, health and social care professionals/ academics working in palliative care, and fifty-two comprised lay persons, including patients and families undergoing palliative care of 18 years and older. The contributions enabled the researchers to identify both an intervention and comparator model of service provision for the main HTA question, as well as to inform subquestions for the assessment of specific aspects (e.g., ethical and sociocultural aspects).The Mathematical and Economic Modelling for Vaccination and Immunisation Evaluation Program in the United Kingdom is the first initiative to have the public participate in mathematical and economic modeling. For this purpose, a public involvement reference group deliberated with the academic researchers about different forms of knowledge, expertise, and evidence. The program indicated that by using deliberation, assumptions of the model could be challenged, and its validity enhanced (24).

Box 5 Understanding the Role of Contextual Factors Specific to a Health Technology
Kleinhout-Vliek et al. (32) studied the use of contextual factors in health coverage decision making for four health technologies (nivolumab, benzodiazepines, smoking cessation, and walking aid with wheels) in Belgium, England, Germany, and the Netherlands. Contextual factors were operationalized as arguments about necessity (e.g., disease severity, unmet medical need, dignity, and human rights) because their perceived validity varies per health technology and because they are elements not easily quantified in clinical and/or economic assessment. These factors are actively used in deliberations, in addition to decision criteria (e.g., effectiveness and cost effectiveness). In some countries, they are part of the set of formalized criteria (individual burden of disease in England, Germany, and the Netherlands; and individual cost considerations in the Netherlands). The number of contextual factors considered varied per country and per case: from six different factors (Germany) to sixteen factors (the Netherlands). Some contextual factors were used implicitly, in the deliberation and/or related documents. Others were made explicit and stated in decision documents. In addition, half of the decisions were taken in other settings than the HTA body, with documents that should provide a justification for the final decision being absent. The latter is important because of transparency and accountability reasons.

Box 6 Value Conflicts
During the 2020 Health Technology Assessment International Global Public Forum, participants agreed that inclusivity, transparency, and impartiality are core values of deliberative processes for drug coverage decision making (5). The participants recognized that these values may conflict as this may also require balancing of stakeholder views, including the conflicted views. It was noted, for example, that an HTA body might be transparent about managing conflicts of interest of stakeholders involved, but still may not achieve impartiality, as this may also require active stakeholder engagement in deliberation. In addition, achieving transparency may require trade-offs over what is kept confidential to protect stakeholder interests. This emphasizes the need to achieve impartiality through other means, such as the development of clear guidance on how evidence will be identified and interpreted.

Example of Committee Composition in Ontario
An example of different committee composition within the same political context based on differing mandates is seen in the province of Ontario (Canada). The Ontario Health Ministry hosts two separate HTA processes for drug and nondrug decision making. Drug funding recommendations in 2018 involved deliberation across sixteen voting experts, representing practicing pharmacists, medical specialists, and health economists, along with an expert in patient experience who participated in regular meetings. There was no broader stakeholder consultation to inform the deliberations. The nondrug deliberation occurred across a group of up to twenty voting members having similar expertise as the drug committee, although this committee also includes experts in ethics, and a minimum of two members contributed to patient and public perspectives, and one of them had current or recent experience in industry. Some deliberative processes are used to provide an opinion (e.g., advice or recommendation).
-If deliberation is used to provide an opinion (e.g., advice or recommendation), who has voting rights?* Although majority-based voting is a common approach to arriving at a collective opinion, alternative procedures may be employed to reduce divergence in viewpoints (e.g., blocking), create more transparency regarding the range of viewpoints around any opinion (e.g., dissenting opinions), or reduce distrust among participants.
-If deliberation is used to provide an opinion (e.g., advice or recommendation)," how is the deliberation to end? (closure)*  a Stakeholders include those interested in the HTA process and/or HTA-informed policy decisions with or without direct involvement in the HTA process.
b General public refer to members of society who have no special role in HTA, but may benefit from the outcomes of the processes.