Background
Stakeholder engagement seems to be increasingly emphasized in the theory and practice of heath technology assessment (HTA). This makes sense, given that HTA can be viewed as centrally featuring stakeholder engagement. For instance, a payer (e.g., a government or private insurance provider), a manufacturer, and an assortment of clinical and other experts can engage one another to assess a technology’s safety, effectiveness, and value for money to inform the payer’s decision of whether to pay for the technology’s use and, if so, at what price. Engagement can extend to other stakeholders, especially patients and their advocates, who stand to benefit from effective technologies, and to members of the public, whose interests include stewarding an efficient and equitable health system. Some research has characterized HTA as a complex adaptive system that largely comprises interactions (both formal and informal) between stakeholders, who have diverse purposes and outlooks (Reference Lopes, Street, Carter, Merlin and Stafinski1).
Many in HTA commend greater engagement of patients to properly target assessments to the health and other outcomes that truly matter to patients. Martin has theorized this in terms of a “technocratic” rationale for engaging patients, namely in helping to achieve greater technical accuracy in assessments and subsequent funding decisions (Reference Martin2). This rationale can extend to other stakeholders, whereby the importance of adequate engagement can be viewed in terms of ensuring technical accuracy, namely getting a picture of a technology’s value that is sufficiently comprehensive and accurate, not partial or skewed.
The technocratic rationale contrasts with a “democratic” rationale, whereby (a) stakeholders are engaged because they have the right to comment on and influence decisions that affect them, and (b) engagement increases the fairness or legitimacy of HTA, considered as a decision-making process (Reference Martin2) (see also (Reference Arnstein3)). Daniels has theorized that legitimacy is enhanced by “embedding HTA in a fair, deliberative process that meets the conditions required by accountability for reasonableness” (Reference Daniels, Porteny and Urritia4). By this, Daniels means that decision makers should justify their decisions (i.e., account for their decisions being reasonable) by appeal to policies or other instruments that have ensured that their decisions are publicly transparent, appealable, revisable in light of new evidence or arguments, and based on considerations that all stakeholders agree are relevant (Reference Daniels, Porteny and Urritia4). The notion of legitimacy that Daniels invokes seems to be largely one of political legitimacy, where a decision maker’s credibility depends on their having few or no acute political and financial interests in a particular decision (Reference Daniels, Porteny and Urritia4). Researchers have found that around the world HTA bodies vary considerably “in how they address legitimacy” (which refers to “the reasonableness of decisions as perceived by stakeholders”), with HTA bodies “still lacking or just starting to develop activities in this area” (Reference Oortwijn, Jansen and Baltussen5). One activity of central importance is “stakeholder involvement, ideally through participation with deliberation” (Reference Oortwijn, Jansen and Baltussen5) (see also (Reference Pinilla-Dominguez and Pinilla-Dominguez6)). Some have called for stakeholders (especially patients) to participate more “evenly,” with power being shared in “an environment of earned mutual trust, respect, openness, and reciprocity” (Reference Sampietro-Colom and Thomas7).
The general public often lacks awareness of HTA (Reference Wortley, Tong and Howard8). But the field of HTA increasingly recognizes the public as a stakeholder. Members of the public may help to communicate public values to decision makers and “balance the needs and demands” of various patient groups (Reference Husereau, Henshall, Sampietro-Colom and Thomas9), providing input on “affordability or prioritisation issues” (Reference Wortley, Tong and Howard8).
Much has been published on why it is important to engage stakeholders, especially patients, and some material has been published on when and how to engage. But less has been published on what countries actually do in terms of engaging HTA stakeholders. Thus, there is need for an overview of current practice, as distinct from theory, and this article aims to meet that need. It reports a scoping review that we conducted as part of the 2023 Australian HTA Policy and Methods Review, whose aim was to examine HTA methods and policies internationally, together with the applicability of those methods and policies to Australian HTA and the possible implications of change in Australia, all with a focus on medicines. The Review was driven by a felt need to promote patient engagement and timelier access to medicines. This article reports findings on which stakeholders are engaged as part of the HTA of medicines and the forms of that engagement.
Methods
The methods used to produce this scoping review are fully described elsewhere (Reference Laka, Carter, Gao, Tamblyn and Merlin10). Evidence was limited to English-language publications identified using PubMed, Embase, the international HTA database maintained by the International Network of Agencies for Health Technology Assessment (INAHTA), the HTA agency websites of mostly high-income countries (listed in Table 2), any regulatory or government websites that HTA agency websites referred to, additional sources recommended during consultation with experts, and forward and backward citation mining. The publication date range was limited to between 1 January 2018 and 1 July 2023 to focus on current practice, though backward citation mining stretched back 10 years to encompass key documents. The selection of publications for inclusion was based on their relevance to the participants–concepts–context (PCC) criteria (see Table 1).
Participants–concepts–context (PCC) domains for the scoping review
Table 1 adapted from (Reference Laka, Carter, Gao, Tamblyn and Merlin10).
Involvement of different stakeholders
Yes 
Partially 
No 
Not reported/no information found.
CADTH = Canadian Agency for Drugs and Technologies in Health (renamed Canada’s Drug Agency in May 2024). INESSS = Institut national d’excellence en santé et en services sociaux. HQO = Health Quality Ontario. IHE = Institute of Health Economics. NICE = National Institute for Health and Care Excellence. NIHR = National Institute for Health and Care Research. See the Supplementary Material for detailed country profiles, which contain all data points used to populate Table 2 and often identify the specific HTA agencies and other organizations concerned.
Publications were further excluded for this article if they did not discuss stakeholder engagement. The scoping review finally included 66 peer-reviewed articles plus 264 documents from the gray literature, as detailed in the PRISMA flow diagram (see Figure 1). A table was created to chart the data concerning whether the selected jurisdictions engaged particular stakeholder groups and, independent of this, whether the jurisdictions engaged at all for any specific or specialized technologies (such as medicines for rare diseases or cancer) or contexts (such as the concerted use of real-world evidence) (see Table 2). A consultation process was undertaken to elicit feedback on the accuracy of the information collated, centrally including outreach to all 52 INAHTA member agencies. Several of those agencies provided detailed feedback. Information gathering concluded in October 2023.
PRISMA flow diagram.
A key limitation of this study was its restriction to English-language publications. On the other hand, a key strength was its inclusion of a sizable gray literature (264 English-language documents). This study:
-
• systematically charts data on the topic (e.g., with the traffic lights of Table 2),
-
• compares the scope and context of key concepts across different jurisdictions to identify common themes and differences, and
-
• identifies trends and, importantly, gaps in both gray and peer-reviewed literatures.
Findings
We were able to populate most of Table 2 based on the included literature (see Table 2). The Supplementary Material contains detailed country profiles. These contain all data points used to populate Table 2 and often identify the specific HTA agencies and other organizations concerned. For instance, the US profile contains data points concerning the Advisory Committee on Immunization Practices, the Agency for Healthcare Research and Quality, the Institute for Clinical and Economic Review, and the Patient-Centered Outcomes Research Institute.
Few jurisdictions reported whether they engaged stakeholders in the case of specific technologies (e.g., highly specialized technologies) or contexts, where distinct pathways sometimes existed apart from a jurisdiction’s general HTA processes. Roughly half (17/27) of the jurisdictions explicitly reported engaging academia as part of their general HTA, though the more ambiguous term “expert” was very often used. Where the term “expert” was used without indication of the type of expert (e.g., clinician or academic), it was categorized under “Other.”
Stakeholder engagement is clearly widespread in HTA. The specific stakeholder most consistently engaged is industry (23/27), often in the form of being permitted or invited to submit applications for the funding of medicines. Clinicians are the next most consistently engaged (22/27), often to provide clinical expertise on the medicine or evidence base. They are followed closely by patients and their representatives (including carers, families, and patient organizations) (21/27), who are often engaged to supplement the evidence base, with reports or commentaries on patient experiences and intervention outcomes that matter to them.
Stakeholder engagement occasionally features as an explicit guiding principle of an HTA organization, as in the case of France’s National Authority for Health (HAS), Canada’s Institute of Health Economics (IHE), and the Canadian Agency for Drugs and Technologies in Health (CADTH) (renamed Canada’s Drug Agency in May 2024). In Latin America, stakeholder engagement is limited but there is agreement on its importance for HTA’s legitimacy and for the protection of “decision makers from potentially distorting external influences,” notably “stakeholders or interest groups with lobbyist or pressure power” (Reference Pichon-Riviere, Soto, Augustovski and Sampietro-Colom11). In general, the rationale underpinning patient and other stakeholder engagement practices may lie more in the technocratic goal of enhancing the evidence base than in the democratic goal of giving affected parties their say (Reference Martin2). However, this is difficult to judge, because the rationale underpinning stakeholder engagement practices is rarely stated, though sometimes it is implied. Statements suggestive of a technocratic goal include the following. Australian guidelines state that the “best available evidence” includes evidence “informed by consumer engagement” (12). CADTH regarded patient engagement as especially important for rare diseases in view of “limited clinical knowledge” (13). Singapore provides guidance on
the contribution that patients and their carers can make [to] … healthcare decision-making in Singapore by providing their experiential knowledge of different medical conditions and health technologies, and explaining which outcomes are most important to them (14)
.The focus is on knowledge. By contrast, the Belgian Health Care Knowledge Centre (KCE) and England’s National Institute for Health and Care Excellence (NICE) allude to a democratic goal in engaging patients for “procedural” reasons (i.e., out of regard for fair process), as well as to enhance evidence (Reference Martin2;(45)Reference Kohn, Dauvrin and Cleemput15;Reference Lopes, Street, Carter and Merlin16). HTA stakeholders appear to generally agree on the technocratic goal of enhancing evidence and on the value of a formal process for gathering and integrating information from patient and public engagement, but they remain uncertain on how to best engage and on how to best use the information obtained (Reference Boothe17).
Appraisal committees (i.e., the committees responsible for making recommendations or decisions about the funding of health technologies) variously include:
-
• clinicians
-
• health economists
-
• patient representatives
-
• members of the public (sometimes referred to as lay people)
-
• ethicists
-
• managers
-
• academics
-
• government agency staff (in an advisory role)
-
• government and professional association representatives
-
• industry representatives.
Industry
Industry is commonly engaged by being permitted or invited to provide a submission for funding. Additional modes of industry engagement include:
-
• offering an advisory service to industry for a fee, on matters of evidence generation and patient engagement (e.g., by England’s NICE) (18). Researchers have called for HTA agencies to better define the early advisory services that they could offer industry (Reference Ibargoyen-Roteta, Galnares-Cordero and Benguria-Arrate19).
-
• conducting early dialogue (Reference Galbraith, Guilhaume and Bélorgey20), presumably about regulatory or funding hurdles and associated evidentiary requirements. Early dialogue or early scientific advice among multiple stakeholders is thought to be especially important with specialized technologies, for example, innovative cancer medicines (Reference Galbraith, Guilhaume and Bélorgey20).
-
• permitting industry and professional organizations to nominate “clinical, patient and commissioning experts,” who help to clarify issues with submitted evidence (21).
-
• inviting industry to respond to the HTA report, including at more than one time point, for example, during draft stages. In Scotland and Europe, industry has been given the opportunity “to comment on the factual accuracy of what is said about their product” by competitors in competitors’ submissions (22;Reference Willemsen, Ettinger and Helmink23).
-
• inviting industry to briefly present in the appraisal committee meeting. Australia’s Pharmaceutical Benefits Advisory Committee (PBAC) found forty-five percent of these presentations to be “informative or moderately informative” and eighteen percent to be “uninformative” (Reference Flowers, Lybrand and Wonder24).
-
• holding an additional meeting if funding is denied but the medicine is deemed to treat “a serious, disabling, or life-threatening condition with no other treatment option,” presumably to resume price negotiation or explore alternate funding pathways (Reference Mulligan, Lakdawalla and Goldman25).
-
• permitting industry to appeal against an unfavorable funding decision or to request further independent review.
-
• permitting or inviting industry to resubmit old applications with new evidence or indications.
Clinicians, academia, and others
The involvement of “experts” is commonly cited as essential in interpreting and supplementing the best available research evidence, though the nature of the expertise required is not always specified. Clinical experts and experts in particular disease areas or scientific methodologies are commonly engaged. Ethicists are occasionally engaged as appraisal committee members or, along with legal experts, as experts for specific technology types. Clinician engagement is regarded as especially important as intervention complexity increases. In Canada, a complex review, such as for “cell and gene therapies” and “first-in-class” products, “involves greater consultation with clinical experts” (26). Singapore, Spain, and the United States have used clinician engagement to guide the prioritization of technologies for assessment. Healthcare providers and “other health system stakeholders” are also often engaged (27).
Patients
Patient engagement is regarded as especially important where clinical knowledge is more limited (for instance, with rare diseases or innovative technologies). Canada’s Institut national d’excellence en santé et services sociaux (INESSS) in Quebec has expressed a general commitment to “further integrating the patient, caregiver and citizen perspectives … especially for innovative therapies” (28). Scotland established a patient and public reference group for the HTA pathway that it created to especially support “innovative approaches” (29).
Patient engagement is also often viewed as serving or enhancing equity. Engagement with patients from diverse backgrounds is commended to help identify diverse equity concerns.
Government departments and HTA agencies in Australia, Germany, Scotland, Singapore, and Wales have dedicated patient engagement staff, committees, or councils. Their roles include:
-
• assisting government to engage patients more effectively
-
• presenting information about patient experiences to the appraisal committee
-
• informing policy
-
• increasing public understanding
-
• collecting new data on patient experiences through small-scale primary research (e.g., interviews and focus groups with patients and carers)
-
• enhancing methods of patient engagement
-
• mentoring patient representatives who sit on appraisal committees
-
• recruiting patients or patient organizations for data collection
-
• supporting patient organizations to provide comment.
Methods for collecting data on patient perspectives include:
-
• canvassing or formally reviewing published literature
-
• personal consultation (Reference Seiffart30)
-
• inviting a small number of patients to test texts for their content and readability (31)
-
• public consultation
-
• conducting appraisal meetings online to facilitate the attendance of patient members (especially during COVID-19) (Taiwan and England’s NICE)
-
• conducting original qualitative research with patients who have “lived experience” of the technology, along with their “families and other caregivers” (32). Canada’s Ontario Health routinely does this. An external contractor conducts focus groups and interviews with patients for Germany’s Institute for Quality and Efficiency in Health Care (31). Spanish HTA agencies have also used patient surveys (Reference Toledo-Chávarri, Triñanes Pego and Rodrigo33).
Europe has mostly engaged patients via online forms and one-on-one conversations (Reference Willemsen, Ettinger and Helmink23), together with group conversations and scoping meetings (Reference Elvsaas, Ettinger and Willemsen34). Internationally, patients are mostly engaged through public consultation: they provide their perspectives on the topic, but direct involvement (e.g., being “at the same table with other stakeholders in working groups”) is less common (Reference Gagnon, Tantchou Dipankui, Poder, Payne-Gagnon, Mbemba and Beretta35). The window of opportunity for patient input is often small, e.g., several days or weeks before and after the assessment report is finished and shared.
A 2014 systematic review found that patients were generally not given direct roles in reducing uncertainties relating to value for money, affordability, or technology adoption or diffusion (Reference Menon, Stafinski, Dunn and Short36). Instead, patients were involved in “activities aimed at generating information on clinical benefit,” which then informed discussions among others on uncertainties (Reference Menon, Stafinski, Dunn and Short36). The Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU) and the US’s Agency for Healthcare Research and Quality have engaged patients to prioritize research projects based on knowledge gaps important to patients (37;Reference Buckley, Ansari, Butler, Williams and Chang38).
Patient recruitment methods span:
-
• the use of patient advocacy networks, including direct outreach to those networks
-
• outreach to patients identified in news articles, social media, and the health service
-
• using websites and social media to invite patient feedback and recruit committee members
-
• symposia and conferences.
England’s NICE uses information from patients in all phases of HTA, including in the scoping stage for development of the Population–Intervention–Comparator–Outcome (PICO) parameters and throughout the fuller assessment and appraisal. Patient engagement in scoping and topic development has also been tried (Reference Garrett, Imaz-Iglesia and Willemsen39). Canada’s CADTH permitted patient groups to comment on draft recommendations (26).
However, despite demonstrated commitments to patient engagement, problems have included:
-
• sporadic and unsystematic engagement, as in Austria (Reference Fried and Wild40)
-
• challenges with recruitment, capacity building (Reference Toledo-Chávarri, Triñanes Pego and Rodrigo33), and timeliness (Reference Elvsaas, Ettinger and Willemsen34)
-
• disagreement and uncertainty on the role and impact of public and patient engagement (Reference Boothe17) (see also (Reference Elvsaas, Ettinger and Willemsen34;Reference Bidonde, Vanstone, Schwartz and Abelson41))
-
• differences in philosophy and priority given to public and patient engagement, including an emphasis on evidence-based principles that functions to exclude the meaningful integration of patient input (Reference Bidonde, Vanstone, Schwartz and Abelson41) (see also (Reference Boothe17)).
To help with the last two problems, researchers have recommended that conflicts “between multiple epistemic traditions” (i.e., conflicting views about what counts as genuine knowledge) be openly acknowledged (Reference Bidonde, Vanstone, Schwartz and Abelson41) (see also (Reference Lopes, Carter and Street42)). To help with recruitment, it has been suggested that a register or pool of relevant patients or patient organizations be created and used (Reference Elvsaas, Ettinger and Willemsen34).
Patient groups commonly report problems in:
-
• recruiting patients (Reference Mercer, Chambers and Mai43)
-
• knowing the impact of their contributions (Reference Lopes, Street, Carter and Merlin16;Reference Mercer, Chambers and Mai43), despite commitments to transparency
-
• having enough resources to prepare submissions.
Researchers have suggested that “the lack of clear reporting on the use of patient group input in deliberations and therefore accountability to patient groups limits progress in patient involvement in HTA” (Reference Wale and Sullivan44). A clearer view of how patient inputs are used will give patients clearer guidance on the kinds of information to provide (Reference Wale and Sullivan44). England’s NICE has a Public Involvement Programme to promote the involvement of patients, families, carers, and the public “regardless of disability, language, or other potential barriers” (45).
The public
The distinct interests of members of the public (e.g., as taxpayers or stewards of a good society and sustainable health system) are rarely differentiated from the interests of patients and their advocates (e.g., as people wanting the best possible health care for themselves and those close to them). For instance, Denmark’s HTA Handbook frequently mentions “citizens,” but confusingly this often refers to “patients as citizens” (Reference Kristensen and Sigmund46). One research study suggests that patient and public member positions on appraisal committees “may have been created without a good deal of consideration for the different contributions they could make,” but many in HTA now see a distinction (Reference Boothe17). Researchers have proposed an updated taxonomy of patient and public groups, emphasizing how these can be conceptually distinguished by their different and potentially divergent interests (Reference Street, Stafinski, Lopes and Menon47). Ireland distinguishes patients from the public but considers only patients as stakeholders by having “a direct interest in the process and outcomes” of an HTA, implying that the general public has, at most, an indirect interest (48).
Modes of engaging the general public include:
-
• the online publishing of policies, procedures, analyses, decisions, meeting summaries, or full assessment reports (there appears to be wide variation internationally in the extent of what is made public)
-
• establishing and running a Layperson Advisory Committee (49) or Citizen Committee for Participation (Korea) to help make funding or other recommendations
-
• conducting appraisal meetings in public. England’s NICE, the All Wales Medicines Strategy Group, the Scottish Medicines Consortium, the US’s Advisory Committee on Immunization Practices, and the US’s Institute for Clinical and Economic Review all hold their meetings in public.
There is evidence that invitations for public comment on HTA reports mostly generate industry comments on methods (Reference Gerlach, Snow and Prioli50). Perhaps partly for this reason, some jurisdictions (such as France, Germany, New Zealand, the Netherlands, and Wales) limit online calls for comment to patient groups (Reference Stafinski, Glennie, Young and Menon51).
Rarely do jurisdictions explicitly discuss engagement of the public, as distinct from patients, even while jurisdictions take care to publicize HTA processes and documentation.
Discussion
Our findings suggest that HTA agencies are increasingly active in stakeholder engagement, especially with patients. Jurisdictions creating or renewing their HTA processes may be turning to the scholarly literature for guidance, then finding and acting on the increasing volume of published material on patient engagement. The observed variation in patient engagement methods may owe to engagement occurring with different goals and at different levels, for example, at the level of informing a single assessment, guiding an entire HTA agency, or setting government policy (Reference Gauvin, Abelson, Giacomini, Eyles and Lavis52). To avoid confusion and ensure that engagement activities are correctly targeted, a clear and cogent taxonomy of patients and other stakeholders (e.g., the public) should be explicitly adopted by governments, HTA agencies, and other stakeholders (Reference Street, Stafinski, Lopes and Menon47). There may also be a role for stakeholders, especially governments and HTA agencies, to develop and communicate a clear policy on which stakeholders they seek to engage, together with the why, when, and how of that engagement. Greater clarity may also be needed on precisely who is responsible for engagement. In 2018, an interview study with industry involvement was conducted regarding different stakeholder expectations in medicines development and use (Reference Husereau, Henshall, Sampietro-Colom and Thomas9). Policymakers and regulators “were thought by others to have a leading role in setting the framework and process” for patient engagement, but the policymakers and regulators themselves did not see this as their role, focusing instead on access to safe medicines (Reference Husereau, Henshall, Sampietro-Colom and Thomas9). The same was true for healthcare professionals, industry members, and researchers: each group thought that patient engagement was somebody else’s responsibility.
Researchers have noted that “enhanced stakeholder engagement requires sustainable resourcing,” which is typically lacking (Reference Sampietro-Colom and Thomas7). Our findings revealed that patient groups, in particular, commonly express concerns about time and resource constraints. Proactive engagement strategies could serve to shift resource burdens from patient groups to other stakeholders. Proactive engagement strategies could also help to address a “lack of information” on the part of patients and the public about HTA (Reference Gagnon, Tantchou Dipankui, Poder, Payne-Gagnon, Mbemba and Beretta35). Additional resourcing would need to build the capacity of patients and the public to engage, as well as the capacity of other stakeholders to proactively engage patients and the public. The resources would need to encompass staffing, skills development, and supports to overcome barriers to engagement, such as any deriving from disability or language. Appropriately resourcing patient groups could also help to reduce the scope for perceiving any undue industry influence on patient groups (Reference Lopes, Street, Carter and Merlin16). Our findings suggest that patient advocacy networks could play a role in facilitating recruitment to engagement activities, and that a pool of potential partners could be created and used to facilitate recruitment. Engagement before finalizing an assessment’s PICO parameters would likely help to ensure that the outcomes assessed include the ones that truly matter to patients. Such engagement features among the options for reform arising from Australia’s Health Technology Assessment Policy and Methods Review (53). A central option for reform is to establish an engagement framework, one element of which would be “adequate resourcing of proactive engagement,” particularly to identify patient “subgroups that do not engage” and “work with them to co-design appropriate engagement approaches” (53). Such a framework could extend to all stakeholder groups. Ontario stands out in its routine conduct of original qualitative research regarding patients’ lived experiences (32). An added advantage of formally conducting such research is to potentially inform appraisal decision makers of social and ethical issues raised by patients, such as impacts on their autonomy and identity (Reference Milverton and Carter54). This would likely have greater implications for some technologies than others. For example, research on patients’ lived experiences of genetic tests would likely reveal various impacts on patients’ autonomy and identity, whereas such research relating to different technologies, such as some medicines, may reveal no such impacts (Reference Milverton and Carter54).
Our findings suggest that governments and appraisal committees should openly acknowledge the potential for conflict between differing epistemic approaches relating to technology evaluation. For example, an evidence-based medicine perspective has the potential to conflict with a perspective focused on knowledge obtained through lived experience of an illness (e.g., what is experienced may not be what is measured in a scientific study) (Reference Lopes, Carter and Street42). HTA committee chairs should be selected or supported to enable deliberations and a broader institutional culture in which both perspectives can be taken seriously. This should help to foster a committee open to patient engagement and insights. Researchers have found that at NICE the chair was critically important in opening a space for dialogue and fostering a broader institutional culture in which “experiential evidence, interpretations and opinions” could be taken seriously alongside “rigorous evaluations and scientific rationality” (Reference Hashem, Calnan and Brown55). An understanding of the complementarity, and not simply conflict, of perspectives may be emerging but not universal.
Appraisal committees should also commit to publicly communicating their consideration of any public and patient engagement, along with any impact of this engagement on decision making. This is warranted in light of stakeholder engagement’s “democratic” rationale (Reference Martin2), but such communication could also prove instrumental to building trust and confidence in the committees and to guiding the public and patients on the sorts of information that committees find helpful. Indeed, this probably extends to all stakeholders, in view of findings of only partial transparency to stakeholders generally (Reference Laka, Carter, Gao, Tamblyn and Merlin10). Australian options for reform include reporting to groups “how their input has been used” (53). Rowe and Frewer theorized that the evaluation of a public engagement procedure should include a “Criterion of influence: The output of the procedure should have a genuine impact on policy” (Reference Rowe and Frewer56). They reasoned as follows:
One of the main complaints about participation methods is that they often have been perceived as ineffectual, simply being used to legitimate decisions or to give an appearance of consultation without there being any intent of acting on recommendations. This results in public skepticism and distrust … [Thus] use of the media to inform the general public about the specific ways in which the output has influenced policy would seem beneficial (Reference Rowe and Frewer56)
.Our findings suggest that patients and other stakeholders should be informed of how their inputs were or were not used in appraisal decision making. Communicating this will improve public engagement under Rowe and Frewer’s influence criterion.
The Australian government recently commissioned a codesign project to improve patient engagement in HTA. In line with our findings, it recommended proactively notifying patients of HTA activities, providing resources and training to support equitable engagement, elevating patient evidence and input within deliberations, and routinely providing insight into how patient input was used (57).
Notable in the included literature was an underemphasis on engaging the public, as distinct from patients. Public engagement is often limited to a matter of transparency and public accountability, whereas the potential scope for public engagement is much larger. There is a public appetite for greater involvement in HTA and for providing input that is “meaningful and useful to the process,” including on “affordability or prioritisation issues” (Reference Wortley, Tong and Howard8). Public representatives on appraisal committees could help to ensure that decision making takes due heed of public values, including public perspectives on value for money. However, Australian research has revealed some public skepticism toward this: “being a lone public voice on a decision-making committee could be ‘intimidating’… the voice of the public was always going to be secondary to other stakeholders” (Reference Wortley, Tong and Howard8). A lone public voice could also be unrepresentative of the broader public. Enhanced public engagement could instead occur in creating or updating a checklist or value framework to assist decision makers to integrate public values, including equity considerations, into their appraisal decision making (Reference Carter, Vogan and Haji Ali Afzali58).
Conclusion
The peer-reviewed and gray literatures on the jurisdictions included in this scoping review suggest the following. The stakeholder groups most consistently engaged as part of HTA are industry then clinicians, followed closely by patients and their representatives. Industry is commonly permitted to provide a submission for funding, though the modes and time points of industry engagement are many. Clinician and patient engagement are regarded as especially important with increased intervention complexity and innovation. Stakeholder engagement is perhaps usually conducted to enhance the collation and interpretation of evidence, not necessarily to increase the legitimacy of the HTA process or to give stakeholders influence over a funding decision that affects them. However, this is difficult to judge, because the rationale underpinning stakeholder engagement practices is rarely stated. Patients are mostly engaged through broader public consultation. Sometimes they work directly with other stakeholders. Problems with patient engagement include challenges with recruitment, time, and resource constraints. Stakeholder groups can also differ in how they view and prioritize public and patient engagement. Differences in philosophy can arise when, for instance, an evidence-based medicine perspective conflicts with, and threatens to override, a perspective focused on knowledge obtained through lived experience of an illness. Governments and appraisal committees should openly acknowledge the potential for conflict between differing epistemic approaches relating to technology evaluation. HTA committee chairs should be selected or supported to enable deliberations and a broader institutional culture in which both perspectives can be taken seriously. Public engagement is often limited to a matter of transparency and public accountability, but the reasons to undertake public engagement are numerous and varied. They include gaining input on affordability and prioritization issues. Decision-making committees should commit to publicly communicating how they considered stakeholder input and any impact that it had on decision making. This could build trust and confidence in the committees and guide the public and patients on the information that committees find helpful. To avoid confusion and ensure that engagement activities are correctly targeted (e.g., at patients or, differently, the general public), a clear taxonomy of stakeholder groups should be explicitly adopted. Greater clarity may also be needed on precisely who is responsible for engagement. Proactive engagement strategies could serve to shift resource burdens from patient groups to other stakeholders. They could also help to inform patients and the public about HTA.
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/S0266462325100494.
Acknowledgments
We thank Ann Single for very insightful comments on the research article on which this article was based. The Australian Government Department of Health and Aged Care contracted Adelaide Health Technology Assessment to conduct the research on which this article was based as part of its Health Technology Assessment Policy and Methods Review (59). We thank the Department for its feedback on the manuscript. The views expressed in this article are those of the authors and not those of the Department of Health and Aged Care or of the Government.
Competing interests
The authors declare none.
Open access
