Health technology assessment (HTA) requires rigorous analytical methods with an emphasis on randomized controlled trials and appraisal of the evidence for benefits, harms, and costs by expert committees, but a focus on clinical outcomes alone may not be sufficient (Reference Drummond, Schwartz, Jonsson, Luce, Neumann and Siebert1). Important information relevant to the clinical use of technology includes patient-relevant health outcomes, access to healthcare services, patient preferences, quality of life, immediate and longer-term consequences (Reference Gagnon, Desmartis, Lepage-Savary, Gagnon, St-Pierre and Rhainds2;Reference Mamzer, Dubois and Saout3). This is in addition to burdens and costs for users of the technologies and overall patient and carer experiences. Patients can provide unique, experiential knowledge to give context to the clinical research, help to overcome uncertainties in clinical evidence, and inform the determination of the value of a technology or service (Reference Facey, Boivin, Gracia, Hansen, Lo Scalzo and Mossman4–Reference Roy, Ganache and Dagenais7).
Patients' views may differ from those of health professionals or researchers and cannot be assumed by other stakeholders (Reference Gagnon, Desmartis, Lepage-Savary, Gagnon, St-Pierre and Rhainds2;Reference Staley and Doherty8;Reference Dipankui, Gagnon, Desmartis, Legare, Piron and Gagnon9). Patient involvement has a key role in the legitimacy of HTAs such that it is important that HTA recommendations have been informed by patient concerns and perspectives (Reference Boothe10). The rights and value of patients participating in HTAs are becoming more widely supported (Reference Wale, Scott, Hofmann, Garner, Low and Sansom11). It is therefore important, for public understanding and knowledge, that the involvement process supports learning through informed conversations (Reference Staley and Barron12) and transparent reporting on HTAs (Reference Deane, Delbecque, Gorbenko, Hamoir, Anton Hoos and Nafria13).
Quality criteria have recently been published for patient engagement in drug development, calling for shared purpose and respect for the resources and energy required for providing patient input. The criteria include representativeness of stakeholders with agreed roles and responsibilities, the resources and capabilities to enable meaningful engagement, transparency in communication and documentation, and continuity and sustainability (Reference Deane, Delbecque, Gorbenko, Hamoir, Anton Hoos and Nafria13). The opportunity for learning from feedback is important for patient advocates and patient groups involved in HTA. Indeed, it is likely that many patient groups do not have a complete understanding of what information they might need to convey, what would be of most use in HTA deliberations, the form that needs to take, and which insights if any could impact the deliberations (Reference Boothe10). The need for learning experiences is often not acknowledged by those who are in a position to provide the feedback (Reference Mathie, Wythe, Munday, Millac, Rhodes and Roberts14).
Boothe (Reference Boothe10) identified that goals for patient involvement in HTA continue to be unclear, without a clear vision of what the involvement should achieve. A lack of alignment on thinking among the different players also exists, making it very difficult to meaningfully evaluate patient involvement. The author used surveys and interviews over a number of years to describe how the different players in HTA appraisals saw the goals of patient group involvement, and how thinking had changed over time, for the Ontario Committee to Evaluate Drugs and the pan-Canadian Common Drug Review (CADTH-CDR), now part of the Canadian Agency for Drugs and Technologies in Health (CADTH) (Reference Boothe10).
The HTAi Patient and Citizen Involvement in HTA Interest Group (PCIG) previously collated the experiences of international patient advocates in providing input to the appraisals of health technologies for HTA bodies. We identified a common lack of clearly-defined purpose and goals of patient input, limited feedback to advocates on the value and use of the patient input, and limited communication to the public of the impact of patient involvement (Reference Scott and Wale15). Only the patient advocates who were patient representatives on an HTA appraisal committee were able to refer to specific positive recommendations where important contributions were made through patient group submissions, for example, on quality of life and more adversely affected sub-populations (Reference Scott and Wale15). In recent years, we have identified patient advocate concerns about how patient involvement in HTA appraisals is reported on, leading to uncertainties about how the input was used. In this commentary, we explore this concern by looking at how three HTA bodies reported on their use of patient involvement in final appraisal committee recommendations and discussion summaries. Our aim was to elucidate how the HTA bodies described their use of patient input in the final recommendations and discussion summaries, and so to observe the situation and view it as unique without imposing any deductive structure upon it. Such insights into how patient input was utilized may help us better understand the goals of patient involvement and how these are being achieved by HTA bodies. We see the need to examine real-world evidence on the value of patient input as is currently being used. Without clear reporting on the use of patient group input patient groups may not have a clear understanding of what information they can best provide to inform deliberations. Providing this evidence may also lead to more open, meaningful discussions and work on the culture and practice around the utilization and reporting of patient input, to encourage learning and evaluation of methods used to gain patient input into HTAs.
We worked with three HTA bodies that have formal processes to collect patient group input through submission templates with carefully defined questions. An important difference among the processes used is that the National Institute for Health and Care Excellence (NICE) and Scottish Medicines Consortium (SMC) ask for the input during the appraisal stage whereas CADTH asks for the input during protocol development. NICE also has a process for involving patient groups in discussions to inform the development of the protocol. At appraisal, NICE invites patient experts generally nominated by patient groups to report on their experiences with the disease and its treatment, and to answer questions. SMC provides the opportunity for technology sponsors to request a Patient and Clinician Engagement meeting (PACE) before its appraisal of orphan or ultra-orphan and end of life medicines; a summary of the meeting is presented at the SMC committee meeting alongside the patient group submissions. We selected two different pharmaceutical interventions for two different medical conditions, as a single example could more easily be considered as an anomaly or specific to a particular health condition.
Methods for Information Gathering
For each of the three HTA bodies, a final committee discussion report summarizes the deliberations of the appraisal committee that lead to the final recommendation (Figure 1). By “final recommendation” we refer to the funding recommendation (CADTH), guidance (NICE), or advice (SMC) from the appraisal committees, together with the summary committee discussion report. SMC incorporates both into a “Detailed Advice Document”; and CADTH Canadian Drug Expert Committee (CADTH-CDEC) into “Recommendations and Reasons” (see Figure 1). As an exploratory study, we descriptively analyzed the documentation for explicit and inferred reference to the different types of patient input, particularly in describing the needs and concerns of patients and caregivers and what they value. Inferred contributions related particularly to the descriptions of the health condition, impact of the disease, and its treatment. Extracts of the text were entered manually into Excel by the first author and checked by the second author. Although the submission templates provide for input under specific headings, no attempt was made to go beyond these descriptors with a thematic qualitative analysis or to give weight to the input beyond what was recorded. We sought feedback from the other members of the HTAi PCIG working group when summarizing this process and their comments were diligently addressed.
Figure 1.Website presentation of the final recommendations and committee discussion reports, and patient-relevant submissions, for each HTA body: Canadian Agency for Drugs and Technologies in Health—Canadian Drug Expert Committee (CADTH-CDEC), Canadian Agency for Drugs and Technologies in Health—pan-Canadian Oncology Drug Review Expert Review Committee (CADTH-pERC), National Institute for Health and Care Excellence (NICE) and Scottish Medicines Consortium (SMC).
We considered two drug appraisals by each of the three HTA bodies (16–21), for treatment of a chronic disease (ustekinumab) and cancer (blinatumomab). Appraisals took place from May 2016 to August 2017. We summarized, for comparison with the publicly available appraisal committee discussion reports, the key points made by patient groups in their templated submissions: on how the condition affects patients' daily lives; experience with symptoms and severity of the condition; how patients are affected by current treatments, and how well they are managing; impact on caregivers; experience with or expectations of a new treatment in terms of benefits, advantages, and disadvantages compared to current treatments; impact on family and caregivers; whether the new treatment is innovative; and any other factors.
HTA Appraisal of a Chronic Disease Treatment
Ustekinumab is a drug for moderately to severely active Crohn's disease in adults whose previous treatment resulted in an inadequate response, a response that was not maintained, or where people could not tolerate or have contraindications to the treatment. Previous treatments were conventional therapy or a TNF-α inhibitor biologic.
Patient Group Templated Submission
The patient groups that provided patient input were: CADTH, Crohn's and Colitis Canada (CCC) and GI (Gastrointestinal) Society, Canada; NICE, Crohn's and Colitis UK; and SMC, Crohn's and Colitis UK. The patient groups stated their sources of information and the submissions were rich with patient quotes (Figure 2). The NICE and SMC submissions were provided by the same UK organization but were substantially different from each other.
Figure 2.Direct patient quotes from patient group templated submissions for moderately to severely active Crohn's disease.
Final Recommendations
For each HTA body, the final recommendations did not directly reference the patient input received, and the cost was a significant factor in the recommendations. We then looked at the reports of the appraisal committee deliberations for reference to patient input.
Final Committee Discussion Reports
For CADTH, the “Recommendations and Reasons” stated that the “Canadian Drug Expert Committee (CDEC) considered patient group submitted information prepared by the Common Drug Review (CDR) on outcomes and issues important to individuals living with Crohn's disease.” The clinical and economic reports provided to CDEC were outside this study remit (Figure 1) and we did not look at any use of the patient group submissions to interpret the clinical trial results and critique the economic model provided to the appraisal committee.
A summary of the patient group submission to NICE was given. Two patient experts presented to the NICE appraisal committee. The committee discussion report focused on the patient experts: “It heard from the patient experts that…”; “The patient experts recounted their experience of”; “both the patient experts had ustekinumab as part of a clinical trial after losing response to TNF-alpha inhibitors, being able to ‘…resume work and everyday activities as well as avoiding the need for surgery’.” One patient expert explained that ustekinumab had left him feeling “wonderfully normal again” while the other highlighted the immense improvement to her quality of life, enabling her to start a family. The patient experts emphasized that maintenance treatment with ustekinumab is a subcutaneous injection rather than an intravenous infusion, which is greatly valued by patients because it means they can take the treatment at home with no need for hospital visits. The key conclusions state: “The committee acknowledged that ustekinumab is a convenient and well-tolerated treatment that has considerably improved the quality of life of the patient experts.”
For SMC, the patient group templated submission was summarized as a bulleted list in a “Summary of patient and public involvement” section of the final appraisal committee discussion report.
Inferred Use of Patient Input
Looking at inferred use of patient group submissions, examples were: “The [NICE] committee understood that Crohn's disease follows an unpredictable, relapsing and remitting course with many debilitating symptoms”; “The [NICE] committee heard that patients fear loss of remission and exacerbations of the disease because of the major impact these have on quality of life… it is very important to have a range of treatment options ….”
HTA Appraisal of a Cancer Treatment
Blinatumomab was the drug under consideration for adults who had relapsed or were refractory to previous chemotherapy treatment.
Patient Group Templated Submission
The patient groups that provided patient input were: CADTH-pCODR, Leukemia & Lymphoma Society of Canada; NICE, Leukaemia Care; and SMC, Leukaemia Care.
The patient groups stated their sources of information. Three patients had direct experience of the new treatment in Canada. In England and Scotland, the patient group was unable to identify any patients with direct experience. The patient group submissions contained few quotes (Figure 3). The NICE and SMC submissions were from the same patient group and their content was similar. For NICE and SMC, the submissions emphasized the importance to patients of being able to have a treatment-free period and the option for some patients to be treated as outpatients, allowing patients to spend more time at home with their families. For CADTH, the patient group submission concentrated on troublesome symptoms and side effects for six people who had not taken the drug and on patient experience for the three who had received blinatumomab.
Figure 3.Direct quotes from patient group templated submissions for previously treated Philadelphia-chromosome-negative acute lymphoblastic leukemia and blinatumomab.
Final Recommendations
The CADTH pan-Canadian Oncology Drug Review (CADTH-pERC) and NICE final recommendations did not refer to specific patient input whilst the SMC Advice statement clearly referred to PACE: “This advice takes account of the views from a Patient and Clinician Engagement (PACE) meeting.”
Final Committee Discussion Reports
For CADTH, the “Summary of pERC Deliberations” states that “The Committee deliberated on the patient advocacy group input, which indicated that patients with ALL value disease control and the management of side effects related to current therapies for ALL. Given improvements in OS and maintenance of QoL, pERC agreed that blinatumomab aligns with patient values overall.”
For NICE, it is the perspective of the patient expert that is given with details about the physical and psychological stresses of failing chemotherapy, and the emotional strain on families and friends: “The committee heard from a patient expert that..”; “The committee heard from the patient expert that patients whose disease responds to blinatumomab can live a relatively normal life during treatment, with minimal side effects.” In some of the text, patient expert and clinical expert views were linked: “…heard from the patient expert and clinical experts that people with ….”
The committee concluded in their deliberations that the availability of an effective treatment that could be delivered primarily in the outpatient setting was hugely beneficial to patients and would have a major impact on their quality of life. A summary of the patient group submission was provided.
For SMC, the patient group submission was referred to and a detailed summary of the submission provided. It was stated that a patient group submission was received from Leukaemia Care, who provided representatives to attend PACE, and that points of the submission were covered under PACE. Detailed information was given on PACE deliberations in relation to “the relatively young age of patients (mid to late thirties) and the high symptom burden associated with this condition and how a diagnosis of relapse is devastating for patients and their families.” At the PACE meeting, “the deep response and high remission rates with blinatumomab, a greater chance of moving to allogeneic haematopoietic stem cell transplantation, improved quality of life and more time spent at home with the family” were noted. It is evident that the PACE feedback was very seriously considered in discussions and was integrated throughout the report.
Inferred Use of Patient Input
Patient input was inferred in the following examples from NICE: “The committee acknowledged that acute lymphoblastic leukaemia does not affect the patient in isolation, but also places emotional strain on their families and friends”; “Living with precursor B-cell acute lymphoblastic leukaemia, particularly when the condition has failed to respond to first line chemotherapy, can have a profound effect on a person's physical and psychological wellbeing.”
Discussion
Our exploration of the final recommendations and summary committee discussion reports for the appraisal of two drugs across three HTA bodies showed clear differences in how the types of patient input—patient group submissions, patient experts, and patient and clinician meetings—were referred to. One possible interpretation is that each type of patient input was considered differently by the appraisal committees. Differences could be caused, for example, by variations in how the different types of input were presented, who presented it, and at what stage of the meeting, and if and how the input resonated with expert members of the appraisal committees. Because no direct feedback was given by HTA agencies to patient groups, it would be very difficult for the patient groups to know how their input was received and how useful it was; which may align with a lack of clear expectations for patient submissions. On the other hand, it could be that report writers are not being asked to specifically capture the utilization of all types of patient input in the deliberations, and a different set of skills may be needed to do this, whilst maintaining the objectivity of the HTA process.
Overall, direct patient input through participation in meetings appeared to be more influential than written patient group submissions. Patient input was utilized by CADTH-pERC to inform alignment with patient values, with little further information on how “alignment with patient values” actually influenced the deliberative process. For NICE, the perspectives of patient experts were referred to in illustrating discussion points, with limited reference to the patient group submissions. For SMC, the PACE feedback was referred to and integrated throughout the summary discussion report. The collective views of patient advocates and clinicians and how they were presented to the appraisal committee would appear to have been highly valued by the committee. Indeed, patient involvement may work best when health professionals and patients enter into dialogue to share their respective expertise and learn from each other (Reference Staley and Barron12).
If experiential knowledge gained from the patient group submissions was being utilized in deliberations, this use may not always be clearly recorded. We did not observe any attempt to relate one form of patient involvement to another form; and any use of experiential patient knowledge to interpret the clinical trial findings was not evident in the reports, including for quality of life data. It has been suggested that better engagement of patients, users, caregivers, citizens throughout the assessment and appraisal process to seek convergence of scientific data on efficiency, safety, and acceptability with contextual and experiential data could be used to strengthen decision making (Reference Roy, Ganache and Dagenais7). For NICE, rich information from patient experts could help overcome the limitations of clinical trials where patient-reported outcomes, patient experience, or quality of life data were missing or the measures were too generic (Reference Hashem, Calnan and Brown22). A study of patient engagement in CADTH-pCODR found that submissions from patient advocacy groups were most impactful when they provided information that was not available, well documented, or easily recognized from other sources. This included information that was not collected in clinical trials, any clinical trade-offs, information on lived experience, and patient and caregiver priorities. The committee gained a better understanding of the realities of a disease and its practical burdens from patient input (Reference Rozmovits, Mai, Chambers and Chan6).
Staley and Barron (Reference Staley and Barron12) have identified the importance of mutual learning as an outcome of patient involvement. Quality interactions can lead to greater understanding and better decisions through respectful conversations with a two-way exchange of knowledge. Patient experts are present during discussions but they may not be charged with the ability, confidence, and opportunity to challenge committee members' assumptions and perspectives in sharing their experiential knowledge. Recording where patient input has an impact supports wider learning as it can fill gaps in the knowledge of other stakeholders and correct or modify assumptions (Reference Staley and Barron12). The overall uncertainty about the specific roles of the different types of patient involvement and how they work together is likely to result in difficulties in strengthening the value of patient involvement, and in informing changes in thinking (Reference Boothe10) as we move toward more person-centered health care. Completing patient group submissions is dependent on the goodwill of patients. If the use of these submissions is not well documented, the ethics of HTA bodies asking for such submissions is questionable (Reference Vanstone, Abelson, Bidonde, Bond, Burgess and Canfield23).
In-person attendance at appraisal committee meetings as observers (Reference Hashem, Calnan and Brown22) and from patient advocates on appraisal committees (Reference Scott and Wale15) has clearly shown that patient input is discussed and that patient experiences and values are important. CADTH researchers were able to track insights contained in patient group submissions to the Common Drug Review before the assessment reports were developed through to the appraisal process and recommendations. They concluded that the points made were included in framing the assessment and in interpreting the evidence, and so were integrated throughout an HTA (Reference Berglas, Jutai, MacKean and Weeks24). On the other hand, in a study of patient experts' perspectives, the authors found that the committee processes could undermine the input from the patient experts, leading them to have a largely peripheral consultative role (Reference Hashem, Calnan and Brown22). The latter authors suggested that a change in culture of the appraisal committees may be needed, and that having patients present to provide their insights increased the level of complexity that committee members faced. Clinicians and HTA professionals can play a major role in excluding, reshaping, or admitting the more subjective experientially based input from patients and carers (Reference Hashem, Calnan and Brown22). An environment is needed where a diversity of views can be readily articulated for patient involvement to meet the objectives of an organization (Reference Gibson, Britten and Lynch25). Boothe (Reference Boothe10) was able to show that more dialogue is now possible with technical experts on the goals, strengths, and challenges with patient involvement, in Canada.
The HTA bodies involved in our study have patient and public involvement teams whose job it is to work with patient groups to support and coordinate patient input into HTAs. Since July 2017, SMC now invites a patient group representative from each submitting patient group to participate during the meeting, and is introducing training sessions for its committee members on working with patient advocates. SMC and NICE appraisal committee meetings are open to the public but with a closed session for decision making; and both produce a public summary document to explain the decisions of the appraisal committees. CADTH is looking at how feedback on the usefulness of the submissions can be given to patient groups. These actions and awareness of the present study may impact on the representation of patient input in the final committee discussion reports.
Limitations of our exploratory study therefore include the following. The appraisals that we studied were from May 2016 to August 2017, and it would be of value to investigate any changes since then. The HTA bodies involved in this study use similar online templates for collecting patient group submissions. These guide the content of the submissions. Our analysis of inferred reference to patient input could suggest that patient group submission content was frequently referred to in this way in the final documentation. This could be the result of how the report authors approached their task, resulting in the lack of visibility on where the content was from. NICE and SMC ask for patient submissions at the appraisal stage whereas CADTH asks for the input earlier to help inform researchers in developing the assessment protocols and throughout the assessment and appraisal processes. For CADTH-pERC, the patient input is incorporated into addressing “patient values.” We analyzed the final committee discussion reports summarizing the deliberations of the appraisal committees. The papers provided to the appraisal committees for their meetings were not part of the analysis. Finally, our investigation is an exploration of only two case studies. The pharmaceuticals were appraised using standard processes and the quality of patient input was not a criterion for selecting the case studies.
Impactful patient involvement in HTAs requires the confidence of patient groups that their input is an important part of the deliberations of appraisal committees, and that the groups can be seen as partners in the appraisal of new technologies. This study identified limited referral to the utilization of patient group templated submissions in the final appraisal committee discussion reports, with greater use of information from patient experts (NICE) and PACE (SMC). Our exploration of how HTA bodies described their use of patient input in the final recommendations and discussion summaries of HTA appraisals highlights that it is important to be able to facilitate improvements in current approaches to the reporting of patient input in HTAs, and to be able to readily identify where and how patient input has provided added value or impact.
We have assumed, by developing templates for patient groups to complete and summarizing this information in appraisal committee papers, that we know the aspects of patient experience that are important for HTA researchers and appraisal committees. Direct comparisons of what knowledgeable patient groups consider important, how they present their information in templates, and what the appraisal committees discuss are lacking in the literature. Our insights suggest that the most meaningful and efficient ways of presenting patient input need further investigation, as do the different types of patient input and the limitations placed on each type. For example, patient experts (NICE) may be in a position to provide illustrative stories and their own words but they are present to answer questions from the appraisal committee, in often intimidating circumstances (Reference Hashem, Calnan and Brown22). Management of real and perceived conflicts of interest of all stakeholders also needs consideration. The work of PACE (SMC) may demonstrate the potential added benefits of medical practitioners and patient advocates working together collaboratively, supported by experienced technical HTA staff, and is worthy of further investigation. A danger exists of added inefficiencies in regulation and reimbursement if the input of patient knowledge is restricted, including into the symptoms of a disease and presentation of clinical trial data on clinical outcomes. This is particularly so in lesser clinically known health conditions (Reference Falchetto26).
Open access
Health technology assessment (HTA) requires rigorous analytical methods with an emphasis on randomized controlled trials and appraisal of the evidence for benefits, harms, and costs by expert committees, but a focus on clinical outcomes alone may not be sufficient (Reference Drummond, Schwartz, Jonsson, Luce, Neumann and Siebert1). Important information relevant to the clinical use of technology includes patient-relevant health outcomes, access to healthcare services, patient preferences, quality of life, immediate and longer-term consequences (Reference Gagnon, Desmartis, Lepage-Savary, Gagnon, St-Pierre and Rhainds2;Reference Mamzer, Dubois and Saout3). This is in addition to burdens and costs for users of the technologies and overall patient and carer experiences. Patients can provide unique, experiential knowledge to give context to the clinical research, help to overcome uncertainties in clinical evidence, and inform the determination of the value of a technology or service (Reference Facey, Boivin, Gracia, Hansen, Lo Scalzo and Mossman4–Reference Roy, Ganache and Dagenais7).
Patients' views may differ from those of health professionals or researchers and cannot be assumed by other stakeholders (Reference Gagnon, Desmartis, Lepage-Savary, Gagnon, St-Pierre and Rhainds2;Reference Staley and Doherty8;Reference Dipankui, Gagnon, Desmartis, Legare, Piron and Gagnon9). Patient involvement has a key role in the legitimacy of HTAs such that it is important that HTA recommendations have been informed by patient concerns and perspectives (Reference Boothe10). The rights and value of patients participating in HTAs are becoming more widely supported (Reference Wale, Scott, Hofmann, Garner, Low and Sansom11). It is therefore important, for public understanding and knowledge, that the involvement process supports learning through informed conversations (Reference Staley and Barron12) and transparent reporting on HTAs (Reference Deane, Delbecque, Gorbenko, Hamoir, Anton Hoos and Nafria13).
Quality criteria have recently been published for patient engagement in drug development, calling for shared purpose and respect for the resources and energy required for providing patient input. The criteria include representativeness of stakeholders with agreed roles and responsibilities, the resources and capabilities to enable meaningful engagement, transparency in communication and documentation, and continuity and sustainability (Reference Deane, Delbecque, Gorbenko, Hamoir, Anton Hoos and Nafria13). The opportunity for learning from feedback is important for patient advocates and patient groups involved in HTA. Indeed, it is likely that many patient groups do not have a complete understanding of what information they might need to convey, what would be of most use in HTA deliberations, the form that needs to take, and which insights if any could impact the deliberations (Reference Boothe10). The need for learning experiences is often not acknowledged by those who are in a position to provide the feedback (Reference Mathie, Wythe, Munday, Millac, Rhodes and Roberts14).
Boothe (Reference Boothe10) identified that goals for patient involvement in HTA continue to be unclear, without a clear vision of what the involvement should achieve. A lack of alignment on thinking among the different players also exists, making it very difficult to meaningfully evaluate patient involvement. The author used surveys and interviews over a number of years to describe how the different players in HTA appraisals saw the goals of patient group involvement, and how thinking had changed over time, for the Ontario Committee to Evaluate Drugs and the pan-Canadian Common Drug Review (CADTH-CDR), now part of the Canadian Agency for Drugs and Technologies in Health (CADTH) (Reference Boothe10).
The HTAi Patient and Citizen Involvement in HTA Interest Group (PCIG) previously collated the experiences of international patient advocates in providing input to the appraisals of health technologies for HTA bodies. We identified a common lack of clearly-defined purpose and goals of patient input, limited feedback to advocates on the value and use of the patient input, and limited communication to the public of the impact of patient involvement (Reference Scott and Wale15). Only the patient advocates who were patient representatives on an HTA appraisal committee were able to refer to specific positive recommendations where important contributions were made through patient group submissions, for example, on quality of life and more adversely affected sub-populations (Reference Scott and Wale15). In recent years, we have identified patient advocate concerns about how patient involvement in HTA appraisals is reported on, leading to uncertainties about how the input was used. In this commentary, we explore this concern by looking at how three HTA bodies reported on their use of patient involvement in final appraisal committee recommendations and discussion summaries. Our aim was to elucidate how the HTA bodies described their use of patient input in the final recommendations and discussion summaries, and so to observe the situation and view it as unique without imposing any deductive structure upon it. Such insights into how patient input was utilized may help us better understand the goals of patient involvement and how these are being achieved by HTA bodies. We see the need to examine real-world evidence on the value of patient input as is currently being used. Without clear reporting on the use of patient group input patient groups may not have a clear understanding of what information they can best provide to inform deliberations. Providing this evidence may also lead to more open, meaningful discussions and work on the culture and practice around the utilization and reporting of patient input, to encourage learning and evaluation of methods used to gain patient input into HTAs.
We worked with three HTA bodies that have formal processes to collect patient group input through submission templates with carefully defined questions. An important difference among the processes used is that the National Institute for Health and Care Excellence (NICE) and Scottish Medicines Consortium (SMC) ask for the input during the appraisal stage whereas CADTH asks for the input during protocol development. NICE also has a process for involving patient groups in discussions to inform the development of the protocol. At appraisal, NICE invites patient experts generally nominated by patient groups to report on their experiences with the disease and its treatment, and to answer questions. SMC provides the opportunity for technology sponsors to request a Patient and Clinician Engagement meeting (PACE) before its appraisal of orphan or ultra-orphan and end of life medicines; a summary of the meeting is presented at the SMC committee meeting alongside the patient group submissions. We selected two different pharmaceutical interventions for two different medical conditions, as a single example could more easily be considered as an anomaly or specific to a particular health condition.
Methods for Information Gathering
For each of the three HTA bodies, a final committee discussion report summarizes the deliberations of the appraisal committee that lead to the final recommendation (Figure 1). By “final recommendation” we refer to the funding recommendation (CADTH), guidance (NICE), or advice (SMC) from the appraisal committees, together with the summary committee discussion report. SMC incorporates both into a “Detailed Advice Document”; and CADTH Canadian Drug Expert Committee (CADTH-CDEC) into “Recommendations and Reasons” (see Figure 1). As an exploratory study, we descriptively analyzed the documentation for explicit and inferred reference to the different types of patient input, particularly in describing the needs and concerns of patients and caregivers and what they value. Inferred contributions related particularly to the descriptions of the health condition, impact of the disease, and its treatment. Extracts of the text were entered manually into Excel by the first author and checked by the second author. Although the submission templates provide for input under specific headings, no attempt was made to go beyond these descriptors with a thematic qualitative analysis or to give weight to the input beyond what was recorded. We sought feedback from the other members of the HTAi PCIG working group when summarizing this process and their comments were diligently addressed.
Website presentation of the final recommendations and committee discussion reports, and patient-relevant submissions, for each HTA body: Canadian Agency for Drugs and Technologies in Health—Canadian Drug Expert Committee (CADTH-CDEC), Canadian Agency for Drugs and Technologies in Health—pan-Canadian Oncology Drug Review Expert Review Committee (CADTH-pERC), National Institute for Health and Care Excellence (NICE) and Scottish Medicines Consortium (SMC).
We considered two drug appraisals by each of the three HTA bodies (16–21), for treatment of a chronic disease (ustekinumab) and cancer (blinatumomab). Appraisals took place from May 2016 to August 2017. We summarized, for comparison with the publicly available appraisal committee discussion reports, the key points made by patient groups in their templated submissions: on how the condition affects patients' daily lives; experience with symptoms and severity of the condition; how patients are affected by current treatments, and how well they are managing; impact on caregivers; experience with or expectations of a new treatment in terms of benefits, advantages, and disadvantages compared to current treatments; impact on family and caregivers; whether the new treatment is innovative; and any other factors.
HTA Appraisal of a Chronic Disease Treatment
Ustekinumab is a drug for moderately to severely active Crohn's disease in adults whose previous treatment resulted in an inadequate response, a response that was not maintained, or where people could not tolerate or have contraindications to the treatment. Previous treatments were conventional therapy or a TNF-α inhibitor biologic.
Patient Group Templated Submission
The patient groups that provided patient input were: CADTH, Crohn's and Colitis Canada (CCC) and GI (Gastrointestinal) Society, Canada; NICE, Crohn's and Colitis UK; and SMC, Crohn's and Colitis UK. The patient groups stated their sources of information and the submissions were rich with patient quotes (Figure 2). The NICE and SMC submissions were provided by the same UK organization but were substantially different from each other.
Direct patient quotes from patient group templated submissions for moderately to severely active Crohn's disease.
Final Recommendations
For each HTA body, the final recommendations did not directly reference the patient input received, and the cost was a significant factor in the recommendations. We then looked at the reports of the appraisal committee deliberations for reference to patient input.
Final Committee Discussion Reports
For CADTH, the “Recommendations and Reasons” stated that the “Canadian Drug Expert Committee (CDEC) considered patient group submitted information prepared by the Common Drug Review (CDR) on outcomes and issues important to individuals living with Crohn's disease.” The clinical and economic reports provided to CDEC were outside this study remit (Figure 1) and we did not look at any use of the patient group submissions to interpret the clinical trial results and critique the economic model provided to the appraisal committee.
A summary of the patient group submission to NICE was given. Two patient experts presented to the NICE appraisal committee. The committee discussion report focused on the patient experts: “It heard from the patient experts that…”; “The patient experts recounted their experience of”; “both the patient experts had ustekinumab as part of a clinical trial after losing response to TNF-alpha inhibitors, being able to ‘…resume work and everyday activities as well as avoiding the need for surgery’.” One patient expert explained that ustekinumab had left him feeling “wonderfully normal again” while the other highlighted the immense improvement to her quality of life, enabling her to start a family. The patient experts emphasized that maintenance treatment with ustekinumab is a subcutaneous injection rather than an intravenous infusion, which is greatly valued by patients because it means they can take the treatment at home with no need for hospital visits. The key conclusions state: “The committee acknowledged that ustekinumab is a convenient and well-tolerated treatment that has considerably improved the quality of life of the patient experts.”
For SMC, the patient group templated submission was summarized as a bulleted list in a “Summary of patient and public involvement” section of the final appraisal committee discussion report.
Inferred Use of Patient Input
Looking at inferred use of patient group submissions, examples were: “The [NICE] committee understood that Crohn's disease follows an unpredictable, relapsing and remitting course with many debilitating symptoms”; “The [NICE] committee heard that patients fear loss of remission and exacerbations of the disease because of the major impact these have on quality of life… it is very important to have a range of treatment options ….”
HTA Appraisal of a Cancer Treatment
Blinatumomab was the drug under consideration for adults who had relapsed or were refractory to previous chemotherapy treatment.
Patient Group Templated Submission
The patient groups that provided patient input were: CADTH-pCODR, Leukemia & Lymphoma Society of Canada; NICE, Leukaemia Care; and SMC, Leukaemia Care.
The patient groups stated their sources of information. Three patients had direct experience of the new treatment in Canada. In England and Scotland, the patient group was unable to identify any patients with direct experience. The patient group submissions contained few quotes (Figure 3). The NICE and SMC submissions were from the same patient group and their content was similar. For NICE and SMC, the submissions emphasized the importance to patients of being able to have a treatment-free period and the option for some patients to be treated as outpatients, allowing patients to spend more time at home with their families. For CADTH, the patient group submission concentrated on troublesome symptoms and side effects for six people who had not taken the drug and on patient experience for the three who had received blinatumomab.
Direct quotes from patient group templated submissions for previously treated Philadelphia-chromosome-negative acute lymphoblastic leukemia and blinatumomab.
Final Recommendations
The CADTH pan-Canadian Oncology Drug Review (CADTH-pERC) and NICE final recommendations did not refer to specific patient input whilst the SMC Advice statement clearly referred to PACE: “This advice takes account of the views from a Patient and Clinician Engagement (PACE) meeting.”
Final Committee Discussion Reports
For CADTH, the “Summary of pERC Deliberations” states that “The Committee deliberated on the patient advocacy group input, which indicated that patients with ALL value disease control and the management of side effects related to current therapies for ALL. Given improvements in OS and maintenance of QoL, pERC agreed that blinatumomab aligns with patient values overall.”
For NICE, it is the perspective of the patient expert that is given with details about the physical and psychological stresses of failing chemotherapy, and the emotional strain on families and friends: “The committee heard from a patient expert that..”; “The committee heard from the patient expert that patients whose disease responds to blinatumomab can live a relatively normal life during treatment, with minimal side effects.” In some of the text, patient expert and clinical expert views were linked: “…heard from the patient expert and clinical experts that people with ….”
The committee concluded in their deliberations that the availability of an effective treatment that could be delivered primarily in the outpatient setting was hugely beneficial to patients and would have a major impact on their quality of life. A summary of the patient group submission was provided.
For SMC, the patient group submission was referred to and a detailed summary of the submission provided. It was stated that a patient group submission was received from Leukaemia Care, who provided representatives to attend PACE, and that points of the submission were covered under PACE. Detailed information was given on PACE deliberations in relation to “the relatively young age of patients (mid to late thirties) and the high symptom burden associated with this condition and how a diagnosis of relapse is devastating for patients and their families.” At the PACE meeting, “the deep response and high remission rates with blinatumomab, a greater chance of moving to allogeneic haematopoietic stem cell transplantation, improved quality of life and more time spent at home with the family” were noted. It is evident that the PACE feedback was very seriously considered in discussions and was integrated throughout the report.
Inferred Use of Patient Input
Patient input was inferred in the following examples from NICE: “The committee acknowledged that acute lymphoblastic leukaemia does not affect the patient in isolation, but also places emotional strain on their families and friends”; “Living with precursor B-cell acute lymphoblastic leukaemia, particularly when the condition has failed to respond to first line chemotherapy, can have a profound effect on a person's physical and psychological wellbeing.”
Discussion
Our exploration of the final recommendations and summary committee discussion reports for the appraisal of two drugs across three HTA bodies showed clear differences in how the types of patient input—patient group submissions, patient experts, and patient and clinician meetings—were referred to. One possible interpretation is that each type of patient input was considered differently by the appraisal committees. Differences could be caused, for example, by variations in how the different types of input were presented, who presented it, and at what stage of the meeting, and if and how the input resonated with expert members of the appraisal committees. Because no direct feedback was given by HTA agencies to patient groups, it would be very difficult for the patient groups to know how their input was received and how useful it was; which may align with a lack of clear expectations for patient submissions. On the other hand, it could be that report writers are not being asked to specifically capture the utilization of all types of patient input in the deliberations, and a different set of skills may be needed to do this, whilst maintaining the objectivity of the HTA process.
Overall, direct patient input through participation in meetings appeared to be more influential than written patient group submissions. Patient input was utilized by CADTH-pERC to inform alignment with patient values, with little further information on how “alignment with patient values” actually influenced the deliberative process. For NICE, the perspectives of patient experts were referred to in illustrating discussion points, with limited reference to the patient group submissions. For SMC, the PACE feedback was referred to and integrated throughout the summary discussion report. The collective views of patient advocates and clinicians and how they were presented to the appraisal committee would appear to have been highly valued by the committee. Indeed, patient involvement may work best when health professionals and patients enter into dialogue to share their respective expertise and learn from each other (Reference Staley and Barron12).
If experiential knowledge gained from the patient group submissions was being utilized in deliberations, this use may not always be clearly recorded. We did not observe any attempt to relate one form of patient involvement to another form; and any use of experiential patient knowledge to interpret the clinical trial findings was not evident in the reports, including for quality of life data. It has been suggested that better engagement of patients, users, caregivers, citizens throughout the assessment and appraisal process to seek convergence of scientific data on efficiency, safety, and acceptability with contextual and experiential data could be used to strengthen decision making (Reference Roy, Ganache and Dagenais7). For NICE, rich information from patient experts could help overcome the limitations of clinical trials where patient-reported outcomes, patient experience, or quality of life data were missing or the measures were too generic (Reference Hashem, Calnan and Brown22). A study of patient engagement in CADTH-pCODR found that submissions from patient advocacy groups were most impactful when they provided information that was not available, well documented, or easily recognized from other sources. This included information that was not collected in clinical trials, any clinical trade-offs, information on lived experience, and patient and caregiver priorities. The committee gained a better understanding of the realities of a disease and its practical burdens from patient input (Reference Rozmovits, Mai, Chambers and Chan6).
Staley and Barron (Reference Staley and Barron12) have identified the importance of mutual learning as an outcome of patient involvement. Quality interactions can lead to greater understanding and better decisions through respectful conversations with a two-way exchange of knowledge. Patient experts are present during discussions but they may not be charged with the ability, confidence, and opportunity to challenge committee members' assumptions and perspectives in sharing their experiential knowledge. Recording where patient input has an impact supports wider learning as it can fill gaps in the knowledge of other stakeholders and correct or modify assumptions (Reference Staley and Barron12). The overall uncertainty about the specific roles of the different types of patient involvement and how they work together is likely to result in difficulties in strengthening the value of patient involvement, and in informing changes in thinking (Reference Boothe10) as we move toward more person-centered health care. Completing patient group submissions is dependent on the goodwill of patients. If the use of these submissions is not well documented, the ethics of HTA bodies asking for such submissions is questionable (Reference Vanstone, Abelson, Bidonde, Bond, Burgess and Canfield23).
In-person attendance at appraisal committee meetings as observers (Reference Hashem, Calnan and Brown22) and from patient advocates on appraisal committees (Reference Scott and Wale15) has clearly shown that patient input is discussed and that patient experiences and values are important. CADTH researchers were able to track insights contained in patient group submissions to the Common Drug Review before the assessment reports were developed through to the appraisal process and recommendations. They concluded that the points made were included in framing the assessment and in interpreting the evidence, and so were integrated throughout an HTA (Reference Berglas, Jutai, MacKean and Weeks24). On the other hand, in a study of patient experts' perspectives, the authors found that the committee processes could undermine the input from the patient experts, leading them to have a largely peripheral consultative role (Reference Hashem, Calnan and Brown22). The latter authors suggested that a change in culture of the appraisal committees may be needed, and that having patients present to provide their insights increased the level of complexity that committee members faced. Clinicians and HTA professionals can play a major role in excluding, reshaping, or admitting the more subjective experientially based input from patients and carers (Reference Hashem, Calnan and Brown22). An environment is needed where a diversity of views can be readily articulated for patient involvement to meet the objectives of an organization (Reference Gibson, Britten and Lynch25). Boothe (Reference Boothe10) was able to show that more dialogue is now possible with technical experts on the goals, strengths, and challenges with patient involvement, in Canada.
The HTA bodies involved in our study have patient and public involvement teams whose job it is to work with patient groups to support and coordinate patient input into HTAs. Since July 2017, SMC now invites a patient group representative from each submitting patient group to participate during the meeting, and is introducing training sessions for its committee members on working with patient advocates. SMC and NICE appraisal committee meetings are open to the public but with a closed session for decision making; and both produce a public summary document to explain the decisions of the appraisal committees. CADTH is looking at how feedback on the usefulness of the submissions can be given to patient groups. These actions and awareness of the present study may impact on the representation of patient input in the final committee discussion reports.
Limitations of our exploratory study therefore include the following. The appraisals that we studied were from May 2016 to August 2017, and it would be of value to investigate any changes since then. The HTA bodies involved in this study use similar online templates for collecting patient group submissions. These guide the content of the submissions. Our analysis of inferred reference to patient input could suggest that patient group submission content was frequently referred to in this way in the final documentation. This could be the result of how the report authors approached their task, resulting in the lack of visibility on where the content was from. NICE and SMC ask for patient submissions at the appraisal stage whereas CADTH asks for the input earlier to help inform researchers in developing the assessment protocols and throughout the assessment and appraisal processes. For CADTH-pERC, the patient input is incorporated into addressing “patient values.” We analyzed the final committee discussion reports summarizing the deliberations of the appraisal committees. The papers provided to the appraisal committees for their meetings were not part of the analysis. Finally, our investigation is an exploration of only two case studies. The pharmaceuticals were appraised using standard processes and the quality of patient input was not a criterion for selecting the case studies.
Impactful patient involvement in HTAs requires the confidence of patient groups that their input is an important part of the deliberations of appraisal committees, and that the groups can be seen as partners in the appraisal of new technologies. This study identified limited referral to the utilization of patient group templated submissions in the final appraisal committee discussion reports, with greater use of information from patient experts (NICE) and PACE (SMC). Our exploration of how HTA bodies described their use of patient input in the final recommendations and discussion summaries of HTA appraisals highlights that it is important to be able to facilitate improvements in current approaches to the reporting of patient input in HTAs, and to be able to readily identify where and how patient input has provided added value or impact.
We have assumed, by developing templates for patient groups to complete and summarizing this information in appraisal committee papers, that we know the aspects of patient experience that are important for HTA researchers and appraisal committees. Direct comparisons of what knowledgeable patient groups consider important, how they present their information in templates, and what the appraisal committees discuss are lacking in the literature. Our insights suggest that the most meaningful and efficient ways of presenting patient input need further investigation, as do the different types of patient input and the limitations placed on each type. For example, patient experts (NICE) may be in a position to provide illustrative stories and their own words but they are present to answer questions from the appraisal committee, in often intimidating circumstances (Reference Hashem, Calnan and Brown22). Management of real and perceived conflicts of interest of all stakeholders also needs consideration. The work of PACE (SMC) may demonstrate the potential added benefits of medical practitioners and patient advocates working together collaboratively, supported by experienced technical HTA staff, and is worthy of further investigation. A danger exists of added inefficiencies in regulation and reimbursement if the input of patient knowledge is restricted, including into the symptoms of a disease and presentation of clinical trial data on clinical outcomes. This is particularly so in lesser clinically known health conditions (Reference Falchetto26).
Acknowledgments
We thank the members of the HTAi Patient and Citizen Involvement in HTA Interest Group Patient Involvement and Education Working Group for their invaluable contributions to this project. We thank the patient organizations that gave SMC permission to share their submissions with the authors (Crohn's and Colitis UK and Leukaemia CARE). Submissions are presented at SMC meetings at a time when the meetings are open to the public. The same patient organizations provided submissions to NICE and these are publicly available on the NICE webpages. This work was presented as part of a panel session at the HTAi 2018 Annual Meeting. The authors thank our peer reviewers for their very helpful comments.
Financial Support
This research was not supported by any specific grant from any funding agency, commercial or not-for-profit sector organization.
Ethics
This study did not directly involve patients and carers. Patient group submissions were available on the websites of CADTH and NICE. SMC sought permission from the patient groups before sharing their submissions with the authors. These groups were Crohn's and Colitis UK and Leukaemia CARE; which also provided submissions to NICE. The patient input was provided by registered patient groups and involved people selected to present the views of patients in their capacity as patient advocates.