Background
In 2020, an international task force put forth a new consensus definition of health technology assessment (HTA) (Reference O’Rourke, Oortwijn and Schuller1). HTA was defined as a multidisciplinary process using explicit methods to determine the value of a health technology to inform healthcare decision making, with a goal of promoting an equitable, efficient, and high-quality health system (Reference O’Rourke, Oortwijn and Schuller1). Traditionally HTA has focused on the clinical and cost-effectiveness of the health technologies being evaluated (Reference O’Rourke, Oortwijn and Schuller1). However, more recently the use of patient-based evidence in HTA, defined as “knowledge that originates directly from patients about their experiences of health, quality of life, health care, health services, and health research” (Reference Staniszewska and Soderholm Werko2), has been gaining attention. There are many features of patient-based evidence that make it well-suited to contributing to HTA. These types of data can speak to issues like which aspects of treatment value are important to consider from the patient’s perspective and help ensure the relevance of the decision outcomes to those who will use the novel interventions under consideration. In addition, the inclusion of patient-based evidence reflects a commitment to partnership with an active role for patients within HTA (Reference Staniszewska and Soderholm Werko2). Patient-based evidence may be derived through quantitative assessments such as surveys, and also through qualitative research (Reference Facey, Hansen and Single3).
Qualitative research describes a set of methodologies that aim to grasp phenomena in a holistic way, to understand the meaning behind these within their own context, and to generate theories to explain observed trends (Reference Charmaz4;Reference Strauss and Corbin5). As such, these methods are well-suited to obtain rich, in-depth information about patients’ experiences, needs, preferences, and attitudes about their care and health (Reference Coast6). Within HTA, they can provide important complementary evidence to the findings from quantitative studies that estimate clinical and cost-effectiveness or health-related quality-of-life (HRQoL) impact (Reference Facey, Hansen and Single3). They can also expand on ad-hoc and anecdotal evidence from individual stakeholders often collected as part of the process of patient input into HTA (Reference Facey, Hansen and Single3).
In recognition of their potential value for informing healthcare decision making, researchers are noting opportunities for expanding the contribution of qualitative methods within health economics and HTA (Reference Coast6–Reference Booth8). At the same time, frameworks to expand patient engagement within HTA – including from agencies such as the National Institute for Health and Care Excellence (NICE) and Canadian Agency for Drugs and Technologies in Health (CADTH) – are being developed (Reference Facey, Hansen and Single3;9;10). However, despite acknowledgment of the potential role for qualitative methods, how widely these methods are presently used to inform patient-based evidence within the HTA submission process is unclear. The objective of this review was to characterize the contemporary use of qualitative data provided as patient-based evidence within HTA submissions, including methods employed, quality of data generated, and the objectives and topics described.
Methods
This review included all submissions to the Technology Appraisals or Highly Specialized Technologies programs at NICE, and to the Common Drug Review (CDR) or Pan-Canadian Oncology Drug Review (PCODR) at CADTH, with recommendations issued between 1 October 2019 and 30 September 2021. At the time of the review, CADTH operated two pan-Canadian single-drug technology assessment processes, with pCODR specifically assessing oncology drugs and CDR assessing all other drugs (11;12). The NICE programs selected for inclusion represent the subset of NICE programs also focused on appraising medicine-based health technologies. The search was implemented within the NICE (https://www.nice.org.uk/guidance/published?ndt=Guidance&ndt=Quality%20standard) and CADTH (https://www.cadth.ca/reimbursement-review-reports) websites on 30 September 2021. The two-year study period was selected to focus on the most contemporary evidence available at the time of initiating the review.
From each identified submission, we extracted the following information: the name of the product, indication, target diseases and therapeutic area, and the outcome of the submission (i.e., whether the product was recommended for reimbursement or rejected). We also extracted details of any patient-based research initiatives described within individual submissions, including who provided input (e.g., patients, caregivers), the topics covered and objectives of the research, methods used for data collection and analysis, and whether the submitted data had been published in a peer-reviewed journal.
Given the wide variation in methods described for collecting patient-based evidence, a taxonomy was developed to help categorize approaches in a meaningful way. Submissions were therefore classified into mutually exclusive categories according to whether they
-
1) provided an explicit description of both qualitative data collection and analysis methods (either stand-alone or in the context of mixed-methods studies);
-
2) included information only on qualitative data collection methods, without a description of how data were analyzed (either stand-alone or in the context of mixed-methods studies);
-
3) described input from a small number of patients (≤5 participants) directly in an ad hoc fashion (either stand-alone or in the context of mixed-methods studies);
-
4) reported quantitative surveys that incorporated (qualitative) free-text comments;
-
5) reported other quantitative data collection methods (without accompanying qualitative methods or use of free-text comments);
-
6) did not directly include patient data (either from quantitative studies, qualitative studies, or based on direct patient feedback); or
-
7) provided insufficient information to determine the methods used.
Drawing on Germeni and Szabo’s proposed framework for qualitative research in HTA (Reference Germeni and Szabo13), we subsequently classified submissions reporting qualitative data collection methods (categories 1 and 2 above) in terms of the purpose that the qualitative research was aiming to serve. Specifically, these purposes included: (i) assessing acceptability and subjective value; (ii) understanding perspectives and providing context; (iii) laying the groundwork for subsequent quantitative exercises; (iv) contributing to economic model development; and (v) reaching groups other methods cannot reach (Figure 1). For subjective value, we considered whether perceptions of safety and effectiveness were comparative or provided for one treatment in isolation.
Germeni and Szabo’s proposed framework for qualitative research in HTA (Reference Germeni and Szabo13).
The CASP checklist for qualitative research (14) was used to appraise the quality of methods and validity of findings, from submissions providing an explicit description of both qualitative data collection and qualitative data analysis methods when they generated patient-based evidence (category 1). The CASP checklist systematically rates whether specific elements in qualitative studies are present (“Yes”), absent (“No”), or unclear (“Can’t tell”), using responses to ten questions that address issues around the validity of results, actual results reported, and helpfulness of results locally. Based on ratings assigned to each submission, the mean number of “yes” responses per submission was tabulated, as was the proportion of submissions with more than 7 “yes” responses.
Targeted literature searches of EMBASE, MEDLINE, and Google Scholar were conducted to ascertain whether the data resulting from initiatives using well-established qualitative methods were published in the peer-reviewed literature. Searches were conducted using keywords from the research topic (e.g., “asthma” and “burden”) and methods (e.g., “qualitative” or “interview”), as well as any details provided regarding the study team who conducted the research. If applicable, grey literature-based report titles were also used to guide these searches.
Results
We identified a total of 231 submissions for which the submission processes were complete and recommendations had been issued between 1 October 2019 and 30 September 2021: 107 (46.3 percent) from NICE and 124 (53.7 percent) from CADTH.
Characteristics of submissions
The most frequent therapeutic areas covered in NICE submissions were oncology (n = 56; 52.3 percent), rare diseases (n = 42, 39.2 percent), rheumatology (n = 7; 6.5 percent), and metabolic diseases (n=7; 6.5 percent). The most frequent therapeutic areas covered in CADTH submissions were oncology (n = 58; 46.8 percent), rare diseases (n = 50, 40.3 percent), and neurology (n = 9; 7.3 percent). For 8 (7.5 percent) NICE submissions and 21 (16.9 percent) CADTH submissions, the outcome of the review process was a decision to not recommend reimbursement. The most frequent reason for a lack of recommendation for reimbursement was that the economic analyses did not demonstrate cost-effectiveness.
Use of qualitative methods in submissions
The methods used to collect patient-based evidence in the reviewed HTA submissions are presented in Figure 2. Of the 107 NICE and 124 CADTH submissions, 83 (NICE) and 119 (CADTH) included patient-based evidence in some form (derived from quantitative studies, qualitative studies, or through direct patient feedback). However, only 14 NICE and 24 CADTH submissions provided an explicit description of systematic qualitative data collection initiatives. One-to-one interviews were the most common method used (reported in 8 NICE and 22 CADTH submissions), followed by focus groups (reported in 6 NICE and 11 CADTH submissions). The use of qualitative data analysis methods was reported even less frequently than the use of data collection methods, observed in 6 NICE and 9 CADTH submissions. All submissions that included details of qualitative data analysis, used thematic analysis. A large number of submissions (31 NICE and 52 CADTH) presented data collected exclusively by survey-based methods involving a combination of closed- and open-ended questions with free text fields. Only one NICE and two CADTH submissions reported exclusive use of qualitative data collection methods (i.e., surveys with free-text fields were not employed).
Categorization of HTA submissions based on methods used to collect patient-based evidence*.
Objectives of qualitative research initiatives
As shown in Table 1, in terms of the objectives of each exercise, the most frequent focus was to understand perspectives and provide context (observed in 100 percent of both CADTH and NICE submissions involving qualitative data collection or analysis). The Asthma Canada advocacy group, for instance, used a mixed-methods approach involving qualitative interviews and a quantitative survey to document the experiences of Canadians living with severe asthma, and the attendant social, financial, and emotional implications of the diagnosis (15). The next most common objective was to understand acceptability and subjective value (in 83.3 percent of CADTH and 64.7 percent of NICE submissions involving qualitative methods). An example addressing both of these objectives is provided by a qualitative study involving one-on-one interviews with Canadians with hemophilia A and their caregivers that aimed to understand the lived experience but also to solicit feedback on patient perceptions of the acceptability and merit of current and future treatments (16). In submissions with qualitative initiatives that described treatment acceptability and subjective value, while patients frequently described perceptions of the clinical effectiveness or HRQoL implications of treatment, comments on comparative effectiveness, safety, or HRQoL impact were rare.
Other broad objectives (i.e., laying the groundwork for quantitative exercises, contributing to economic models or reaching groups other methods cannot reach) were infrequent focuses of the qualitative research initiatives reviewed. An example of a qualitative initiative contributing to an economic model comes from the submission for Luxturna (voretigene neparvovec; Novartis) where focus groups were used to understand the costs of blindness, including indirect costs, from the patient perspective; and key insights on the types of costs to consider within the economic analyses were taken from the findings of those focus groups (17). The percentage of submissions focusing on rare diseases was similar or slightly higher among the subset of appraisals involving qualitative data collection or analysis methods (50.0 percent for NICE and 41.7 percent for CADTH) compared with the overall set of appraisals reviewed (39.2 percent for NICE and 39.5 percent for CADTH) only one appraisal specifically called out disease rarity to support their use of qualitative methods. That appraisal that included a Canadian interview-based study in viral keratoconjunctivitis reported that qualitative methods were used because of the small sub-population indicated and to understand heterogeneous patient experiences (18).
Quality appraisal and publication of qualitative research
The quality of methods and validity of findings of the initiatives from the 6 NICE and 9 CADTH submissions describing both qualitative data collection and analysis methods were evaluated using the CASP checklist. On average, methodologic descriptions were more comprehensive in CADTH submissions compared with those submitted to NICE; nonetheless, most descriptions of the methodology and results provided within the submissions lacked adequate reporting of one or more key study design elements (Figure 3). The mean number of yesses on the CASP checklist for the 6 NICE submissions using qualitative analysis was 7.0. and 33 percent of submissions had >7 yes responses. The mean number of yesses on the CASP checklist for the 9 CADTH submissions using qualitative analysis was 7.8, and 78 percent of submissions had >7 yes responses. Aspects of the CASP checklist where submissions were most often deficient (i.e., rated “no” or “unclear”) included adequately considering the relationship between researcher and participants, considering ethical issues, and providing sufficiently rigorous data analysis.
Percentage of NICE and CADTH submissions meeting CASP checklist quality assessment criteria.
Publication of qualitative studies was infrequent. Two manuscripts and one poster were identified of studies related to CADTH submissions, and one manuscript and two posters were identified related to NICE submissions.
Discussion
This review sought to summarize the contemporary use of qualitative patient-based evidence contributing as supportive evidence within the HTA process. Within the submissions reviewed, the use of systematic qualitative data collection methods was infrequent, and the use of systematic methods for both collection and analysis of qualitative data was even rarer. Furthermore, the description of the studies to generate qualitative patient-based evidence presented within the submissions was often inadequate. Many of the qualitative data collection exercises lacked explicit descriptions of the methods informing data analysis, or when included, the terms used to describe analysis methods were often used imprecisely (Reference Kiger and Varpio19). Thematic analysis was exclusively used, and no other approaches (such as framework analysis or interpretative phenomenological analysis) were identified. Expanding the types of qualitative approaches used may serve to broaden the potential applicability of these methods within HTA.
This review also documents the purposes for which qualitative methods are presently being used in providing patient-based evidence in this aspect of the HTA process. Given the wide acknowledgment of the usefulness of qualitative methods for understanding the meaning individuals attach to experiences (Reference Charmaz4;Reference Strauss and Corbin5), it is not surprising that these methods were widely applied to understand perspectives and provide context from patients or caregivers. Gathering in-depth data on acceptability and potential uptake challenges from treatment-experienced individuals was also a common objective. However, while the patient-perceived clinical or HRQoL treatment impact was frequently discussed, understanding how a patient considers the new versus an existing treatment (i.e., patient-based evidence of comparative effectiveness) was not regularly described. Only a small number of initiatives were identified that laid the groundwork for quantitative exercises or informed the development of economic models (17;20), despite these being commonly cited health economic applications of qualitative research (Reference Coast6). Only one study particularly called out the value of qualitative methods for reaching populations that other methods cannot (18). This is in spite of qualitative methods being well-suited to rare disease research, where small patient populations can make the use of quantitative methods more challenging (Reference Germeni, Vallini, Bianchetti and Schulz21). It is particularly surprising given that approximately 40 percent of submissions included in this review were for rare disease products. At present, how important decision makers find patient-based research initiatives focusing on these objectives is unclear. Formalized guidance that points researchers and patient groups to the topics most useful to investigate using qualitative methods would be helpful for ensuring patient-generated evidence is useful to complement the ongoing quantitative research for HTA. While HTA stakeholder feedback is being sought to help define, for example, how quantitative preference studies can inform HTA (Reference van Overbeeke, Forrester, Simoens and Huys22), similar guidance for qualitative data is lacking.
The qualitative initiatives submitted as supportive evidence for HTA often scored low on the CASP checklist. Aspects that were particularly problematic included consideration of the relationship between researcher and participants, discussion of ethical issues, and methods for qualitative analysis. To understand how these limitations to reporting identified here compared to qualitative work done in other contexts, we juxtaposed them with some recently published systematic reviews of qualitative evidence (Reference Butterworth, Wood and Rowe23–Reference Schober and Abrahamsen25). In general, summary estimates of CASP scores presented within published qualitative evidence syntheses tended to be higher than those calculated for the initiatives within submissions to HTA. Most qualitative evidence in the published reviews scored highly for discussion of ethical issues. However, consistent with our research, two areas where published qualitative studies also tended to be rated as deficient were in providing adequate descriptions of methods of analysis, and consideration of the relationship between the researcher and participant. It is worth noting that the methods of the studies used to generate patient-based evidence for HTA tended to be described in a more expanded fashion in the few identified publications (Reference Wiley, Khoury and Snihur26) versus what was presented in the submissions themselves. This potentially suggests that these methodological issues may in fact have been considered by the research team but were not fully documented within the HTA submission. It also highlights an opportunity for patient groups and manufacturers to clarify the reporting of their qualitative work through reference to the CASP or other available checklists (14), prior to submitting these data for review.
An additional method by which the perceived credibility of patient-based research – both quantitative and qualitative – can be assessed is through peer-reviewed publication (Reference Facey, Hansen and Single3). However, publication of the reviewed research submitted to NICE and CADTH was uncommon. We recognize that publication of research findings is a time-consuming process, and the timelines for this may not align with those for data submission to healthcare decision makers. It is possible that publication of some of these research projects is pending, or that the findings of the research were published but not identified by the study team in our targeted search. However, it is also important to acknowledge the documented potential barriers to publishing qualitative findings, including the systematic favoring of more “striking” research findings (e.g., time-lag bias), or reviewers’ reported lack of confidence in the quality or description of qualitative methods (Reference Toews, Booth and Berg27;Reference Toews, Glenton and Lewin28). Despite these challenges, the rigors of having undergone the peer review process, in combination with the use of qualitative research reporting checklists or design-specific reporting frameworks, could help refine methodologic descriptions that would also enhance the quality of qualitative research submitted to HTA.
Strengths of the review include the use of a well-accepted and commonly-used checklist to assess the quality of the reporting of included studies, and the comprehensive approach to reviewing and characterizing a large sample of submissions to two leading HTA agencies with a strong focus on patient engagement. In addition to the logistic convenience of their reviews being published in English, we selected NICE and CADTH because both HTAs focus on patient input into aspects of their processes. However, given that these two agencies are very explicit in how they assess the economic value of interventions, it is not surprising that there is still a large focus on the quantitative aspects required for informing cost-effectiveness analyses. How these findings reflect the situation with HTA agencies with less of a focus on economic aspects (like the Institute for Quality and Efficiency in Healthcare (IQWIG) in Germany, for example) is unknown. Similarly, because processes for patient input and engagement are HTA agency-specific, how the findings of this review would extend to HTA agencies outside of Canada, England, and Wales is unclear. Other limitations include that the syntheses presented in this review are based on data presented within the HTA submission. This may not adequately reflect the level of rigor within the studies themselves if described in a different context (e.g., within manuscripts that had undergone the peer review process). The ratings of the research initiatives informing the HTA submissions were assigned by a single reviewer and these aspects may be interpreted differently by other reviewers. There was variability in the level of description and data provided across submissions; and details provided must be considered in the context of differing requirements for CADTH and NICE submissions with respect to patient evidence. Finally, it is important to acknowledge the potential impact of COVID-19 on the findings of this review, as part of the review period (but not necessarily individual study data collection periods) coincided with the timing of the pandemic. While the pandemic certainly created challenges to research in general, it also offered some advantages to qualitative research methods (use of teleconferencing tools, easier and more widespread participant engagement (Reference Cornejo, Bustamante, Del Rio, De Toro and Latorre29)). These changes in methodology are persisting even as the pandemic abates. An interesting future direction would be to conduct a follow-up review to assess how the extent and quality of qualitative research contributing to HTA changes over the coming years.
Conclusions
Both quantitative and qualitative methods may be used to provide patient-based evidence for HTA (Reference Facey, Hansen and Single3), and the choice of approach and method should be dictated by the research question and needs of the stakeholders, rather than philosophical or ideological grounds (Reference Murphy, Dingwall, Greatbatch, Parker and Watson7). In addition to the need for quantitative evidence clearly outlined (30), NICE identifies and acknowledges the importance of qualitative evidence for informing HTA (Reference Booth8). Similarly, CADTH is a leader with respect to the level of patient engagement and input into various parts of their HTA processes, and they provide evidence on their website of situations where patient evidence helped shape HTA decisions (9). However, at present, there is little direction from decision makers about the types of qualitative evidence most useful to submit, or how these data should be synthesized and communicated (Reference Booth8;9).
In the 107 NICE and 124 CADTH HTA submissions reviewed within the present study, although inclusion of patient-based evidence was common (occurring in 83 NICE and 119 CADTH submissions), use of formal qualitative methods of collection was infrequent, described in 14 NICE and 24 CADTH submissions. Use of formal methods of analysis was even rarer, occurring in only 6 NICE and 9 CADTH submissions. When these were included, reporting tended to be brief and/or inconsistent.
While interest in providing patient-based evidence for HTA is increasing, we feel that the focus should be on collecting and analyzing these data in a systematic and rigorous way to help ensure their usefulness and credibility. As well, promoting methodologically-sound qualitative research should be done in the same way as quantitative research. Certainly, publication of findings – which is viewed as a key component in generating patient-derived evidence (Reference Facey, Hansen and Single3) – will be important to furthering this goal; and also the use of well-accepted and commonly-used checklists to aid in study design, transparency, and reporting. Additionally, we would suggest focusing the research on topics that quantitative methods are less well-suited to address. The in-depth insight that qualitative methods can provide can be used to help illuminate how patients’ experiences, attitudes, and preferences (Reference Coast6) will affect their adoption of the health technologies under consideration. In our view, such topics could include using qualitative methods to inform the development of quantitative patient preference studies, and understanding comparative safety and efficacy from the perspective of the patients who have experience with the health technology. In addition, qualitative methods can aid in demonstrating the patient relevance of specific parameter inputs considered within an economic model for that new technology. We would also encourage researchers to make use of the breadth of available qualitative methods – considering also ethnography, or grounded theory for example – to extend on research conducted using thematic analysis. At the same time, a better understanding of the influence of qualitative research on decision making should be investigated, potentially through observations of deliberative processes. To complement investigations of the deliberative process, research to understand HTA stakeholder preferences around qualitative data, and where they see key opportunities would be helpful; as has been previously performed for quantitative preference studies for example (Reference van Overbeeke, Forrester, Simoens and Huys22;Reference Huls, Whichello, van Exel, Uyl-de Groot and de Bekker-Grob31), would be helpful. Going forward, guidance from decision makers around key topics or areas for investigation may be helpful for ensuring that the patient voice is clearly heard, while the results of well-conducted qualitative studies fill specific gaps in knowledge to inform HTA.
Competing interest
The authors declare none.
Open access
