Background: The Changing Context of HTA

The process of deciding which health technologies to adopt has become increasingly important in how countries allocate limited resources for patient health benefit. As the World Health Organization states, countries face complex choices in deciding how to direct their finite health budgets to meet the priority health needs of their populations (1;2). Such decisions require a range of evidence to ensure that they are complete and consider all relevant aspects of a decision, including information about social, ethical, and quality-of-life issues. In this time of COVID-19, with the rush to rapid evidence generation, it is even more important that patient-based evidence is considered within health technology assessment (HTA) and its central role recognized. HTA has been defined as

More recently, a new definition has been published, which highlights the importance of determining value, which could theoretically include the patient perspective.

Of importance is the additional note that highlights the potential for patients to influence the dimensions and the assessment of value.

As a process of decision making, HTA sits within a wider context of global health care and research, underpinned by the cornerstone of high-quality evidence. Countries are at different points in the development of their HTA systems and experience a range of pressures. In a Western context, we observe significant financial and philosophical effort placed on creating systems and infrastructures that focus on the appraisal of high-quality evidence for HTA. There has been a focus on producing, synthesizing, and disseminating primarily quantitative clinical and economic research evidence in HTA (5;6), mainly focusing on evidence of clinical and cost-effectiveness, particularly in countries with long histories of HTA development (Reference Stephens, Handke and Doshi7). Although this has produced very valuable knowledge for HTA, the focus on clinical and economic forms of data in initiatives such as rapid HTA has contributed to the relative neglect of more patient-focused forms of knowledge and evidence. Ultimately, who should decide what forms of knowledge are most appropriate in rapid HTA? In this paper, we argue that clinical, economic, and patient-based evidence are all forms of knowledge vital for HTA. However, there are indications of a dilution of effort when including patient-based evidence. For example, when conducting a rapid HTA, according to the EUnetHTA Core Model, there is no need to fill in the domain of “Patients and Social aspects,” which is a requirement for a full HTA (8). When these lesser demands are imposed on a rapid HTA alongside the trend for more pragmatic thinking in policy making, there is a risk of creating an HTA process that omits areas of real importance to patients and could be judged as failing in its original intention.

The utility of clinical and economic forms of evidence, on which HTA predominantly draws, can be produced out of the context of people's lives, and so could miss important societal context and nuance. Sometimes, this can relate to funder or agency stipulations that narrow inclusion criteria for a review, reducing the potential for developing a rich understanding of the complete impact of a technology.

Health trends in Western and in lesser developed countries have also created ripples, shifts, and changing demands in relation to the type and content of evidence needed for HTA. These changes include aging populations, with a move away from acute and infectious diseases to chronic multimorbidities with patients experiencing continued and frequent contact with services and interventions (2). Although infectious diseases still dominate in terms of the causes of death in developing countries, as the economies of these countries grow, noncommunicable diseases will become more prevalent, joining Western countries in such health challenges. This will be due largely to the adoption of Western lifestyles and their accompanying risk factors—smoking, high-fat diet, obesity, and a lack of exercise (9). More recently the COVID-19 pandemic has created additional challenges for health systems (10). Such trends mean patients have ongoing and frequent contact with health interventions and services, experiencing these services in the context of their wider lives. These changes introduce the potential for a wider set of patient-important factors that should be considered during an appraisal of a health technology, beyond clinical and economic factors.

The related field of patient-reported outcomes has developed in response to the acknowledgment that patient perspectives of health status are important and make an important contribution to HTA. However, patients are rarely involved as active collaborators in the development of patient-reported outcomes (PROMs) and, so, uncertainty exists as to which outcomes are patient important rather than researcher important (Reference Haywood, Brett, Salek, Marlett, Penman and Shklarov11;Reference Staniszewska, Haywood, Brett and Tutton12).

More recently, there has been a focus on the coproduction of knowledge, which has the potential to change the power dynamic of research (Reference Hickey, Brearley, Coldham, Denegri, Green and Staniszewska13) and ultimately of future HTA. This trend in partnership reflects the greater involvement of patients and the public in HTA and health research, as part of a broader democratization of societal systems, with people and communities collaborating with professionals to generate evidence of benefit to them (Reference Crowe, Fenton, Hall, Cowan and Chalmers14;Reference Boyle and Harris15). We have seen the development of instrumental and substantive ethical rationales that support the involvement of patients in HTA (Reference Sandman, Bond, Hoffman, Facey, Ploug Hansen and Single16).

This strengthening of the patient role in the generation of evidence and the broader societal transformations provide a changing context for HTA, with the potential to create new types of questions. These questions may be more concerned with issues of how patients live with a condition, their experiences of a condition, an intervention or a technology, their views on the effectiveness of a technology in relation to outcomes of importance to them, and whether a technology is acceptable and relevant to them within the context of their lives. With these changes in the nature of questions about an intervention, device, medicine, or technology, there has been a developing awareness of the need for research designs to respond more intuitively, in order to provide the most appropriate evidence for HTA. One example of such a change in research design to accommodate a growing recognition of the complexity of evidence is illustrated by the Medical Research Council (MRC) complex interventions framework (17). Complex interventions are defined as interventions with several interacting components; they “present a number of special problems for evaluators, in addition to the practical and methodological difficulties that any successful evaluation must overcome.” Many of the additional limitations or biases relate to the difficulty of standardizing the design and delivery of the interventions, their sensitivity to features of the local context, the organizational and logistical difficulty of applying experimental methods to service or policy change, and the length and complexity of the causal chains linking intervention with outcome. Despite its developing insight into complex interventions, and the significant potential to cocreate knowledge, the MRC Complex Interventions Guidance is still predicated on clinical and economic forms of knowledge and lacks active collaborative involvement of patients and the public in its development (17).

In considering patient-based evidence, we define the patient as an individual with a disease or disorder who is using some aspect of the healthcare system because of this disease or disorder rather than a community member who holds a public interest but has no commercial, personal, or professional interest in the HTA process (Reference Street, Stafinski, Lopes and Menon18).

Evidence

With the changing context of HTA, it is timely to review the nature of evidence required as the cornerstone for HTA (Reference Merlin, Weston and Tooher19). Evidence as a concept in health has been defined in a range of ways. The Concise Oxford English Dictionary (2016) (20) gives a number of definitions for evidence, including the following examples of relevance to patient evidence.

1. (1) Of things: To serve as evidence for; to attest, prove.

2. (2) Of persons: To support by one's testimony, attest (a fact or statement).

3. (3) To establish by evidence; to make evident, demonstrate, prove.

4. (4) To give evidence, appear as a witness.

Although dictionary definitions focus on legalistic aspects, the concept of evidence in health care has focused on notions of proof and rationality (Reference Rycroft-Malone, Seers, Titchen, Harvey, Kitson and McCormack21). A key factor underpinning evidence is the need for validation and verification to ensure high quality through scrutiny (Reference Davies, Nutley, Smith, Davies, Nutley and Smith22). Although we now acknowledge the importance of qualitative forms of evidence, historically, evidence was assumed to be research based and quantitative (Reference Sackett, Richardson, Rosenberg and Haynes23). This approach has inevitably impacted on the types of research questions that lend themselves to a randomized controlled design such as those concerned with clinical efficacy judged by clinical parameters. This focus on quantifiable research-based evidence informed the development of evidence-based medicine, and, thus, evidence-based practice, defined by Sackett and colleagues (Reference Sackett, Rosenberg and Gray24) as “the conscientious, explicit and judicious use of current best evidence about the care of individual patients.” Sackett and colleagues (Reference Sackett, Rosenberg and Gray24) stated that the practice of evidence-based medicine means integrating individual clinical expertise (which includes a consideration of patients’ preferences) with the best available external clinical evidence from systematic research. By best available clinical evidence, they referred to clinically relevant research, often from the basic sciences of medicine, but “especially from patient-centered clinical research,” although the exact nature of this form of evidence was not specifically defined. Although we support Sackett and colleagues' (Reference Sackett, Rosenberg and Gray24) initial emphasis, we are aware that they did not consider how patients’ values, perspectives, or experiences could be formally integrated into the clinical or economic evidence base. Rather they suggested that patient perspectives should be considered in a clinical encounter, almost relegating this to a form of social interaction rather than evidence. This perspective is interesting, particularly as the etymology of the word “evidence” is rooted in the concept of experience, relating to what is manifest and obvious (Reference Upshur, Morse, Swanson and Kuzel25). The broad field of qualitative research has evolved somewhat separately but is now increasingly embedded within HTA. Qualitative synthesis is now routinely conducted at some HTA agencies (Reference Werkö, Sweden, Facey, Ploug Hansen and Single26–Reference Majid and Weeks28) and for developing guidelines (29). However, the evolving context of HTA, in relation to epidemiological and societal changes, creates fundamental challenges for the paradigm of science and evidence that underpins HTA. Interestingly, the “spaces” in which to have such macro debates about the nature of evidence for HTA are relatively rare, making fundamental paradigm change difficult to achieve. It is important that these spaces are accessible to all who wish to participate. We argue that the time has come to recognize our need to reconceptualize evidence and consider the potential for new ways of thinking that extend and enrich our evidence for HTA, particularly in these times of COVID-19, where the need for patient-based evidence has been highlighted in a number of ways, including the need for experiential forms of evidence to create core-outcome sets (30). We argue that this further reinforces the need to embed patient-based evidence into HTA.

Patient-Based Evidence

Patient-based evidence was first described in a paper that reported the development of a patient evidence base in chronic fatigue syndrome, focusing on the experiences of the condition and health services as a key source of evidence for practice (Reference Staniszewska, Crow, Badenoch, Edwards, Savage and Norman31). Initial thoughts on its nature were presented in a chapter as part of a book focused on patient involvement in HTA (Reference Staniszewska, Werkö, Facey, Ploug Hansen and Single32). In this paper, we take this initial conceptualization forward and place it in a broader context. We propose that patient-based evidence represents evidence or knowledge that originates directly from patients about their experiences of health, quality of life, health care, health services, and health research. Conceptually, it could include not only experiences, but also perceptions, needs, or attitudes about their care and health. It can contain both cognitive and affective elements, reflecting patient narratives that can describe an event or a situation (cognitive) and how they felt about it (affective). It can include both the content of care (e.g., dimensions of experience or health status) and the process of receiving care or the process of health changes. Patient-based evidence conceptually underpinned the development of the Warwick Patient Experience Framework (WaPEF), which identified seven key dimensions of patient experience and directly informed the NICE Patient Experience Guidance and Quality Standard (Reference Staniszewska, Boardman, Gunn, Roberts, Clay and Seers33). These included the patient as an active participant, the responsiveness of services as an individualized approach, continuity of care and relationships, lived experience, information, communication, and support.

In the spirit of conceptual coproduction, each dimension of WaPEF (Reference Staniszewska, Boardman, Gunn, Roberts, Clay and Seers33) was reviewed by the NICE Patient Experience Guideline Development Group which included six patients. This demonstrates the future need for patient-based evidence to be developed using a coproduction approach, reflecting a more explicit partnership model, with patients involved at key points of decision making to ensure that the concept fits the reality of peoples’ lives. Coproduced patient-based evidence could become a key quality criteria in HTA, which would reflect a tangible desire to truly place the patient at the heart of methodological advancement.

Probably the best developed type of patient-based evidence, conceptually and methodologically, is the field of quality of life, or patient-reported outcomes, which has made considerable conceptual and methodological progress in attempting to capture health-related forms of patient-based evidence (Reference Haywood, Brett, Salek, Marlett, Penman and Shklarov11;Reference Staniszewska, Haywood, Brett and Tutton12). However, patient-reported outcome measures may still capture a researcher's construct of health, rather than a patient's, because they have often been constructed without patients as collaborators (Reference Staniszewska, Haywood, Brett and Tutton12). HTA agencies have also made some progress in including patient evidence in their assessments. For example, in the Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU), there is often a chapter on patient-based evidence in an HTA report and there is a chapter specifically on evaluation and synthesis of studies using qualitative methods of analysis in the SBU Handbook of Assessment of methods in health care and social services (34). However, at an international level, this is not a consistent trend. The implications of this for HTA are important because it represents an omission and, thus, a distortion of the evidence base that currently underpins HTA.

In summary, patient-based evidence appears to be a complex, multifaceted construct, with a range of possible content of great importance and relevance to a full HTA. However, it still represents a marginalized form of evidence, the “poor cousin,” underdeveloped conceptually, theoretically, and methodologically, and often criticized for these deficits when researchers struggle to integrate in within other forms of evidence in HTA. The current situation has not changed substantively since Culyer and Rawlins commented that “how it [patient experience] may be integrated into more complete appraisals have [sic] scarcely been addressed by scholars, let alone implemented by agencies” (Reference Culyer and Rawlins35). Although some progress has been made, there is still much to do.

However, there are signs of hope. Elements of patient-based evidence are already used in HTA, as outlined earlier, and there is significant potential for its development, from a nascent idea, into a theoretically informed, conceptually clear construct that is well understood and utilized within HTA. This could represent a significant paradigm shift in thinking. To achieve this, we might draw on the work of Thomas Kuhn, who, in 1962, published his seminal work The Structure of Scientific Revolutions (Reference Kuhn36). Kuhn challenged the world's current conception of science, which he saw as a steady progression of the accumulation of new ideas. In a series of reviews of past major scientific advances, Kuhn showed that this viewpoint was wrong. Science advanced the most by occasional revolutionary explosions of new knowledge, each revolution triggered by the introduction of new ways of thought so large that they must be called "new paradigms." In creating greater conceptual clarity, it may be possible to organize patient-based evidence into a typology that could be integrated into existing forms of evidence. Alternatively, there may be a more radical version where we need to review all forms of evidence for HTA. We recognize that although we pose a significant challenge for HTA, our intention is to finally create equity between clinical, economic, and patient-based evidence by drawing on Kuhn's theory of how best to advance science in the spirit of helpful disruption.

Why Include Patient-Based Evidence in HTA?

There are a range of reasons for the inclusion of patient-based evidence in HTA. The first reason concerns the nature of knowledge or evidence required for HTA. Earlier accounts of HTA in this paper highlight the importance of more social aspects of HTA, for example, the experience of a health technology or whether it meets a patient health need. It could be argued that addressing patient need should philosophically represent the “absolute essence” of why we undertake HTA, to make decisions for patient or public benefit, alongside the wider system and political imperatives that impact HTA. Clinical and economic forms of evidence make key contributions to HTA, but they do not tell the whole story from a patient's perspective. If we consider the common view of validity that states that high-quality research should include all aspects of a concept under scrutiny (Reference Streiner and Norman37), then patient-based evidence becomes a critical part of high-quality HTA, reflecting Culyer and Rawlins’ concept of “complete appraisals” (Reference Culyer and Rawlins35). Now that we have introduced the nascent concept of patient-based evidence, our future direction should be to continue its exploration and clarification, rather than rejecting it because of its poor conceptual development thus far. To reject it would be to risk maintaining a narrow view of evidence that does not fulfill the concept of a complete HTA and is less responsive to the societal and epidemiological changes.

Exploring the concept of research validity in PROM development inevitably triggers a wider consideration of the role of patients in its methodological aspects. For example, face and content validity offer great potential for the coproduction of knowledge, though they are rarely thought of in this way, with researchers often rapidly assessing these qualities as checks prior to psychometric evaluation, rather than seeing them as opportunities for patient partnership in the creation of knowledge. For example, this could include coproducing the identification of the dimensions of health that a patient-reported outcome measure should include as a form of patient-based evidence. Is the dimension of health relevant to the patient group, or is it more important for researchers or clinicians? Are the emerging dimensions and items acceptable to patients and do they reflect how patients conceptualize an illness? Will the resulting patient-reported outcome measure be appropriate for the patient population? Does it capture the variability in symptoms that patients understand the best? Reporting the coproduction of a patient reported outcome measure may provide it with a legitimacy for a patient population, moving it on from the traditional research constructs of validity, reliability, and responsiveness and takes us into the territory of patients starting to identify patient-important methodological features, as forms of community validity (Reference Staniszewska, Haywood, Brett and Tutton12).

In addition to enhancing the quality of research for HTA, we can argue that a close involvement of patients represents ethical practice. Sandman et al. (Reference Sandman, Bond, Hoffman, Facey, Ploug Hansen and Single16) outline key reasons for patient involvement in HTA, including ensuring relevance to healthcare goals and needs and capacity building for patient empowerment. They also identify substantive rationales including ensuring fairness and legitimacy through democratic participation and fairness in terms of respect for autonomy and equity.

Moreover, the integration of patient evidence reflects the need to be democratically accountable, particularly in health systems funded by tax payer's money. Such accountability provides important legitimacy and enhances the trustworthiness of HTA from the public point of view.

Finally, the importance of patient-based evidence is reinforced when we consider that the aim of HTA is not just to generate knowledge for its own sake, but to change the practice and provision of care, so the topics chosen have to be of importance to patients, communities, and the wider society.

The Application of Patient-Based Evidence

Despite the poor conceptualization and limited recognition of patient-based evidence, there are important examples of its application and use in HTA and related areas, although it has not always been described using the term patient-based evidence.

SBU has produced several HTAs for the past 10 years that provide examples of patient-based evidence. The topics range from living with loss of teeth and edentulousness (38), chronic pain (39), patient involvement and participation in treatments on ADHD, autism spectrum disorders, and psychosis/schizophrenia (40–42), self-harming patients’ experiences and perceptions of professional care and support (43;Reference Lindgren, Svedin and Werkö44), experiences of living with fetal alcohol spectrum disorders (45;Reference Domeij, Fahlstrom, Bertilsson, Hultcrantz, Munthe-Kaas and Gordh46), experiences and perceptions of unaccompanied children and youth (47) to experiences of care in the fields of Endometriosis (48), Myalgic encephalomyelitis and Chronic Fatigue Syndrome (ME/CFS) (49), Psychological and psychosocial interventions in forensic psychiatric care (50), traumatic Brain Injury (51), Eating Disorders (52), and Pharmacological treatment of common pain conditions in older persons (53). All these assessments, thus, include forms of patient-based evidence originating from published scientific studies that have gone through the process of relevance and quality appraisal before the stage of synthesis (at which only studies of moderate and low risk of bias remain), and they concern how people perceive and experience their condition, their health, and their quality of life and what these mean to them. Sometimes, the SBU has included the patients’ own experience of their participation and sometimes also their families' or next-of-kin experiences. The evidence in the reports is formed by the synthesis of experiences, the scientific studies having gone through the similar assessment methods as those of clinical or economic evidence, that is, systematic and structured assessments of relevance and quality and robust scientific methods of synthesis, hence the use of the term "evidence." In all types of synthesis, the same prerequisite is necessary, that is, that primary studies have been conducted with high methodological quality and that all studies are assessed systematically with the appropriate tools or templates for the proper study design. This also applies here. Most of these examples of HTAs at SBU mainly consist of synthesis of studies using qualitative methods of data collection and analysis, but not all. Studies with mixed methods and RCTs have also been included, but to a far lesser extent, given the nature of the research questions. The examples of patient-based evidence from the SBU span over 10 years, with methodological development alongside its use. For example, qualitative evidence synthesis has developed in the last 10 years. A new checklist for assessing the quality of qualitative primary research studies is used, as well as a tool to assess the methodological limitations of qualitative evidence synthesis, with GRADE-CERQual used to assess risk of bias (54). Patient-based evidence is now systematically included in the SBU HTA reports, as the range of examples has shown. One example of how the inclusion of patient-based evidence adds value became apparent early on when some SBU conclusions drawing on patient-based evidence (PBE)-evidence led to actions in practice; this was in the case of the Autism report (41), where the County Councils and Regions (that are responsible for providing health care in Sweden) decided to regularly pay particular attention to siblings. PBE evidence in the SBU-report showed they might otherwise experience problems with social relationships and sometimes might even be exposed to intimidating and violent behaviour. A number of challenges of capturing the impact of HTA exist (Reference Rosén and Werkö55), and there is still a need for conducting more research studies within HTA processes in order to better develop the understanding of how PBE can contribute to decision making.

Patient Involvement in the Coproduction of Patient-Based Evidence

In calling for a greater focus on patient-based evidence in HTA, we need to coproduce the concepts, theory, and methods that underpin it. The term "coproduction" was developed to highlight the potential relationships that could exist between the producers and “clients” when it was realized that the production of a service was difficult without the active participation of those intended to receive it (Reference Ostrom, MCGinnis and Ostrom56). Although patients are vital as suppliers of patient-based evidence, through their inclusion as subjects in trials or in qualitative experience studies, they also have a fundamental role to play as active collaborators in shaping the nature of patient-based evidence, including concepts and methods (Reference Stephens and Staniszewska57). Coproduction at the heart of conceptual and methodological development for HTA requires the creation of new ways of working, supported by research cultures and systems that facilitate coproduction. This means its relevance and applicability in HTA require further refinement, although it has intuitive appeal as a way to foster partnership, reciprocity, and openness. From a research perspective, coproduction could contribute to constructing complete knowledge for HTA. We suggest that the guidance developed by the National Institute of Health Research could provide a useful starting point for HTA by identifying the key principles of coproduction. These include sharing of power, including all perspectives and skills, respecting and valuing the knowledge of all those working together, reciprocity, and building and maintaining relationships (Reference Hickey, Brearley, Coldham, Denegri, Green and Staniszewska13).