Hostname: page-component-89b8bd64d-7zcd7 Total loading time: 0 Render date: 2026-05-08T18:01:26.351Z Has data issue: false hasContentIssue false
  • English
  • Français

Can we use existing guidance to support the development of robust real-world evidence for health technology assessment/payer decision-making?

Published online by Cambridge University Press:  02 November 2022

Gorana Capkun Open the ORCID record for Gorana Capkun [Opens in a new window] ,
Sorcha Corry Open the ORCID record for Sorcha Corry [Opens in a new window] ,
Oonagh Dowling Open the ORCID record for Oonagh Dowling [Opens in a new window] ,
Fatemeh Asad Zadeh Vosta Kolaei Open the ORCID record for Fatemeh Asad Zadeh Vosta Kolaei [Opens in a new window] ,
Shweta Takyar Open the ORCID record for Shweta Takyar [Opens in a new window] ,
Cláudia Furtado ,
Páll Jónsson Open the ORCID record for Páll Jónsson [Opens in a new window] ,
Diane Kleinermans ,
Laurie Lambert Open the ORCID record for Laurie Lambert [Opens in a new window]  and
Anja Schiel Open the ORCID record for Anja Schiel [Opens in a new window]
...Show all authors
Gorana Capkun*
Affiliation:
Novartis Pharma AG, Basel, Switzerland
Sorcha Corry
Affiliation:
Novartis Oncology, East Hanover, NJ, USA
Oonagh Dowling
Affiliation:
Novartis Pharma AG, Basel, Switzerland
Fatemeh Asad Zadeh Vosta Kolaei
Affiliation:
Novartis Pharma AG, Basel, Switzerland
Shweta Takyar
Affiliation:
Novartis Pharma AG, Basel, Switzerland
Cláudia Furtado
Affiliation:
HTA Division, INFARMED; National School of Public Health, New University of Lisbon, Lisbon, Portugal
Páll Jónsson
Affiliation:
Data and Real World Evidence, NICE, London, UK
Diane Kleinermans
Affiliation:
Drugs Reimbursement, INAMI, Brussels, Belgium
Laurie Lambert
Affiliation:
CADTH, Ottawa, ON, Canada
Anja Schiel
Affiliation:
Lead Methodologist in Regulatory and Pharmacoeconomic Statistics, NoMA, Oslo, Norway
Karen Facey
Affiliation:
FIPRA, Belgium and Usher Institute, University of Edinburgh, Edinburgh, UK
*
*Author for correspondence: Gorana Capkun, E-mail: gorana.capkun@novartis.com
Rights & Permissions [Opens in a new window]

Abstract

Advances in the digitization of health systems and expedited regulatory approvals of innovative treatments have led to increased potential for the use of real-world data (RWD) to generate real-world evidence (RWE) to complement evidence from clinical trials. However, health technology assessment (HTA) bodies and payers have concerns about the ability to generate RWE of sufficient quality to be pivotal evidence of relative treatment effectiveness. Consequently, there is a growing need for HTA bodies and payers to develop guidance for the industry and other stakeholders about the use of RWD/RWE to support access, reimbursement, and pricing. We therefore sought to (i) understand barriers to the use of RWD/RWE by HTA bodies and payers; (ii) review potential solutions in the form of published guidance; and (iii) review findings with selected HTA/payer bodies. Four themes considered key to shaping the generation of robust RWE for HTA bodies and payers were identified as: (i) data (availability, governance, and quality); (ii) methodology (design and analytics); (iii) trust (transparency and reproducibility); and (iv) policy and partnerships. A range of guidance documents were found from trusted sources that could address these themes. These were discussed with HTA experts. This commentary summarizes the potential guidance solutions available to help resolve issues faced by HTA decision-makers in the adoption of RWD/RWE. It shows that there is alignment among stakeholders about the areas that need improvement in the development of RWE and that the key priority to move forward is better collaboration to make data usable for multiple purposes.

Keywords

real-world datareal-world evidenceguidancehealth technology assessmentpayers

Information

Type
Article Commentary
Information
International Journal of Technology Assessment in Health Care , Volume 38 , Issue 1 , 2022 , e79
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0), which permits unrestricted re-use, distribution and reproduction, provided the original article is properly cited.
Copyright
© The Author(s), 2022. Published by Cambridge University Press

Introduction

Real-world data (RWD) has been used for decades by regulators for pharmacovigilance purposes and by HTA for contextualization of evidence to a specific health system setting, to extrapolate outcomes and input to economic modeling. With the advancement of digitization in health systems and expedited regulatory approvals of innovative treatments, there is greater potential for the use of RWD to generate real-world evidence (RWE) to complement evidence from clinical trials. However, many health technology assessment (HTA) bodies and payers have voiced concerns about the ability to develop RWE of sufficient quality to be pivotal evidence of relative treatment effectiveness, and argue that randomized controlled trials (RCTs) should remain the key evidence base (Reference Oortwijn, Sampietro-Colom and Trowman1).

HTA is based on an evidence-based medicine paradigm, with a foundation of critical review of meta-analyses and RCTs. Experience with appraisal of RWD is often more limited, and different expertise is required for the generation of evidence based on RWD. For example, the generation of RWE may involve linking of data from a range of data sources and consider the impacts of creating retrospective definitions of basic aspects of a study, including patient eligibility, baseline characteristics, outcomes, and time windows. Whilst regulators are extending their interest in, and guidance for, the use of RWE to include the consideration of issues related to treatment effectiveness in a clinical practice setting, and appreciate the opportunity to decrease uncertainty in their decision-making, many national HTA bodies have not yet established clear guidance for industry on what RWE they would accept and how it will be appraised (2;3). The Registry Evaluation and Quality Standards Tool (REQueST) from European Network for Health Technology Assessment (EUnetHTA) supports the evaluation of clinical registries for use in HTA (4), but there is no other joint HTA guidance relating to RWD or RWE, nor are registries the sole source of RWD. As the availability of RWD from healthcare systems, patients, and other sources increases, so will the need for HTA bodies and payers to develop clear guidance for industry and other stakeholders about the use of RWD/RWE to support access, reimbursement, and pricing decision-making for highly innovative technologies.

This commentary presents work undertaken to (i) understand barriers to the use of RWD/RWE by HTA bodies and payers; (ii) review potential solutions in the form of existing guidance published by individual HTA bodies, payers, regulators, multi-stakeholder collaborations, and academic groups; and (iii) share the initial opinion on the topic from selected HTA/payer bodies.

Known/published barriers to RWD/RWE uptake by HTA bodies and payers

To better understand the concerns surrounding the use of RWD/RWE raised by HTA bodies and payers, a scoping literature review of PubMed was conducted in April 2021 (see Supplementary Materials) to identify articles published in English, including interviews or group work involving several HTA/payer experts concerning the use of RWD/RWE. This initially focused on articles that included HTA bodies in Europe and then those outside Europe. Five key articles from multi-stakeholder groups involving a range of HTA bodies in Europe were identified (1;5–8) from the HTAi Policy Forum, IMI-GetReal, and RWE4Decisions. In addition, one article solicited German and other European stakeholder views on the value and challenges of RWE post-approval, and relevant RWD collection requirements (Reference Sievers, Joos and Hiligsmann9).

Evidence from this scoping review suggested that the value of RWD, and the RWE it generates, is already accepted and used in support of HTA. RWE provides information about the incidence, prevalence, and natural history of the disease, compliance/adherence to treatment, quality of life, health system resource use, drug utilization, costs, and can support the development of transition probabilities for economic models. However, the evidence also highlighted concerns related to the use of RWE to demonstrate treatment effects. Key issues were extracted from each paper, as shown in Table 1. These included a range of topics about the quality of RWD, data infrastructure and access issues, transparency in curation and analysis, use of appropriate statistical methodology, transferability/generalizability of RWD/RWE, and the mistrust of conclusions made based on RWD. In addition, issues related to the lack of stakeholder collaboration were raised in terms of alignment of RWE requirements pre- and post-licensing (e.g., differences between HTA, payer, and regulators), and clarity about when RWE may be acceptable.

Table 1.

Issues regarding the use of RWD/RWE reported by HTA/Payers in the EU

CEA, comparative effectiveness assessment; HTA, health technology assessment; REA, relative effectiveness assessment; RWD, real-world data; RWE, real-world evidence.

These issues were grouped into themes that were considered key to shaping the generation of robust RWE for HTA and payers: (i) data (availability, governance, and quality); (ii) methodology (design and analytic); (iii) trust (transparency and reproducibility); and (iv) policy and partnerships. Papers involving views of HTA bodies and payers outside Europe (in Canada and USA) were then reviewed and issues extracted (data available on file) (10–18). There was a remarkable similarity in the issues raised, that confirmed themes identified in Europe which are presented as the pillars in Figure 1.

Figure 1.

Known/published barriers to RWD/RWE uptake by HTA bodies and payers. EU, European Union; HTA, health technology assessment; RCT, randomized controlled trial; RWD, real-world data; RWE, real-world evidence.

Potential solutions to address HTA bodies/payer challenges with use of RWE

Recognizing that other fields, such as pharmacoepidemiology, have developed guidance about the generation of RWE that might be applicable to HTA, a further targeted literature review was undertaken to identify potential solutions to overcome the issues raised by HTA bodies and payers. This included a search of key RWD/RWE initiatives, HTA and regulatory websites, EUnetHTA, Google, and PubMed. In total, ninety-three publications from forty-one organizations or collaborations (regulators, academics, professional societies, expert collaboratives, and individual HTA bodies) were identified that presented guidance on the use of RWD or development of RWE in particular settings. These aligned well with the four pillars showing the commonality of issues across stakeholders.

Table 2 presents each of the four pillars and potential guidance that may be available for use or adaptation to help resolve issues faced in HTA decision-making. The key aspects of these publications are presented in the following sections.

Table 2.

Initiatives providing solutions for the challenges and barriers to RWD/RWE adoption for HTA bodies and payers, stratified by the four identified pillars; (i) data (availability, governance, and quality), (ii) methodology (design and analytic); (iii) trust (transparency and reproducibility); and (iv) policy and partnerships

Note: Tools leveraged such as GRADE (Grading of Recommendations Assessment, Development and Evaluation) that address quality generally rank all non-randomized studies as ‘low quality’, regardless of the study quality.

AHRQ, body for healthcare research and quality; AMCP, academy of managed care pharmacy; CanREValue, Canadian real-world evidence for value of cancer drugs; CER, comparative effectiveness research; EHDEN, European Health Data Evidence Network; EMA, European Medicines Body; ENCePP, European Network of Centres for Pharmacoepidemiology and Pharmacovigilance; EU, European Union; EUnetHTA, European Network for Health Technology Assessment; FDA, food and drug administration; GRACE, Good ReseArch for Comparative Effectiveness; HMA, heads of medicines bodies; HTA, health technology assessment; i ~ HD, European Institute for Innovation through Health Data; IMI, innovative medicines initiative; IQWiQ, institute for quality and efficiency in health care; ISPE, international society for pharmaceutical engineering; ISPOR, professional society for health economics and outcomes research; NICE, National Institute for Health and Care Excellence; NPC, NATIONAL PHARMACEUTICAL COUNCIL; OHDSI, observational health data sciences and informatics; OMOP, observational medical outcomes partnership; OPERAND, observational patient evidence for regulatory approval and understanding disease; PE, pharmacoepidemiological; RCT DUPLICATE, randomized controlled trials duplicated using prospective longitudinal insurance claims: applying techniques of epidemiology; RECORD, REporting of studies conducted using observational routinely-collected data; REPEAT, reproducible evidence: practices to enhance and achieve transparency; RWE, real-world evidence; SPACE, structured preapproval and postapproval comparative study; STRATOS, STRengthening analytical thinking for observational studies; STROBE, STrengthening the Reporting of OBservational studies in Epidemiology; WHO, World Health Organization.

Data (availability, governance, and quality)

The availability, governance, and quality of data were addressed in detail by many publications, particularly by collaborative groups. Initiatives led by HTA bodies and payers included EUnetHTA’s REQueEST tool for evaluating registries for HTA use (4), and the German national Institute for Quality and Efficiency in Health Care (IQWiG) guidance on the analysis of routine practice data for benefit assessment (19). ISPOR’s Task Force on Retrospective Databases checklist was created to assess issues unique to database studies, such as data reliability and validity (20). European collaboratives such as European Health Data Evidence Network (EHDEN) have identified data sources and developed approaches to support data quality and harmonization (e.g., via a common data model). Furthermore, regulators, research collaboratives and academics have published several checklists, guides, and reporting standards relating to various data aspects including relevance, reliability, fitness for use, quality, and privacy (see Table 2). All groups encourage pre-planning and transparency of approaches.

Methodology (design and analytic)

Methodological issues relating to design and analytics of real-world studies were considered in a number of the publications identified. Best practice guidance was published on the design and analysis of observational/non-randomized studies for comparative effectiveness research (CER) that is, GRACE, ISPE, STROBE, ISPOR, ENCePP, and PCORI (Table 2). Statistical methodologies for informing CER are widely reported (NICE 2015 (21), ISPOR 2009 (Reference Berger, Mamdani, Atkins and Johnson22), ISPOR 2012 (Reference Berger, Dreyer and Anderson23), GRACE (Reference Dreyer, Schneeweiss and McNeil24), AHRQ (25), and IMPACT HTA W6 (Reference Kent, Salcher-Konrad and Boccia26)) and often include approaches to identifying and mitigating bias and confounding (EUnetHTA 2015 (27), ISPOR 2009 part I (Reference Berger, Mamdani, Atkins and Johnson22), AHRQ (25), ENCePP Methods (28), STRATOS (Reference Sauerbrei, Abrahamowicz and Altman29), and NICE 2015 (21)). A comprehensive overview of existing guidance, frameworks, and checklists is provided by Jaksa et al. (Reference Jaksa, Wu and Jonsson30), as well as ISPOR, NICE, and EUnetHTA. The latter two checklists are HTA-specific and are summarized below.

As part of its robust statistical methodological guidance document, NICE described how it evaluates the quality of an RWD analysis, on treatment effect generally, and in the context of cost-effectiveness, using existing tools and checklists. These include ISPOR 2003; ISPOR-AMCP-NBC 2013 (Reference Kreif, Grieve and Sadique31). The underlying assumptions of statistical methods are often overlooked by HTA reviewers and helpfully described by NICE DSU (2015), with a supportive algorithm to aid the appropriate method selection (21).

EUnetHTA’s guidance on the internal validity of non-randomized studies on interventions similarly includes a critical review of tools and checklist assessing risk of bias, recommending ACROBAT-NRSI (A Cochrane Risk of Bias Assessment Tool) and RoBANS (Risk of Bias Assessment Tool for Non-randomized Studies) (27). In 2019, EUnetHTA developed the REQueST tool to support the evaluation of methodological information, essential registry standards, and additional requirements. Accompanying the tool is a ‘vision paper’ which explores the options for the long-term delivery, use and sustainability of REQueST beyond EUnetHTA Joint Action 3 (4). ISPOR and AHRQ also provide a checklist and guidance on registries, respectively. An extensive list of available checklists on design and analyses are available in the appendix of both documents (21;27).

Other key multi-stakeholder initiatives concerning methodology include IMPACT HTA (work package 2 and 6 specifically) (Reference Kent, Salcher-Konrad and Boccia26) and IMI-GetReal. Please see Table 2 for further details.

Trust (transparency and reproducibility)

Several groups have developed publications focused on trust, transparency, and reproducibility. The multi-stakeholder RWE Transparency Initiative (ISPOR/ISPE/NPC/Duke-Margolis) reported practical recommendations for establishing a culture of transparency for the analysis and reporting of RWD/RWE studies, including the creation of an RWE registry that partnered with the Open Science Foundation to promote a more widespread culture of registering RWE study (Reference Orsini, Berger and Crown32). For the reproducibility of RWE studies, a structured template for planning and reporting on RWE studies (STaRT-RWE) has been developed (Reference Wang, Pinheiro and Hua33). The majority of organizations providing solutions related to trust in the generation and use of RWD/RWE were non-HTA body/payer specific and developed recommendations without HTA involvement.

Policy and partnerships

Identified policy and partnership-related barriers to the adoption of RWE by HTA bodies and payers included the lack of harmonization on policies, evidence requirements, and the lack of coordination at international level between HTA bodies and payers for RWD collection, acceptance, context of acceptance, and relevance. Collaborative initiatives including ISPOR/ISPE, ISPOR-AMCP-NPC; OHDSI, EHDEN, GetReal, CanREValue, and RWE4Decisions have all worked to develop solutions related to policy and partnerships. The Observational Health Data Science and Informatics (OHDSI) program was established in 2014 as an interdisciplinary partnership to bring out the value of health data through large-scale analytics, producing open-source solutions. OHDSI collaborates with the EHDEN (European Health Data and Evidence Network) Academy to support work related to data quality and provide education for all those working on RWD (Reference Kent, Burn and Dawoud34). RWE4Decisions, works at the policy level and is payer-led, but seeks to foster partnership among stakeholders to explore what RWD can be collected for innovative technologies that meet the needs of patients and healthcare systems, and to ensure efficient use of RWD/RWE to inform HTA body and payer decisions. A recent US Food and Drug Administration (FDA) and European Medicines Agency (EMA) collaboration developed a roadmap for international collaboration on RWE using COVID-19 as the model case (35;36) and FDA-Reagan-Udall-Friends of Cancer-Aetion developed the COVID-19 Evidence Accelerator, which involved the collaboration of leading experts in health data aggregation and analytics with the aim of sharing insights, comparing results, and answering key questions to inform the collective COVID-19 response (37).

Discussion

The four pillars

The findings of the literature reviews were presented to a panel of HTA experts at an RWE4Decisions webinar in October 2021 (38). The panelists agreed with the four pillar themes, noting that they addressed both policy-related issues and issues relating to processes for individual HTAs. The pillar relating to data (availability, governance, and quality) was considered as paramount by all panel members and methodology was also considered key. The improvement of transparency relating to RWE study conduct, including registration of RWE study designs and analysis plans, for example via the ISPOR portal, was also supported by the panel members. In terms of policy, the development of the European Health Data Space (39) was agreed as an important step. The panel considered the development of partnerships among stakeholders to be of high importance, particularly initiatives that bring together data sources that may be relevant to HTA, within a trusted research environment, such as EMA’s work on the Data Analysis and Real World Interrogation Network (DARWIN EU) (40).

In addition to the four main pillars, education to upskill HTA bodies and the need for senior-level HTA/payer commitment to provide resources for work in the field of RWE were seen as essential to underpinning infrastructure. Education and expertise links closely to the themes of methodology, trust, and partnership. There is a clear need for the upskilling of all stakeholders involved in the collection, curation, analysis, and appraisal of RWD/RWE. This includes education of and engagement with clinicians who collect RWD in the real-world healthcare setting, particularly those who manage disease registries (such as the European Reference Networks for rare diseases), or those who contribute to the assessment and documentation of outcomes as part of Outcomes Based Managed Entry Agreements (OBMEA). Education is often required to explain the objectives and information needs of HTA, and better engagement facilitates the identification of opportunities for collaboration to generate or provide access to RWD. Multi-stakeholder dialogues focused on planning for RWE were seen as an important aspect of education for all stakeholders to discuss the advantages and disadvantages of different RWD sources that could complement planned clinical studies or be used for OBMEA.

In terms of senior-level commitment and resourcing, the recent strategic initiative taken by the National Institute for Health and Care Excellence (NICE) was noted (41). The NICE Strategy 2021–2026 consists of four pillars, one of which is ‘leadership in data, research and science’, including using RWD to resolve gaps in knowledge and drive forward access to innovations for patients. This commitment has been demonstrated through the NICE leadership of Work Package 6 in the IMPACT HTA project “methodological guidance on the analysis and interpretation of non-randomized studies to inform health economic evaluation” (Reference Kent, Salcher-Konrad and Boccia26). The resulting guidance has been used by NICE to inform their RWE Framework that was launched in June 2022 (42).

Can existing guidance be used by HTA bodies and payers?

The extent of existing available guidance from trusted resources was not known to all panel members, and several publications seemed highly relevant to address challenges faced in HTA, either in its present form, or with adaptation. Without detailed review of all available documents, it was unclear whether gaps remain, but the following areas were considered key for HTA and may be a focus for the development of HTA-specific guidance and policy inputs in the future.

All panel members agreed that RCTs should be undertaken to demonstrate relative efficacy, whenever possible, but they cannot provide all the information needed for HTA body and payer decision-making. Moreover, they agreed that RWE can provide important complementary evidence that can help resolve HTA/payer uncertainties. To provide robust RWE, systematized and transparent methods of data curation are needed that take account of data provenance (e.g., clinical registries, health claims data, patient-reported outcomes) and processes as well as the original purpose of data collection and the associated limitations that imposes. Interoperability and linking between different data sources are critical issues within the context of confidentiality governance legislation, which may be interpreted differently across EU member states or different data owners. Data discoverability is another important consideration; the signposting of high-quality data sources and how they can be accessed is essential. These issues have been addressed by many of the research collaboratives identified, and more recently by regulators, but HTA bodies and payers also have an important role to play in driving the development, access, and use of RWD.

Existing guidance to support protocol-driven high-integrity RWE studies may be sufficient for HTA purposes in terms of how to (i) clearly document data sources, (ii) how relevant patients will be identified, processes for curation of data according to definitions of exposure (treatment), outcomes and covariates, methods for analysis and appropriate sensitivity analyses, but this needs more detailed review. The publication of RWE protocols underpins the methodological pillar but is also an essential element in transparency and building trust.

It is widely acknowledged that greater collaboration is needed among HTA bodies and payers across jurisdictions to anticipate when RWD/RWE may be needed in an OBMEA post the initial HTA/payer decision and to align those RWD requirements, at least to a core data set (6;43). This alignment needs to extend to national and transnational regulatory agencies to consider their post-authorization data collection requirements. Collaborative approaches should acknowledge the needs of different decision-makers, data availability for specific diseases, member state data infrastructures, and in HTA body, clinical and patient capacities. Alignment on the quality of RWD generation for the pre-submission phase, and agreement on the core data set and the use of the same data systems for OBMEAs would help avoid duplication of efforts and be respectful to patients. The costs associated with gathering and analyzing RWD/RWE are also substantial and the role of each stakeholder, particularly industry must be agreed.

For all aspects, sharing of cases where RWE had been critically assessed in HTA or small pilot or demonstration projects were seen as valuable by the panel members.

Conclusions

In order to create robust RWE to inform HTA/payer decision-making it is helpful to consider four key pillars of (i) data (availability, governance, and quality); (ii) methodology (design and analytic); (iii) trust (transparency and reproducibility); and (iv) policy and partnerships, underpinned by education and senior level HTA/payer commitment. These themes are in alignment with those raised by other stakeholders. Furthermore, the guidance and tools developed to address issues related to these four pillars provide a strong foundation for HTA bodies and payers to enhance the use of RWE to inform decision-making. Data quality is considered as the highest priority, with a cultural shift needed so that HTA bodies and payers take a more leading role in discussions about RWD to ensure their needs are considered in data constructs (such as disease registries), and data enquiry systems (such as DARWIN EU), alongside input to larger policy initiatives, such as creation of the European Health Data Space (39). A range of methodological guides exist that could contribute to transparency and trust. However, to generate decision-grade RWE, investment is needed, not only in data infrastructure but also in human resources, as the generation and use of RWE requires knowledge and capacity development in all stakeholder groups. A collaborative approach is needed to ensure we do not ‘reinvent the wheel’ and make best use of existing expert guidance regarding the development of RWE that will ultimately improve patient outcomes and optimize treatment use. Existing and emerging collaborations such as RWE4Decisions, EUnetHTA, and the GetReal Institute play a crucial role in both signposting to existing guidance and in the development of bespoke guidance that can support the use of RWD/RWE in HTA and payer decision-making. Such guidance could be used by individual bodies to create their own policies and processes.

There is also a need to publicly document examples that show how the quality of RWD has been evaluated for HTA purposes (e.g., using EUnetHTA REQueEST), appraisals where RWE has been critically assessed, conditional data collection agreements (both national and across border – for example in BENELUXAI, FINOSE or across the nations of the UK or Canadian provinces) and reports of how additional data collection has informed decision-making and could have been improved. Such examples and existing guidance should be leveraged by HTA bodies to co-create, with multi-stakeholder input, seminal HTA guidance on RWD/RWE that is translatable and adaptable to the local context. Finally, we encourage HTA/payer experts to join DARWIN EU, European Health Data Space, and other health data infrastructure discussions to ensure that data, which are of particular interest to payers, such as healthcare utilization and cost data, are collected.

Acknowledgments

The authors would like to acknowledge Jo de Cock (INAMI), Niklas Hedberg (TLV, Sweden), and Aisling O’Leary (NCPE, Ireland) for their valuable contributions to the HTA panel discussions. The authors thank Rebecca Hornby, PhD, and Gemma Carter, PhD, of Oxford PharmaGenesis, Oxford, UK, for providing medical writing support, which was sponsored by Novartis Pharma AG.

Supplementary material

To view supplementary material for this article, please visit https://doi.org/10.1017/S0266462322000605.

Funding statement

This study was funded by Novartis Pharma AG.

Conflicts of Interest

G.C. is an employee and shareholder of Novartis Pharma AG. S.C. is an employee and shareholder of Novartis Oncology. O.D. is an employee of Novartis Pharma AG. F.K. is an employee and shareholder of Novartis Pharma AG. S.T. is an employee of Novartis Pharma AG. C.F. is an employee of INFARMED. P.J. is an employee of the National Institute of Health and Care Excellence and has received grants and financial support from the Innovative Medicines Initiative. D.K. and L.L. have no conflicts of interest. A.S. is an employee of the Norwegian Medicines Agency and acted as Vice-Chair on the EUnetHTA 21 Committee for Scientific Consistency and Quality Joint Scientific Consultations. K.F. is an employee of FIPRA and received funds from Novartis Pharma AG for the preparation of this manuscript. K.F. has also received a grant from the University of Edinburgh, consultancy/advisory board fees from the Canadian Agency for Drugs and Technologies in Health, Edinburgh Innovations, the Belgian Institut National d’Assurance Maladie-lnvalidite, the National Institute of Health and Care Excellence, Novartis Pharma AG and Pfizer, and honoraria from Merck and the GetReal Institute.

Footnotes

G.C., S.C., O.D., F.A.Z.V.K., and S.T.: Joint first authorship.

References

Oortwijn, W, Sampietro-Colom, L, Trowman, R. How to deal with the inevitable: Generating real-world data and using real-world evidence for HTA purposes – From theory to action. Int J Technol Assess Health Care. 2019;35:346–350.Google ScholarPubMed
U.S. Food and Drug Administration . Real-world evidence [Internet] [cited 2022 Mar 23]. 2022. Available from: https://www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence.Google Scholar
European Medicines Agency . A vision for use of real-world evidence in EU medicines regulation [Internet] [cited 2022 Mar 23]. 2021. Available from: https://www.ema.europa.eu/en/news/vision-use-real-world-evidence-eu-medicines-regulation.Google Scholar
EUnetHTA. Vision paper on the sustainable availability of the proposed Registry Evaluation and Quality Standards Tool (REQueST) [Internet] [cited 2022 Mar 23]. 2019. Available from: https://www.eunethta.eu/wp-content/uploads/2019/10/EUnetHTAJA3_Vision_paper-v.0.44-for-ZIN.pdf?x16454.Google Scholar
Oortwijn, W. Real-world evidence in the context of health technology assessment processes-from theory to action [Internet] [cited 2022 Mar 23]. 2018. Available from: https://www.researchgate.net/publication/331154538_Real-world_evidence_in_the_context_of_health_technology_assessment_processes-from_theory_to_action.Google Scholar
Facey, KM, Rannanheimo, P, Batchelor, L, Borchardt, M, de Cock, J. Real-world evidence to support Payer/HTA decisions about highly innovative technologies in the EU—Actions for stakeholders. Int J Technol Assess Health Care. 2020;36:459–468.CrossRefGoogle Scholar
Makady, A, Ham, RT, de Boer, A, et al. Policies for use of real-world data in health technology assessment (HTA): A comparative study of six HTA agencies. Value Health. 2017;20:520–532.Google ScholarPubMed
Makady, A, van Veelen, A, Jonsson, P, et al. Using real-world data in health technology assessment (HTA) practice: A comparative study of five HTA agencies. Pharmacoeconomics. 2018;36:359–368.Google ScholarPubMed
Sievers, H, Joos, A, Hiligsmann, M. Real-world evidence: Perspectives on challenges, value, and alignment of regulatory and national health technology assessment data collection requirements. Int J Technol Assess Health Care. 2021;37:e40.Google ScholarPubMed
Berger, ML, Sox, H, Willke, RJ, et al. Good practices for real-world data studies of treatment and/or comparative effectiveness: Recommendations from the Joint ISPOR-ISPE special task force on real-world evidence in health care decision making. Value Health. 2017;20:1003–1008.CrossRefGoogle ScholarPubMed
Pearson, SD, Dreitlein, B, Towse, A, Hampson, G, Henshall, C. Understanding the context, selecting the standards: A framework to guide the optimal development and use of real world evidence for coverage and formulary decisions [Internet] [cited 2022 Mar 23]. 2018. Available from: https://icer.org/wp-content/uploads/2020/11/ICER-RWE-Framework-Companion-White-Paper-03282018.pdf.Google Scholar
Hampson, G, Towse, A, Dreitlein, WB, Henshall, C, Pearson, SD. Real-world evidence for coverage decisions: Opportunities and challenges. J Comp Eff Res. 2018;7:1133–1143.Google ScholarPubMed
Malone, DC, Brown, M, Hurwitz, JT, Peters, L, Graff, JS. Real-world evidence: Useful in the real world of US payer decision making? how? when? and what studies? Value Health. 2018;21:326–333.CrossRefGoogle ScholarPubMed
Clausen, M, Mighton, C, Kiflen, R, et al. Use of real-world evidence in cancer drug funding decisions in Canada: A qualitative study of stakeholders’ perspectives. CMAJ Open. 2020;8:E772–E778.CrossRefGoogle ScholarPubMed
Chan, K, Nam, S, Evans, B, et al. Developing a framework to incorporate real-world evidence in cancer drug funding decisions: the Canadian real-world evidence for value of cancer drugs (CanREValue) collaboration. BMJ Open. 2020;10:e032884.CrossRefGoogle ScholarPubMed
Polisena, J, Jayaraman, G. Use of real-world data and evidence for medical devices: A qualitative study of key informant interviews. Int J Technol Assess Health Care. 2020;36:579–584.Google ScholarPubMed
Bullement, A, Podkonjak, T, Robinson, MJ, et al. Real-world evidence use in assessments of cancer drugs by NICE. Int J Technol Assess Health Care. 2020;10:1–7.Google Scholar
Lee, W, Dayer, V, Jiao, B, et al. Use of real-world evidence in economic assessments of pharmaceuticals in the United States. J Manag Care Spec Pharm. 2021;27:5–14.Google ScholarPubMed
IQWiG. Concepts for the generation of routine practice data and their analysis for the benefit assessment of drugs according to §35a Social Code Book V (SGB V). IQWiG Reports – Commission No. A19–43 [Internet] [cited 2022 Mar 23]. 2020. Available from: https://www.iqwig.de/download/a19-43_routine-practice-data-for-the-benefit-assessment-of-drugs_rapid-report_v1-0.pdf.Google Scholar
ISPOR. A checklist for retrospective database studies [Internet] [cited 2022 Mar 23]. 2003. Available from: https://www.ispor.org/heor-resources/good-practices/article/a-checklist-for-retrospective-database-studies.Google Scholar
NICE Decision Support Unit . NICE DSU Techincal Support Document 17: The use of observational data to inform estimates of treatment in technology appraisal: Methods for comparative individual patient data [Internet] [cited 2022 Mar 23]. 2015. Available from: http://nicedsu.org.uk/wp-content/uploads/2016/03/TSD17-DSU-Observational-data-FINAL.pdf.Google Scholar
Berger, ML, Mamdani, M, Atkins, D, Johnson, ML. Good research practices for comparative effectiveness research: defining, reporting and interpreting nonrandomized studies of treatment effects using secondary data sources: The ISPOR good research practices for retrospective database analysis task force report–part I. Value Health. 2009;12:1044–1052.CrossRefGoogle ScholarPubMed
Berger, ML, Dreyer, N, Anderson, F, et al. Prospective observational studies to assess comparative effectiveness: The ISPOR good research practices task force report. Value Health. 2012;15:217–230.CrossRefGoogle ScholarPubMed
Dreyer, NA, Schneeweiss, S, McNeil, BJ, et al. GRACE principles: Recognizing high-quality observational studies of comparative effectiveness. Am J Manag Care. 2010;16:467–471.Google ScholarPubMed
Agency for Healthcare Research and Quality . Developing a protocol for observational comparative effectiveness research: A user’s guide [Internet] [cited 2022 Mar 23]. 2013. Available from: https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/observational-cer-protocol_research.pdf.Google Scholar
Kent, S, Salcher-Konrad, M, Boccia, S, et al. The use of nonrandomized evidence to estimate treatment effects in health technology assessment. J Comp Eff Res. 2021;10:1035–1043.CrossRefGoogle ScholarPubMed
EUnetHTA . Internal validity of non-randomised studies (NRS) on interventions [Internet] [cited 2022 Mar 23]. 2015. Available from: https://www.eunethta.eu/wp-content/uploads/2018/01/Internal-validity-of-non-randomised-studies-NRS-on-interventions_Guideline_Final-Jul-2015.pdf.Google Scholar
European Network of Centres for Pharmacoepidemiology and Pharmacovigilance . ENCePP guide on methodological standards in pharmacoepidemiology [Internet] [cited 2022 Mar 23]. 2021. Available from: https://www.encepp.eu/standards_and_guidances/methodologicalGuide.shtml.Google Scholar
Sauerbrei, W, Abrahamowicz, M, Altman, DG, et al. STRengthening analytical thinking for observational studies: The STRATOS initiative. Stat Med. 2014;33:5413–5432.CrossRefGoogle ScholarPubMed
Jaksa, A, Wu, J, Jonsson, P, et al. Organized structure of real-world evidence best practices: Moving from fragmented recommendations to comprehensive guidance. J Comp Eff Res. 2021;10:711–731.Google ScholarPubMed
Kreif, N, Grieve, R, Sadique, MZ. Statistical methods for cost-effectiveness analyses that use observational data: A critical appraisal tool and review of current practice. Health Econ. 2013;22:486–500.CrossRefGoogle ScholarPubMed
Orsini, LS, Berger, M, Crown, W, et al. Improving transparency to build trust in real-world secondary data studies for hypothesis testing&–why, what, and how: Recommendations and a road map from the real-world evidence transparency initiative. Value Health. 2020;23:1128–1136.CrossRefGoogle Scholar
Wang, SV, Pinheiro, S, Hua, W, et al. STaRT-RWE: Structured template for planning and reporting on the implementation of real world evidence studies. BMJ. 2021;372:m4856.Google ScholarPubMed
Kent, S, Burn, E, Dawoud, D, et al. Common problems, common data model solutions: Evidence generation for health technology assessment. Pharmacoeconomics. 2021;39:275–285.CrossRefGoogle ScholarPubMed
European Medicines Agency . European Commission, EMA and FDA agree new priorities to strengthen their collaboration on medicines [Internet] [cited 2022 Mar 23]. 2020. Available from: https://www.ema.europa.eu/en/news/european-commission-ema-fda-agree-new-priorities-strengthen-their-collaboration-medicines.Google Scholar
European Medicines Agency. Advancing international collaboration on COVID-19 real-world evidence and observational studies [Internet] [cited 2022 Mar 23]. 2021. Available from: https://www.ema.europa.eu/en/news/advancing-international-collaboration-covid-19-real-world-evidence-observational-studies.Google Scholar
Collaboration: FDA-Reagan-Udall-Friends of Cancer-Aetion . COVID-19 evidence accelerator [Internet] [cited 2022 Mar 23]. 2022. Available from: https://evidenceaccelerator.org/.Google Scholar
RWE4Decisions . The use of real-world evidence for decision-making: Next steps to develop HTA guidance [Internet] [cited 2022 Mar 23]. 2021. Available from: https://rwe4decisions.com/event/the-use-of-real-world-evidence-for-decision-making-next-steps-to-develop-hta-guidance/.Google Scholar
European Commission . European health data space [Internet]. 2022. Available from: https://eceuropaeu/health/ehealth-digital-health-and-care/european-health-data-space_en.Google Scholar
European Medicines Agency . Data analysis and real world interrogation network (DARWIN EU) [Internet] [cited 2022 Mar 23]. 2021. Available from: https://www.ema.europa.eu/en/about-us/how-we-work/big-data/data-analysis-real-world-interrogation-network-darwin-eu.Google Scholar
NICE. The NICE strategy 2021 to 2026 [Internet] [cited 2022 Mar 23]. 2021. Available from: https://static.nice.org.uk/NICE%20strategy%202021%20to%202026%20-%20Dynamic,%20Collaborative,%20Excellent.pdf.Google Scholar
NICE. NICE real-world evidence framework [Internet] [cited 2022 Mar 23]. Available from: https://view.officeapps.live.com/op/view.aspx?src=https%3A%2F%2Fwww.nice.org.uk%2FMedia%2FDefault%2FAbout%2Fwhat-we-do%2FReal-world-evidence%2FRWE-framework-consultation-framework.docx&wdOrigin=BROWSELINK.Google Scholar
RWE4Decisions. Generating real-world evidence in outcomes-based managed entry agreements: Two fictitious case studies: RWE4Decisions case studies workshops – June 2021[Internet] [cited 2022 Mar 23]. 2021. Available from: https://rwe4decisions.com/wp-content/uploads/2021/11/RWE4Decisions-workshops-2021-report-final.pdf.Google Scholar
Figure 0

Table 1. Issues regarding the use of RWD/RWE reported by HTA/Payers in the EU

Figure 1

Figure 1. Known/published barriers to RWD/RWE uptake by HTA bodies and payers. EU, European Union; HTA, health technology assessment; RCT, randomized controlled trial; RWD, real-world data; RWE, real-world evidence.

Figure 2

Table 2. Initiatives providing solutions for the challenges and barriers to RWD/RWE adoption for HTA bodies and payers, stratified by the four identified pillars; (i) data (availability, governance, and quality), (ii) methodology (design and analytic); (iii) trust (transparency and reproducibility); and (iv) policy and partnerships

Supplementary material: File

Capkun et al. supplementary material

Capkun et al. supplementary material

Download Capkun et al. supplementary material(File)
File 75.2 KB