1. Introduction
Post-market surveillance (PMS) systems are in place to collect relevant data in a systematic manner for medical devices and involve activities conducted by manufacturers once a product enters the market, to collect information on its quality, performance and safety (European Union, 2017). For regulators, the surveillance system is a tool to take action if there are safety concerns and risks of continued use of the medical device, which outweigh the benefits (WHO, 2021). This paper's objectives are (1) to investigate current efforts and future opportunities to systematically use real-world data (RWD) and real-world evidence (RWE) in PMS to support decision-making of digital health technologies (DHTs) and (2) to develop a framework to set out guiding principles (i.e. good practice guidelines) informed by the literature review and Delphi consensus-building exercise. Our research is timely because this is a new policy area where no country has yet established a well-designed, robust system, in part because digital health policy frameworks are a work in progress in many countries. In this paper, DHTs refer to health technologies as defined by the National Institute for Health and Care Excellence (NICE), including applications (apps), programmes and software used in the health and care system, which may be standalone or combined with other products such as medical devices or diagnostic tests (NICE, 2022). The definition is similar in scope to the one proposed by the World Health Organization in their guidelines on the use of digital health interventions. It refers to ‘a broad umbrella term encompassing eHealth, including mHealth or mobile wireless technologies for health, as well as emerging areas, such as the use of advanced computing sciences in ‘big data’, genomics and artificial intelligence’ (WHO, 2019).
A robust system of PMS has implications on the data needs for evidence generation and evidence thresholds, but also on how to learn from the technologies and inform policy once the DHT is on the market. In this paper, RWD and RWE are based on the US Food and Drugs Administration (FDA) definition (Box 1).
RWD and RWE definitions
Source: FDA Definition of RWD and RWE: https://www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence.
There are untapped benefits from large volumes of data being created. Data could inform providers and patients, feed back into decision-making similar to the health technology assessment (HTA) to inform regulation or set evidence thresholds. It could monitor the technology once it is on the market and could be used to inform upstream design of the DHT, or to understand the degree of generalisability of the DHT. However, it remains unknown whether digital technologies/consultations uncover more need, hence raising the burden. There could be implications on workload for general practitioners with a relatively young/healthy population that could be serviced quickly via online consultation (telemedicine), leading to equity concerns and commissioning/payment considerations. Online consultations attract younger, healthier patients while those who are older/or with long-term conditions are likely to be less comfortable attending remote consultations (Salisbury et al., Reference Salisbury, Quigley, Hex and Aznar2020; Sounderajah et al., Reference Sounderajah, Clarke, Yalamanchili, Acharya, Markar, Ashrafian and Darzi2021; Wieringa et al., Reference Wieringa, Neves, Rushforth, Ladds, Husain, Finlay, Pope and Greenhalgh2022). For example, informing policy could systematically embed user feedback from providers and patients and on how to better mitigate the digital divide (e.g. disparities in the use of technologies among underserved populations). Unlike DHTs, there are ‘pharmacovigilance’ policies and systems for pharmaceutical products that are partly harmonised in Europe for the monitoring and evaluation of medicines once they are on the market, which also helps to understand their effects on different patient profiles and patient sub-groups.
One of the current challenges is the sheer volume of data arising from DHTs. Data come from a variety of sources, with no current international framework on guiding principles for data access and analytics, creating by default, global databases (e.g. an app may contain data from different countries). Related to large volumes of data is how to use data and to support interoperability at various levels (e.g. DHTs, healthcare settings, national borders) as raised in the EU4Digital's eHealth Network on patients' rights in cross-border healthcare. The initiative aims to connect national authorities responsible for eHealth issues to help share policy surrounding eHealth interoperability and standardisation (European Union, 2021). The extent to which real-time data are accessible to support analytics or evidence generation remains an open question.
This paper investigates the opportunities for systematically embedding RWD and RWE in PMS to support decision-making of DHTs drawing on five country experiences (Estonia, Finland, Germany, Italy and United Kingdom with a focus on England). Decision-making in this paper considers decisions taken at different levels in the system, such as by patients, healthcare workers or national and regional institutions (e.g. payers or HTA bodies). We then develop a framework including guiding principles informed by the literature review and Delphi exercise, through an international consensus-building exercise. The consensus framework will bring benefits in decision-making to support nationally and internationally coherent decision-making in PMS. It will assist health authorities in setting out clarity in policy scope, role of institutions in collection of RWD, regulators to set requirements using RWE and digital health developers in their compliance.
Our research explores policy issues around RWD and RWE for DHTs to support their PMS and we propose ‘digital health technology vigilance’ similar in concept to pharmacovigilance to avert potential adverse effects; we argue that this is currently missing in country policies and will be a key aspect for DHTs in the future (CIOMS, 2022). Country policies that include DHT vigilance in a systematic way in their surveillance will improve our understanding of the risks and functionalities of DHTs. A robust PMS system with information on RWD will facilitate the benefits for decision-making including monitoring for quality, safety and adverse events to inform users, governments and digital health developers.
2. Methods
2.1 Literature review
A review of the literature was conducted, inspired by the scoping review method, to map policy issues related to RWD and RWE for DHTs. This included a review of relevant literature, including grey literature in five select countries: Estonia, Finland, Germany, Italy and the United Kingdom (UK).
Countries were selected based on the extent of the application of RWD and RWE in the health system (Table 1). The countries are not a representative sample but rather our interest was to reflect a mix of models of healthcare delivery, including different levels of decentralisation, financing healthcare digitalisation, adoption of digital technologies and population perceptions in regard to collection, use and sharing of data (Ferré et al., Reference Ferré, de Belvis, Valerio, Longhi, Lazzari, Fattore, Ricciardi and Maresso2014; Habicht et al., Reference Habicht, Reinap, Kasekamp, Sikkut, Aaben and van Ginneken2018; Keskimäki et al., Reference Keskimäki, Tynkkynen, Reissell, Koivusalo, Syrjä, Vuorenkoski, Rechel and Karanikolos2019; Vehko et al., Reference Vehko, Ruotsalainen and Hyppönen2019; Blümel et al., Reference Blümel, Spranger, Achstetter, Maresso and Busse2020; OECD et al., 2021; Anderson et al., Reference Anderson, Pitchforth, Edwards, Alderwick, McGuire and Mossialos2022).
Country characteristics: Estonia, Finland, Germany, Italy and the United Kingdom (England)
The literature search focussed on the following aspects: the countries' existing national frameworks and policy objectives, national efforts, ongoing activities and opportunities to further embed RWD and RWE into decision-making. These aspects were applied to each country to guide the search and gain information of the national context. Some countries have had policies in place for several years, but the study focussed on the past decade (2012–2022) to capture relevant and recent developments in digital health policy. An important contribution of our approach was that beyond publications in English, the literature search was conducted in the country's native language (Estonian, Finnish, German, Italian), across relevant national databases and sources of grey literature to discover relevant legal documents, reports and research articles (Table 2).
Literature search for review
2.2 2.2 International consensus-building exercise
The Delphi was informed by findings from the review. In the literature, we identified what was missing from current country practices related to measuring and monitoring of DHTs and what potential the countries have to make measuring and monitoring more systematic. We formed initial guiding principles based on our interpretation of the reviewed documents and then these guiding principles were further developed with the expert panel in the Delphi consensus exercise.
The second part of the methodology employed a consensus-building exercise using the Delphi method. The purpose of Delphi was to develop a framework based on guiding principles to embed RWD and RWE systematically to support decision-making for policy makers, providers and patients. The Delphi rounds involved a range of international experts identified through academic, clinical and research networks.
2.3 Expert recruitment and consent
Experts in DHTs, including experts and researchers affiliated with national and regional institutions, payers, healthcare professionals and experts from patient representative organisations were identified in each of the review countries. To achieve a broad range of views, and country representation beyond our country focus, members of the European Health Policy Group (EHPG) were also invited to participate in the wider clinical, academic and practitioner community. The EHPG consists of a wide range of European experts with affiliations in health policy, research and clinical settings. Experts were contacted via the academic institutions of the study authors. A total of 496 experts were contacted.
2.4 Ethical approval
This study was approved by the London School of Economics and Political Science (approval number 101617). The study's pre-registration can be accessed on the Open Science Framework (https://osf.io/y4bk6). Participation in the Delphi study was voluntary and all participants provided informed consent to participate via email.
2.5 Data collection
The Delphi method was employed through online questionnaires over three rounds between 30 September 2022 and 15 November 2022. The study team agreed to construct the questionnaire in English. In each round, the Delphi questionnaire had two sections. The first part contained building block principles as key elements of a digital health policy framework. The second part contained principles that related to leveraging RWD and RWE for PMS and DHT vigilance.
The experts were asked to rate the importance for each principle using a five-point Likert scale: 1 = exclude, 2 = low priority, 3 = medium priority, 4 = high priority, 5 = essential to include. Additionally, the questionnaire collected demographic information on participants (geographic location, age, gender and occupation).
In the first round Delphi, consenting experts accessed the background information of the study via email, which included a link to the online questionnaire on Qualtrics. Experts were asked to rate and comment on the proposed principles. The given response time was 14 days and two reminders were sent before the first round closed.
Based on the comments from the first round Delphi, the study team modified the principles and established new principles for the subsequent two rounds. In the second and third round, all experts who had provided their consent were invited with their unique personal electronic link to access the questionnaire via Welphi (an online survey platform specifically designed for Delphi consensus-building exercises). Experts were asked to rate and comment on the proposed principles. Experts received a summary of anonymised ratings and comments from each of the previous rounds. This information was provided so experts could consider whether they wished to rank items differently. Experts were given 8 days to complete the second round and 5 days to complete the third round. In both rounds, two reminders were sent before the round closed.
2.6 Analysis
The analysis assessed the distribution of responses to determine the pooled level of agreement for each principle. Consensus was reached for a principle when at minimum 75% of the experts had given a rating of at least four, indicating that the inclusion of the principle in the proposed framework was agreed as a high priority. Guidelines for the Delphi survey technique recommend defining consensus as 75% is acceptable level of agreement (Hasson et al., Reference Hasson, Keeney and McKenna2000). In the literature, some studies have defined consensus as ranging between 75 and 80% (Hasson et al., Reference Hasson, Keeney and McKenna2000; Keeney et al., Reference Keeney, McKenna and Hasson2011; Jünger et al., Reference Jünger, Payne, Brine, Radbruch and Brearley2017). Thereafter, sensitivity analysis was conducted. The analysis was informed by all comments received alongside the distribution of responses including the mean, median, interquartile range (IQR), standard deviation and variance. Analysis was conducted using Excel.
3. Results
3.1 Literature review
The country experiences highlight examples of policies to guide decision-making, but the depth and scope of these policies varies (see Appendix 1 for country summaries). Estonia and Finland have invested and developed digital health-related policies for several years since the 1990s. Germany and Italy are the most recent arrivals, in terms of significant innovations regarding digital health policies, within the past three years. The United Kingdom falls somewhere in the middle.
National efforts have focussed on a range of different activities including reimbursement, data collection efforts and monitoring of quality. Estonia set out reimbursement requirements for some DHTs. Finland's well-established infrastructure supports the flow of digital-related data to national repositories which collect some aggregate information on DHT usage. Germany recently developed an explicit approach for fast-track approvals of digital health applications. Italy set out a long-term vision to collect and harness large amounts of RWD including some digital health-related data on telemedicine. England in the United Kingdom systematically collects aggregate RWD on primary care visits (face-to-face and virtual) to support outcomes, commissioning and regulate quality (Table 3).
Summary of findings from Estonia, Finland, Germany, Italy and the United Kingdom (England)
Although the pandemic catalysed a shift in demand towards the uptake of DHTs offering potential sources of RWD and RWE, there are opportunities to further embed and develop approaches for using RWD and RWE systematically. Sources of RWD could bring further insights on patient experience and support clinical decision-making as seen in the example of remote monitoring of patients with cardiac-related conditions in Estonia. National aggregate data collections do not yet offer ways to distinguish the type of DHT, nor offer systematic information on the uptake among underserved patient groups, but in Finland, the existing nationwide health data resources could provide better opportunities for them. The example from Italy suggests that large amounts of RWD require alignment with data protection but also conditions to create an open and accepting environment to foster expertise in DHTs. The examples of Germany and the UK stress the need for ongoing data collection to inform the product's benefits, quality, safety and reimbursement once the DHT is available on the market.
Country experiences with RWD and RWE arising from DHTs are in their infancy. These examples highlight opportunities and challenges that have implications for post-marketing surveillance but also for a notion we term as DHT vigilance. Adopting a total product lifecycle (TPLC) approach may better align the RWD and RWE data needs and requirements for decision-makers in policy and regulatory circles, manufacturers, providers and patients because it adopts a wholistic approach in the product's lifecycle from design to PMS (Figure 1). The Delphi consensus exercise builds on these findings, with guiding principles that could inform a country's digital health policy framework.
Different evidence needs arising from RWD and RWE.
Source: CDRH Transparency: Total Product Life Cycle (TPLC) | FDA.
3.1.1 Results of the Delphi consensus-building exercise
Of the invited experts, 44 completed round one (response rate 9%). Of them, 24 (67%) completed round two, and similarly, 24 (67%) completed round three. Table 4 shows the demographic characteristics of experts in each round. Close to half or more were aged between 40 and 59 years, with a fairly even split between male and female experts in the first two rounds. Most participating experts came from the countries focussed on in this study: Finland, Germany, Italy, United Kingdom and Estonia. The experts came from a range of settings with most coming from scientist/academic research, healthcare professionals and national/regional institutions.
Demographics of experts
Summary of main findings. A total of 16 out of 20 principles reached consensus. Consensus was reached on five out of 15 principles in the first round. Feedback from round one informed the re-wording of principles and three additional principles in the Delphi questionnaire. In the second round Delphi, consensus was reached on a further eight out of 16 principles. Feedback from round two led to re-wording principles and removing one principle. Consensus was reached on three out of seven principles in the third round (see Appendix 2). Consensus ranged from 75 to 87%.
There are eight building block principles for a DHT framework presented in decreasing order of level of agreement (Table 5). They set out a range of considerations on explicit definitions of DHTs, institutional requirements and a strategy around the collection and use of RWD and RWE including for when a DHT is on the market (see Appendix 2).
Consensus reached on building block principles
Eight principles are proposed to set out how to consider where RWD and RWE could inform decision-making with respect to PMS and DHT vigilance presented in decreasing order of level of agreement (Table 6).
Consensus reached on leveraging RWD and RWE for post-market surveillance and digital health technology vigilance
In summary, the first set of guiding principles identified areas to inform the development and digital health policy framework. The second set of principles extends these notions with respect to leveraging RWD and RWE once the DHT is on the market. The intersection between the two brings to the fore their interdependence with respect to PMS and a concept we propose as DHT vigilance. This paper argues that taking a TPLC approach is essential to frame these considerations.
The agreed consensus speaks to the need for key principles to set out a clear strategy, clarity around the application and use of RWD and RWE to support an agile environment drawing on data collections and evidence that are manageable and appropriate to inform decision-making. The agreement on guiding principles reflects their importance across a cross-section of key stakeholders concerning RWD and RWE. This is an important finding on its own that reflects a convergence in thinking among a group of stakeholders with a range of expertise.
Sensitivity analysis. We conducted sensitivity analysis on our findings (see Appendix 2). The threshold level of agreement was adjusted by five percentage points downwards and upwards to test the robustness of the results. With a 70% level of agreement (instead of 75%), the main findings stayed the same; the remaining four that were not included were still below 70% (between 56 and 65%). With an 80% level of agreement (instead of 75%), eight principles were removed: three building block principles and five from the PMS and DHT vigilance principles. The remaining principles with 80% agreement still reflect the priorities in terms of policy scope, institutional role and data collection requirements.
Our intention was to reach consensus among experts from five main affiliations: national/regional institution, payer, healthcare professional, scientist/academic research, patient/patient organisation. The experts contacted in the study countries, and via the EHPG and in academic institutions reflect this wide mix. A breakdown of responses by affiliations with sufficient sample size (five or more) was available for three out of the five categories. The sensitivity analysis shows that there is broad consensus across most of the guiding principles (Appendix 2).
4. Limitations
It is important to note the limitations in our analysis. Our review focussed on the most relevant evidence and was not systematic in design, so the results of the review are not exhaustive. However, the review was guided by the study team's pre-existing expertise on the subject, which promoted the construction of an overall picture of the national activities obtained with the chosen search strategy. The Delphi study was conducted over a month and so there may be key experts in the five countries that may have been missed. Nevertheless, every effort was made to gather a cross-section of views from the stakeholder groups in these country settings. Contact with all the country experts including introductory emails were sent, if necessary, in their native languages to describe their role and expectations. Experts who agreed to participate were informed that the survey and instructions were administered in English. There may be differences in interpretation of the statements by the experts, but we do not expect that differences in interpretation of the statements will have a large impact on our findings. When looking across affiliations, there is broad consensus with most of the guiding principles which further supports our study's results.
5. Discussion and conclusion
This paper set out to understand how RWD and RWE arising from DHTs are used in decision-making. The findings from the review of five countries filled this knowledge gap, identifying opportunities for each country and their progress with respect to digital health policymaking. The review informed the development of guiding principles to embed RWD and RWE systematically for DHTs to inform decision-making. The proposed principles were reached using the Delphi consensus-building technique among an international group of experts in the field. Consensus was reached for 16 out of 20 principles, which represent building blocks related to policy scope, institutional role and data collection and principles to promote PMS and DHT vigilance (Figures 2 and 3). For the four where consensus was not reached, the distribution of responses, however, reflects their relative high importance (responses had a median value of four) (Appendix 2). Currently, the principles are ranked in order of agreement. A next step for research will be to consider reflecting their importance and relationship with respect to the principles of economics and role and motivation for government intervention in the market of DHTs.
Building block principles to support a total product lifecycle approach.
Principles for post-market surveillance and digital health technology vigilance.
In conclusion, the three main contributions of this paper include first a more nuanced understanding of RWD and RWE with respect to DHTs in the countries reviewed. Second, this is the first study to offer the development of guiding principles to improve digital health policymaking for RWD and RWE introducing the notion of DHT vigilance. Third, the findings from the Delphi exercise highlight the need for greater focus on RWD and RWE across Europe. Considerations include clear criteria for decision-making, institutional involvement in overseeing how and where RWD and RWE are appropriate for decision-making. International involvement and engagement would be useful given the large amounts of data being collected, along with flexibility around evidence needs for the most appropriate study designs and a digital health strategy that is agile given the fast-paced nature of this sector.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S1744133123000208.
Acknowledgements
The authors would like to express their gratitude to all participants of the Delphi consensus-building exercise. The authors acknowledge the support of Oriana Ciani and Francesco Longo in earlier stages of the paper's research.
Financial support
L. V. received funding from the Strategic Research Council at the Academy of Finland (grants 327145 and 352501 for the DigiIN Project), without any influence on the design of the study, collection, analysis and interpretation of the data, or the conclusions.
Open access
