Introduction/Background
In health technology assessment (HTA), the determination of value is dependent on the dimensions and the perspectives considered (Reference O’Rourke, Oortwijn and Schuller1). The patient perspective is one important aspect of HTA and can be represented through the direct involvement of patient representatives or through review of patient-based evidence (Reference Facey, Bertelsen, Wale, Single, Facey, Ploug Hansen and Single2). It is recommended to perform robust research into patients’ needs, preferences, experiences, and patient participation for including the patient perspective in value determination. However, it is not yet well defined how such research can be integrated into the specific national or regional HTAs to fulfill the high evidence expectations. Some HTA bodies rely on participatory processes through surveys, templates, hearings, or representation by patient organizations in HTA committees to account for the patient perspective (Reference Facey, Ploug Hansen and Single3–Reference Livingstone, Verdiel and Crosbie5). In contrast to evidence that can be critically assessed, patient input arising from participation tends to be more topical, able to provide additional insights during any deliberation throughout the HTA, and is generally strong in reflecting the local context such as service variation or special access conditions.
While participatory processes for involving patients in HTA are evolving and are applied in various forms by HTA bodies (Reference Facey, Ploug Hansen and Single3; Reference Facey, Bedlington and Berglas4), limitations of the current involvement procedures have also been noted (Reference Livingstone, Verdiel and Crosbie5–Reference Scott and Wale8). The complexity and individualization of involving patients puts a high burden on the resources of HTA bodies as well as of patient organizations or patient advocates. Dedicated, well-trained personnel as well as the capacity for timely delivery of the information requested by the HTA body are needed. In current frameworks, patients may be underrepresented due to location, symptoms or severity of illness, language, cognitive function, length or progress of illness, rarity of disease, or lack of recognition particularly if they are not represented by a patient organization. Finally, the uncertainty of the representativeness of individual qualitative input somewhat devalues it in comparison to the quantitative data-based evidence.
Several approaches are currently used in HTA to integrate the patient perspective including studies of Patient Reported Outcomes in the context of clinical studies, Patient Reported Experience studies, or surveys collecting information on real-life experiences relating to the disease or the delivery of care (Reference Facey, Ploug Hansen and Single3). It has been suggested that an additional solution to improve the representativeness and comprehensiveness may be non-interventional, observational analysis of ‘natural’ communication between patients or caregivers in social media (Reference Street, Farrell, Facey, Ploug Hansen and Single9). As patient and caregiver communities are open to broad participation from all levels of society, and are often used to discuss own experiences, seek help from people in a similar situation, and get advice for the daily living or coping with one’s own health or disease, the communication in social media may be a wider source for understanding patient experiences in a more representative, qualitative or quantitative manner than through direct active input from individuals or selected populations through testimonies, interviews or surveys. Research on social media, henceforth referred to as Social Media Research, such as social media listening or sentiment analysis, may yield insights into patients’ needs, expectations, and experiences from a wider group of patients in a more standardized or objective fashion being less dependent on selective individual experiences (Reference Cook, Mullins and Gautam10–Reference Pomputius14). On the other hand, Social Media Research may introduce new biases such as relying on input from digitally active patients who may predominantly discuss negative experiences or from sources aiming to manipulate the discussion among users (Reference Cook, Cave and Holtorf13; Reference Fadaee, Huynh and Towfigh15).
While some attention has been given to the detection of infective diseases or adverse events through Social Media Research, little work has been done yet on its use for HTA purposes. Street and Farrell suggested that analysis of patient communication in social media may help to retrieve information on patient experiences in “hard-to-reach” patient types and that social media may be a more rapidly accessible source of information on patient experiences (Reference Street, Farrell, Facey, Ploug Hansen and Single9). However, they also caution that important ethical and methodological challenges with collecting patient views and experience via social media need to be addressed.
To fill in the gap in understanding the conduct of Social Media Research and its use in HTA, Patient and Citizen Involvement Group of Health Technology Assessment international (HTAi; www.htai.org/pcig) initiated a multi-stakeholder project to explore whether Social Media Research can generate evidence which fills current information gaps relating to patient experiences, patient needs, or patient values and can deliver additional value to HTA.
This initiative driven by a team of diverse stakeholders with patient involvement expertise (HTA practitioners, industry, patient advocates, health policy experts) aimed to collaboratively examine and describe the current environment related to the use of Social Media Research in HTA.
Methods
Definitions
Social media are mobile and web-based highly interactive platforms via which individuals and communities share, cocreate, discuss, and modify user-generated content (Reference Kietzmann, Hermkens, McCarthy and Silvestre16).
For the purposes of this report, Social Media Research is defined as research with data originating from any social media platform including, but not limited to: Twitter, Facebook, YouTube, Instagram, Reddit, blogs, and chatrooms or forums (closed, password-protected, or open, non-password-protected). The research may refer to large quantitative data mining/modeling methods through to more qualitative in-depth analyses. Social Media Research generally aims to reveal new insights into information sharing or policy discussions, to understand personal experiences and opinions or sentiments from the individual or patient community perspective, to conduct epidemiological studies, to identify and detect adverse events or other clinically relevant reports, or to observe online behavior.
Project process
The PCIG subcommittee (core project team) consisted of patient advocates and patient engagement experts from HTA agencies (4), patient organizations (3), academia (3), and industry (2 pharmaceutical, 1 medical devices).
Step 1: Framing the issue
The project started with a framing workshop. Informed by the respective expertise and experience of each of the participants, the PCIG project team defined in collaborative work sessions using an Online Collaboration Whiteboard the expected deliverables, properties, and requirements for the use of evidence generated through Social Media Research in HTA from five perspectives: Patients, HTA agencies, Research, Industry, and the Public. The resulting whiteboard contributions were clustered by themes (grounded theory approach), discussed, and then revised based on the consensus in the discussion.
The different perspectives were challenged and complemented through individual discussions with independent stakeholders including experts from research (1), HTA agencies (5), industry (5), or patient organizations (14) during the time from March to June 2021 following a discussion guide (Supplementary File 1). All patients were members of patient organizations and had various levels of experience with social media as moderators or users. The independent stakeholders were selected from a long list of contacts that were proposed by the team members after prioritization to achieve a broad coverage of stakeholder types, healthcare systems, disease areas, and age groups for the patients. The final selection was determined by the willingness to participate in these discussions.
The inputs relating to the acceptability, the perceived risks, expected governance, and feasibility of using Social Media Research for informing HTAs were recorded after each discussion in an Excel spreadsheet and summarized for each stakeholder group by one reviewer.
Step 2: Literature reviews
Three subjects emerged from the stakeholder input in the framing phase that required a more in-depth review of the current state of the art, and which were therefore explored through scoping literature reviews:
-
A. Robustness of Social Media Research Methods
-
B. Case examples for Social Media Research in healthcare decision making
-
C. Privacy / Legal / Ethical Considerations (reported in a separate publication (17))
For each of the scoping reviews, a list of specific keywords was defined by the three members of each review team and then applied to a search in CINAHL, PubMed, EMBASE, and/or Google scholar. The retrieved long list of articles was screened by title for inclusion or exclusion by the review team members. This resulted in the inclusion of 41 (Privacy / Legal / Ethical Considerations), 75 (Social Media Research Methods), and 23 (Use of Social Media Research in Decision Making) papers, respectively. Each team member extracted relevant information according to agreed-on criteria. The overall scoping summary was consolidated through a consensus of the respective three team members.
The entire process and findings were summarized and discussed in a combined final report, which was circulated for consultation in the PCIG community and to selected stakeholders who had not been involved in the project before. The report, revised after the consultation, serves as a foundation for this manuscript.
Results
Expectations for social media research for HTA purposes
The co-creatively defined stakeholder expectations and requirements for Social Media Research guided the subsequent project flow including their validation through discussions with independent stakeholders and the three literature reviews. The stakeholder perspectives and expectations are summarized in Table 1.
Table 1. Stakeholder perspectives and expectations for social media research or a social media research platform as co-created in two team workshops and refined through stakeholder discussions
Abbreviations: HTA, health technology assessment; R&D, research and development.
The main expectations formulated by the stakeholders were that Social Media Research can systematically reveal patient-relevant outcomes, expectations, preferences, experiences, unmet needs, and impact of disease or treatment on patients’ life without putting additional burden on patient communities. Thereby, it is expected to form a source for identifying the unmet needs of the patient in alignment with HTA process, which can inform HTA researchers on a variety of aspects including the suitability of the COAs or PROs (to be) used in clinical trials, subgroup or regional divergence, changes of patient experiences over time, or matching of terminology used by patients and professionals (linguistic interface). The advantages expected from Social Media Research include that the research finds the patient conversations in their own language (instead of patients having to find the platform or survey or other means of contributing to the HTA process and learning the vocabulary applied there), that it does not create additional work for patients, and that it allows for continuous observation of changes in disease area as communicated by patients or caregivers.
On the other hand, the strongest concerns emerging from the stakeholder perspectives mainly related to two topics: (i) Legal and ethics requirements including privacy protection, and (ii) anticipated risks related to the data and research robustness.
Within the Legal and Ethical Requirements, the consideration of existing ethical frameworks and legislation for Social Media Research, the handling of data from different types of sources, and security issues seemed particularly important.
Anticipated research and data-related challenges included data biases or robustness of methods, potential abuse of system or data, manipulated or fraud data, validity of the evidence, and resource- and scope-related challenges such as lack of time and expertise in HTA agencies.
Stakeholder discussions
Individual discussions were held with 25 stakeholders representing different roles or perspectives in the HTA process (5 from HTA agencies, 14 patient advocates, 1 academic, 5 from industry). The summary of the results of all discussions is included in Supplementary File 2.
The stakeholders generally confirmed the expectations outlined originally by the project team. Only few additional expectations or concerns were raised. Specifically, both patient and industry advocates emphasized that any guidance, data, and analytics should be accessible to all stakeholders including patient organizations, academic research, industry, and HTA researchers.
Likewise, patients and industry postulated that with acceptance of best practices for Social Media Research, HTA agencies should also formally accept this evidence as admissible for HTA or in specific HTA process steps (e.g., early scientific advice, discussion of relevant clinical trial endpoints, lifecycle management).
Across the board, the stakeholders advocated that evidence generated by Social Media Research is only seen as additional complementary information and should not replace any of the current forms of evidence or input. The findings should always be validated through other means.
On the other side, all stakeholders were concerned, that currently there is only inadequate and unsatisfactory protection in Social Media Research against manipulation or abuse of the system (e.g. snooping, astroturfing, misrepresentation of certain populations, false sites, disinformation, content or research generated through artificial intelligence, etc.).
Patient advocates were unsettled about using closed-groups communication without the consent of the participants. They warned that even if intimate information is openly shared among patients, some patients may not want to share personal information for research purposes.
HTA practitioners cautioned that HTA agencies typically neither initiate nor conduct new research but mostly review and appraise the evidence published or submitted in dossiers by the innovator or by patient organizations.
Credibility of social media research for HTA
All stakeholders saw potential value in Social Media Research. Patient advocates identified validation by other types of evidence (e.g., testimonials, surveys) as an important complement to strengthen the credibility of Social Media Research. The HTA stakeholders were more hesitant and emphasized that Social Media Research would always be assessed in specific contexts for its potential value. Such research could be used to identify themes and inspire new ideas, with the intention of follow-up with more robust methods to further explore those themes. For example, results from Social Media Research could inform the early scientific advice for certain health technologies or the initial phases of an HTA (scoping).
There was a high awareness that Social Media Research is already being performed in academic settings as well as driven by industry, platform owners, or patient organizations. Current use of Social Media Research as recalled by the stakeholders included info-demiology (‘internet-epidemiology’, the science of distribution and determinants of information in an electronic medium, specifically the Internet, or in a population, with the ultimate aim to inform public health and public policy (Reference Eysenbach18; Reference Eysenbach19)), info-veillance (public health surveillance through internet-based data sources containing unstructured information from multiple origins (Reference Eysenbach18; Reference Barros, Duggan and Rebholz-Schuhmann20)), pharmacovigilance, applications in market research, or sentiment research.
Opportunities and risks of social media research for HTA
Possible advantages or opportunities of Social Media Research were identified by all stakeholders and are summarized in Table 2. On the other hand, several risks or concerns related to Social Media Research are reported in Table 3.
Table 2. Perceived advantages of social media research identified by stakeholders in the framing phase
Abbreviations: HTA, health technology assessment; PICO, patient–indication–comparator–outcomes.
Table 3. Perceived risks of social media research identified by stakeholders in team and stakeholder discussions
Scoping reviews
Social media research methods
Of 135 references of potential interest (published between 2016 and April 2021), 75 were included in the analysis. The key findings from the analysis are summarized in Table 4 and, more detailed, in the online Supplementary File 3.
Table 4. Summary of findings of the literature search on methods for social media research (N = 75)
Some potential use cases for Social Media Research with insights useful for HTA were reported such as info-veillance, info-demiology, pharmacovigilance, public-perception research, and patient need identification. Most of the current Social Media Research is observational in nature and is assisted by some form of advanced analytic techniques to avoid resource-intense manual browsing and identification. Some challenges related to the accuracy of automated procedures for Social Media Research, data pattern identification, as well as data sourcing remain to be addressed. Due to the seemingly public nature of the data as well as novel research pathways, ethical review is rarely done – an area that warrants further guidance.
In addition, such research is done by a range of researchers with diverse professional backgrounds and value frameworks such as IT-related sciences, clinical research, or behavioral/social sciences.
Our findings are consistent with the recommendations from other recent reviews (Reference Bour, Schmitz and Ahne21; Reference Polisena, Andellini and Salerno22), which concluded that “the outcomes of these applications should be considered exploratory” (in the context of sentiment analysis) (Reference Polisena, Andellini and Salerno22) but also suggested that the current body of research should be used to inform further development of more structured and consistent methods as well as guidance (Reference Bour, Schmitz and Ahne21). Therefore, a second, separate output from this project is an initial guidance or framework to support future producers and users of Social Media Research in quality assurance published in this same journal (Reference Holtorf, Danyliv and Krause17).
However, constant updating will be needed to cater for new developments. For example, some of the limitations listed in Table 4 are already being addressed through newer approaches such as improved recognition of fake reports (Reference Fu, Peng and Liu23) or using pretrained models. Nevertheless, despite advanced methods, diligence in external validation and model re-calibration before implementation are still recommended for ensuring clinical meaningfulness (Reference Celi, Cellini and Charpignon24).
Use of social media (research) in HTA
Of 580 potentially relevant references from the years 2016–2022, 23 were retained for inclusion after screening of title, abstract, and full text. Key findings of the analysis are summarized in Table 5. A more in-depth report is available in the Supplementary File 4.
Table 5. Results of the literature review on case examples of the use of social media research for HTA or healthcare decision making (n = 23)
Abbreviations: COPD, chronic obstructive pulmonary disease; CDC, center for disease control; FDA, food and drug agency.
No specific case was identified within the time frame of the search where Social Media Research has directly impacted HTA.
However, social media discussions have impacted healthcare-related decisions in not-evidence-based ways: Fadaee et al. warned that interest-driven negative anti-hernia mesh communication in social media -- largely driven by lawyers in the USA (three of the five top tweeters were linked to law firms involved in mesh-based lawsuits) -- might provoke a negative FDA decision, as it has happened before with pelvic/vaginal mesh (Reference Fadaee, Huynh and Towfigh15).
The FDA, academic researchers, and the European WEB-RADR consortium have started to use Social Media Research for adverse event detection. WEB-RADR concluded that methods and data are not yet robust enough for standard use in pharmaco-vigilance (Reference van Stekelenborg, Ellenius and Maskell25) and strongly recommended to always corroborate findings from Social Media Research through other means.
In other examples, sentiment analysis revealed information on people’s attitudes and motivations with the purpose of informing public health policies or communication. Pharmaceutical industry has used Social Media Research to better understand patient experiences and needs and to inform the early dialog processes with HTA agencies for the choice of patient-relevant endpoints (Reference Cook, Cave and Holtorf13). Although the data may have been included in the submission dossiers for reimbursement, how these data are in fact used by different HTA agencies, is unclear.
Many potential applications of Social Media Research in healthcare policy-making, −implementation or -communication are emerging, but not much of this emerging evidence is currently routinely used in HTA and no example has been identified, where HTA agencies have commissioned Social Media Research.
Discussion
For the question asked at the outset of this work, there seems to be a broad consensus that Social Media Research has the promise and potential of revealing new and complementary information about patient experiences, needs, and behaviors. This evidence may help to test the potential benefit of new health technologies to the patients, which may in turn help to make more patient-relevant decisions about health technologies (26).
However, Social Media Research typically depends on secondary use of private communication data. Due to the rapid developments in the pathways of communication in social media and likewise, the means and methods for research using these data, not all risks to the data producers or the people they are communicating about are known. Special care should hence be taken for their protection throughout the lifecycle of Social Media Research. More detailed guidance emerged from the literature review of the ethical and legal aspects of Social Media Research and is published separately (Reference Holtorf, Danyliv and Krause17).
Since there are high expectations for the value and use of Social Media Research, it appears appropriate for the HTA community to proactively engage in the creation of a framework in which Social Media Research can be considered and used within HTA.
However, due to evolving methods for retrieval and analysis and the changing properties of the data sources no best practice has been defined yet. Examples of the use in policy and decision making are rare and currently, HTA agencies seem to be curious but careful if not hesitant in utilizing such evidence when evaluating new technologies. Reasons for such hesitancy are manyfold (e.g., lack of competence for qualitative research in the assessment team, current local HTA regulation driven by almost exclusive consideration of RCT-based evidence in some countries, distrust in robustness of methods, distrust relating to lack of justice and representativeness, distrust in quality of data, etc.). Reliable means are needed to determine the robustness of methods and meaningfulness of the results and thereby, facilitate using the evidence from Social Media Research for HTA. These should be addressed in a future guidance for researchers and HTA practitioners on Social Media Research to gain more experience with this type of evidence and address the potential resistance. An attempt to initiate such guidance is published separately in the same issue of this journal (Reference Holtorf, Danyliv and Krause17).
At this stage, however, considering that the methods, means, and data for such research are constantly refined and advanced, with continuing uncertainty around their robustness, it seems premature to develop a standard approach and hence, unlikely to happen soon. In contrast, we advise an individual ethical review of each research concept and plan to judge whether the expected benefits of the research will outweigh the potential risks or harms to those who have communicated on the social media (Reference Holtorf, Danyliv and Krause17).
Despite the increasing automatization of methods, most of the examples reviewed for this report still involve large amounts of ‘manual’ work which requires considerable expertise and resources. While HTA researchers are intensively trained for assessing and appraising clinical trial-based data and studies, they may be less experienced with qualitative data analysis and evidence (6; 8).
Consequently, future work should focus on guiding (i) Social Media Research, (ii) the use of the evidence originating from Social Media Research in HTA, and (iii) on learning from the experiences with the evidence generated through Social Media Research.
Limitations
Only limited resources were available for this initiative and the report can only be seen as an approximation to the subject rather than providing a final assessment. With increasing experience in Social Media Research and its use for guiding policy making, new opportunities might be recognized, which were not included in this work.
The scoping reviews are pragmatic and not exhaustive. However, sufficient saturation was reached in all three themes of the scoping reviews to inform and guide the recommendations provided by this report.
There was no expert on ethics or legal aspects among the team members. Due to the importance of these issues, it is recommended to include such experts in any future work. In addition, although being a multi-stakeholder team, the representation of some stakeholders was limited due to the small size of the team. For example, there was only one full-time academic researcher on the team. However, we hope that this paper is a starting point to bring more researchers (in all their diversity) in and start working intensively on this urgent subject, together with other stakeholders.
The work in this report mostly focused on the use of Social Media Research for assessing new technologies and less on the use of Social Media Research to inform the priorities or communication strategies of HTA agencies that aim at improving public and patient acceptance and perceptions of the value of HTA.
Recommendations
Best practices and a framework should be defined to help researchers, patient communities, and HTA practitioners or committees to ensure and determine the credibility of the results of Social Media Research in the context of a specific HTA.
To initiate this, a separate report proposes a set of principles for planning, conducting, and reporting the results of Social Media Research aiming to maximize the relevance and applicability and, in addition, to allow for assessing the legal and ethical integrity and protecting the individuals who communicate about themselves or are affected by the communication of others (Reference Holtorf, Danyliv and Krause17).
In addition, pilot programs including the planning, conduct and reporting of a Social Media Research study addressing a subject of interest to one or several HTA agencies should be organized in a multi-stakeholder consortium to test and improve the guidance initiated with this project. Stakeholders who should be part of such a consortium include HTA practitioners, case-relevant patient organization representatives, consumer representatives, academic researchers in outcomes research and health economics, methodological-analytical experts, and experts in patient-based evidence as well as ethicists.
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/S0266462323002593.
Acknowledgements
We would like to thank the stakeholders who shared their opinions, observations, and critical considerations relating to social media research with us for their invaluable input. The Steering Committee of the Patient and Citizen Involvement in HTA Interest Group (PCIG), particularly the Chair Ann Single, has continuously reviewed the progress of the work and given input or made helpful connections in her network. The following people have given input in specific phases of the project: Lindsey Lockhart and Jennifer Dickson (Scottish Medicines Consortium, Healthcare Improvement Scotland), and Jane Tsai (Vice CEO of the Formosa Cancer Foundation, Taiwan). We thank all additional stakeholders from different regions around the world who have reflected and challenged our concepts during the individual discussions: Belgium: Zsofia Bakonyi, Alexandra Moutet; Bulgaria: Stanimir Hasardzhiev; Canada: Christina Sit; Germany: Birgit Bauer; Hungary: Yvette Jakab; Ireland: Donna Walsh; Taiwan: Jane Tsai, Brenda Yen, Wingtin Li, Eric Liou Huan-Ruei; Switzerland: Christopher M de Rudolf, Sanja Njegic, Ina Meyer, Angels Costa, Guy Golan; UK incl. Scotland: Claire Donaghy, Brendan Barnes, Gill Halpin, Sally Hatton, Natalie Frankish; Ukraine: Oresta Piniazhko; US: John Lewis, Michael Mittelman, Elisabeth Oehrlein. Finally, we are grateful to the people who reviewed and commented on the report during the consultation phase. Funding Statement: The research underlying this paper received no specific grant from any funding agency, commercial or not-for-profit sectors. The writing of the manuscript was financially supported by Novartis (Switzerland).
Competing interest
The work presented in this manuscript is the result of a project of the Patient and Citizen Involvement in HTA Interest Group (PCIG) of Health Technology Assessment International (HTAi; www.htai.org) which has received no funding. As a multi-stakeholder team, the authors come from different type of organizations: HTA (Y. Venable, M. Pierre, L.-Y. Huang), Patient Organizations (A. Silveira Silva, Donna Walsh), Pharma & Med-Tech Industry (A. Danyliv, A. Krause, A. Hanna), Academia (J. Mattingly), and Consultancy (A.-P. Holtorf).
Open access
