Real-world evidence (RWE) is a useful supplement to a product's evidence base especially for medical devices, which are often unsuitable for randomized controlled trials. Generally, RWE is analyzed retrospectively (for example, healthcare records), which lack granularity for health-economic analysis. Prospective collection of RWE in hospitals can promote device-specific endpoint assessment. The advent of the General Data Protection Regulation (GDPR) requires a privacy-by-design approach. This work describes a workflow for a GDPR-compliant device-specific RWE collection as part of quality improvement initiatives (QII).

A literature review identifies relevant clinical and quality markers as endpoints to the investigated technology. A panel of experts grade these endpoints on their clinical significance, privacy sensitivity, analytic value, and feasibility for collection. Endpoints meeting a predefined cut-off are considered quality markers for the QII. Finally, an RWE data collection app is designed to collect the quality markers using either longitudinal, pseudonymized data or single time-point anonymized data to ensure data protection by design.

Using this approach relevant clinical markers were identified in a GDPR-compliant manner. The data collection app design ensured that patient data were protected, while maintaining minimum requirements on patient information and consent. The pilot QII collected data on over 5,000 procedures, which represents the largest single data set available for the tested technology. Due to its prospective nature this programme was the first to collect patient outcomes in sufficient quantity for analysis, while previous studies only recorded adverse events.

GDPR and RWE can co-exist in harmony. A design approach, which has data protection in mind from the start can combine high quality RWE collection of efficacy and safety data with maximum patient privacy.

The use of surrogate outcomes in health technology assessment (HTA) is increasing and methods for validating surrogate relationships have been published. However, these may not be fully applicable to ultra-rare diseases due to challenges such as scarcity of evidence and heterogenous populations. This study reviews and summarizes the use of surrogate outcomes and committee's considerations in the evaluations within the National Institute for Health and Care Excellence's (NICE) Highly Specialised Technology (HST) programme, which was established in 2013 in response to the challenges associated with the assessment of ultra-rare diseases.

All HST evaluation documents published before November 2020 were reviewed. Data extracted included surrogate outcomes used, rationales, the committee's considerations on the validity and generalizability of the surrogate relationships, related uncertainties, and other factors considered in decision-making.

Seven out of the eighteen published HST topics used surrogate outcomes. The rationale for most of the surrogate relationships focused on biological plausibility. Common concerns raised by the committee included the generalizability of the surrogate relationship to the condition of interest, the lack of validation, and inability to prove or quantify the magnitude of benefits associated with the surrogate relationships. In some topics, other aspects of the evidence and clinical/patient expert's opinions were also considered by the committee.

The use of surrogate outcomes is common in NICE HST evaluations and the challenges in supporting surrogate relationships with more than biological plausibility are recognized. However, our review indicates that, the committee considers more than just biological plausibility and will take into account other related factors.

Literature searching for evidence on apps in bibliographic databases is challenging because they are often described with inconsistent terminology. Information Specialists from the United Kingdom's National Institute for Health and Care Excellence (NICE) have developed validated search filters for retrieving evidence about apps from MEDLINE and Embase (Ovid) reliably.

Medical informatics journals were hand-searched to create a ‘gold standard’ set of app references. The gold standard set was divided into two sets. The development set provided the search terms for the filters. The filters were validated by calculating their recall against the validation set. Target recall was >90%.

A case study was then conducted to compare the number-needed-to-read (NNR) of the filters with previous non-validated MEDLINE and Embase app search strategies used for the ‘MIB214 myCOPD app’ NICE topic. NNR is the number of references screened to find each relevant reference.

The MEDLINE and Embase filters achieved 98.6 percent and 98.5 percent recall against the validation set, respectively. In the case study they achieved 100 percent recall, reducing NNR from 348 to 147 in MEDLINE and from 456 to 271 in Embase.

The novel NICE health apps search filters retrieve evidence on apps from MEDLINE and Embase effectively and more efficiently than previous non-validated search strategies used at NICE.

Human screening of title and abstracts in a systematic literature review (SLR) is labor intensive and time-consuming. In many instances, thousands of citations may be retrieved; the vast majority excluded upon screening. Text-mining semi-automates and accelerates screening by identifying patterns in relevant and irrelevant citations, as labelled by the screener. One such text-mining tool, Abstrackr, uses an algorithm within an active-learning framework to predict the likelihood of citations being relevant. The objective of this study was to assesses the performance of Abstrackr for title and abstract screening in an SLR of treatments for relapsed/refractory diffuse large B-cell lymphoma.

Citations identified from searches of electronic databases were imported to Abstrackr. An investigator-selected database of terms indicating relevance of title and abstract to the research question were uploaded. These terms were partly informed by the SLR inclusion/exclusion criteria. Citations deemed most relevant by Abstrackr were screened first (screening prioritization). Screening was carried out until a maximum prediction score of 0.4 or less, based on previous experience in the literature, was reached. Remaining citations were deemed unlikely to be relevant and did not undergo screening (screening truncation). Separately, a single-human screener screened all citations using Covidence.

A total of 7,723 citations and 154 initial terms were uploaded to Abstrackr. Of these citations, 2,572 (33 percent) were screened before a prediction score of 0.39 was reached. Compared to single-human screening (conducted on all citations), the workload saving associated with Abstrackr was 5 days. A total of 451 (6 percent) citations proceeded to full-text screening; ten (0.1 percent) were included in the final evidence base. No citations predicted to be irrelevant by Abstrackr were included in the final evidence base.

Text-mining tools such as Abstrackr have the potential to reduce workload associated with title and abstract screening, without missing relevant citations.

Drugs for relapsing-remitting multiple sclerosis (RRMS) are costly and not included in the National List of Essential Medicines of Thailand yet. This study aims to conduct an economic evaluation of high-cost drugs for RRMS.

The Markov model was used to estimate lifetime costs and quality-adjusted life years (QALYs) gained. The treatment options include Interferon beta-1a (IFN) and Teriflunomide (TERI) (first-line), Fingolimod (FIN) and Natalizumab (NATA) (second-line), and Alemtuzumab (ALEM) (third-line) compared with usual care. The effectiveness of drugs was retrieved by network meta-analysis. The probability of health state transition was obtained from primary data. Treatment-related costs were derived from the national database. Other costs and utilities were obtained from the study in Thai RRMS patients.

The lowest lifetime costs option was usual care (THB2 million) (USD65,808), while the highest QALY gained option was IFN-NATA-ALEM (8.6 QALY gained). All treatment options were not cost-effective compared with usual care at the threshold of THB160,000 (USD5,300) per QALY gained. However, the option of IFN-NATA-ALEM yielded the lowest incremental cost-effectiveness ratio (ICER), which was THB4.4 million (USD144,778) per QALY gained.

High-cost drugs were not cost-effective; nonetheless, the IFN-NATA-ALEM option could increase QALY gained with the lowest additional budget. The government should negotiate the price of IFN to decrease by eighty percent.

Extrapolation methods are commonly used to model the cost-effectiveness of health technologies beyond observed data. Reassessing cost-effectiveness estimates using updated clinical trial data has the potential to reduce uncertainty and optimize decision-making. We present a case study based on percutaneous repair (PR) with the Mitraclip system, a technology to treat severe secondary mitral regurgitation (MR). For the study purpose, we considered the COAPT trial that evaluated the effectiveness of adding PR to medical treatment versus medical treatment alone.

We developed a time-varying Markov model to assess the cost-effectiveness of PR. Clinical inputs were based on reconstructed individual patient data from the COAPT trial results reported at 2 years, and at 3 years.

We developed parametric modeling for overall survival (OS) and heart failure hospitalizations (HFH) to obtain clinically plausible extrapolations beyond observed data. We adopted the French perspective and used a 30-year time horizon. We expressed incremental cost-effectiveness ratios (ICERs) as cost per quality-adjusted life year (QALY).

Based on 2 year-data, preferred parametric models for OS and HFH were exponential and log-logistic respectively, yielding an ICER of EUR21,918/QALY and >0.5 probability of PR being cost-effective (EUR50,000/QALY threshold).

Updated analyses at 3 years showed a change of OS trajectory for PR that justified the use of piecewise modelling, yielding an updated ICER that went up to EUR77,904/QALY (base-case), and to a minimum of EUR58,175/QALY (scenario analysis). Using data at 3 years, PR had <0.5 probability of being cost-effective.

In this case study, the availability of updated survival analyses of the main trial is likely to have some impact on decision-making and/or pricing discussion as part of health-technology assessment (HTA). We aim to provide further updated analyses as 4 years results of the COAPT study become available.

More broadly, original technology appraisals are frequently undertaken when mid/long-term follow-up trial data may be lacking. Our example suggests the need for continuous HTA review as new clinical data are released.

The idea of using extracorporeal cytokine adsorption therapy (ECAT) is to remove cytokines from the blood in order to restore a balanced immune response. Yet, it is unclear as to whether the use of ECAT improves patient-relevant outcomes. Hence, the aim of this article is to synthesize the currently available evidence with regard to a potential clinical benefit of ECAT used in cardiac surgery or sepsis.

We conducted an updated systematic review summarizing the body of evidence with regard to a potential clinical benefit of ECAT. The study followed the PRISMA statement and the European Network for Health Technology Assessment (EUnetHTA) guidelines. The quality of the individual studies and the strength of the available evidence was assessed using the Cochrane risk of bias tool (v.1) and the GRADE approach respectively. Mortality, organ function, length of stay in the intensive care unit and length of hospitalization, as well as adverse events, were defined as critical outcomes.

For the preventive treatment of ECAT in patients undergoing cardiac surgery, we found very low-quality inconclusive evidence for mortality (5 randomized controlled trials (RCTs), n = 163), length of stay in the intensive care unit (5 RCTs, n = 163), and length of hospitalization (3 RCTs, n = 101). In addition, very low-quality inconclusive evidence was found for (serious) adverse events (4 RCTs, n = 148). For the therapeutic treatment of ECAT in patients with sepsis/ septic shock, we found very low-quality inconclusive evidence for mortality up to 60-day follow-up (2 RCTs, n = 117), organ function (2 RCTs, n = 117) and length of stay in the intensive care unit (1 study, n = 20). Similarly, very low-quality inconclusive evidence was found for (serious) adverse events (2 RCTs, n = 117). There are currently eighteen ongoing RCTs on the use of ECAT.

There is a lack of reliable data on the clinical benefit of using ECAT as an add-on treatment preventively in cardiac surgery and therapeutically in patients with sepsis or septic shock. While theoretical advantages are anticipated, the current available evidence is inconclusive and was not able to establish the efficacy and safety of ECAT in combination with standard care in the investigated indications. In light of the available RCTs, we strongly recommend the consideration of studies with patient-relevant endpoints and adequate statistical power, instead of investing further research funds on small studies that may not shed more light onto the potential clinical benefit of ECAT. The results of ongoing RCTs are awaited to guide the decision on whether further research funds should be invested in ECAT research or to conclude that the intervention may not show clinical benefits for patients.

The sequential use of alternative treatments for chronic conditions represents a complex, dynamic intervention pathway; previous treatment and patient characteristics affect both choice and effectiveness of subsequent treatments. Evidence synthesis methods that produce the least biased estimates of treatment-sequencing effects are required to inform reliable clinical and policy decision-making. A comprehensive review was conducted to establish what existing methods are available, outline the assumptions they make, and identify their shortcomings.

The review encompassed both meta-analytic techniques and decision-analytic modelling, any disease condition, and any type of treatment sequence, but not diagnostic tests, screening, or treatment monitoring. It focused on the estimation of clinical effectiveness and did not consider the impact of treatment sequencing on the estimation of costs or utility values.

The review included ninety-one studies. Treatment-sequencing is usually dealt with at the decision-modelling stage and is rarely addressed using evidence synthesis methodology for clinical effectiveness. Most meta-analyses are of discrete treatments, sometimes stratified by line of therapy. Prospective sequencing trials are scarce. In their absence, there is no single best way to evaluate treatment sequences, rather there is a range of approaches, each of which has advantages and disadvantages and is influenced by the evidence available and the decision problem. Due to the scarcity of data on sequential treatments, modelling studies generally apply simplifying assumptions to data on discrete treatments. A taxonomy for all possible assumptions was developed, providing a unique resource to aid the critique of decision-analytic models.

The evolution of network meta-analysis in HTA demonstrates that clinical and policy decision-making should account for the multiple treatments available for many chronic conditions. However, treatment-sequencing has yet to be accounted for within clinical evaluations. Economic modelling is often based on the simplifying assumption of treatment independence. This can lead to misrepresentation of the true level of uncertainty, potential bias in estimating the effectiveness and cost effectiveness of treatments and, eventually, the wrong decision.

Health technology assessment (HTA) can play a key role in evidence-based decision-making. However, HTA requires resources that might be lacking in low-income settings. To test the feasibility of adapting existing evidence as part of the HTA process, this project evaluated the effectiveness and economic impact of breast cancer screening programs for women over 40 years in the West Bank, where mammography screening is provided for free in governmental clinics.

We conducted a search for systematic reviews, HTAs and guidelines in electronic databases. We included the most recent global systematic review and meta-analysis that fulfilled our inclusion criteria. The European Network for Health Technology Assessment (EUnetHTA) adaptation toolkit was used to guide adaptation and undertake a budget impact analysis of the economic impact of mammography screening. We build capacity by working as a team of HTA experts and first-time HTA researchers. The results were disseminated to raise awareness for HTA.

The European Commission Guidelines on Breast Cancer Screening were identified as most recent global systematic review with meta-analyses, out of 2,365 references. The adapted evidence may inform policies on screening in the West Bank. Our experience is that adaption requires extensive skills and resources, including finding, assessing, and adapting relevant evidence. The EUnetHTA toolkit is useful, but also adds to the workload. Furthermore, local stakeholder engagement is important in topic selection, to access information, and to contextualize global evidence to the local setting.

This study is currently ongoing, but preliminary findings show that producing an HTA by adapting existing evidence in resource-limited settings is feasible. There is a need for nuanced guidance on transferability of evidence from other settings. Future studies should investigate innovative methods to optimize the adaption process. Capacity building in adaptation is important to ensure the production of quality HTA products. Inclusion of local team members and stakeholders is important for future development of HTA in the region.

A national act (Order SSI/1356/2015) regulating Post-Launch Evidence Generation (PLEG) studies was set in Spain in 2015. These PLEG studies are to inform decisions about technologies already included in the Benefit Portfolio of the Spanish National Health System (SNHS) in order to confirm/exclude/modify their terms of use. Once a PLEG is established the selected hospitals provide the technology according to a common protocol and register outcomes until the required sample size is reached.

The PLEG studies are prospective, observational and single arm studies on safety, effectiveness and cost-effectiveness of a technology in real practice. The technology is selected because of the identification of an evidence gap, usually through a health technology assessment (HTA) report made by an agency of the Spanish Network of HTA Agencies (RedETS). The execution of a PLEG is assigned to one of the RedETS Agencies, which is responsible of delivering annual reports and a final report when the objectives are reached.

The following six PLEG studies, all of them on medical devices, have been launched in Spain so far, i) Endobronchial valve for patients with persistent air leak; ii) Biodegradable esophageal stent; iii) Percutaneous mitral valve repair system by clip; iv) Left Atrial Appendage Closure Device; v) Sensor-based glucose monitoring systems for children with type 1 diabetes mellitus; vi) Left ventricular assist devices for destination therapy. Five studies will finish their data collection by the end of 2020 or during 2021.

A new national procedure using PLEG has been made available in Spain facilitating the use of real-world evidence to inform national decision-making on the financing of selected technologies due to uncertainties about their effectiveness, safety, cost-effectiveness and organizational impact. The studies are requiring a high amount of coordination tasks, as they are involving an average of 21 hospitals each. The usefulness and suitability of this procedure to achieve its objectives must be evaluated once their results are available.

Clinical decision support systems (CDSS) are being developed to support evidence-based antibiotic prescribing and reduce the risk of inappropriate or over-prescribing; however, adoption of CDSS into the health system is rarely sustained. We aimed to understand the implementation challenges at a macro (policymakers), meso (organizational) and micro-level (individual practices) to identify the drivers of CDSS non-adoption.

We have adopted a mixed-method study design which comprised of: (i) systematic review and meta-analysis to assess the impact of CDSS on appropriate antibiotic prescribing, (ii) Online survey of clinicians in Australia from hospitals and primary care to identify drivers of CDSS adoption and (iii) in-depth interviews with policymakers to evaluate policy-level challenges and opportunities to CDSS implementation.

CDSS implementation can improve compliance with antibiotic prescribing guidelines, with a relative decrease in mortality, volume of antibiotic use and length of hospital stay. However, CDSS provision alone is not enough to achieve these benefits. Important predictors of clinicians’ perception regarding CDSS adoption include the seniority of clinical end-users (years), use of CDSS, and the care setting. Clinicians in primary care and those with significant clinical experience are less likely to use CDSS due to a lack of trust in the system, fear of comprising professional autonomy, and patients’ expectations. Lack of important policy considerations for CDSS integration into a multi-stakeholder healthcare system has limited the organizational capacity to foster change and align processes to support the innovation.

These results using multiple lines of evidence highlight the importance of a holistic approach when undertaking health technology management. There needs to be system-wide guidance that integrates individual, organizational and system-level factors when implementing CDSS so that effective antibiotic stewardship can be facilitated.

The clinical data is increasing at a considerably higher rate than the capacity of the healthcare system and clinicians to manage this data. Digital tools such as clinical decision support systems (CDSS) provide opportunities for evidence-based patient care by intelligently filtering and presenting the information required for clinical decision making at the point of care. Despite the success of pilot projects, CDSS have had limited implementation in broader health systems. We aimed to identify challenges faced by policymakers for CDSS implementation and to provide policy recommendations.

We conducted eleven semi-structured interviews with Australian policymakers from state and national committees involved in digital health activities. The data were analyzed using reflexive thematic analysis to identify policy priorities.

Our findings indicate that fragmentation of care processes and structures in the digital health ecosystem is one of the main impediments to delivering coordinated care using CDSS. Five themes for policy action were identified: (i) establishing a shared conceptual framework for user-centered design of CDSS that is aligned with stakeholders’ priorities, (ii) maintaining the right balance between the customization and standardization of systems, (iii) developing mutually agreed semantic interoperability standards at the local, state and national level, allowing generation and exchange of information across the health system without changing its context and meaning, (iv) reorienting organizational structures to build capacity to foster change, and (v) developing collaborative care models to avoid conflicting interests between stakeholders.

Findings highlight the importance of developing system-wide guidance to establish a clear vision for CDSS implementation and alignment of organizational processes across all levels of health care. There is a need to build a shared policy framework for modelling the innovative activities such as CDSS implementation across the digital health landscape which minimizes the operational and strategic fragmentation of different organizations.

The German health technology assessment (HTA) rejected additional benefit of alectinib for second line (2L) ALK+ NSCLC, citing possible biases from missing ECOG performance status data and unmeasured confounding in real-world evidence (RWE) for 2L ceritinib that was submitted as a comparator to the single arm alectinib trial. Alectinib was approved in the US and therefore US post-launch RWE can be used to evaluate this HTA decision.

We compared the real-world effectiveness of alectinib with ceritinib in 2L post-crizotinib ALK+ NSCLC using the nationwide Flatiron Health electronic health record (EHR)-derived de-identified database. Using quantitative bias analysis (QBA), we estimated the strength of (i) unmeasured confounding and (ii) deviation from missing-at-random (MAR) assumptions needed to nullify any overall survival (OS) benefit.

Alectinib had significantly longer median OS than ceritinib in complete case analysis. The estimated effect size (Hazard Ratio: 0.55) was robust to risk ratios of unmeasured confounder-outcome and confounder-exposure associations of <2.4.

Based on tipping point analysis, missing baseline ECOG performance status for ceritinib-treated patients (49% missing) would need to be more than 3.4-times worse than expected under MAR to nullify the OS benefit observed for alectinib.

Only implausible levels of bias reversed our conclusions. These methods could provide a framework to explore uncertainty and aid decision-making for HTAs to enable patient access to innovative therapies.

The COVID-19 pandemic led to a significant surge in clinical research activities in the search for effective and safe treatments. Attempting to disseminate early findings from clinical trials in a bid to accelerate patient access to promising treatments, a rise in the use of preprint repositories was observed. In the UK, NIHR Innovation Observatory (NIHRIO) provided primary horizon-scanning intelligence on global trials to a multi-agency initiative on COVID-19 therapeutics. This intelligence included signals from preliminary results to support the selection, prioritisation and access to promising medicines.

A semi-automated text mining tool in Python3 used trial IDs (identifiers) of ongoing and completed studies selected from major clinical trial registries according to pre-determined criteria. Two sources, BioRxiv and MedRxiv are searched using the IDs as search criteria. Weekly, the tool automatically searches, de-duplicates, excludes reviews, and extracts title, authors, publication date, URL and DOI. The output produced is verified by two reviewers that manually screen and exclude studies that do not report results.

A total of 36,771 publications were uploaded to BioRxiv and MedRxiv between March 3 and November 9 2020. Approximately 20–30 COVID-19 preprints per week were pre-selected by the tool. After manual screening and selection, a total of 123 preprints reporting clinical trial preliminary results were included. Additionally, 50 preprints that presented results of other study types on new vaccines and repurposed medicines for COVID-19 were also reported.

Using text mining for identification of clinical trial preliminary results proved an efficient approach to deal with the great volume of information. Semi-automation of searching increased efficiency allowing the reviewers to focus on relevant papers. More consistency in reporting of trial IDs would support automation. A comparison of accuracy of the tool on screening titles/abstract or full papers may help to support further refinement and increase efficiency gains.

This project is funded by the NIHR [(HSRIC-2016-10009)/Innovation Observatory]. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.

Physician preference items (PPIs) are high-cost medical devices on which clinicians express firm preferences with respect to a particular manufacturer and a specific product. The aim of this research is to understand what are the most important factors, as well as their relative importance, in the choice of new PPIs (that is, hip or knee prosthesis) adoption on behalf of orthopaedic clinicians in Italy.

Based on a literature review and clinical experts’ opinions, we identified a number of key factors (for example, health technology assessment (HTA) recommendation) and their corresponding levels (for example positive HTA recommendation). We administered an online survey to hospital orthopaedists using two experimental techniques for preference elicitation (that is, discrete choice experiment (DCE) and case 1 best-worst scaling (BWS)). BWS data were analysed through descriptive statistics (that is, best-minus-worst score) and conditional logit model. A mixed logit model was applied to DCE data, and a willingness-to-pay (WTP) was estimated. All analyses were conducted using Stata 16.

A total of ninety orthopaedists (95% male; mean age: 52.8 years) were enrolled in the survey. In BWS, the most important factor was ‘clinical evidence’, followed by ‘quality of products’, ‘HTA recommendations’ and ‘previous experience’, while the least important was ‘cost’. DCE results suggested that orthopaedists prefer high-quality products with robust clinical evidence, positive HTA recommendation and affordable cost, and for which clinicians have a consolidated experience of use and a good relationship with the sales representative. The WTP for a high-quality product was estimated at EUR1,733, and for a good relationship at EUR2,843.

This is the first study aimed at analysing the multidimensionality of clinician's decision-making process in selecting new PPIs in orthopaedics in Italy. Despite the quality of products being declared as one of the most important dimensions in BWS, when other factors populate a hypothetical DCE scenario, physicians are not willing to accept quality at any cost (for example, high quality and very bad support from the producer or with uncertain clinical evidence).

Early identification of innovative medicines is crucial for timely health technology assessment (HTA) and efficient patient access. The National Institute for Health Research Innovation Observatory (NIHRIO) identifies, monitors and notifies key HTA stakeholders in England of ‘technologies’ (innovative medicines) within three to five years of regulatory approval. Increasing numbers of innovative medicines and significant uncertainties in clinical and regulatory pathways are major challenges in the monitoring and notification process. An active monitoring framework using pre-defined predictive criteria has previously been developed. This framework provides a standardized and consistent process, but is highly resource-intensive, requiring manual review of individual records.

Using the previous active monitoring framework, a scoring matrix was calculated and used to prioritize individual technologies using available data in the NIHRIO database: estimated regulatory timelines, regulatory awards/designations, innovative medicine type (for example gene therapies) and clinical trial phase, completion dates and results. A threshold for automatic and manual reviewing of technologies was developed and tested by NIHRIO analysts.

The scoring system identified approximately ninety percent of technologies meeting the threshold for semi-automated reviewing. The review period for these technologies are set automatically according to predefined criteria depending on data availability. The review periods are updated automatically until the record reaches the threshold that triggers manual reviewing. The remaining ten percent had estimated regulatory timelines necessitating the need for manual reviewing and early engagement with companies to verify regulatory timelines and/or notify HTA stakeholders.

Preliminary analysis indicates that each technology is routinely and automatically updated. The semi-automatic updating represents a significant improvement in the efficiency of the monitoring of the large volume of technologies on the NIHRIO database. Ongoing work is being undertaken to further refine, pilot and test the system.

This project is funded by the NIHR [(HSRIC-2016-10009)/Innovation Observatory]. The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care.

There is increasing pressure to rapidly shape policies and inform decision-making where robust evidence is lacking. This work aimed to explore the value of soft-intelligence as a novel source of evidence. We deployed an artificial intelligence based natural language platform to identify and analyze a large collection of UK tweets relating to mental health during the COVID-19 pandemic.

A search strategy comprising a list of terms relating to mental health, COVID-19 and the lockdown was developed to prospectively identify relevant tweets via Twitter's advanced search application programming interface. We used a specialist text analytics platform to explore tweet frequency and sentiment across the UK and identify key topics of discussion for qualitative analysis. All collated tweets were anonymized.

We identified 380,728 tweets from 184,289 unique users in the UK from 30 April to 4 July 2020. The average sentiment score was fifty-two percent, suggesting overall positive sentiment. Tweets around mental health were polarizing, discussed with both positive and negative sentiment. For example, some people described how they were using the lockdown as a positive opportunity to work on their mental health, sharing helpful strategies to support others. However, many people expressed the damaging impact the pandemic (and resulting lockdown) was having on their mental health, including worsening anxiety, stress, depression, and loneliness.

The results suggest that soft-intelligence is potentially a useful source of evidence. The approach taken to identify and analyze this data may offer an efficient means of establishing key insights from the ‘public voice’ relating to critical health issues. However, there are still various limitations to consider concerning the technology and representativeness of the data. Future work to explore this type of evidence further, and how it might formally support decision-making processes, is recommended.

This project is funded by the NIHR [(HSRIC-2016-10009)/Innovation Observatory]. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.

Synthesis of clinical effectiveness is a well-established component of health technology assessment (HTA) combining data from multiple trials to obtain an overall pooled estimate of clinical effectiveness, which may inform an associated economic evaluation. Time-to-event outcomes are often synthesized using effect measures from Cox proportional hazards models assuming a constant hazard ratio over time. However, where treatment effects vary over time an assumption of proportional hazards is not always valid. Several methods have been proposed for synthesizing time-to-event outcomes in the presence of non-proportional hazards. However, guidance on choosing between these methods and the implications for HTA is lacking.

We applied five methods for estimating treatment effects from time-to-event outcomes, which relax the proportional hazards assumption to a network of melanoma trials, reporting overall survival: restricted mean survival time, an accelerated failure time generalized gamma model, piecewise exponential, fractional polynomial and Royston-Parmar models. We conducted a simulation study to compare these five methods. Simulated individual patient data was generated from a mixture Weibull distribution assuming a treatment-time interaction. Each simulated meta-analysis consisted of five trials with varying numbers of patients and length of follow-up across trials. For each model fitted to each dataset, we calculated the restricted mean survival time at the end of observed follow-up and following extrapolation to a 20-year time horizon.

All models fitted the melanoma data reasonably well with some variation in the treatment rankings and differences in the survival curves. The simulation study demonstrated the potential for different conclusions from different modelling approaches.

The restricted mean survival time, generalized gamma, piecewise exponential, fractional polynomial and Royston-Parmar models can all accommodate non-proportional hazards and differing lengths of trial follow-up within an evidence synthesis of time-to-event outcomes. Further work is needed in this area to extend the simulation study to the network meta-analysis setting and provide guidance on the key considerations for informing model choice for the purposes of HTA.

The Scottish Medicines Consortium (SMC) conducts early health technology assessment (HTA) of new medicines on behalf of NHSScotland. Evidence from patients and carers on end-of-life and orphan medicines is gathered during Patient and Clinician Engagement (PACE) meetings. The output is a consensus statement describing a medicine's added value from the perspective of patients/carers and clinicians, which is used by SMC committee members in decision-making. This study compared the importance of factors in the PACE statement to PACE participants and committee members.

A survey of ninety-eight PACE participants (consisting of forty-two patient group (PG) representatives and fifty-six clinicians) investigated the importance of quality of life (QoL) themes (family/carer impact, health benefits, tolerability, psychological benefit, hope, normal life, treatment choice and convenience) identified from an earlier thematic analysis of PACE statements. The findings from PG representatives and clinicians were compared, and the overall results were further compared with those from a previous survey of committee members (n = 26).

Among PACE participants who responded (twenty-six PG representatives and fourteen clinicians), 100 percent rated ‘health benefits’ and ‘ability to take part in normal life’ as important / very important. ‘Convenience of administration’ and ‘treatment choice’ received the lowest rating with fifteen percent and nineteen percent respectively of PG representatives versus seven percent of clinicians rating each as very important. ‘Hope for the future’ received the most diverse response with fifty-eight percent of PG representatives and fourteen of clinicians rating this as very important.

In general, PACE participants rated importance of QoL themes higher than committee members (n = 21) but the rank order was similar. Differences between the proportion of PACE participants and committee members who rated themes important/very important was greatest for ‘treatment choice’ (sixty-seven percent versus twenty percent respectively) and ‘hope for the future’ (eighty-two percent versus fifty-three percent).

The findings demonstrate some alignment between PACE participants’ and committee members’ responses, supporting the value of the PACE output in decision-making. Areas for further research are highlighted.