The National Institute for Health and Care Excellence (NICE) intends to increasingly use real-world evidence in developing guidance. To increase trust in such evidence, NICE has developed a framework for developing and assessing real-world evidence studies, including understanding the value of the selected data source for the decision problem.

Starting with published high-quality studies about data quality, we developed a conceptual model of the elements needed to understand the quality of a data source. Results from a literature search were then mapped to the model. We used this to design a structured reporting tool, the data Suitability Assessment Tool (dataSAT), and tested it in several cases studies. Additionally, we engaged with internal and external stakeholders to obtain feedback on the tool and revised it accordingly.

DataSAT covers provenance of the data, assessment of data quality, and the data’s relevance to the research question. For data provenance, information is requested about the data source independent of the study’s interests, including the purpose, setting, dates of operation, funding, data specification, and management and quality assurance plans for the data sources. Data quality is covered by quantitively assessing the completeness and accuracy of the following key study elements to inform critical appraisal of the study: population inclusion and exclusion criteria; intervention; comparator; and outcomes and key covariates. The findings on data sources and data quality are then interpreted in terms of relevance to the decision problem. This includes relevance to the population in the United Kingdom, the treatment pathway and care setting, the availability of key study elements, time-related factors such as length of follow up, and the effects of sample size and missing data on the validity of findings.

DataSAT allows summary information on source data, including quality and relevance, to be reported in a structured manner, enabling decision makers to better understand how the data influence the robustness of analyses used in health technology assessment. This helps increase trust in the use of real-world evidence.