Big data can facilitate novel study designs

Some would argue that the gold standard for evidence is the well-conducted randomised controlled trial (RCT). Yet, in many situations it is challenging or even impossible to conduct this type of study. For example, for rare outcomes such as suicide following an episode of self-harm, it can be difficult to ensure that sample sizes are adequate or that the follow-up period is long enough to detect the possible beneficial effects of an intervention.

In this context, large routine electronic datasets may help in assessing which types of intervention are beneficial, even if an RCT is not possible. For example, Erlangsen and colleagues (Reference Erlangsen, Lind and Stuart2015) used whole-population data from Denmark to assess the role of a psychosocial intervention in reducing subsequent completed suicide risk in a national sample of people who had self-harmed. The authors used propensity scores – an approach that can be used in observational data that involves matching on variables that predict outcomes, which leads to a replication of the balance that is normally achieved in well-conducted RCTs. By achieving this balance in a cohort of individuals who had self-harmed (with data on 5678 individuals who had received the psychosocial therapy following a self-harm episode and 17 034 individuals who had not) the investigators were able to establish that receiving a psychosocial intervention that focused on suicide prevention after an initial episode of self-harm reduced the risk of repeated self-harm episodes and death by any cause 1 year after the index event and was also associated with a reduced risk of repeated self-harm, death by suicide and deaths by any cause 5–20 years after the intervention.

Studies such as this are clearly a powerful example of how routine electronic data on large samples might be applied to contexts where standard RCTs may not be feasible. Erlangsen and colleagues do caution that, despite the sophisticated methodologies employed, selection bias, as well as a lack of more detailed information on what the ‘psychosocial therapy’ entailed, are potential limitations for their study (Erlangsen Reference Erlangsen, Lind and Stuart2015), yet work like this highlights the possibilities of big data in making important contributions to the mental health evidence base.

Big data can enhance study recruitment

Recruitment to clinical trials can be difficult, with specific challenges relating to the recruitment of people with mental disorders (Howard Reference Howard, de Salis and Tomlin2009). This may partly be a function of clinicians acting as gatekeepers, to the extent that patients may not be offered the opportunity to participate in studies, even if they wish to do so (Callard Reference Callard, Broadbent and Denis2014; Patel Reference Patel, Oduola and Callard2017). Details of one innovative system to enhance the recruitment of patients into research studies, which was developed in partnership with patients and carers and is based on an anonymised electronic health record system, is highlighted Box 1. Despite the challenges outlined in Box 1, the ‘consent for contact’ (C4C) system is an extremely innovative example of what may be possible using large electronic health record resources, with models developed in partnership with patients and carers.

Big data can enhance RCTs

Embedding well-designed RCTs within everyday clinical practice, where electronic health records have fully replaced paper-based systems for medical note-keeping (Gulliford Reference Gulliford, van Staa and McDermott2014; McIntosh Reference McIntosh, Stewart and John2016), so that there can be ‘randomisation at point of routine care’ (van Staa Reference van Staa, Goldacre and Gulliford2012) is another innovative study design that is yet to be fully realised in psychiatry. As mental health trusts increasingly move towards fully electronic medical records, such a ‘mixed design’ which intermeshes the clinical trial with automated data collection (including data on outcomes such as adverse events, has obvious logistic and cost advantages. For example, potential participants for research trials may be followed up through the data routinely noted on the electronic health record, which may be of particular value for adverse events (van Staa Reference van Staa, Goldacre and Gulliford2012). This type of study design has already been employed in trials of antibiotic prescriptions and stroke prevention (Gulliford Reference Gulliford, van Staa and McDermott2014) and it might also be suitable for studies of mental health interventions.

Big data can enhance health records through data linkage

Unlinked data from health records may be missing important information, which could hamper analyses. Frequently, important indicators of health outcomes and important sociodemographic variables such as ethnicity are poorly recorded or of variable quality (Bhopal Reference Bhopal, Fischbacher and Povey2011). Linked datasets (Box 2) allow the possibility of bringing in information from various sources to create large cohorts or datasets of individuals with less common conditions on a scale that is difficult to otherwise achieve in traditional epidemiological studies. This is partly because traditional epidemiological studies may be hampered by challenges of recruitment, loss to follow-up/attrition and falling participation rates (Knudsen Reference Knudsen, Hotopf and Skogen2010). The linkage of data to routine sources additionally helps to ‘plug the gap’ if important indicators, such as self-ascribed ethnicity, can be brought in via the linkage (Bhopal Reference Bhopal, Fischbacher and Povey2011). For example, a linkage of health data to census records in Scotland (SHELS) highlighted ethnic minority mental health inequalities specific to the devolved Scottish context (Bhopal Reference Bhopal, Fischbacher and Povey2011; Bansal Reference Bansal, Bhopal and Netto2014). Traditional studies using unlinked routine data would not have been able to achieve this, as ethnicity was not routinely recorded in Scottish health records at the time of the study. In England, the linkage of death certificate information to records from mental health trusts have informed our understanding of premature mortality in severe mental illness (Chang Reference Chang, Hayes and Perera2011; Das-Munshi Reference Das-Munshi, Chang and Dutta2017) as well as conditions such as chronic fatigue syndrome (using the CRIS register; Roberts Reference Roberts, Wessely and Chalder2016). The linkage in these examples enabled a sample size that allowed sufficiently powered analyses.

Big data cannot replace statistical analysis

A common misconception when presented with data collected for the entire population is that we no longer need to be concerned about the statistical significance of our findings. Statistical theory presupposes that the data we are analysing can be treated as a random sample of the overall population of interest. Therefore, it is often assumed that if we know the health outcomes for the entire population then we no longer have to worry about burdensome statistical calculations. We could instead simply give the percentage of people with, say, a diagnosis of schizophrenia who were exposed to some risk factor and the percentage not exposed and assume that this covers everything. However, research is rarely about what has already occurred. Instead, we intend that research findings are relevant for future situations and allow us to develop some overarching theory. Even the most complete population data will only ever comprise a subset of all possible instances of the phenomenon of interest and therefore statistical methods are needed to account for this.

Big data cannot predict the future

This brings us to arguably the most common example of ‘big data hubris’: that the more data we collect the more likely we will be able to accurately predict future events. A good example of this, that is often cited, is the case of Google Flu Trends (Box 3). Although it appears that Google has abandoned this project, others believe that this is far from the end of the story. One wide-ranging review argues that similar algorithms could be successful, although they would require constant updating and improvement and should ideally be used alongside other epidemiological tools (Lazer Reference Lazer, Kennedy and King2014).

Big data cannot make up for the absence of theory

Along with the initial wave of enthusiasm about big data came the idea that ‘text mining’ of large datasets to find relevant patterns was methodologically valid in itself. Concerns about causality and the reasoning behind these algorithms were seen as irrelevant as long as the algorithms worked, as was the case with Google Flu Trends (Box 3) (Mayer-Schönberger Reference Mayer-Schönberger and Cukier2013). This is the logic behind ‘machine-learning’ approaches. In machine learning a ‘training’ dataset is used to develop an algorithm from a large set of often arbitrary variables. This algorithm is then applied to another set of test data until an optimum predictive tool is arrived at. This kind of ‘black box’ approach is therefore essentially atheoretical – the authors do not need to know why the algorithm works, simply that it does work when applied to the test data. It is not hard to see how this could be attractive to mental health research, where many fundamental questions about aetiology remain unanswered. Instead of trying to determine the mechanism that might lead to, say, increased rates of schizophrenia among migrants, a simpler approach might be simply to arrive at a predictive tool by determining patterns in available data. In fact, in the case of Google Flu Trends, the algorithms themselves were never made public, so it was impossible to determine why they were ever successful in the first place, or why they subsequently underperformed. Although this approach has enormous advantages in many fields, for example in machine translation and text mining, it is highly problematic in epidemiological research, as Google Flu Trends demonstrated.

Big data cannot always be taken at face value

Unlike research data, administrative data seldom come with documentation explaining how the data were collected or how categories used in the coding were arrived at. For example, if we are interested in rates of mental disorder for different ethnic groups, with survey data we can determine whether ethnicity is self-reported or not, as well as the categories used in the original questionnaire. However, if we look at health records, in the UK, it is often impossible to say who provided the ethnic classification or how it was originally coded. This could be a problem if we tried to determine ethnic health differences between different areas, but were unable to distinguish between differences in coding methods and underlying health differences.

Often, the way that administrative data are presented suggests a completeness and objectivity that can be misleading if taken at face value. For example, just because a field is presented in general practitioner (GP) data for a diagnosis of depression this does not mean that it can be useful if we wish to determine prevalence (Box 4). It is possible, however, in some instances to use a hybrid screening approach to make up for this. For example, for rare disorders such as psychosis more accurate diagnostic coding has been achieved using a combination of clinical expertise and machine-learning techniques to process detailed data from health records documenting symptoms (Patel Reference Patel, Jayatilleke and Broadbent2015; Gorrell Reference Gorrell, Oduola and Roberts2016). Often, however, we do not have detailed symptom data. It is therefore important to be aware that all data are created in a context, whether social, administrative, technical or clinical. If we fail to take this into account, we risk misinterpreting the data we have collected (Hennekens Reference Hennekens, Buring and Mayrent1987; Prince Reference Prince, Stewart and Ford2003).

Big data alone cannot solve complex analysis problems

Large datasets of population health records can help solve one of the major challenges of psychiatric research, by providing adequately powered samples of the population of interest. However, the challenges of data analysis do not become easier simply because more data are collected. In fact, the larger the dataset the greater the potential complexity to be accounted for in the analysis. With a small well-designed trial or survey potential confounding variables, i.e. patterns in the data that could obscure our findings, can often be easily accounted for. However, population health records do not come with any such safeguards and are easily open to misinterpretation due to our failure to account for these patterns. For example, we could misinterpret spatial patterns by failing to account for differences in contextual risk factors such as urbanicity (see above), as well as differences in the reporting practices of mental health trusts in different parts of the country. Similarly, temporal patterns could also confound our results, such as changing ICD diagnostic categories. For example, with the change from ICD-9 to ICD-10 the latter showed a much higher sensitivity for dementia, which could easily be misinterpreted as an increase in prevalence if we were examining trends using health records data alone (Quan Reference Quan, Li and Duncan Saunders2008).

To account for this often requires a quite different analysis approach to the statistical methods used with smaller, more theoretically determined, samples. Where a well-designed RCT could potentially be analysed using routine techniques such as a t-test, for whole-population data more complex multilevel modelling and Bayesian analysis are often necessary. Although this is becoming easier with the widespread adoption of more advanced statistical methods, these still remain beyond the expertise of many researchers.

Big data cannot make research more replicable

In recent years, increasing concern has been raised about the ‘replicability crisis’ in scientific research, and particularly psychological research. For example, in one recent poll of 1500 scientists 70% had failed to reproduce another scientist's experiment and around 50% had failed to reproduce one of their own experiments (Baker Reference Baker2016). Often, examples are given of small-scale psychology experiments yielding interesting findings that consistently fail to be replicated. It could be argued that big data is one solution to this problem, as more data means results are more generalisable and therefore replicable. However, this is to misunderstand the nature of the problem. It is typically not the size of the dataset that is at issue, but the potential for spurious results in those situations where the researcher is faced with a multitude of different possible interpretations of the data. As datasets become larger and more complex, the number of potential subgroups to be analysed, analysis methods used and alternative categorisations to be adopted increases exponentially. For the unscrupulous researcher this could mean simply re-running the analysis by trying every possible combination of these until the results fit the required statistical significance (or ‘P-value’) – a practice known as p-hacking (Gelman Reference Gelman and Loken2014). This has reached the point where the American Statistical Association (ASA) recently felt compelled to issue a formal statement about the correct use of P-values (Wasserstein Reference Wasserstein and Lazar2016). For the ASA, the recent expansion in the use of large complex datasets for research, while expanding the possibilities for novel research, increases the risk of erroneous conclusions being made from the data. This may not even be deliberate; it is possible that, faced with many different analysis possibilities, the researcher may, whether consciously or not, be inclined towards the one more likely to give the desired result given the data they are presented with. It is very difficult to rule this out, although one solution is to make the analysis process more transparent (Box 5). For some types of research this can be achieved by reporting in advance the protocol for future studies, along with details of the analysis method. However, this is not necessarily applicable or helpful for many descriptive studies, where the ultimate focus may not be predetermined.

Big data cannot answer questions for which data have not already been collected

Big data is, by definition, data collected for purposes other than research and therefore does not always fit the research questions we wish to ask. For example, with diagnoses recorded over time for the purposes of clinical care, and not aetiological research, it is often very difficult to determine exactly how date of diagnosis relates to onset. Similarly, if we rely on big data alone it becomes very difficult to do research on disorders that have not already come to the attention of mental health services. In such situations, a reliance on routine health-systems data may under- or overestimate the actual prevalence of mental disorders. For such situations, cross-sectional surveys based in the community still have a major role to play. So, while traditional methods such as surveys, RCTs and qualitative studies allow us to determine what data are collected, with big data there is a danger that we neglect those research topics for which we do not already have available data (Schofield Reference Schofield, Das-Munshi and Becares2017b). This has particular relevance to mental health research, where social factors are often inextricably linked to the aetiology and progress of mental disorders (van Os Reference van Os, Kenis and Rutten2010; Reininghaus Reference Reininghaus and Morgan2014). A reliance on big data alone risks a circularity in the way research is conducted, as studies framed within a biomedical model, using data collected from medical records alone, remove the possibility that social factors might be included in the aetiology of mental disorder.