The difficulties of adapting the EBM framework to a systems thinking and complexity approach

Systems thinking and complexity approaches are essential in health systems research, guidelines for complex interventions (e.g., community-based care), complex target health conditions (e.g., mental and developmental disorders, multi-morbidity) and complex practice contexts where decisions are made under conditions of high uncertainty such as in primary care and mental health care (Sturmberg et al. Reference Sturmberg, O'Halloran and Martin2010; Salvador-Carulla et al. Reference Salvador-Carulla, Garcia-Alonso, Gibert and Vazquez-Bourgon2013; Huckel Schneider et al. Reference Huckel Schneider, Sturmber, Gillespie, Wilson and Lukersmith2016 (in press)). Complexity in clinical situations requires reasoning and judgment that may rely on broader sources of knowledge than the unidimensional guideline recommendation. In addition, a systems thinking approach is not intended to generate a single and definite solution to a PICO question, but to add new information to the knowledge base to improve decision making from a formative/summative perspective to organisational learning (Hargreaves, Reference Hargreaves2010; Snilstveit et al. Reference Snilstveit, Oliver and Vojtkova2012).

The lack of a philosophy of science approach to guideline development

We have revised this problem in a series of prior papers where we advocate for the extended multi-domain approach to scientific knowledge (Salvador-Carulla et al. Reference Salvador-Carulla, Fernandez, Madden, Lukersmith, Colagiuri, Torkfar and Sturmberg2014; Fernandez et al. Reference Fernandez, Sturmberg, Lukersmith, Madden, Torkfar, Colagiuri and Salvador-Carulla2015; Lukersmith et al. Reference Lukersmith, Hopman, Vine, Krahe and McColl2016). The definition of a series of key terms related to the development of this model is provided in Table 1. First, it is necessary to consider the differences between the three stages of scientific knowledge: discovery, corroboration and implementation (Schurz, Reference Schurz2014) and their contribution to guideline development. The traditional approach to EBM implicitly assumed that the analysis and ordering of the existing knowledge base of discovery and corroboration could be automatically translated, implemented and applied in practice. However, knowledge translation and implementation research is emerging in response to the need to identify the determinants, processes and outcomes of translating (corroboration) research into practice (Rabin et al. Reference Rabin, Brownson, Haire-Joshu, Kreuter and Weaver2008; Peters et al. Reference Peters, Adam, Alonge, Akua Agyepong and Tran2013).

Table 1.

Concepts and definitions of an extended multi-domain approach to scientific knowledge ^a

^a These formal definitions are partly based on the unified approach to the philosophy of science (Schurz, Reference Schurz2014) and have been described in Salvador-Carulla et al. Reference Salvador-Carulla, Fernandez, Madden, Lukersmith, Colagiuri, Torkfar and Sturmberg2014, Salvador-Carulla & Symonds Reference Salvador-Carulla and Symonds2016 and Lukersmith et al. Reference Lukersmith, Hopman, Vine, Krahe and McColl2016.

Second, it is important to provide operational definitions of the different domains of scientific knowledge (experimental evidence, observational evidence, contextual evidence, expert knowledge and experiential knowledge), the areas of scientific research relevant to understand recommendations (e.g., framing of scientific knowledge as a new research area of expert knowledge) (Salvador-Carulla et al. Reference Salvador-Carulla, Fernandez, Madden, Lukersmith, Colagiuri, Torkfar and Sturmberg2014), and logical inferences apart from deduction and induction, and which are extremely relevant to recommendations’ development such as abduction and mean-ends inferences (Table 1).

Alternatives to the EBM pyramid for ranking the hierarchy of research evidence

The traditional EBM hierarchy (Howick et al. Reference Howick, Chalmers, Glasziou, Greenhalgh, Heneghan, Liberatir, Moschetti, Phillips and Thornton2011) graphically portrayed as the EBM pyramid places a higher value on deductive inference from experimental evidence although systematic reviews rely heavily on induction. In the EBM model, randomised controlled trials are considered the highest quality, whilst observational evidence is de-valued, and context, expert ‘opinion’ and consumers’ experience are factually excluded as relevant sources of knowledge in the systematic reviews and the subsequent recommendations (Rosner, Reference Rosner2012). Expertise is not only important as a source of knowledge but also to generate recommendations. The ‘gold standard’ for guideline methods requires the inclusion of expert clinicians and experiencers consumer groups in the guideline development team. Yet, their contribution to the recommendations is typically not made transparent. Paradoxically the recent attempts to refine and improve the pyramid model (e.g., EBHC pyramid 5.0) (Alper & Haynes, Reference Alper and Haynes2016) insist on the analysis of the quality of evidence based on its internal validity and not on its external validity which is critical in complex questions which are context-dependent (Fernandez et al. Reference Fernandez, Sturmberg, Lukersmith, Madden, Torkfar, Colagiuri and Salvador-Carulla2015).

It is clear that the methods to formulate guideline recommendations need to accommodate the complexity, uncertainty and variability of health care practice. In our opinion, there needs to be a new framework for the development of guidelines if they are to be ‘fit for purpose’, feasible, useful and assimilated into practice. The framework should incorporate all the advances in the application of the knowledge-base of discovery to clinical practice made by EBM, but it also requires repeatable and explicit methods to integrate other scientific knowledge types described in Table 1. As guidelines are implementation tools, a new framework must consider scientific knowledge derived from implementation research (context and impact analysis). We have taken the first step and developed a preliminary model to frame an extended multi-domain approach to scientific knowledge that is graphically represented as a ‘Greek temple’. Figure 1 illustrates the movement from the unidimensional approach to broader extended multi-domain model of scientific knowledge to generate clinical guidelines and recommendations.

Fig. 1.

Moving from the pyramid model of scientific knowledge to the General framework: Greek temple model of scientific knowledge.

The five columns of scientific knowledge used across discovery, corroboration and implementation research, represent the domains of scientific knowledge described in Table 1. We consider that although experimental evidence derived from discovery and corroboration is a basic pillar for developing guidelines, other domains should be considered in guideline development and they cannot be merged together in a unidimensional ranking. Separate analyses should be provided to summarise the experimental knowledge and the observational knowledge using the criteria already available (e.g., EBHC pyramid 5.0) (Alper & Haynes, Reference Alper and Haynes2016), to provide a scientific framing together with information on context analysis, expert knowledge on appropriateness/adequacy, and the experiential knowledge on acceptability. For example, electroconvulsive therapy may show a high-efficacy and cost-effectiveness for major depressive disorder (Milev et al. Reference Milev, Giacobbe, Kennedy, Blumberger, Daskalakis, Downar, Modirrousta, Patry, Vila-Rodriguez, Lam, MacQueen, Parikh and Ravindran2016), and at the same time high differences in local availability (context), and in the judgment by expert professionals (appropriateness) and by consumers (acceptability). It is extremely difficult to display this complexity in a single dimension. This case may be better represented by a multi-dimensional profile than a single statement followed by the strength of the recommendation. However, much more work needs to be done to develop and test transparent, rigorous and repeatable guideline development methods to integrate and package all scientific knowledge on the topic of concern in an acceptable, applicable and practical way.

Health condition and practice context:

‘Intellectual developmental disorder’ or ‘IDD’ shows unsolved differences in its terminology, standard diagnostic criteria and assessment (Bertelli et al. Reference Bertelli, Munir, Harris and Salvador-Carulla2016a , Reference Bertelli, Munir and Salvador-Carulla b ); as well as in its care delivery system, planning and policy (Salvador-Carulla & Symonds, Reference Salvador-Carulla and Symonds2016). Adults with IDD often have health conditions that differ from those of the rest of the population, but there are few PICO-type studies for this group of patients. Most of the scientific knowledge regarding this population is rated as low-quality expert opinion within the EBM framework. Even when the same health conditions are present in the IDD population as the general population, evidence-based recommendations are not necessarily inclusive of adults with IDD because of the wide variability from person to person of factors such as level of disability (i.e., mild to profound) and aetiology, that could affect the outcome of interventions. Moreover, many adults with IDD experience co-morbid physical, behavioural and mental health conditions (for which separate evidence-based guidelines might exist but do not include the IDD population). The practical usability of guidelines for adults with IDD is highly limited due to conflicting recommendations and the risk of engaging in multiple recommended interventions (e.g. the use of medications in comorbid and complex cases).

A clear example is provided by the use of antipsychotic medications for problem behaviours in persons with IDD. Antipsychotics are often prescribed by family physicians in Ontario (Cobigo et al. Reference Cobigo, Ouellette-Kuntz, Lake, Wilton, Lunsky, Klein-Geltink and Yates2013). Available studies provide insufficient evidence to support such off-label prescribing practices (Deb et al. Reference Deb, Unwin and Deb2015). In addition, EBM neither elucidates factors that influence clinical decisions to start or continue such prescriptions nor does it assist clinicians or consumers in determining, whether the benefits of prescribing antipsychotic medications outweigh its burdens and risks (e.g., sedation and metabolic syndrome).

Problems in the implementation of the IDD Canadian guidelines

Authors updating the IDD Canadian guidelines recognised several methodological problems in the former version (2011). First, most of the recommendations were based on expert consensus only, which is regarded as the lowest quality of evidence in EBM. Second, the strength of a recommendation was assumed to be the same as the quality of the best supporting research evidence. For instance, this assumption was not conducive to judging that level I evidence (of the highest quality in EBM) could ever warrant a weak recommendation for action, because of factors relating to the context of care or patient or caregiver preferences and values, for example, just as level III evidence (of the lowest quality) might sometimes warrant a strong recommendation. As a result, there was very little in the 2011 Guidelines that could be proposed as a strongly recommended action. Third, for the level III evidence on which most of the 2011 Guidelines were based, the authors did not consider whether or not the experts had based their views on experiences from comparable medical contexts or had taken into account the complexities of caring for a heterogeneous patient population among adults with IDD with varying health needs, experiences, preferences and values. Fourth, the link between recommendations and framing values, such as a commitment to patient-centred care, adaptations to practices to accommodate the specific needs of people with IDD, and a commitment to supported decision-making, was not required by an EBM approach to developing the guidelines.

The implementation rate of any regular preventive care assessment of adults with IDD in primary care settings in Ontario, Canada, was found to be low (Ouellette-Kuntz et al. Reference Ouellette-Kuntz, Cobigo, Balogh, Wilton, Lunsky, Lunksy, Klein-Geltink and Yates2013) . In a subsequent study regarding implementing health checks for people with IDD in different primary care settings, many system and other factors that influence their successful implementation were identified (Durbin et al. Reference Durbin, Selick, Casson, Green, Spassiani, Perry and Lunsky2016). The EBM-knowledge on which prior versions of the IDD guideline recommendations were based was silent on these contextual factors, and the authors sought to explicitly consider other types of knowledge and clinical context in the updated version of the guidelines.

Outline of the methods for the development of the 3rd version of the IDD Canadian Guidelines

Given these and other limitations of a strictly EBM approach to guideline development, several alternatives were undertaken. First, the authors supplemented the knowledge base for guidelines by incorporating other sources of knowledge related to context, expert knowledge, and consumer and carer knowledge, as described above. These sources of knowledge were especially useful for deciding which guidelines to revise, delete or add based on the cumulative experience of Canadian primary care providers who had implemented the 2011 Guidelines. By making these additional categories of knowledge explicit, the authors allowed for some recommendations for action for which there was no ‘evidence’ according to the EBM model. It is important to note that the addition of context knowledge was made possible by the previous work of the Healthcare Access Research and Developmental Disabilities (H-CARDD) programme. This is a partnership of scientists, policymakers, and clinicians which uses administrative data to provide information on the health of a cohort of adults with IDD relative to other adults in Ontario. The key findings of H-CARDD have been published in the Atlas of the Primary care of Developmental Disabilities in Ontario (Lunsky et al. Reference Lunsky, Klein-Geltink and Yates2013; Salvador-Carulla & Symonds, Reference Bertelli, Munir, Harris and Salvador-Carulla2016).

Second, the authors ranked quality according to different criteria for each of the types of knowledge profiled. For example, empirical studies were ranked ‘level I’ if they were randomised control trials or systematic reviews and ‘level II’ for other types of studies, whereas expert knowledge was classified as ‘level I’ if they had been generated by an explicit consensus process and ‘level II’ if they had not. By profiling the different domains of knowledge considered (rather than simply ranking the quality of knowledge as an aggregate), the authors enabled clinicians to make their own judgments regarding recommended actions based on what they deemed most relevant to particular patients and their context of care.

Third, the authors took a complexity approach to their proposed action for each guideline by providing ‘advice’ or guidance that is relevant for individual patient care rather than being based abstractly on generalised ‘evidence’. Three different levels of recommendation have been established for every domain:

1. (1) ‘Recommend’: always apply. The primary care provider should regard the guideline as a basic standard of care in Canada.

2. (2) ‘Consider’: indicated to primary care providers that it should be applied at the physician's discretion. The professional would need to take into account factors specific to each patient and context before deciding to implement the guideline.

3. (3) ‘Aspire’: authors endorsed the action as an ideal/future standard of care that should be applied if possible but acknowledged that advocacy and changes to contexts and systems of care might be necessary for clinicians to have the capacity and resources to apply such a guideline routinely.

By distinguishing among these types of advice, the authors were able to reduce the number of guideline recommendations as basic standards of care. They were also able to add new guidelines that could be regarded as aspirational but were excluded from the 2011 Guidelines (which focused only on what was judged to be attainable at that time within the Canadian health care system). The authors were also able to flag those guidelines for which additional particular factors in varying contexts of care and in different patients would play a significant role in clinical decision making.

Finally, framing values were incorporated into the guidelines themselves by adding to the General Issues section of the 2011 Guidelines some guidance regarding approaches to care. In addition, the authors revised each guideline with the view of ensuring consistency with these framing values. They also sought to promote good concrete clinical decisions by acknowledging the importance of considering additional particular factors for each patient prior to implementing the action they were to consider, such as the patient's strengths and vulnerabilities in decision making, and regard for patient and caregiver input and participation in decisions concerning the proportion of benefit to harm for the patient of the intervention being considered.