ToC is useful when interventions are complex

Complex health interventions contain multiple interacting components, causal strands, feedback loops and emergence of unexpected outcomes (Craig et al. Reference Craig, Dieppe, Macintyre, Michie, Nazareth, Petticrew, Mitchie, Nazareth and Petticrew2008). Interventions can be implemented in multiple locations with different governance structures, increasing the complexity of even simple interventions. ToC focuses on the key, essential outcomes of the programme while allowing flexibility in how this is achieved across different contexts. The complex interventions developed and evaluated by PRIME were district-level mental healthcare plans in five countries (Hanlon et al. Reference Hanlon, Fekadu, Jordans, Kigozi, Petersen, Shidhaye, Honikman, Lund, Prince, Raja, Thornicroft, Tomlinson and Patel2016). We used the ToC map to summarise a complex system of multiple causal pathways across three levels of the health system (Breuer et al. Reference Breuer, De Silva, Shidaye, Petersen, Nakku, Jordans, Fekadu and Lund2016a). We then mapped intervention packages onto these causal pathways, for example, a detection package at community level, a psychosocial treatment package at facility level or a mental health awareness package at district level. Each of these packages had the same function across countries, but the form differed as they were developed for different countries according to resources, evidence, need, feasibility and acceptability. This allowed comparability across settings but flexibility to ensure the intervention packages were fit for purpose (Hawe et al. Reference Hawe, Shiell and Riley2004).

ToC workshops can help to develop a contextualised mental healthcare plan with stakeholder buy-in

Implementers, researchers, policy makers and service users each have their own implicit understanding of how and why a complex intervention works and what outcome it will achieve (Peters, Reference Peters2014). Including a variety of stakeholders in ToC workshops allows all stakeholders to co-develop the ToC and make explicit the steps along the causal pathway (Andersen, Reference Andersen2004).

The PRIME ToC development process was extensive and included a cross-country ToC workshop, 2–4 workshops in each of the five PRIME countries and revision of the cross-country ToC (Breuer et al. Reference Breuer, De Silva, Shidaye, Petersen, Nakku, Jordans, Fekadu and Lund2016a). This resulted in one ToC map per country and a ToC across all five countries. The ToCs were influenced by each other and built on prior work of the PRIME consortium including a draft framework that outlined the three levels of the health system at which PRIME planned to intervene (Lund et al. Reference Lund, Tomlinson, de Silva, Fekadu, Shidhaye, Jordans, Petersen, Bhana, Kigozi, Prince, Thornicroft, Hanlon, Kakuma, McDaid, Saxena, Chisholm, Raja, Kippen-Wood, Honikman, Fairall and Patel2012). We found that ToC workshops assisted us in developing a logical ToC map which formed the basis of a contextualised mental health care plan (Breuer et al. Reference Breuer, De Silva, Fekadu, Luitel, Murhar, Nakku, Petersen and Lund2014). The presence of mental health specialists, researchers, policy makers, district-level health planners and management and service providers during the ToC development ensured that the resulting ToCs and mental healthcare plans were relevant, feasible and that the barriers and facilitating factors supporting this programme were clearly articulated. The presence of stakeholders also ensured their buy-in as they were able to contribute to the conceptualisation of the programme as well as highlight potential problems prior to the development of a detailed implementation plan. Chibanda et al. (Reference Chibanda, Verhey, Munetsi, Cowan and Lund2016), who used ToC to develop a counselling intervention for common mental disorders in Zimbabwe, found that early engagement helped to build rapport with stakeholders who provided detailed contextual information. This increased the likelihood that the intervention would be successful.

Although ideally the ToC should be owned by all stakeholders, this is often difficult in practice (Sullivan & Stewart, Reference Sullivan and Stewart2006). Ownership of the PRIME ToCs most closely resembles elite ownership (Breuer et al. Reference Breuer, De Silva, Fekadu, Luitel, Murhar, Nakku, Petersen and Lund2014). According to (Sullivan & Stewart, Reference Sullivan and Stewart2006), this is ownership by a small group of leaders (including community leaders) who are responsible for implementing the programme. In PRIME, this is due to multiple reasons: (1) the large number of stakeholders involved in the workshops in countries [median 15 (interquartile range 13–22)] making extended consultation difficult; (2) the finalisation of the ToC by PRIME researchers after the workshop; (3) hierarchies within the health service making participation in the workshops uneven (despite our attempts to mitigate this) and (4) the lack of beneficiaries of the programme who attended the ToC workshop (Breuer et al. Reference Breuer, De Silva, Fekadu, Luitel, Murhar, Nakku, Petersen and Lund2014).

ToC workshops are resource intensive

ToC workshops are resource intensive. These include human resources to plan, facilitate and attend the workshop, and the financial costs of conducting the workshop. To our knowledge, there has been no formal comparison of costs between ToC and other methods to develop and evaluate complex mental health interventions. However, from our experience, there are both higher costs and greater stakeholder input into the development of the ToC compared with other methods of developing a ToC, such as qualitative interviews or document review. Therefore, it is important to determine the level of complexity of the interventions and the extent to which local knowledge and stakeholder buy-in is useful so that a balance between resources and buy-in can be decided on a priori by the research team.

ToCs need champions to drive their development and implementation

ToCs need champions who understand the ToC approach, can drive the ToC development and implementation during the life of the programme (Lee, Reference Lee2014). This includes organising and facilitating the workshops, finalising the resultant ToC map, getting further stakeholder input where required, finalising the evidence base and indicators for the ToC map and ensuring that the evaluation design of the programme measures the indicators outlined in the ToC. In PRIME, we were the ToC champions who led the work across all five countries and were supported by ToC champions in each country who led and facilitated the country workshops and developed the country ToCs.

Despite having champions, there has been no formal revision of the PRIME ToC in any country yet. There are three likely reasons for this: (1) data collection for the evaluation of the programme has recently been completed and the analysis is underway; (2) the research teams had competing priorities; and (3) no formal ToC revision of the ToC across and within countries was included in the workplan of the programme. In future projects, we would recommend a formal revision process of the ToC at key points in the process, for example, after piloting and after initial implementation of the programme, and after the final summative evaluation. For example, Asher et al. (Reference Asher, Fekadu, Hanlon, Mideksa, Eaton, Patel and De Silva2015) used the findings of her pilot study to revise her ToC for a community-based rehabilitation intervention for people living with schizophrenia in Ethiopia prior to implementing the intervention in a cluster randomised controlled trial.

The approach to ToC development should remain flexible

If the instructions on how to develop, portray or use a ToC become too prescriptive, ToC runs the risk of becoming yet another monitoring and evaluation tool that is used superficially in order to satisfy the requirements of funding agencies. Prinsen and Nijhof describe how logframes were initially developed often with stakeholders using a problem or objective tree. However, now they are largely templates standardised by funding agencies for completion (Prinsen & Nijhof, Reference Prinsen and Nijhof2015). In PRIME, because we developed these ToCs in addition to our formal monitoring and evaluation requirements from our funder, there was flexibility in how the ToCs were developed, which helped to ensure that they represented the causal pathways to change.

ToC can provide a framework for evaluation and assist with identifying indicators for measurement

Once a causal pathway of short-, medium- and long-term outcomes has been developed, indicators are developed for each of these outcomes. This measures the achievement of each step along the causal pathway and distinguishes between implementation failure, where the programme was not implemented as intended, or theory failure, where the programme did produce the expected outcomes (Patton, Reference Patton2008). In PRIME, we used the ToC to identify common indicators for each ToC outcome across each of the five PRIME countries. This in turn informed the design of evaluation, which allowed the programme to be compared across all PRIME sites (De Silva et al. Reference De Silva, Rathod, Hanlon, Breuer, Chisholm, Fekadu, Jordans, Kigozi, Petersen, Shidhaye, Medhin, Ssebunnya, Prince, Thornicroft, Tomlinson, Lund and Patel2016).

ToC indicators may need to be prioritised to account for time and resource constraints

It is unsurprising that the ToC of complex programmes are complex with many outcomes, causal pathways and feedback loops. The ToC maps can result in a comprehensive evaluation plan, which is made up of multiple study designs collecting various types of data. If resources are limited, it may be necessary to prioritise key indicators within the ToC so that the study designs are focused on collecting data on the most important steps along the causal pathway.

ToC does not prescribe a data collection method

ToC provides a framework to identify the pathways to impact, but it does not prescribe the type of data collection or analysis (Connell & Kubisch, Reference Connell, Kubisch, Fulbright-Andersen, Kubisch and Connell1998). This allows for the use of a wide range of quantitative and qualitative data collection and analysis techniques such as surveys, in depth interviews, document reviews, cohort studies, nested randomised controlled trials or programme observation (Breuer et al. Reference Breuer, De Silva, Shidaye, Petersen, Nakku, Jordans, Fekadu and Lund2016a). Data for the PRIME ToC indicators were collected using four study designs: repeat community surveys, repeat facility detection surveys, cohort studies and case studies (De Silva et al. Reference De Silva, Rathod, Hanlon, Breuer, Chisholm, Fekadu, Jordans, Kigozi, Petersen, Shidhaye, Medhin, Ssebunnya, Prince, Thornicroft, Tomlinson, Lund and Patel2016).

Combining data to evaluate indicators for short-, medium- and long-term outcomes is possible

In a recent systematic review, we found that many ToC data analysis techniques evaluate each outcome separately and do not combine the evaluation of the short-, medium- and long-term outcomes of the ToC (Breuer et al. Reference Breuer, Lee, De Silva and Lund2016b). Methods that show promise for an integrated analysis of indicators in the ToC for complex mental health interventions are comparative case studies (Mookherji & LaFond, Reference Mookherji and LaFond2013), qualitative comparative analysis (Kane et al. Reference Kane, Lewis, Williams and Kahwati2014) or statistical approaches such as path analysis or structural equation modelling (Adedokun et al. Reference Adedokun, Childress and Burgess2011). Comparative case studies allow in-depth comparison of cases but become difficult to compare as cases increase. A conceptual limitation of structural equation modelling and other statistical techniques is the reduction of contextual factors to variables. ToC, as part of the broader school of theory-driven enquiry, is interested in ‘what works for whom in what circumstances’. A reductionist statistical approach may be useful in understanding key causal mechanisms but may obscure some of the complexity related to the context and other factors which co-vary between contexts. In PRIME, we have used a qualitative comparative approach to analyse data from ToC indicators from Nepal and will present this in a subsequent paper. Qualitative comparative approach is a case-orientated approach that uses set theory and Boolean logic to understand patterns across cases (Kane et al. Reference Kane, Lewis, Williams and Kahwati2014) and holds promise for analysing ToC indicators in mental health services research.

A ToC from a programme can be used as a heuristic device that can be adapted for other similar contexts

Where a ToC has already been developed, it can be used to inform other ToCs for similar programmes (Funnell & Rogers, Reference Funnell and Rogers2011). Our PRIME ToC had a lot of similarities to ToCs developed in other similar mental health programmes, for example, the Friendship Bench in Zimbabwe and the RISE project in Ethiopia (De Silva et al. Reference De Silva, Breuer, Lee, Asher, Chowdhary, Lund and Patel2014; Asher et al. Reference Asher, Fekadu, Hanlon, Mideksa, Eaton, Patel and De Silva2015; Breuer et al. Reference Breuer, De Silva, Shidaye, Petersen, Nakku, Jordans, Fekadu and Lund2016a; Chibanda et al. Reference Chibanda, Verhey, Munetsi, Cowan and Lund2016). ToCs developed for one programme could be used as a heuristic device for other programmes. Care should be taken to ensure that the requisite contextual information and stakeholder input is obtained during this process to ensure that the ToC is adequately adapted to the new setting.