Impact statement
Globally, nearly a quarter of all years lived with disability due to mental disorders occurring before the age of 25 (Kieling et al., Reference Kieling, Buchweitz, Caye, Silvani, Ameis, Brunoni, Cost, Courtney, Georgiades, Merikangas, Henderson, Polanczyk, Rohde, Salum and Szatmari2024). Yet, help-seeking rates for mental health problems among children and adolescents remain low (Reardon et al., Reference Reardon, Harvey, Baranowska, O’Brien, Smith and Creswell2017). The Community Case Detection Tool (CCDT) is an evidence-based tool developed for trusted and respected community members to facilitate community-level proactive detection of mental health needs and promote help-seeking at available care (van den Broek et al., Reference van den Broek, Agondeze, Greene, Kasujja, Guevara, Tukahiirwa, Kohrt and Jordans2024).
This proof-of-concept study evaluates an optimization strategy of the CCDT, called CCDT+, designed to enhance the quality of detection and effectiveness in promoting help-seeking. The CCDT+ consists of a dashboard that presents actionable outcomes for data-driven supervision and integrates motivational interviewing techniques, along with behavioural nudges, into the training of community members using the CCDT to encourage help-seeking.
The CCDT+ significantly improved detection accuracy, with 2.34 times higher odds compared to standard CCDT. Additionally, areas using the CCDT+ saw a 2.05-fold increase in mental health service utilization. Qualitative findings showed that the CCDT+ was perceived to improve work efficiency, effectiveness, quality and boosted motivation. Access issues to real-time data for supervisors and gaps in coordination between service providers and gatekeepers were the main reported barriers.
The CCDT+ introduces an embedded quality-improvement process for mental health detection tools and shows promise in enhancing the accuracy of referrals over time and in real time. Optimization strategies like the CCDT+ can contribute to the more effective use of scarce resources, which is especially important given the limited availability of mental health services in most low- and middle-income countries (LMICs) (Patel et al., Reference Patel, Saxena, Lund, Kohrt, Kieling, Sunkel, Kola, Chang, Charlson, O’Neill and Herrman2023).
Introduction
Globally, nearly a quarter (24.85%) of all years lived with disability caused by mental disorders occur before the age of 25 (Kieling et al., Reference Kieling, Buchweitz, Caye, Silvani, Ameis, Brunoni, Cost, Courtney, Georgiades, Merikangas, Henderson, Polanczyk, Rohde, Salum and Szatmari2024). Despite this important window for detection and access to care, rates of help-seeking for mental health problems among children and adolescents remain low (Reardon et al., Reference Reardon, Harvey, Baranowska, O’Brien, Smith and Creswell2017). Children often rely on others to identify mental health problems, access services and continue the use of care (Godoy et al., Reference Godoy, Mian, Eisenhower and Carter2015). Children in low- and middle-income countries (LMICs) are disproportionately affected in terms of access to mental health care due to limited financial and human resources, lack of policies and services focusing specifically on child and adolescent mental health, and a paucity of accurate tools to support identification and screening of mental health conditions among children (Babatunde et al., Reference Babatunde, van Rensburg, Bhana and Petersen2019). Despite the growing availability of effective mental health interventions for children in LMICs (Venturo-Conerly et al., Reference Venturo-Conerly, Eisenman, Wasil, Singla and Weisz2023), only a limited number have been brought to scale (Jordans and Kohrt, Reference Jordans and Kohrt2020). Even where services are available, demand-side barriers – such as a low perceived need for care, under-detection, stigma and a preference to handle the problem by oneself – further hinder help-seeking for mental health problems (Andrade et al., Reference Andrade, Alonso, Mneimneh, Wells, Al-Hamzawi, Borges, Bromet, Bruffaerts, De Girolamo, De Graaf, Florescu, Gureje, Hinkov, Hu, Huang, Hwang, Jin, Karam, Kovess-Masfety, Levinson, Matschinger, O’Neill, Posada-Villa, Sagar, Sampson, Sasu, Stein, Takeshima, Viana, Xavier and Kessler2014; Kazdin, Reference Kazdin2019). In children and adolescents, detecting mental health problems is particularly challenging due to varying developmental stages and a wide range of normal behaviours throughout these stages, which make it difficult for caregivers to identify behaviours that indicate a need for care (Kazdin, Reference Kazdin2019). These challenges are exacerbated in conflict-affected and low-resourced settings, where daily disruptions and the burden on gatekeepers may hinder early identification.
The community case detection tool (CCDT; also known as ReachNow) has been developed to address demand-side barriers to mental health care for children and adolescents by facilitating community-level proactive detection of mental health care needs and promoting help-seeking. The tool was developed with and for community gatekeepers – trusted and respected community members without specialized training in mental health – and can be used in daily routine activities (Jordans et al., Reference Jordans, Kohrt, Luitel, Komproe and Lund2015, Reference Jordans, Luitel, Lund and Kohrt2020; van den Broek et al., Reference van den Broek, Ponniah, Jeyakumar, Koppenol-Gonzalez, Kommu, Kohrt and Jordans2021, Reference van den Broek, Hegazi, Ghazal, Hamayel, Barrett, Kohrt and Jordans2023). It presents common symptoms of childhood psychological distress through contextualized easy-to-understand illustrated vignettes. Previous studies have demonstrated the accuracy and effectiveness of the tool: the positive predictive value (PPV) of the tool found was 0.67 in Sri Lanka, 0.69 in Uganda and 0.77 in occupied Palestinian territories (van den Broek et al., Reference van den Broek, Ponniah, Jeyakumar, Koppenol-Gonzalez, Kommu, Kohrt and Jordans2021, Reference van den Broek, Hegazi, Ghazal, Hamayel, Barrett, Kohrt and Jordans2023, Reference van den Broek, Agondeze, Greene, Kasujja, Guevara, Tukahiirwa, Kohrt and Jordans2024). Furthermore, in the locations where the CCDT was used, a significant 17-fold increase in the utilization rate of mental health care services among children aged 6–18 years was found, compared to routine detection and mental health awareness-raising activities (van den Broek et al., Reference van den Broek, Agondeze, Greene, Kasujja, Guevara, Tukahiirwa, Kohrt and Jordans2024).
Given the limited availability of mental health care services in most LMICs (Patel et al., Reference Patel, Saxena, Lund, Kohrt, Kieling, Sunkel, Kola, Chang, Charlson, O’Neill and Herrman2023), it is important to ensure that tools to detect children in need of those services have a low false positive rate so that scarce resources can be used most optimally. Establishing the accuracy of tools to detect mental health problems in new contexts is a resource-intensive process. Even after validation, standardized tools like the PHQ-9 still often yield high rates of false positives, with PPVs ranging from 0.17 to 0.37 in South Africa, 0.23 in Kenya and 0.31 in Nepal (Luitel et al., Reference Luitel, Rimal, Eleftheriou, Rose-Clarke, Nayaju, Gautam, Pant, Devkota, Rana, Chaudhary, Gurung, Åhs, Carvajal-Velez and Kohrt2024; Marlow et al., Reference Marlow, Skeen, Grieve, Carvajal-Velez, Åhs, Kohrt, Requejo, Stewart, Henry, Goldstone, Kara and Tomlinson2023; Tele et al., Reference Tele, Carvajal-Velez, Nyongesa, Ahs, Mwaniga, Kathono, Yator, Njuguna, Kanyanya, Amin, Kohrt, Wambua and Kumar2023). Furthermore, without leveraging routine data, the accuracy levels of these instruments remain the same. High rates of false positives can cause unnecessary discomfort for children and risk overburdening available services.
Digital dashboards have emerged as increasingly common tools for monitoring service quality and optimizing outcomes (Bickman, Reference Bickman2008; Randell et al., Reference Randell, Alvarado, Elshehaly, McVey, West, Doherty, Dowding, Farrin, Feltbower, Gale, Greenhalgh, Lake, Mamas, Walwyn and Ruddle2022). These dashboards use data visualisation techniques to summarize data and provide insight into key metrics in an easy-to-understand format. Furthermore, these key metrics can be used to inform supervision and enhance supervision effectiveness (Randell et al., Reference Randell, Alvarado, Elshehaly, McVey, West, Doherty, Dowding, Farrin, Feltbower, Gale, Greenhalgh, Lake, Mamas, Walwyn and Ruddle2022).
This study is a proof-of-concept study of an optimisation strategy for the CCDT, the CCDT+, developed to monitor and improve the quality of detection and effectiveness of help-seeking promotion. The CCDT+ includes a dashboard presenting actionable outcomes for data-driven supervision and integrates motivational interviewing (MI) techniques and behavioural nudges in the community gatekeeper training and supervision sessions to promote help-seeking. The objectives of this study are to (1) assess the added value of the CCDT+ in improving accuracy and service utilization outcomes compared to the standard CCDT, and (2) evaluate implementation outcomes of the CCDT+.
Methods
Study design
This mixed-methods study was conducted from January to May 2023 in Palorinya refugee settlement located in Obongi District in the West Nile region in Uganda. Uganda accommodates over 1.5 million refugees and asylum seekers and is one of the world’s leading hosts for refugees (UNHCR, 2023). There are 14 formal refugee settlements in Uganda, each sub-divided into administrative units called ‘zones’. Despite being entitled to several services – such as education, healthcare and employment – refugees often face a multiplicity of risk factors for adverse mental health outcomes, including social isolation and loss of livelihoods (Stark et al., Reference Stark, Meinhart, Hermosilla, Kajungu, Cohen, Agaba, Obalim, Knox and Onyango Mangen2024). The prevalence of mental health problems among children and adolescents has been reported to reach 23% (Opio et al., Reference Opio, Munn and Aromataris2022). Palorinya refugee settlement was established in 2016 and is divided into five zones with a total population of 127,000 during the time of this study, an estimated 43% of whom are aged between 5 and 17 years. Majority of refugees are from South Sudan (UNHCR, 2022).
The CCDT+ was integrated into ongoing programs of an international humanitarian organization, War Child, and a national mental health care provider, the Transcultural Psychosocial Organization (TPO) Uganda. This study was conducted in all five zones. Two neighbouring zones were combined as one. The median zone population size was 36434.5 (IQR 25828.5, 37671).
This proof-of-concept study comes after a stepped wedge cluster randomized trial (SW-CRT) that evaluated the effectiveness of the standard CCDT in Uganda from January till November 2022 (van den Broek et al., Reference van den Broek, Agondeze, Greene, Kasujja, Guevara, Tukahiirwa, Kohrt and Jordans2024). During the SW-CRT, the CCDT was sequentially rolled out across 28 zones in five refugee settlements over a period of nine months. These settlements encompassed Bidi Bidi, Kyaka II, Kyangwali, Omugo and Rhino. The proof-of-concept study presented here follows the same procedures in a similar setting and population, and the comparative data used in this study is drawn from the SW-CRT conducted immediately prior to this study.
Participants
Participants included trusted and respected community gatekeepers trained in using the CCDT+, children and adolescents detected by these gatekeepers, and one clinical psychologist and two social workers contracted by TPO. Similar to the SW-CRT, the number of gatekeepers per zone was based on the total zone population size, applying a ratio of one gatekeeper for every 3000 residents. Gatekeepers were selected by War Child through their established networks and existing working relationships, taking into account their roles and positions within the community. Inclusion criteria for gatekeepers were individuals aged 18 years or older who were trusted and respected members of the community, actively involved in child wellbeing, and with access to families. Examples of such gatekeepers included youth club leaders, village health team members and intervention facilitators. Children and adolescents participating in this study included all children aged 6–18 who were detected by gatekeepers as matching with the CCDT. Only those who subsequently sought help at TPO were included in our sub-sample for analysing the main outcomes related to the accuracy of detected cases and service utilization.
Procedures
Standard CCDT
The CCDT was developed based on the adult Community Informant Detection Tool (CIDT) (Jordans et al., Reference Jordans, Kohrt, Luitel, Komproe and Lund2015). The tool consists of two illustrated vignettes printed on a single sheet of paper. Each vignette presents a case story and six illustrations of a child experiencing common internalizing or externalizing problems, including symptoms related to depression, anxiety and oppositional defiant disorder. At the end of each vignette, a short decision algorithm supports gatekeepers to gauge the resemblance, frequency and intensity of symptoms observed, and to determine the follow-up action. See Figure 1. In case of a match with the tool, the gatekeeper is advised to engage in a dialogue with the caregivers to encourage help-seeking to a known and available mental health service. The vignettes are culturally adapted through input from potential gatekeepers and national mental health care workers, blind back-translations and focus group discussions (FGDs) with potential gatekeepers to assess appropriateness and acceptability. The tool uses colloquial language and non-stigmatizing local idioms of distress to support proactive detection of symptoms by people without specialized training in mental health, and by using daily observations.
The Community Case Detection Tool.
The standard CCDT training is two days and focuses on the basics of child and adolescent mental health, use of the tool, child safeguarding and ethical considerations. Gatekeepers (n = 177) in the SW-CRT participated in the standard CCDT training delivered by a clinical psychologist (n = 4 in total) and a project officer based in each settlement. Gatekeepers used the tool during their daily routine activities and promoted help-seeking for children and adolescents matching with one of the vignettes. They provided information about how to access mental health care services, assigned a study ID and recorded de-identified detection data in a logbook (i.e., date of detection, age, gender, vignette used and location). Upon accessing the mental health care services, routine intake data was collected (i.e., date of intake, age, gender, mental health assessment outcome and location). Monthly supervision sessions were organized by the psychologist and a project officer based in their settlement.
Optimisation strategy: CCDT+
The CCDT+ is an enhanced version of the standard CCDT. It combines the standard CCDT (i.e., the tool for gatekeepers to support proactive community-level detection and help-seeking promotion) with an optimisation strategy consisting of: (i) MI techniques combined with behavioural nudges used by gatekeepers to promote help-seeking; and (ii) a digital dashboard for supervisors with key metrics around help-seeking and the accuracy of detection. Gatekeepers received a 2.5 day training by a trained supervisor in the standard CCDT training, plus an additional half-day session focusing on the MI techniques and behavioural nudges. MI is used as a collaborative conversation technique to enhance an individual’s own motivation and commitment to change and was originally developed as a treatment for individuals with substance use disorders (Miller and Rollnick, Reference Miller and Rollnick2013). MI has been extended to treat other mental health problems and health behaviours such as medication adherence for chronic illness. Furthermore, MI has also been effectively used as a pre-treatment intervention to increase motivation to seek help and engage in further assistance (Lawrence et al., Reference Lawrence, Fulbrook, Somerset and Schulz2017). Three MI techniques were integrated in the gatekeeper training: (i) asking open questions, (ii) affirming, and (iii) reflective listening. In addition, gatekeepers were trained in delivering in-person reminder messages as behavioural nudges to further encourage help-seeking among those that were detected. Nudges are based on behavioural economic theory and are used as strategies to alter an individual’s behaviour in a predictable manner without prohibiting any choices or significantly altering their economic incentives (Thaler and Sunstein, Reference Thaler and Sunstein2008). Reminders are an example of a low-cost behavioural nudge and have been effectively applied to promote other health-related decisions such as vaccination uptake (Dai et al., Reference Dai, Saccardo, Han, Roh, Raja, Vangala, Modi, Pandya, Sloyan and Croymans2021). This combination of MI and behavioural nudges aims to first increase motivation and intentions to seek help among those detected, followed by targeted reminders to support the transition from intentions to action.
Gatekeepers used the CCDT, MI and behavioural nudges during their daily routine activities to detect children and promote help-seeking. Caregivers of children detected were encouraged to seek help and received a referral card from the gatekeeper with information about how to contact and reach TPO. Mental health services provided by TPO included group interventions such as Journey of Life, Cognitive Behavioural Therapy, individual specialized care or referral to other service providers.
Fortnightly data-driven supervision meetings led by two social workers were organized for gatekeepers by a project officer. These social workers, supervised by a clinical psychologist, were each responsible for gatekeepers in two zones. The supervisors (two social workers and a clinical psychologist) had access to the CCDT+ dashboard on a tablet or laptop. This dashboard combines detection data collected by gatekeepers and routine intake data collected by the mental health service providers (TPO) and provides the following actionable insights: (1) the number and location of CCDT-detected cases, (2) which CCDT-detected cases sought help and accessed care using a client ID, and (3) the accuracy of the CCDT-detected cases that sought help. A supportive supervision approach was followed, which is a collaborative and non-hierarchical approach to supervision. It fosters open communication, joint problem-solving and skill-building, allowing gatekeepers to discuss challenges, and receive constructive feedback based on the data presented on the dashboard (McBride and Travers, Reference McBride and Travers2021). The supervisors were trained by the research team in the gatekeeper training materials and received two days of training in using the CCDT+ dashboard to supervise gatekeepers. See Figure 2 for a screenshot of the dashboard.
Screenshot of the CCDT+ dashboard – overview page (mobile and desktop version).
The dashboard enables data-driven supervision and was used by supervisors to identify areas for quality improvement and to strengthen the capacity of gatekeepers in terms of accuracy of detection and effectiveness in help-seeking promotion. Prior to each supervision meeting, supervisors accessed the dashboard to record key observations based on the trends in the data. With data linked to individual gatekeeper IDs, supervisors provided feedback to groups of gatekeepers as well as more targeted support to individual gatekeepers. The following outcome metrics were shown on the dashboard for quality improvement and capacity strengthening:
-
1) Service utilization. Calculated as the proportion of children and adolescents detected by gatekeepers that utilized TPO’s mental health care services. If detected cases had not sought help within four to eight weeks after being detected, supervisors would share the client IDs with individual gatekeepers and revisit the MI techniques and reminder methods with the gatekeeper. The four- to eight-week window was chosen to provide enough time to seek help (four weeks after detection) while also respecting the right not to seek help (beyond eight weeks after detection).
-
2) Accuracy expressed as the PPV. PPV was calculated as the proportion of children and adolescents detected through the CCDT who were considered as needing mental health care services. The need for services was based on the information gathered during the clinical interview conducted by TPO using structured mental health symptom checklists. A PPV below 75% served as a prompt for supervisors to provide additional capacity strengthening with (individual) gatekeepers by revisiting the content of the vignettes. This PPV threshold was chosen because a PPV lower than 75% indicates that more than one in four children did not meet the criteria to receive services, and therefore potentially overburdening the health system and causing discomfort among children.
The dashboard was developed through three steps including (1) a hackathon with data scientists to develop a minimum viable product; (2) the development of proof-of-concept version based on multiple feedback rounds with the research team; and (3) two rounds of online user testing in Uganda and adaptations with three clinical supervisors and a coordinator from TPO as potential end-users of the dashboard.
Consent procedures
Gatekeepers, social workers and the clinical psychologist provided written informed consent for participating in the research activities. Children and adolescents under the age of 18 provided written assent, and their caregivers provided written informed consent to share data on mental health service utilization with the research team for study purposes.
Outcomes and measures
The outcomes used to assess the added value of the CCDT+ compared to the standard CCDT included: (1) the PPV of the CCDT+, and (2) mental health care services utilization during the implementation of the CCDT+. Both outcomes were operationalized and measured the same way as in the SW-CRT evaluating the effectiveness of the CCDT. The PPV was defined as the proportion of children and adolescents detected who were considered as needing mental health care services (i.e., true positive). The primary reference criterion for a true positive was an indication for treatment as assessed by a mental health care provider. The secondary reference criterion was the presence of any mental health condition matching the CCDT or severe distress as assessed by a mental health care provider. Mental health care utilization was defined as: (i) the count of new cases, that is, children and adolescents aged 6–18 years, who are seeking mental health care services for the first time, and (ii) the count of re-entry cases, seeking mental health care services after a lapse of at least six months, assuming the CCDT facilitated their re-entry to care. These data were extracted and tabulated monthly using TPO’s routine mental health case registration form.
The implementation outcomes included the perceived acceptability, appropriateness, feasibility and usability of the CCDT+ by gatekeepers and supervisors. Acceptability was defined as the perception of whether various elements of CCDT+ were agreeable, palatable or satisfactory (Proctor et al., Reference Proctor, Silmere, Raghavan, Hovmand, Aarons, Bunger, Griffey and Hensley2011). This was assessed using the 4-item Acceptability of Intervention Measure (AIM) (Weiner et al., Reference Weiner, Lewis, Stanick, Powell, Dorsey, Clary, Boynton and Halko2017). Appropriateness was defined as the perceived fit, relevance or compatibility of the CCDT+ (Proctor et al., Reference Proctor, Silmere, Raghavan, Hovmand, Aarons, Bunger, Griffey and Hensley2011) and assessed using the 4-item Intervention Appropriateness Measure (IAM) (Weiner et al., Reference Weiner, Lewis, Stanick, Powell, Dorsey, Clary, Boynton and Halko2017). Feasibility was defined as the extent to which various elements of CCDT+ can be successfully used (Proctor et al., Reference Proctor, Silmere, Raghavan, Hovmand, Aarons, Bunger, Griffey and Hensley2011) and assessed using the 4-item Feasibility of Intervention Measure (FIM) (Weiner et al., Reference Weiner, Lewis, Stanick, Powell, Dorsey, Clary, Boynton and Halko2017). Usability was defined as the extent to which various elements of the CCDT+ could be used by gatekeepers and supervisors to achieve specified goals with effectiveness, efficiency and satisfaction and was assessed using the 10-item Intervention Usability Scale (IUS) (Lyon et al., Reference Lyon, Pullmann, Jacobson, Osterhage, Achkar, Renn, Munson and Arean2021). These implementation science measures were adapted for use in Uganda and administered in English, Juba Arabic and Bari. The adaptation process included an initial review of the items, forward and blind back-translation, cognitive interviewing and pilot testing. These surveys were administered post-implementation with the clinical psychologist (n = 1), social workers (n = 2) and all gatekeepers (n = 45).
Qualitative feedback regarding these implementation outcomes was gathered post-implementation, through key-informant interviews (KIIs) with the clinical psychologist (n = 1), social workers (n = 2), gatekeepers (n = 8) and three FGDs with gatekeepers (n = 27 in total). Gatekeepers for the FGDs were purposively selected based on their level of participation (e.g., active and less active in using the tool and in supervision meetings). These were conducted in a central place in the community, by the trained project officer coordinating the training and supervision sessions. Topics included experiences in using the dashboard, organizing and participating in supervision sessions, using the MI techniques and reminders, and challenges and recommendations. See Supplementary Material S1 for the sample characteristics and topic guides.
Analyses
Statistical analyses
We estimated the added value of the CCDT+ on improving the PPV and mental health care service utilization outcomes compared to the standard CCDT. This involved comparing the PPV and mental health service utilization rates in Palorinya during CCDT+ implementation with those of five other refugee settlements in Uganda where standard CCDT was in place, using data from the SW-CRT for the latter.
We compared the PPV of detected cases between the SW-CRT and current study data over four months post-CCDT implementation period using logistic regression accounting for clustering within zones using a sandwich estimator. We compared the mental health care service utilization between the SW-CRT and current study data using a negative binomial regression model with a population size offset.
For both, the comparison data was restricted to the data collected during the same post-CCDT implementation timeline as the CCDT+ implementation period in Palorinya (i.e., four months post-CCDT+ implementation data in Palorinya were compared to the first four months of post-CCDT implementation data in the comparison settlements).
The distribution of usability, feasibility, acceptability and appropriateness indicators collected during post-interviews are presented as descriptive analyses. We explored whether these indicators varied by gatekeeper type using Kruskal–Wallis tests.
Qualitative analyses
A pragmatic approach to analysing the qualitative data was used, in line with the applied nature and aim of this study to gather experiences and feedback about the CCDT+ as an optimization strategy. We used a modified framework method (Ramanadhan et al., Reference Ramanadhan, Revette, Lee and Aveling2021; Ritchie and Spencer, Reference Ritchie and Spencer2002), with a hybrid inductive and deductive approach to the analysis. The process included familiarization, open-coding and thematic framework development. All transcripts were indexed based on the framework, charted in NVivo version 12 and interpreted per theme. A more detailed description of the process can be found in Supplementary Material S1, and the completed COREQ (consolidated criteria for reporting qualitative research) checklist can be found in Supplementary Material S2 (Tong et al., Reference Tong, Sainsbury and Craig2007).
Results
During the proof-of-concept period, 45 gatekeepers (33% female) were trained in the five zones in Palorinya. Gatekeepers detected 1026 children and adolescents as matching with the CCDT. On average, detected children and adolescents were 12.18 years of age (SD = 3.63) and 58.38% were male. Of the 1026 detected cases, 801 (78.1%) utilized TPO’s mental health care services for the first time or re-entered after not having sought help for at least six months. Among the group that sought help (n = 801), 656 children and adolescents were indicated to be in need of mental health care based on the clinical interview (PPV = 0.82; 95% CI: 0.79, 0.84), and 670 were diagnosed with a mental health condition corresponding to the CCDT or experienced severe distress (PPV = 0.84; 95% CI: 0.81, 0.86). The odds of accurate case detection (among children who utilized care for the first time or re-entered) was significantly higher in zones where the CCDT+ was implemented when compared to zones using standard CCDT. More specifically, there was a 2.34-fold increase in the odds of accurate case detection among children who utilized treatment based on the indication for treatment criterion (95% CI: 1.41, 3.83). Similarly, there was a 5.53-fold increase in the odds of accurate case detection among children who utilized treatment based on the diagnostic outcome criterion (95% CI: 3.94, 7.76). See Table 1.
Positive predictive value of the CCDT+ vs. CCDT
PPV = positive predictive value
1 One observation is missing information on the indication for treatment.
There was a 2.05-fold increase in the rate of mental health services utilization over time in the CCDT+ zones as compared to the zones that implemented the standard CCDT (95% CI: 1.09, 3.83). We observed a significant decline in utilization over time, which did not appear to differ across study conditions (IRR = 1.06, 95% CI: 0.70, 1.60). Similarly, case detection also declined over time in both conditions (IRR = 0.80, 95% CI: 0.59, 1.08). The rate of detection over time is 1.54 times higher in CCDT+ zones, however, this difference was not significant (95% CI: 0.62, 3.81). Settlement-specific utilization rates can be found in Supplementary Table S1.
The levels of acceptability, appropriateness, feasibility and usability of the CCDT+ as reported by gatekeepers and supervisors were high, see Supplementary Table S2. There were no significant differences in implementation outcomes by gatekeeper type.
Qualitative findings regarding the implementation of the CCDT+ were around; (1) work efficiency and effectiveness, (2) professional development, (3) perceived impact on work quality, and (4) role and expectations. The main findings, themes and key quotes are presented in Table 2.
Key themes regarding the implementation of the CCDT+
Theme 1. Work efficiency and effectiveness
Supervisors found the dashboard useful for daily tasks, particularly for guiding community outreach efforts, monitoring gatekeepers’ performance and identifying areas needing attention during supervision. The insights presented in the dashboard combined with feedback provided by gatekeepers – such as reasons for individuals not seeking help – allowed for more efficient outreach scheduling by the supervisors. Furthermore, supervisors observed an increase in help-seeking during the period of implementation, which was a motivating factor for supervisors. The main challenges supervisors experienced were related to the technological aspects of the dashboard. Issues such as data errors and limited access to the dashboard due to license issues impacted follow-ups and outreach planning. Gatekeepers perceived the MI techniques and reminders as enhancing their effectiveness in promoting help-seeking. Additionally, the information shared by supervisors enabled gatekeepers to plan their mobilization efforts more precisely. One related key recommendation from gatekeepers was to improve coordination between gatekeepers and service providers to ensure that gatekeepers can share up-to-date information about when and where services will be available.
Theme 2. Professional development
Supervisors and gatekeepers both valued the feedback loops from supervisor to gatekeeper and gatekeeper to supervisor as key motivators in their work. It was seen as confirming the positive outcomes of their efforts and enhancing their sense of accomplishment and effectiveness. Supervisors appreciated the use of the dashboard as a new skill they learned, which enhanced their supervision capabilities. In addition, having access to this type of data was seen as unique for teams implementing projects. Gatekeepers valued both positive and negative feedback, this boosted their confidence, kept them motivated and minimized mistakes. Ongoing capacity strengthening during the supervision meetings helped gatekeepers recall forgotten aspects of the training and addressed new questions that came up from practical implementation. The supervision meetings provided a supportive environment where challenges were openly discussed and practical solutions were developed. This opportunity to receive and provide peer support was another important element for gatekeepers.
Theme 3. Work quality
The dashboard enabled supervisors to identify trends and inconsistencies in the data nearly in real time. Supervisors used this to continue capacity-strengthening activities with gatekeepers in a group and allowed for more precise and individual training if certain areas had to be improved by specific gatekeepers. After conducting these sessions, supervisors noticed increases in true positive rates. Gatekeepers played an active role in setting the agenda for the supervision meetings. The additional training during the supervision sessions was appreciated by gatekeepers, not only to correct mistakes but also to refresh certain skills and practice.
Theme 4. Role and expectations
The dashboard aligned well with the work of supervisors. For gatekeepers, the activities aligned particularly well with those who were already conducting household visits. The main challenge with reminding people to seek help and the more frequent interaction between gatekeepers and families was that families often asked for details regarding the care that was provided, which gatekeepers did not know due to confidentiality measures. Gatekeepers therefore sometimes struggled to provide satisfactory answers. Additionally, families sometimes expected material goods and questioned gatekeepers when these were not provided, which posed a challenge for the gatekeepers and affected their status within the community. Despite the role as a gatekeeper being voluntary, gatekeepers appreciated the small transportation refunds and breakfast provided. This minimal compensation was crucial for their motivation and ability to support their own families. It was recommended to increase the transport refund based on distance, provide relevant material goods and organize more frequent meetings in central locations.
Discussion
The gap between the need for mental health care among children and adolescents and its provision is a global issue. Given the scarcity of mental health resources in most LMICs, optimization strategies are essential to monitor and improve the quality of evidence-based detection tools. These strategies can contribute to a more efficient use of limited resources. In this proof-of-concept study, we evaluated the CCDT+, an optimization strategy for a tool developed to detect children in need of mental health care and promote help-seeking.
In areas where the CCDT+ was implemented, the PPVs were high and consistent across both reference criteria: needing mental health services (PPV=0.82) and the presence of any mental health condition matching the CCDT or severe distress (PPV=0.84). Furthermore, the odds of accurate detection were significantly higher, in fact, more than two times as high in zones using the CCDT+ compared to those using the standard CCDT. This suggests that the CCDT+ reduces false positives and alleviates unnecessary burden on mental health services and discomfort for children. A key element of the optimization strategy was the data-driven supervision which included ongoing feedback for (individual) gatekeepers about the percentage of children they detected who met criteria for mental health services out of the total number detected. If more than one in four children did not meet the criteria to receive services, individual gatekeepers received extra training during supervision. This ongoing feedback could have improved the accuracy of detection and reduced the number of false positives.
Comparing the PPV found in this study with that of traditional mental health screening tools suggests that the CCDT+ may be more accurate in detecting mental health conditions. The PPV of the PHQ-9 for instance was reported as 0.23 in Kenya and 0.17–0.37 in South Africa (Marlow et al., Reference Marlow, Skeen, Grieve, Carvajal-Velez, Åhs, Kohrt, Requejo, Stewart, Henry, Goldstone, Kara and Tomlinson2023; Tele et al., Reference Tele, Carvajal-Velez, Nyongesa, Ahs, Mwaniga, Kathono, Yator, Njuguna, Kanyanya, Amin, Kohrt, Wambua and Kumar2023). However, caution is needed in this comparison, as we are comparing the accuracy against a broad range of diagnoses, whereas symptom checklists are often evaluated against specific diagnoses. Furthermore, existing tools require validation to establish local cutoffs – a time-consuming process, and after validation, the false positive rate often does not change with ongoing use. The optimization strategy presented here is an embedded quality-improvement process for mental health detection tools which has the potential to enhance the accuracy of referral over time and in real-time. The quality-improvement aspect was also appreciated by supervisors and gatekeepers. For supervisors, the CCDT+ not only allowed them to monitor the performance of specific gatekeepers but also facilitated more precise, individualised training, potentially an important factor in boosting the accuracy results discussed above. According to gatekeepers, feedback on performance, creating ongoing learning opportunities, having access to a supportive group of peers and receiving regular updates on their work served as key motivators.
We observed an overall 2-fold increase in the rate of mental health services utilization, while no significant difference in the case detection rates was observed between study conditions. This is an important finding, as other existing tools only focus on the identification of symptoms and lack an integrated help-seeking component. Our results suggest that the combination of data-driven supervision, the use of MI techniques and behavioural nudges by gatekeepers may have facilitated the transition from intentions to actual help-seeking behaviours among those detected. While this proof-of-concept demonstrates the promise of the use of MI techniques by key community members (Lawrence et al., Reference Lawrence, Fulbrook, Somerset and Schulz2017; Naar-King et al., Reference Naar-King, Outlaw, Green-Jones, Wright and Parsons2009), the effectiveness of the CCDT+ and which components are active or which dose leads to the best outcomes, will need to be evaluated using more rigorous research designs.
An important consideration in the design of the dashboard was to avoid over-detection and we therefore did not assign a threshold or target for the number of children detected. A steady rate of detection with improved accuracy and help-seeking rate in this study was therefore regarded as a positive, expected finding. Another anticipated outcome of the optimization strategy was to find a sustained or even improved impact of the CCDT over time. While this held true for accuracy outcomes, we noted a decline in mental health utilization over time, like the standard CCDT. The observation of this decline in both conditions suggests that after a certain period, the majority of cases in a given area may have been identified and sought assistance.
The qualitative findings indicated several areas for strengthening the CCDT+. First, close collaboration between gatekeepers, who mobilize families and service providers, who organize outreach services, became increasingly important with the implementation of behavioural nudges. Gatekeepers emphasized the need for up-to-date information on when and where services would be available. Second, families frequently requested information about the care provided, which gatekeepers were unable to share due to confidentiality protocols. To address this need, we recommend future initiatives that aim to promote help-seeking to include a feature enabling gatekeepers to give families broad, non-confidential updates on care progress. Finally, supervisors stressed the importance of having continuous, real-time access to detection and utilization data. Replacing paper-based detection data with digitally collected data could be one way to improve access to real-time information.
Several limitations merit attention when interpreting the results of this study. Although the comparison data was drawn from the same project, from a similar setting in Uganda, following similar procedures, the data were technically collected separately, using a different study design and at a different time point (up to 12 months earlier). Additionally, the CCDT+ gatekeeper training was a half day longer compared to the standard CCDT training. The accuracy findings relied on routinely collected data and included only children who sought help. Furthermore, the supervisors using the dashboard were also responsible for assessing mental health outcomes used for accuracy testing, potentially introducing confirmation bias. Another limitation is that we could only report the accuracy of cases that sought help; thus, false positives might have self-selected themselves out of this study. Finally, proactive case detection needs to be accompanied by accessible, quality mental health services. In this study, a partnership with TPO Uganda, a national mental health care provider, was established to support service provision; however, assessing the quality of care delivered was beyond the scope of this study.
Conclusions
Implementing optimization strategies that monitor and improve the quality of evidence-based detection tools can contribute to more efficient use of mental health care resources. The CCDT+ shows promise as an embedded quality-optimization process that integrates data-driven supervision with MI techniques and behavioural nudges to enhance the detection of mental health problems among children and promote help-seeking. This proof-of-concept study indicates that the CCDT+ may not only improve the accuracy of detection but also enhance the effectiveness of help-seeking promotion among children compared to the standard CCDT. Furthermore, it highlights some important areas for improvement. Further research is needed to evaluate the effectiveness of the different elements of the CCDT+ and the techniques used.
Abbreviations
- AIM
-
Acceptability of Intervention Measure
- CCDT
-
Community Case Detection Tool
- CCDT+
-
Community Case Detection Tool+ (optimization strategy)
- CIDT
-
Community Informant Detection Tool
- FGD
-
Focus group discussion
- FIM
-
Feasibility of Intervention Measure
- IAM
-
Intervention Appropriateness Measure
- IUS
-
Intervention Usability Scale
- KII
-
Key-informant interview
- PPV
-
Positive Predictive Value
- TPO Uganda
-
Transcultural Psychosocial Organization Uganda
Open peer review
To view the open peer review materials for this article, please visit http://doi.org/10.1017/gmh.2024.150.
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/gmh.2024.150.
Data availability statement
The data will be available after article publication from the principal investigator at: mark.jordans@warchild.net. Data sharing requests will be assessed by a data use team, comprising of the principal investigators Prof Mark Jordans and Dr Rosco Kasujja, and investigators Dr M. Claire Greene, Myrthe van den Broek and Sandra Agondeze and shared after a data sharing agreement has been signed.
Acknowledgements
We thank all gatekeepers, children and their families for spending time in this study and acknowledge all contributions of staff from War Child Uganda and TPO Uganda. War Child and TPO Uganda were supported by Sint Antonius Stichting Projects (SAS-P) [Grant number: SAS-P-21103-UG]. The foundation had no role in the study design and implementation. We would also like to thank Deloitte for their support in developing the initial version of the dashboard through Deloitte Impact Foundation.
Author contribution
MJ and MvdB were responsible for the funding acquisition. Conceptualization of the study was done by MvdB and MJ and all authors were involved in designing and conducting the research, including (procedures related to) data collection. Management and coordination were done by SA, AFG, OI, MvdB and MJ. MCG was responsible for all formal analysis, preparation and creation of data presentation. SA, AFG, MvdB, MCG and MJ accessed and verified the data. All authors were responsible for the decision to submit the manuscript. The writing and preparation of the draft manuscript was done by MvdB and all authors were part of the reviewing and editing process. Overall supervision was done by MJ.
Financial support
War Child was supported by Sint Antonius Stichting Projects (SAS-P) [Grant number: SAS-P-21103-UG]. The foundation had no role in the study design and implementation.
Competing interest
None declared.
Ethics statement
This study was reviewed and approved by Makerere University School of Health Sciences Research and Ethics Committee (MAKSHSREC-2022-416) and Uganda’s National Council for Science and Technology (HS2609ES). Gatekeepers, social workers and the clinical psychologist participating in this study provided written informed consent. Service level consent and assent to document and release de-identified routine mental health service utilization data with the research team was obtained for study purposes from caregivers, adolescents and children detected by the CCDT that sought help.
Open access
Comments
Subject: Manuscript Submission for Publication
September 9, 2024
Dear Professor Judy Bass and Professor Dixon Chibanda,
On behalf of the author group, I would like to submit our manuscript titled ‘Data-driven supervision to optimize the effectiveness of proactive case detection for mental health care among children: a proof-of-concept study’ for publication in Cambridge Prisms: Global Mental Health.
In our manuscript we present the findings of a proof-of-concept study of the CCDT+, an enhanced version of the Community Case Detection Tool (CCDT), a tool developed for key community members to support proactive community-level detection and enhance help-seeking among children in need of mental health care. The CCDT+ includes a dashboard presenting actionable outcomes used for data-driven supervision and integrates Motivational Interviewing techniques and behavioural nudges in the training of community members using the CCDT to promote help-seeking. This mixed-methods study, conducted from January to May 2023 in the Palorinya refugee settlement in Uganda, assessed the added value of the CCDT+ in improving detection accuracy and mental health service utilization compared to the standard CCDT, which was recently evaluated in a stepped wedge cluster randomized trial (van den Broek et al. 2024).
The results demonstrate that among the group that sought help (n=801), 656 children and adolescents were indicated to be in need of mental health care based on the outcomes of a clinical interview (PPV=0.82; 95% CI: 0.79, 0.84). Furthermore, the CCDT+ significantly enhanced detection accuracy, with a 2.34-fold increase in the odds of accurate detection and a 2.05-fold increase in the rate of mental health service utilization over time, compared to the standard CCDT.
The CCDT+ introduces an embedded quality-improvement process for mental health detection tools and shows promise in enhancing the accuracy of referral over time and in real-time. Optimization strategies like the CCDT+ can contribute to the more effective use of scarce resources, which is especially important given the limited availability of mental health services in most low- and middle-income countries (LMICs) and the growing global mental health crisis (Patel et al., 2023).
We believe that our study seamlessly aligns with the focus of Cambridge Prisms: Global Mental Health and contributes a valuable and promising innovation to the mental health care gap, a matter of worldwide concern for policymakers, clinicians, and researchers.
All named authors have approved the manuscript for submission. The content of this manuscript has not been published elsewhere and will not be submitted to any other journal while under consideration by Cambridge Prisms: Global Mental Health. We hope you will find our manuscript interesting and suitable for publication.
Yours sincerely,
Prof. Dr. Mark J. D. Jordans
Director Research and Development, War Child Alliance
Amsterdam Institute for Social Science Research, University of Amsterdam
Mark.jordans@warchild.net
References
van den Broek M, Agondeze S, Greene C, Kasujja R, Guevara AF, Tukahiirwa RK, Kohrt BA and Jordans MJD (2024) A community case detection tool to promote help-seeking for mental health care among children and adolescents in Ugandan refugee settlements: a stepped wedge cluster randomised trial. The Lancet Child & Adolescent Health 8, 571–579. https://doi.org/10.1186/ISRCTN19056780.