CLINICIAN’S CAPSULE
What is known about the topic?
Emergency clinicians are not specifically trained to assess and manage mental health emergencies; thus, they have wide practice variation while determining disposition for children with mental health crises.
What did this study ask?
What is the inter-user reliability of a new emergency psychosocial assessment and management guiding tool, HEARTSMAP, and the impact of implementing it in a pediatric emergency department?
What did this study find?
The tool demonstrated strong inter-rater reliability between PED clinicians and was associated with high rates of outpatient resource connectivity.
Why does this study matter to clinicians?
HEARTSMAP can provide emergency clinicians with reliable and comprehensive assessment and management strategies for youth presenting with mental health-related concerns
INTRODUCTION
More than 15% of North American adolescents experience mental health concerns1, Reference Kirby and Keon2; however, youth struggle to access and receive care because of prolonged wait times and lack of communication between care providers.3 As a result, pediatric emergency departments (PEDs) face increasing numbers of youth presenting with mental health concerns.Reference Cloutier, Kennedy and Maysenhoelder4-Reference Newton, Ali and Johnson6 Across North America, mental health-related presentations are the fastest growing segment of PED visits, accounting for 3.4% of emergency department (ED) visits in the United StatesReference Larkin, Claassen, Emond, Pelletier and Camargo7-Reference McCaig and Nawar10 and increasing to 3%–7% per year since 2002 in Canada.Reference Newton, Ali and Johnson6, Reference Mapelli, Black and Doan11
Assessments of these youth in PEDs are challenging and often conducted by a variety of clinicians (students, residents, pediatricians, and specialists), resulting in significant practice variation in assessment, mental health service referrals, and disposition.Reference Reder and Quan12 Furthermore, unfamiliarity with multifactorial components affecting risk acuity may lead clinicians to over-rely on emergency psychiatric consultation, prolonging the length of stay (LOS) in an already strained ED.Reference Mapelli, Black and Doan11
The American Academy of Pediatrics identified a need for clinical screening tools to better detect mental illness in the ED.Reference Dolan and Fein5 To address this, Cappelli et al.Reference Cappelli, Gray and Zemek13 developed “HEADS-ED,” a tool in which clinicians rate the severity of psychosocial concerns across seven sections to determine whether youth require immediate psychiatric intervention or can be redirected to an on-site mental health team. A three-point Likert scale is used for each section and triggers psychiatric intervention for a cumulative score of seven or more or if suicidality exceeds a threshold score. Though a Psychiatric Times article stated that HEADS-ED would incorporate resource recommendations in the future,Reference Cappelli14 it currently does not provide guidance as to the types or urgency of services required and does not distinguish psychiatric from social or behavioural needs.Reference Cappelli15 Therefore, the dichotomous outcome of HEADS-ED does not provide sufficient management support for EDs without an on-site mental health team, in which ED physicians are solely responsible for determining disposition and community referrals.Reference Dolan and Fein5
To overcome these limitations, we created HEARTSMAP, an online platform providing the capacity to embed a complex referral algorithm (Figure 1). There are ten sections to help facilitate a full but efficient emergency psychosocial assessment: Home, Education and activities, Alcohol and drugs, Relationships and bullying, Thoughts and anxiety, Safety, sexual health, Mood and behaviour, Abuse, and Professionals and resources. Sample questions and free-form textboxes for notes are provided for each section. Sections are scored from 0 to 3: 0 for no concerns; and 1, 2, or 3 for mild, moderate, or severe concerns, respectively. HEARTSMAP differs in structure from HEADS-ED in that it has two additional sections, Sexual health and Abuse, to help determine appropriate adolescent health or social work needs. Furthermore, HEARTSMAP distinguishes thought disorders from abnormal behaviour resulting from other mental health issues (e.g., mood disorders), by separating the “Emotions, behaviours, and thought disturbance” section of HEADS-ED into “Thoughts and anxiety” and “Mood and behaviour,” allowing clinicians to differentiate between and grade the severity of these two psychiatric symptoms separately. HEADS-ED has a “Discharge resources” section to capture existing care plans, and HEARTSMAP users must indicate whether other professionals or services are currently in place for each section, acknowledging that varying resources may be established for different issues. Scores from each HEARTSMAP section map to at least one of five domains: 1) social, 2) function, 3) youth health, 4) psychiatry, and 5) abuse. Management recommendations based on scoring patterns and cumulative domain scores, as well as an indication of services already in place, capture both the severity and urgency of need. In addition, HEARTSMAP guides management, distinguishing between psychiatric and other psychosocial needs including crisis response teams, social work, youth health specialists, substance abuse services, and detoxification programs or redirection to already established care providers.
Figure 1. HEARTSMAP recommendation logics
The goal of this study was to evaluate: 1) the psychometric properties of HEARTSMAP; and 2) the impact of a pilot implementation of HEARTSMAP on patient outcomes and system flow.
METHODS
Objectives and outcome measures
Our primary objective was to first evaluate the inter-rater reliability and predictive validity of the HEARTSMAP tool. For predictive validity, we looked at an acute psychiatric consultation in the PED for consideration of hospitalization, as PED disposition is the predominant clinical decision made in this context. Psychiatric hospitalization within 30 days of the PED index visit (including the index visit) was the primary outcome measure for reliability and predictive validity, although analyses were conducted for all other levels and types of management recommendations triggered by HEARTSMAP. A psychiatric hospitalization was defined as an admission of a patient to the psychiatric ward or under the psychiatrist on a medical ward. Following validation of HEARTSMAP, we performed a pilot implementation. Our objectives related to implementation were to measure the impact of HEARTSMAP on 1) patient outcomes; 2) proportion receiving psychiatric consultation in the PED; 3) the LOS (defined as the time between subject arrival and disposition) for PED mental health-related visits; and 4) access to community mental health services following the PED presentation.
Setting
The study was conducted at British Columbia Children’s Hospital (BCCH) in Vancouver, the only tertiary PED in British Columbia, Canada. BCCH receives >40,000 visits annually, of which approximately 1,000 are related to mental health. Our PED is primarily staffed by Royal College–certified pediatric emergency medicine (PEM) subspecialists with nurse practitioner (NP) support. At BCCH, admission to the psychiatric ward requires an emergency psychiatric consultation, and if obtained, the final disposition (admission v. discharge) is determined by the psychiatrist. The time period for the retrospective validation study was between 1 October to 30 November 2013, and 1 February to 31 March 2014. The prospective implementation study occurred between 14 October to 19 December 2014, and 1 February to 31 March 2015. These time periods coincided with local peak volumes of mental health-related visits and were chosen to capture the highest number of cases while minimizing research assistant coverage.
Participants
To measure the psychometric properties of HEARTSMAP and to evaluate the impact of HEARTSMAP on the PED flow and system utilization, data were collected from two populations: a retrospective and prospective cohort.
Our retrospective cohort consisted of a random sample of mental health-related PED visits of patients aged <17 years. Using an administrative database containing records of all BCCH PED visits, mental health-related presentations were identified from chief complaints and discharge diagnoses. Inclusion criteria included terms such as depression, anxiety, suicidal ideation, self-harm, substance abuse, mood, eating, behavioural, and psychotic disorders or their permutations in either the chief complaint or discharge diagnosis. Only cases containing complete psychosocial assessments (for which HEARTSMAP could be applied) were included. Exclusion criteria included patients who were medically unstable for a full PED psychosocial assessment (e.g., severe intoxication or psychosis). Health records with mental health-related chief complaints or discharge diagnoses were identified. Using a random number generator, records were assigned a subject number and then were sorted chronologically. Charts were retrieved in chronologic order until the sample size was met. Data were obtained from 104 medical records that met the inclusion criteria.
Our prospective cohort included youth aged <17 years who were identified by the triage nurse as presenting to the BCCH PED for a mental health-related complaint and placed in the mental health assessment room. These usually included youth with chief complaints including depression, anxiety, suicidal ideation, self-harm, substance abuse, mood, eating, behavioural, psychotic disorders, or their permutations. Exclusion criteria included patients who were medically unstable for a full psychosocial assessment and unaccompanied patients from whom consent could not be obtained. Individual clinicians including PED physicians, clinical trainees, and NPs prospectively evaluated enrolled subjects using HEARTSMAP.
The University of British Columbia Children’s and Women’s Health Centre of British Columbia Research Ethics Board granted approval. A waiver of consent was obtained for retrospective subjects. Written informed consent was received from a parent/legal guardian for prospective subjects.
Study protocol
Study members reviewed health records pre-implementation for eligible subjects and collected the following variables: 1) demographic information (age, gender, and triage acuity, as well as the first three characters of their residential postal code linked to neighbourhood average total household income as a proxy measure of socioeconomic status); 2) clinical information relevant to the HEARTSMAP sections; and 3) measures of PED flow, utilization, disposition, and return visit information. Data were entered into a standardized online form using Research Electronic Data Capture (REDCap) software to create a clinical vignette. REDCap is a secure web-based application designed to support data capture in clinical research studies.Reference Harris, Taylor and Thielke16 Clinician evaluators including two emergency physicians, an NP, and a registered nurse independently applied HEARTSMAP to all retrospective cases. All evaluators had access only to clinical vignettes and were blinded to management decisions and outcomes. Data analysis pertaining to inter-rater consistency and predictive validity were assessed before piloting the tool in the PED.
As part of the pilot implementation of HEARTSMAP, all PEM clinicians including retrospective chart evaluators underwent orientation to the tool consisting of in-person training over two academic half days. Those who were not in attendance were provided access to online materials and video recordings.Reference Doan and Black17 In-person training included familiarizing clinicians with the tool itself, demonstrating its use, and then having clinicians apply the tool to case examples. Once the clinicians accurately demonstrated their competency by accurately scoring each section for three online cases, they were given access to the clinical tool.
During the pilot implementation period, clinicians on shift were encouraged to use the HEARTSMAP to conduct their initial assessment. Research assistants were present to remind physicians of the tool and facilitated its use by providing clinicians with an iPad to complete their assessment. Research assistants also approached patients with mental health-related complaints and obtained consent to collect the same data as the retrospective cohort with the addition of the subjects’ HEARTSMAP scores, triggered recommendations, and access to community services to address mental health problems. While recommendations for the tool could be accessed by the treating PED clinician, the final management and disposition decisions were left to the discretion of that clinician. Clinicians involved in the retrospective review did not contribute to prospective cases.
Analytic approach
Cohen’s kappa (κ) statistic was used to measure inter-rater agreement between the clinicians’ evaluations of the retrospective cases. Specifically, we assessed the agreement of the HEARTSMAP-triggered recommendations relating to service referrals, namely social (social worker), youth health (adolescent medicine), and psychiatry. For psychiatry, we evaluated agreement for each level of resources offered: 1) ED psychiatry consultation; 2) outpatient crisis response team referral; and 3) less acute community mental health team referrals.
The predictive validity for the tool, reported as sensitivity and specificity in identifying subjects requiring a psychiatric admission or a PED return visit, was evaluated for both retrospective and prospective cohorts. Psychiatric hospitalization could occur only if an emergency psychiatric assessment was obtained in the PED.
Hypothesizing a κ value of 0.7, with a 95% confidence interval (CI) spanning 0.2 (±0.1), the sample size required for the inter-user agreement was 80 cases.
We evaluated the impact of implementing HEARTSMAP on PED flow, utilization, and patient outcomes by comparing our retrospective to our prospective cohort. The retrospective (pre-implementation) cohort served as the control group.
We used descriptive statistics to summarize results of other outcomes. Continuous variables (e.g., LOS) are presented as medians and inter-quartile range and age as mean and CIs. Binary or categorical variables including psychiatry consultation (yes or no), disposition outcomes, and return visits are presented as proportions with 95% CIs.
We used the Mann-Whitney U test to compare the LOS and wait time for the subjects between study periods, as these were non-normally distributed. Patient outcomes were compared using the chi-square test. All statistical tests were conducted using IBM SPSS Statistics (v. 23).
RESULTS
One hundred forty-six randomly selected patient charts were reviewed, and 104 subjects met the inclusion criteria for the retrospective validation phase. During the pilot implementation of HEARTSMAP, we approached 83 patients, of whom 70 subjects consented.
Demographic information for the study subjects is presented in Table 1. The mean age, sex, and triage acuity were comparable between both cohorts. The triage acuity distribution for the whole PED during both periods remained consistent.
Table 1. Study population demographic and visit characteristics
CTAS=Canadian Triage and Acuity Scale.
* CTAS is a tool used in Canadian emergency departments to triage patients into five categories based on the acuity and severity of presenting symptoms. CTAS Level 1=resuscitation; Level 2=emergent; Level 3=urgent; Level 4=less urgent (semi-urgent); Level 5=non-urgent.
In the retrospective cohort, we found substantial agreement between pediatric emergency physician reviewers regarding triggering recommendations for acute psychiatric consultation (κ=0.7), as well as moderate agreement for outpatient crisis response team referrals (κ=0.4) and Child and Youth Mental Health team for non-urgent psychiatric referrals (κ=0.4) (Table 2). HEARTSMAP had a sensitivity of 76.2% (95% CI: 62.8–89.5%) for predicting admission or ED return visit and a specificity of 64.8% (95% CI 54.6–75.1%).
Table 2. Retrospective intra- and inter-professional tool reliability regarding consultation of services
NP=nurse practitioner; PED=pediatric emergency department; RN=registered nurse (bedside nurse).
During the pilot implementation of HEARTSMAP, we approached 83 patients, of whom 70 consented, but only 62 had HEARTSMAP applied. Failure to utilize HEARTSMAP in 8/70 subjects reflects that clinicians had not universally adopted HEARTSMAP as their standard clinical assessment approach, despite ongoing pilot implementation and evaluation activities. Among subjects for whom HEARTSMAP was applied, the tool recommended acute psychiatric consultation in 46 (74%) patients. As clinicians were able to decide final management and disposition regardless of the recommendations based on HEARTSMAP, 39 (63%) subjects received psychiatric consultations, and 13 (21%) were admitted by the psychiatry team at their index visit. In addition, 8 (13%) subjects returned to the PED within 30 days of their index visit, of whom four were subsequently admitted for psychiatric management. In total, 17 subjects were admitted (either at their index presentation or within 30 days of it). All 17 (100%) admitted subjects had received a HEARTSMAP assessment indicating a recommendation for urgent psychiatry consultation at their index PED visit. Patient outcomes, disposition, and measures of flow are reported in Table 3. Of note, the ED clinician did not consult psychiatry on one subject despite the HEARTSMAP recommendation, and this subject returned twice and was admitted on the third presentation.
Table 3. Measures of ED flow and patient disposition for patients with mental health complaints evaluated using HEARTSMAP
CI=confidence interval; ED=emergency department; IQR=inter-quartile range; LOS: length of stay; PED=pediatric emergency department; WT=wait time.
* Statistically significant
† Total hospitalization within 30 days of index visit including those at index visit
The median LOS among discharged youth with mental health complaints increased slightly from 288.0 to 297.0 minutes (p=0.89) in the pre- to post-implementation periods. For admitted patients, the median LOS increased from 311.0 to 413.0 minutes (p=0.05). In contrast, the median LOS for the whole PED population remained constant at 160.0 minutes through both periods.
There were eight subjects who consented to the study for whom HEARTSMAP was not applied. Of these eight, one chose to leave the PED before being seen by a physician. Four patients received psychiatry consultation, and three of these patients were admitted at their index presentation. One patient returned to the ED within 30 days of the initial presentation and was subsequently admitted for further psychiatric management. Flow measures were similar to the cohort that had received HEARTSMAP.
At the two-week follow-up, all patients who had HEARTSMAP-triggered recommendations for further community services had accessed those resources. Table 4 shows the types of resources accessed following the ED visit, showing a wide range of types and level of acuity of services.
Table 4. Community resources accessed by patients enrolled in the pilot HEARTSMAP implementation study
* Includes parental counselling, family counselling, and family preservation services
DISCUSSION
HEARTSMAP is a new reliable youth mental health assessment and management tool for PED clinicians and was highly sensitive for identifying youth requiring psychiatric admission in our prospective cohort. We found a strong inter-rater agreement of 0.69 among physicians with regards to psychiatric consultation. This demonstrates the reliability of HEARTSMAP for triggering a psychiatric assessment in the ED that is helpful for standardizing a mental health assessment in settings in which providers may have varying levels of training and mental health expertise. The impact of the tool on ED flow was mixed, with a slight increase in median LOS for youth requiring psychiatric consultation and admission.
The inter-rater agreement of HEARTSMAP was similar to that reported by Capelli et al. for HEADS-ED, who found an agreement of 0.79 between crisis intervention workers and research assistants in their initial validation study.Reference Cappelli, Gray and Zemek13 In their most recent work comparing assessments between crisis intervention workers and PED physicians, interclass correlation coefficients for sections pertaining to psychiatric issues and dispositions were slightly lower (Emotions and Behaviour: 0.208, Suicidality: 0.529, and Discharge and Resources: 0.292).Reference Cappelli, Zemek and Polihronis18 In that study, the authors used prospectively enrolled subjects, but our validation phase used extracted clinical vignettes. Despite these methodological differences, clinical vignettes have been found to provide similar results to the use of standardized patients and were comparable to those of real clinical cases.Reference Peabody, Luck and Glassman19, Reference Veloski, Tai, Evans and Nash20
In our prospective study, 100% of the patients who were ultimately admitted for further psychiatric care were identified by HEARTSMAP as requiring ED acute psychiatric consultation, although the reviewers were not blinded to the triggered recommendations. We were not able to capture how frequently HEARTSMAP might have impacted the decision-making of PED clinicians. It is possible that the recommendations triggered by the use of the tool altered the PED clinician’s management of the patient. This might have potentially led to the consultation of psychiatry in the ED for some patients who might otherwise have been discharged or conversely led to outpatient referrals for patients whom a clinician might otherwise have requested an acute psychiatric consultation. In either case, our results suggest that the implementation of HEARTSMAP resulted in safe recommendations. HEARTSMAP recommended an ED psychiatric consultation rather than outpatient assessment and care for all patients who were admitted for psychiatric treatment. No patient for whom HEARTSMAP recommended discharge to community services returned or were redirected to the ED within 30 days of their index visit and admitted to hospital.
With regard to patient follow-up, 100% of subjects subsequently connected with mental health resources within 30 days of their PED visit during the prospective cohort study period. However, in conjunction with the ED implementation of HEARTSMAP, an emergency outpatient follow-up service was created for families with unmet mental health needs (the “LINK” clinic) that was incorporated as an option within the outpatient HEARTSMAP tool recommendations. The success in outpatient follow-up, therefore, likely reflects the combined effects of the tool-based recommendations and this rapid-access resource. Previous studies of mental health follow-up from the PED reported significantly lower rates: a study by Sobolewski et al.Reference Sobolewski, Richey, Kowatch and Grupp-Phelan21 found a 66% follow-up rate for those referred to mental health services after PED discharges; and Bridge and colleaguesReference Bridge, Marcus and Olfson22 described a 43% mental health follow-up rate among youth discharged from the ED after self-harm. Given the success of our subjects connecting with resources after their PED visit, similar initiatives may benefit other centres with low rates of patient connectivity to community mental health resources.
While we attempted to evaluate the impact of HEARTSMAP on PED flow and resource utilization, the pilot implementation sample size was relatively small and did not allow many observed trends to reach statistical significance. We observed a significantly longer post-implementation LOS among patients requiring psychiatry consultation or hospitalization despite similar proportions of patients receiving psychiatry consultation in the control group (68.4% [95% CI 58.1–77.6%] and 61.4% [49.0–72.8%], respectively). It is possible that use of the HEARTSMAP tool contributed to this observed increase in LOS. However, as the tool makes recommendations only after all sections are completed, any additional LOS from the use of the tool would apply equally to patients with a HEARTSMAP recommendation of discharge with an outpatient referral. Many external factors affect LOS for those requiring admission or consultation, including bed availability and psychiatry work load, and our findings are likely multifactorial.
LIMITATIONS
Limitations include a small prospective sample size, which was associated with our small effect size on measures of ED flow, thus preventing us from drawing definitive conclusions regarding the impact of HEARTSMAP on secondary outcomes because of a lack of power. Unaccompanied youth and those in the care of the ministry could not consent to the study, and patients who were too medically unstable for a full psychosocial assessment (e.g., severe intoxication and psychosis) were excluded. These two populations may represent a vulnerable group at increased risk for hospitalization for whom we could not evaluate the impact of HEARTSMAP. The study was conducted at a single PED staffed with PEM subspecialists, thus limiting the generalizability of findings to other ED settings. Finally, comparisons between our retrospective and prospective cohorts with regard to measures of flow must take into consideration the environmental and time-constraint differences between the use of the tool in research (non-clinical) and clinical environments.
While HEARTSMAP shows promise in identifying patients requiring acute psychiatric evaluation and admission, it is not intended to predict which youth will experience suicide completion or attempt following a PED visit. Overall, low rates of suicide among adolescents (0.0049% annually across North America) make it difficult to identify sensitive predictive risk factors, and previous attempts have only contributed a small additional predictive value above random chance.Reference Ribeiro, Franklin and Fox23-Reference Gould, Greenberg, Velting and Shaffer25 Large multi-centre studies would be required to examine the relationship between PED assessment for suicide risk and completed suicide as an outcome.
CONCLUSION
HEARTSMAP is a new standardized PED mental health tool that facilitates psychosocial assessments and provides risk-based and resource-specific management strategies, differentiating among psychiatric, social, and behavioural needs. The tool demonstrated strong inter-rater reliability between PED clinicians and was associated with high rates of outpatient resource connectivity. Future studies are needed to validate these findings further and to substantiate flow and utilization effects across a range of ED settings.
Competing interests
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors. None declared.
