The ability to set policy for disaster response relies on data to inform decision making. The greater the health consequences, the more important quality data is for ensuring appropriate resource allocation. A crisis in mental health among adolescents following the onset of the COVID-19 pandemic has been well documented; however, there is a paucity of well-designed studies that can help direct future responses to mental health crises. In a review of available studies by Theberath,Reference Theberath, Bauer and Chen1 it is clear that COVID-19 has had disproportionate effects on the mental health outcomes in some populations and that there is a need for additional research on groups with underlying risk factors.
To examine the needs and resources available in under-resourced communities in the US, Children’s Health Fund’s national network of pediatric practices sought to describe how the need and availability of resources evolved during the pandemic across its network. Unfortunately, this effort was stymied by the lack of available data and poor prioritization by network practices. This paper describes the planned study design and the consequent limitations in data that prevented enacting this study. In this way, this report provides a case example and lessons learned in attempting to collect and utilize patient and practice data to inform resource allocation following the COVID-19 pandemic. This report intends to help institutions and organizations proactively prepare to organize and package the most critical data needed during a health crisis. In addition to providing a case example of resource limitations that hinder the creation of informed policy during disaster, this paper demonstrates a process for evaluating system failures during disaster response.
Methods
The study utilized electronic health record data available through the national network of Children’s Health Fund’s partner programs. Children’s Health Fund supports a coalition of pediatric programs throughout the US that provide quality health care to children and families living in under-resourced communities. Founded in 1987 as a response to the crisis of children living in homeless shelters in New York City, today the network is composed of 24 partner programs operating in 15 states, Washington, D.C., and Puerto Rico. The national network is unique because it is composed of health care providers who work at the community level, building trust and providing resource-intensive care that children in these communities would not otherwise receive. Data and experiences from these providers are used to direct interventions and advocate for policy change. This includes work to increase access to medical care after disastersReference Brito2 and in those facing transportation barriers.Reference Grant, Goldsmith and Gracy3
With the onset of the mental health crisis that developed in the wake of the COVID-19 pandemic, Children’s Health Fund sought to estimate the change in prevalence of adolescent depression before and after the start of the COVID-19 pandemic, along with an estimation of the resources available to manage this need. The plan was to extract and analyze results of the Patient Health Questionnaire (PHQ) from network sites, as well as demographic and practice-level data, mirroring a similar effort in a patient population with private insurance.Reference Mayne, Hannan and Davis4 The PHQ-2 is a standardized screening tool that is used widely by pediatricians to screen for depression in patients 12 years of age and older. It consists of 2 questions which serve as a subset of the longer source instrument known as the PHQ-9. When the responses to the first 2 questions are “no,” the test is scored as negative signifying less concern for depression. Scores of the PHQ-2 can range from 0-6, with a score of 3 or greater considered positive and prompts the administration of the remaining 7 questions of the PHQ-9. Using the longer PHQ-9 allows for more accurate categorization of the severity of depressive symptoms as being mild, moderate, moderately severe, or severe.Reference Kroenke, Spitzer and Williams5 A positive response on the PHQ-9 screening test requires clinical confirmation to arrive at a definitive diagnosis of depression, but the PHQ-9 is used in practice and research because of its high predictive value.
The analysis was to measure the extent to which adolescent PHQ screening results changed during the COVID-19 pandemic. PHQ screening results were to be evaluated by analysis of covariance, controlling for site and seasonality, for pre-COVID-19 and COVID-19 pandemic periods. This statistical technique, utilizing repeated measures analysis, was needed due to seasonal variability in suicidal ideation and presentation to outpatient clinics, as well as other factors that influence screening rates and responses. A secondary analysis would have examined whether baseline mental health influences seen at different national network programs affected change in PHQ results.
In addition to collecting data on depression screening, data on mental health staffing was to be collected from programs within Children’s Health Fund’s national network through a standardized questionnaire sent to program leadership or human resources. The intent was to determine if the availability of mental health staffing changed across the pandemic and if staffing was adjusted to meet the demand for services. The analysis plan was to compare the data from the pre-COVID-19 period (January-December 2019) to the COVID-19 pandemic period (June 2020-May 2021) to determine if staffing adjusted to meet the demand. Because data were not collected specifically for the proposed study and no one had access to the subject identifiers linked to the data, the study is not considered human subject research.
Results
An initial recruitment of sites took place at the December 2021 annual meeting of Children’s Health Fund’s national network in New Orleans, LA; subsequent email communications assessed interest in and commitment to provision of data. Of the 24 network sites, 15 were collecting PHQ data and therefore eligible for inclusion in the study. Seven of these were unable to participate due to lack of sufficient human resources for data reporting or other more pressing priorities. Of the remaining 8 sites, inconsistent data recording and availability of funding to support data extraction prevented 6 from contributing data. Only 2 of the 24 sites had the data, human resources, financial support, and interest needed to contribute. The 2 sites were deemed insufficient for generalizability, and therefore the project was terminated; no data from the proposed project were collected. Instead, we used a STROBE flow chart, traditionally used to examine patient-level non-inclusion in research,Reference von Elm, Altman and Egger6 to better understand the causes of program non-participation (Figure 1).
CHF Mental Health.
Discussion
The COVID-19 pandemic had a significant negative effect on children’s mental health as demonstrated in the clinic setting by a study comparing PHQ-2 scores before and during the pandemic in a population of adolescents seen within the Children’s Hospital of Philadelphia clinic system.Reference Mayne, Hannan and Davis4 In that study, most patients had private insurance and access to acute and preventive pediatric care. An attempt to replicate this study within more resource-constrained communities of Children’s Health Fund’s national network was unsuccessful due to modifiable factors that should be considered during times of stability in anticipation of requirements for data during disasters.
Heterogeneity in use of standardized screening measures was unexpectedly high across our sites. While the PHQ is a commonly used tool for screening and follow-up of adolescent mental health concerns, its utilization was unreliable due to: 1) not using the PHQ consistently for mental health screening in the primary care setting, 2) initiation of screening only after it became apparent that mental health screening of adolescents was necessary during the COVID-19 pandemic, 3) cessation of screening after the pandemic began due to shifts in priorities in these resource-constrained sites, or 4) screening intermittently before and after the pandemic began due to highly variable prioritizations in these chronically under-resourced sites. Establishing baseline screening processes for mental health assessment prior to disasters is a priority. Ensuring data retrievability and data compatibility beforehand would ensure that mental health can be monitored over time.
Even more concerning is the inability to obtain staffing information. Most sites did not have the resources to pull staffing data, and even if they could, they could only provide information on current mental health staffing rather than historic trends. In addition, each site uses their mental health staff differently: some only provide screening and triage with referral to outside community resources for care, some only provide short-term therapy, and a minority provide on-going mental health care. Thus, it is impossible to assess expected productivity in mental health care. Further, clinical data and human resource data are siloed, making it impossible to assess the relationships between need and capacity, especially when staff provide care for multiple subpopulations.
Rapid funding for disaster response is often available but takes clinical efforts and organizational prioritization away from the chronic needs of communities. This change in prioritization affected 5 of our programs to the extent that they were unable to participate. This constitutes a “multiple hazard” scenario in which focus on the first hazard (COVID-19) diminished the ability of teams to address additional hazards, including determination of need for mental health resources. Such cascading multi-hazard scenarios were predicted for natural disasters co-occurring with COVID-19Reference Quigley, Attanayake and King7 and we now understand that the additional hazards of poverty and racism significantly contributed to morbidity.Reference Parolin and Lee8 During disaster response, a focus on measuring mental health and other chronic disease states should be maintained, particularly for marginalized populations, so that deterioration in conditions outside of the direct effect of the disaster can be identified and addressed.
Conclusion
Unfortunately, this problem of insufficient data to set health priorities and policy is not unique despite calls for enhanced mechanisms for data collection and sharing by the World Health OrganizationReference Dye, Bartolomeos and Moorthy9 and identification of these concerns as a priority by the U.S. Government Accountability Office.10
The lack of investment within communities makes development of interoperable infrastructures a lower priority as acute needs remain unmet prior to disasters. In addition to obtaining general resources to prevent inequities in susceptibility during crises, additional resources should be directed toward preparing for data collection needed for resource allocation for future crises that will disproportionately impact these communities.
Author contribution
AG extracted data, created the figure, and facilitated workgroup sessions. JH developed the initial study proposal and reviewed and interpreted the data. BH and SS interpreted the data within the context of practices in under-resourced communities. JC drafted the manuscript and interpreted data in the context of disaster response.
All authors edited the manuscript and approved the final version for submission.
Competing interests
None.
Open access
