Public health emergencies (PHEs, which are events, whether natural or human-made, that pose a risk to health—for example, hurricanes, pandemics, and terrorist attacks) can negatively impact behavioral health (BH). For some individuals, BH impacts, defined as new or worsening mental health conditions or substance use disorders, can be severe and long-lasting.Reference Rodriguez and Kohn1 After a PHE, approximately 1 in 4 adults will develop clinically significant post-traumatic stress syndrome and anxiety, and about 1 in 5 will develop depressive symptoms.Reference Newnham, Mergelsberg and Chen2 Over time, symptoms of post-traumatic stress syndrome typically decrease, but depression and anxiety can persist. Both the direct impacts of the COVID-19 pandemic and the necessary public health measures to control its spread could have exacerbated the need for BH services. For instance, depression symptoms in children and adolescents increased during the pandemic, particularly among girls,Reference Madigan, Racine and Vaillancourt3 and drug overdose mortality was higher in 2020 than in previous years.Reference Cartus, Li and Macmadu4
Currently, no dedicated surveillance systems exist to monitor BH during and after PHEs.Reference Lyerla and Stroup5 Even outside of the PHE context, in lieu of having a comprehensive surveillance system for BH, public health officials rely on a patchwork of surveys and surveillance systemsReference Reeves, Strine and Pratt6 that have limitations, including long delays between data collection and release, nonrepresentative sampling, and lack of validated BH measures.Reference Lyerla and Stroup5, Reference Azofeifa, Stroup and Lyerla7
With improved monitoring and surveillance of BH precursors and impacts, public health officials can understand BH trends and better anticipate potential impacts of PHEs, identify acute increases in BH needs, evaluate the effectiveness of BH interventions, position resources to prevent adverse BH impacts from future PHEs, and begin to address long-standing BH inequities that may emerge in PHEs.
To plan for, respond to, and recover from PHEs, public health agencies and their BH partners need timely local data on BH needs to develop tailored public health interventions. However, little is known about the strengths and limitations of existing data sources, indicators of population-level BH, and methodological approaches for conducting BH surveillance before, during, and after PHEs. Furthermore, there is limited guidance on how public health agencies can use BH surveillance data to better prepare for and respond to PHEs.
To fill these gaps, we conducted a mixed-methods study to answer 2 primary research questions (RQs):
RQ 1: What indicators, data sources, and analytic approaches could be used for BH surveillance in the PHE context?
RQ 1.1. What types of BH indicators are needed for surveillance in the PHE context?
RQ 1.2. What existing data sources contain these BH indicators, are available at the substate level, and are updated frequently enough to be useful in the PHE context?
RQ 1.3. How can public health agencies access and analyze these data, and what are the potential strengths and limitations of the different data sources?
RQ 2: How could better BH surveillance before, during, and after PHEs support decision-making and public health action?
RQ 2.1. How can these data be used to inform public health decision and action?
RQ 2.2. What do public health officials perceive as best ways to use BH analyses to inform decisions and public health action?
Methods
We conducted a literature review and environmental scan to identify data sources, indicators, and analytic approaches that could be used for BH surveillance in the PHE context. We then conducted exploratory analyses and interviews with public health officials to examine the utility of a subset of these data sources for BH surveillance in PHEs (Figure S.1). The corresponding author’s Human Subjects Review Committee exempted this study from further review (2019-0920-AM03).
RQ 1: Identifying BH Indicators, Existing Data Sources, and Analytic Approaches
RQ 1.1: Review of systematic reviews and meta-analyses to identify indicators
To develop a conceptual model of potential types of BH indicators for surveillance in the PHE context, we conducted a keyword search of the PubMed, PsycINFO, and Web of Science databases (Supplemental Online Appendix Table S.1). To focus on the most current information available within a rapidly changing landscape, we limited the search to English-only systematic reviews and meta-analyses from the prior 5 years: 2014 to November 2019 (the date the search was conducted). We screened resulting references for relevance and systematically abstracted information from included articles.
RQ 1.2 and 1.3: Environmental scan to identify data sources and analytic methods
We then performed an environmental scan consisting of a second literature review and interviews with representatives of organizations that collect or aggregate data relevant to BH (e.g., Poison Control Centers, the National Retail Data Monitor), representatives from public health agencies, and experts in public health and BH. The purpose was to identify data sources that contained relevant BH indicators and analytic methods that have been used to examine those indicators.
Literature search. We searched the peer-reviewed literature published in English from 2010 to 2020 using PubMed and PsycINFO and the search terms in Supplemental Online Appendix Table S.2. We also conducted searches for grey literature on Google Scholar for selected data sources using search terms tailored to each source (Supplemental Online Appendix Table S.2). After screening articles for relevance, we again systematically abstracted information from included articles.
Findings from the search were used to narrow down the list of data sources to a subset that was assessed to be promising for BH surveillance in the PHE context, defined as meeting the following 3 criteria: available (1) at the sub-state level, (2) more frequently than once a year, and (3) for most jurisdictions in the United States.
Informational interviews with organizations that collect or aggregate data relevant to BH. We held 21 informational interviews lasting 45 to 60 minutes with representatives of organizations that collect or aggregate data relevant to BH, focusing on the subset of promising data sources. We identified these individuals through targeted web searches and chain-referral sampling. After obtaining verbal consent, we used a semi-structured discussion guide (Supplement 1) to assess how public health agencies could access and analyze these data, and strengths and limitations of the data sources. We analyzed our detailed notes using a qualitative descriptive approach.
In-depth interviews with public health agencies and experts. We conducted 11 interviews lasting 45 to 60 minutes with state and local public health agencies and experts in public health and BH, selected for their experience with these data sources and analytic approaches for BH surveillance. After obtaining verbal consent, we used a semi-structured discussion guide (Supplement 1) that covered challenges with BH surveillance, potential data sources, and analytic methods. Interviews were audio-recorded and transcribed. Transcripts were coded in Dedoose for common themes.
Findings from both the informational and in-depth interviews informed the design of our exploratory analyses (RQ 2.1).
RQ 2: Examining the Utility of the Promising Data Sources
RQ 2.1: Exploratory analyses
To answer RQ 2.1, we conducted exploratory analyses of the promising data sources. We obtained datasets for locations that had experienced one or more PHEs, had sufficient population sizes for robust analyses, and were in different regions of the country. Some of these sources were publicly available, whereas others required data requests and data use agreements. For each source, we followed a structured process to examine the utility of the data source for BH surveillance in PHEs (Supplemental Online Appendix S.3). We documented our qualitative assessments at each step, which we integrated with interview findings described below (RQ 2.2) to arrive at our final assessment of the sources. Each source required slightly different analytic methods, which are presented elsewhere.Reference Fischer, Landis and Acosta8–Reference Acosta, Faherty and Weden11
RQ 2.2: Interviews with public health agencies and data experts
We conducted 60-minute in-depth interviews lasting 45 to 60 minutes with 37 state, tribal, and local public health representatives; subject matter experts in public health and BH; and data source experts. We identified interview participants through chain-referral recommendations and the literature search for RQs 1.2 and 1.3. After obtaining verbal consent, we used a semi-structured discussion guide (Supplement 2) to gather input on how findings from the exploratory analyses (RQ 2.1) could inform public health action, focusing on the utility of these findings for BH surveillance in PHEs. Interviews were audio-recorded and transcribed, and transcripts were coded in Dedoose for common themes.
Results
RQ 1: Identifying BH Indicators, Existing Data Sources, and Analytic Approaches
RQ 1.1: A conceptual model of BH indicators
Through the literature review and environmental scan, we identified approximately 40 types of BH indicators that could be surveilled in the PHE context. We developed a conceptual model that groups these indicators into 3 categories (Figure 1):
-
• Upstream indicators: community strengths, vulnerabilities, and other social determinants of BH that may influence the likelihood that a future PHE will impact population-level BH. These include risk and protective factors such as unemployment or job loss, housing instability, and social cohesion, several of which have been linked to BH.Reference Hergenrather, Zeglin and McGuire-Kuletz12-Reference Linton, Leifheit and McGinty14
-
• Midstream indicators: early signs of BH impacts after the PHE has occurred for which there is still time to intervene and mitigate their effects. These include “calls for help” when BH is declining, such as to poison control centers (PCCs) for intentional ingestions, or measures of coping strategies such as taking over-the-counter sleep aids for insomnia associated with stress or depression.Reference Ballard, Vande Voort and Bernert15
-
• Downstream indicators: measures of a PHE’s later impacts on BH. These include emergency department (ED) visits for a BH crisis, calls for emergency medical services (EMS) to respond to an overdoseReference Slavova, Rock and Bush16 or BH emergency, or deaths related to overdose21 or suicide.17
Conceptual framework of indicators of community strengths and vulnerabilities and behavioral health impacts of public health emergencies.Reference Rodriguez and Kohn1, Reference Newnham, Mergelsberg and Chen2
*Promising data sources for which we conducted exploratory analyses
**Promising data sources that we assessed through the environmental scan and in-depth interviews but for which we did not conduct exploratory analyses
1Abbreviations: BH: behavioral health; PHE: public health emergency; UI: unemployment insurance; OTC: over-the-counter; PCC: Poison Control Centers; EMS: emergency medical services; ED: emergency department
2Promising data sources that could be repurposed for BH surveillance during and after PHEs are bolded
Not all the indicators in Figure 1 are available in the data sources discussed next. However, they are included for completeness, as public health agencies may have access to local data sources that are not available in other jurisdictions.
RQ 1.2: Existing data sources
The literature review, environmental scan, and formative interviews yielded 27 data sources containing one or more of the BH indicators we identified but no single dedicated data source for monitoring BH in the PHE context.
Applying the 3 criteria described previously to those 27 sources (i.e., available (1) at the substate level, (2) more frequently than once a year, and (3) for most jurisdictions in the United States) yielded 8 data sources that were considered promising for use in BH surveillance during and after PHEs:
-
• unemployment insurance claims
-
• calls to 2-1-1 centers
-
• calls to poison control centers for intentional ingestions
-
• over-the-counter medication sales of sleep aids
-
• pharmacy data on psychotropic medication fills
-
• EMS activations
-
• ED visits, and
-
• web searches for BH-related terms.
These sources contained indicators in the upstream, midstream, and downstream categories.
Two of these sources, EMS activations and ED visits, had national dashboards or well-established national surveillance programs, so we did not obtain these data for primary data analysis and instead used in-depth interviews (RQ 2.2) to solicit examples of how they have been used for BH surveillance in PHEs.
Next, web search data (i.e., aggregated through the Google Trends website) was excluded from the list of promising data sources after our exploratory analyses failed to detect a BH signal in these data, and we found that the data source did not have sufficient spatiotemporal granularity to be useful to state and local public health officials.
Finally, based on the in-depth interviews, we added calls to the 9-8-8 Suicide and Crisis Lifeline to the above list of promising data sources, bringing the total back to 8. The 9-8-8 dialing code launched in July 2022 and therefore did not “go live” in time for us to obtain data for analysis.
Table 1 summarizes how to access the data sources, their spatio-temporal granularity, and their strengths and limitations. Bolding indicates the 5 sources with which we conducted our own exploratory analyses (unemployment insurance claims, calls to 2-1-1 centers, calls to poison control centers for intentional ingestions, over-the-counter medication sales of sleep aids, and pharmacy data) (Supplemental Appendix S.3). Our assessment of EMS activations, ED visits, and calls to 9-8-8 was based on the environmental scan (RQ 1.2 and RQ 1.3) and semi-structured interviews (RQ 2.2).
Summary of the most promising data sources for behavioral health surveillance in the context of public health emergenciesa
a Adapted from Acosta et al.(Reference Acosta, Faherty, Landis, Baker, Fischer, Gandhi, Levin, Dastidar, Weden and Ayer19).
Abbreviations: BH, behavioral health; CDC, Center for Disease Control and Prevention; ED, emergency department; EMS: emergency medical services; ICD-10, International Classification of Diseases, 10th Revision; NEMSIS, National Emergency Medical Services Information System; NPDS, National Poison Data System; NRDM, National Retail Data Monitor; NSSP, National Syndromic Surveillance Program; OTC, over-the-counter; PCC, Poison Control Centers; PHE, public health emergency; UI, unemployment insurance.
Overall, existing data sources lacked data on sociodemographic factors that would be necessary to examine BH inequities. Only unemployment insurance claims and ED visit data consistently contained information on age, gender, and race of the claimant or patient. Age and gender were available in the data on poison control center (PCC) calls and prescription medication fills; however, race was not. Only gender was consistently collected from callers to 2-1-1 centers. Age, race, and gender were inconsistently included or not available in data on sleep aid sales, EMS activations, and suicide and crisis hotline calls.
Although other data sources contain BH indicators (e.g., those compiled by the Council of State and Territorial Epidemiologists through an initiative to strengthen surveillance of mental health and substance use),Reference Hopkins, Landen and Toe18 these indicators are not available frequently enough to be useful in the PHE context (Supplemental Online Appendix Table S.4).
RQ 1.3: Analytic approaches
We identified over 100 studies that described possible analytic approaches to monitor BH indicators (Table 2).Reference Acosta, Faherty, Landis, Baker, Fischer, Gandhi, Levin, Dastidar, Weden and Ayer19 Most studies used retrospective and observational repeated cross-sectional or longitudinal analytic approaches aggregating information across space (e.g., census tract) and/or time (e.g., weekly), rather than relying on prospective data collection using a counterfactual or comparison group. Analytic methods varied based on the question(s) to be explored or the indicator(s) to be monitored; the type(s) and volume of data available for analysis; the timing in relation to a PHE; and the available resources, capacity, and technical proficiency that the PH agency and its partners bring to this work. Commonly used methods for time series analyses of BH indicators included data-smoothing techniques followed by statistical modeling such as interrupted time series analyses or ITSA, difference-in-difference, and ARIMA (i.e., autoregressive integrated moving average) modeling.
Illustrative publications that use the promising data sources to monitor behavioral healtha
a For a complete list of publications identified through our environmental scan, see Acosta et al.(Reference Acosta, Faherty, Landis, Baker, Fischer, Gandhi, Levin, Dastidar, Weden and Ayer19).
b The studies included for this data source differ from the others in that they demonstrate an association between unemployment (the indicator of interest) and behavioral health or public health emergencies.
RQ 2: Examining the Utility of Existing Data Sources
RQ 2.1: Uses of these data sources to inform public health decisions and action
Table 3 summarizes methods and key findings from our exploratory analyses using 5 of the promising data sources in the context of PHEs. Full details of these analyses can be found elsewhere.Reference Acosta, Faherty, Landis, Baker, Fischer, Gandhi, Levin, Dastidar, Weden and Ayer19 Across the multiple locations we examined and different PHE types, we found that changes in indicators of BH risks and impacts of PHEs can be detected in these data sources, suggesting the utility of these data to inform public health decision-making.
Summary of exploratory analyses of five of the most promising data sources for behavioral health surveillance in the public health emergency context
a Adapted from Acosta et al.Reference Acosta, Faherty, Landis, Baker, Fischer, Gandhi, Levin, Dastidar, Weden and Ayer19
Abbreviations: ARIMA, autoregressive integrative moving average; COVID-19, Coronavirus Disease 2019; CPA, change-point analysis; ITSA, interrupted time series analysis; OTC, over-the-counter; PHE, public health emergency; SSRI, selective serotonin reuptake inhibitor.
Overall, we observed anomalies in the indicators we examined across most of the promising data sources for the start of the COVID-19 pandemic, but anomalies in these indicators around more acute, localized PHEs were more difficult to detect. For instance,
-
• in Los Angeles County, unemployment insurance claims peaked just after California’s business closure mandate in April 2020 and again in September 2020, coinciding with severe wildfires. In June 2020, SSRI fills began to increase.
-
• in Los Angeles County, we detected 4 periods of anomalous activity in over-the-counter sleep aid sales, coinciding with the 4 large COVID-19 waves there.
-
• in Broward County, Florida, calls to 2-1-1 after Hurricane Irma and the pandemic declaration showed a larger percent increase over baseline for men than for women. Calls by women remained elevated for a longer period than by men after the hurricane. The opposite was found after the pandemic declaration.
-
• by early 2021, calls to PCCs for intentional exposures among youth in Dallas County, Texas, increased after the pandemic declaration, closing the longstanding gap for intentional exposures between youth and adults. We detected an increase in calls for intentional exposures in this county during the summer surge in COVID-19 cases but not shortly after the pandemic declaration. The winter storm of February 2021 was not associated with an increase in calls to PCCs for intentional ingestions.
RQ 2.2: Public health officials’ perceptions of the utility of these analyses to inform public health action
Public health agency representatives identified several public health actions that could result from timely BH surveillance efforts.
-
• Before a PHE: Based on surveillance of upstream BH indicators, public health agencies and their partners could develop protective interventions, advocate for policies, and allocate resources to mitigate BH risks.
-
• During and after a PHE: With better surveillance of mid- and downstream BH indicators, public health agencies and their partners could alert emergency management services, BH, and other health-care providers to prepare for possible surges in demand for services and could advocate for resources to respond to those needs.
To increase their confidence in potential surveillance signals and to inform public health action, public health practitioners reported the need to compare findings across multiple indicators. Even among jurisdictions that have explored the integration of various data streams into their BH surveillance efforts, such as Washington State20 and North Carolina,21 interviewees noted some persistent challenges. More research is needed to (1) understand to what extent anomalies detected in these data sources correlate with “true” changes in population prevalence of BH needs and (2) use a signal in one data source (e.g., unemployment claims) to predict when other downstream signals might emerge (e.g., increases in ED visits for BH).
Discussion
Our study did not identify any existing dedicated surveillance systems to monitor BH in the context of PHEs.Reference Lyerla and Stroup5 However, there are a limited number of data sources that public health agencies in the United States could repurpose for timely BH surveillance at the substate level. We were able to detect “signals” of population-level BH shifts in these data sources, even though these data were not originally collected for this purpose. The magnitude and duration of anomalies in the BH indicators varied by PHE, age, and gender. Consistent with our conceptual model, changes in time series of BH indicators occurred at different time points in the PHE context, with upstream indicators peaking first followed by mid- and downstream indicators.
Existing data sources were limited and primarily allowed for monitoring of lagging, rather than leading, BH indicators. Aside from unemployment claims, the data sources we identified had limited information about structural inequities and well-being, which are key factors related to BH. Only a few data sources consistently collect the sociodemographic data needed to examine PHEs’ inequitable impacts on subpopulations, including at the intersection of race, gender, and age.
Additional data sources with the spatiotemporal granularity required to be useful in the PHE context are needed to more comprehensively monitor upstream social determinants of health that influence BH in the short and longer term. Efforts to modernize and transform public health data systems in the United StatesReference Christopher, Zimmerman and Chandra22, Reference Thorpe, Chunara and Roberts23 are underway to improve the consistency and completeness of data on sociodemographic factors to allow for subgroup analyses and better target public health interventions. These initiatives will benefit from close coordination among public health and BH stakeholders at federal, state, and local levels to clearly define roles and responsibilities around data collection, analysis, reporting, and action.
Emerging technologies and advanced analytic methods also offer opportunities for continued progress in strengthening BH surveillance, during both PHEs and routine times. For example, wastewater surveillance is increasingly being applied in jurisdictions across the country to monitor trends in levels of opioids in wastewater.Reference Duvallet, Hayes and Erickson24 As these emerging technologies come into more widespread use, they may provide important complementary data for more integrated, timely, equitable, and actionable BH surveillance during PHEs.Reference Margevicius, Generous and Taylor-McCabe25
To use these data for public health action, additional capabilities and partnerships are necessary. Platforms for aggregating and visualizing BH data are advancing. For instance, the National Emergency Medical Services Information System has added BH indicators to their dashboard,26 and the National Retail Data Monitor has added over-the-counter sleep aids to its data categories. However, partnerships that include key local organizations from the BH and public health systems are necessary for data interpretation and dissemination.
Experts noted the importance of triangulating among different data sources. However, even after comparing findings across different data sources and using statistical methods to test for significant shifts in BH indicators, decision-makers may face uncertainty around the best course of action to take given the findings. Public health officials will need to acknowledge the gaps in existing data when co-interpreting findings with the affected populations and conveying results and recommendations to different audiences.
Limitations
This work has some limitations. First, the literature review, although rigorous in its approach, was not designed to identify all relevant articles on this broad topic, so it is possible that there are other BH indicators that could be considered for surveillance in the PHE context. Second, the exploratory analyses are intended to be preliminary examinations of the promising data sources we identified. They were conducted retrospectively, using selected past PHEs, including hurricanes, wildfires, a severe winter storm, and the onset of the COVID-19 pandemic, rather than as the PHEs unfolded, which would have allowed us to draw conclusions about their utility for BH surveillance in real time. In addition, the findings from these analyses may not be generalizable to other PHEs and to other locations.
Conclusions
Effectively monitoring and addressing BH needs, both during a PHE and in routine times, depends on timely BH surveillance data with sufficient geographic granularity to inform just-in-time decision-making. Although our study did not identify a dedicated surveillance system to monitor BH in the context of PHEs, several existing data sources could be repurposed by public health agencies to strengthen their BH surveillance efforts in the PHE context.
Implications for Public Health Practice
In the near term, while data modernization efforts are underway, public health officials could use the data sources identified in this study, supported by a tool kit with instructions on how to access and analyze each of them.Reference Acosta, Faherty, Landis, Baker, Fischer, Gandhi, Levin, Dastidar, Weden and Ayer19 These data modernization efforts are an opportunity to ensure that BH indicators are integrated into these changing systems so that they could be monitored and surveilled to inform the BH response to PHEs. Public health officials also could focus on leveraging existing partnerships between BH and public health systems or forming new ones, and they could access existing data aggregation platforms such as the National Emergency Medical Services Information System that are increasingly incorporating BH indicators into their dashboards. Public health workforce development efforts could focus on building data science, analytic, modeling, and informatics skills that specifically apply to BH.
Over the longer term, efforts to modernize public health data systems in the United States could explicitly include strategies to update core data and surveillance infrastructure for BH so that public health officials can monitor emerging BH impacts of PHEs, detect and intervene earlier on population-level BH concerns during and after PHEs, and access technology that reduces the collection and reporting burden for BH indicators.
With continued efforts to understand BH risks, needs, and impacts for the overall population as well as key subpopulations, public health action to combat the current BH crisis can be more tailored, proactive, and equitable.
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/dmp.2024.127.
Data availability statement
Selected data may be made available upon request.
Acknowledgments
The authors thank Shira Fischer, Rachel Landis, Margaret Weden, Lawrence Baker, Priya Gandhi, Jonathan Levin, Lindsey Ayer, Jessi Espino, Michael King, Deborah Gould, Robyn Cree, Rebecca Leeb, Amy Schnall, Lakshmi Radhakrishnan, Katelyn Pugh, LaReina LaFlair, and Laurie Martin for their contributions to this study. They thank Hilary Peterson for her assistance with preparing this manuscript for submission, Liisa Hiatt for project management support, and the experts who provided their insights through interviews and participation in virtual convenings.
Author contribution
Conceptualization: Faherty, Acosta; data acquisition, curation, and analysis: Faherty, Acosta; writing, original draft: Faherty, Acosta; writing, review and editing: Vagi, Leinhos, Soler; funding acquisition: Faherty, Acosta; project administration: Faherty, Acosta, Leinhos.
Funding statement
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Research reported in the manuscript has been funded by the US. Centers for Disease Control and Prevention (CDC), an Agency of the Department of Health and Human Services, under CDC contract 75D30119C06926. The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
Competing interest
The author(s) declare none.
Ethical Standards
RAND’s Human Subjects Protection Committee determined this study (2019-0920-AM03) to be exempt from further committee review.
Open access
