Introduction
Sport-related concussion is common among youth in the United States, resulting in 1.5–3.0 million reported incidents each year.Reference Bryan1 Concussion symptoms are diverse, idiosyncratic, may be impacted by pre-morbid conditions, and can be difficult to alleviate.Reference Dillard2 Researchers and practitioners have developed evidence-based practice recommendations for concussion prevention, injury identification,Reference Caron3 treatment of symptoms,Reference Leddy4 and the return of young people back to functioning in school and sport.Reference DeMatteo5 Some established best practices in concussion management have now been codified into state law, with all 50 states and the District of Columbia enacting some form of youth concussion management legislation.Reference Thompson6 However, these laws differ across states and their implementation is not well documented, with early research in this area indicating high levels of variability both across and within states.Reference Baugh7
Variability in implementation of school-based policies has been observed previously at the state, district, school, and even student level.Reference Shifrer8 Evaluation of these inconsistencies revealed that variability in implementation is not random, but rather systematic and associated with school and student level factors such as age,Reference Achilles9 race,Reference Bradshaw10 socioeconomic status,Reference Herbers11 gender,12 and the training levels of staff and educators.Reference Darling-Hammond13 Inequity in the application of school-based policies is termed disproportionality and has been documented in several school-based policies including identification of students for gifted and talented programs,14 school discipline practices,15 identification and application of special education services and support plans in school,16 and baseline neuropsychological testing for concussion management.Reference Campbell17 Given these patterns, employing a disproportionality lens when assessing implementation concussion management policy is warranted.
Previous research by our group has demonstrated that variability in concussion policy implementation exists.18 This study extends that work using a disproportionality lens to: (a) evaluate the relationship between implementation of concussion management policies and the sociodemographic variables of students in Massachusetts high schools, and (b) assess whether disparities in the implementation of concussion management policies in Massachusetts high schools are systematic, thereby mirroring previously observed disproportionality patterns.
Hypotheses
Based on existing disproportionality literature in school-based policy implementation, we anticipated that the extent of schools’ concussion policy implementation would be related to the sociodemographic makeup of the school’s student population and that the differences in implementation would be systematic and represent systemic disparities such that schools with more marginalized students would have lower rates of policy implementation.
Methods
A cross-sectional survey was employed. Data on concussion implementation practices were collected via surveys of high school nurses and data regarding school sociodemographic factors were collected through publicly available databases.
Participants
Surveys were sent to a single school nurse employed in each eligible high school in Massachusetts. School nurses were selected for the current study through collaboration with the public health office, who identified school nurses as the personnel charged with implementing the established legislation given their consistency across high schools. That is, all high schools in Massachusetts are required to have a school nurse on staff. Other staff that may support concussion management (athletic trainers) are not mandated and therefore would not be a consistent stakeholder across schools. Our team has historically discussed concussion legislation with other stakeholders including athletic trainers19 and athletic directors.Reference Howland20 School nurses were included in the study if they were employed at least part-time at a public high school that had a student population >100 students. This sample frame was selected because private schools and special purpose public schools with less than 100 students often operate differently (e.g., different legislative requirements, no extracurricular sport) than ordinary public schools.
Data Collection
Following IRB approval from the University of Massachusetts Boston (IRB Protocol Number: 2019146) and the Boston University Medical Campus (IRB Protocol Number: H-38653), eligible schools were identified using the Massachusetts Department of Early and Secondary Education’s website. There were 344 public high schools with >100 students. However, due to data collection complications, all 344 schools were not included in the sample. Specifically, Boston Public School District was not included (n = 33) due to changes to research regulations that required additional institutional review board review through the Boston Public School District. The research team moved forward with this review board submission; however, in combination with the COVID-19 pandemic, data collection was never able to be initiated. Additionally, n = 7 identified schools were no longer in operation at the time of data collection. Therefore, N = 304 public high schools were included in the sample frame. The nurse in each school was identified through collaboration with regional school nurse consultants, who support school nurses in their region by serving as a liaison to the Massachusetts Department of Public Health. The state of Massachusetts has six school regions (West, Southwest, Boston, North, Northeast). We worked with all 6 regional school nurse consultants to identify a nurse stakeholder in each school. When nurse consultants did not have direct contact information for a school nurse, we called the school individually and requested the email address of their primary school nurse.
In Fall 2019, surveys were sent to the N = 304 school nurses via email. The emails were forwarded to each nurse individually by their regional nurse leader. The email included a full consent document, an overview of the study, and a link to the Qualtrics survey. Two additional email notices were sent as follow-up reminders to complete the survey (February 2020, April 2020). Nurses were reminded to ignore subsequent emails if they had completed the survey previously and duplicate surveys were removed from the sample. Initial emails were sent in January 2020 and data collection ended in April 2020.
Measures
Nurse demographic questionnaire. A brief questionnaire was included at the start of the survey that determined whether the nurse worked in a private or public school, the type of school in which they were employed (charter vs. traditional public), the presence of extracurricular sport in their school, involvement in the Massachusetts Interscholastic Athletic Association (MIAA), employment practices in the school (number of full-time nurses and athletic trainers), and the years of experience the nurse had in their current school.
Concussion management implementation survey. In collaboration with colleagues at the Boston Medical Center Injury Prevention Center (JH, HH, JC), the lead author (CWH) developed a survey that assessed concussion management practices. The concussion management practices were delineated into three categories: (1) neurocognitive baseline testing, (2) mandated practices from Massachusetts youth concussion legislation, and (3) interdisciplinary care practices. The baseline neurocognitive category consisted of four questions that assessed the modality of baseline testing in the school (i.e., onsite, online), and who received baseline testing (e.g., all students, some student-athletes, all student-athletes). The mandated practices category asked nurses to identify which of the 25 practices listed in the Massachusetts regulations21 were consistently implemented (>90% of the time) in their school. The interdisciplinary team category first inquired whether or not an integrated team or forum was used to support concussion management at the nurse’s school and if nurse stated “yes” then asked the nurse to identify the stakeholders involved in their team (e.g., teachers, parents, athletic trainer) and provide a rating between 1 and 10 that represented the level of coordination between the team members (1 = no coordination, 10 = complete integration and collaboration). This paper reports only the mandated practices category as the purpose is to examine factors associated with disparities in the way that legislated and other policies are applied across schools.
School demographic variables. To examine the relationship between implementation scores and school sociodemographic variables, publicly available school-level data were collected from the Department of Elementary and Secondary Education (DESE) website (https://prfiles.doe.mass.edu/) and the National Center for Education Statistics database (https://nces.ed.gov/). Variables analyzed included racial identities of students, number of teachers employed, student enrollment, student to teacher ratio, percentage of students identified as English language learners (ELL), the school’s composite performance index (CPI), and the school’s economic disadvantage rate (EDR). The school’s CPI is a score between 0–100 that represents the extent to which students are progressing or have attained proficiency on the state-wide assessments. The EDR of a school is a construct defined by the DESE which denotes the percentage of students in a school who participate in one or more state-administered programs (e.g., Supplemental Nutrition Assistance Program (SNAP); the Transitional Assistance for Families with Dependent Children (TAFDC); the Department of Children and Families’ (DCF) foster care program, and MassHealth (Medicaid)). Where possible, school level demographics were matched to nurse responses according to the name of the school provided on the survey by the nurse. Although not all respondents provided their school’s name, and therefore some demographic data were not matched to any nurse response, the author collected school demographic data for all schools contacted in the current study (N = 304). All collected data available upon request.
Analyses
Describe Current Concussion Management Practices. For each school, implementation scores were created by summing responses to questions on the mandatory regulations (max score of 25), such that a higher score indicates more practices being implemented on a consistent (>90% of the time) basis. Measures of central tendency, variability in implementation scores, and visual inspection of the distribution of responses across policies were used to assess for patterns in implementation. A comprehensive discussion of mandated and non-mandated implementation practices across Massachusetts high schools is provided in a previous publication.22
Assess Relationship Between Implementation & Demographic Variables. To assess whether there was a relationship between implementation scores and sociodemographic profiles of high schools, Pearson product moment and Spearman’s rank order correlations were calculated for continuous and categorical/dichotomous variables, respectively. Demographic variables of interest included the distribution of the racial identities of the student population, the school’s EDR, percentage of ELL students, student population size, number of teachers, student to teacher ratio, and the school’s CPI.
Evaluate Whether Differences in Implementation were Systematic. To assess whether differences in concussion management implementation represent systematic disparities across schools, a k-means clustering analysis and group-wise comparisons were conducted. To separate schools into meaningful groups, a K-means clustering analysis was employed. K-means clustering is an iterative unsupervised machine learning technique designed to separate data into K groups based on an identified parameter, where K is the predetermined number of groups as decided by the researcher. In the current study, the parameter used to partition individual nurse responses into distinct groups was their total implementation score. To establish the optimal number of clusters (i.e., number of centroids) we followed established practices including the elbow method,Reference Thorndike23 gap statistic,Reference Tibshirani24 and average silhouette method.Reference Kodinariya and Makwana25 Once the optimal number of centroids was determined, the K-means clustering algorithm worked in an iterative process to create groups that maximized similarity of scores within the group while optimizing difference (i.e., Euclidean distance) between groups.Reference Yuan and Yang26 The iterative process ended when convergence was achieved which was defined as two successive iterations that result in less than or equal to 0.01 movement in centroids. To assess differences in sociodemographic profiles of the established groups, group-wise comparisons were conducted. Due to violation of equal variance (significant F-test), and unequal sample sizes, nonparametric Man-Whitney U tests were used to assess for differences between the established groups. All analyses were conducted using R-Studio and R statistical software.27
Results
Participants
Of the 304 nurses who were contacted to participate in this study, 201 provided eligible survey responses (response rate: 66%), of which 45 were removed due to the absence of a school identifier (i.e., school name). As such, N = 156 (51%) schools were included in the current analyses. Nurses from all 6 regions of the state were represented by respondents, however there was an underrepresentation of the Boston region as the Boston Public School District did not participate in the study. Table 1 presents the demographic data of the 156 schools that had a nurse who responded to the survey and provided their school’s name and the 148 comparison schools that did not have a nurse who responded or did not provide a school identifier. Comparison across these groups indicated significant differences with respect to the mean percentage of students identified as English Language Learners (ELL; p = .025), African American/Black (p = .013), and White (p = .024) such that on average those who responded to the survey had significantly less students who identified as ELL and African American/Black and a significantly higher percentage of White students in their school. These findings should be considered when interpreting the results of the study.
Table 1. Demographic variables for schools that responded to survey (responder) and those that did not (non-responder)
Relationships Between Implementation & School Demographic Variables
Given previously observed variability in concussion policy implementation within Massachusetts high schools,28 we assessed the relationship between policy implementation and school sociodemographic variables. Results of the Pearson product moment correlation analyses revealed significant negative correlations between implementation scores and several school demographic variables (See Figure 1). Specifically, significant negative correlations were observed between the implementation score and the percentage of students with a marginalized racial identity (r = -0.167, p = .038), the percentage of ELL students (r = -0.213, p = .008), and school EDR (r = -0.208, p = .009). Except for the number of full-time athletics trainers, all sociodemographic variables were negatively associated with implementation scores such that as schools’ percentage of ELL students, Black/African American students, and EDR increased, their implementation score decreased. The number of full-time athletic trainers was positively associated with implementation of non-mandated best practices such that as the number of full-time AT’s increased so too did the best practices implementation score. Finally, regarding categorical variables, one significant relationship emerged between implementation of mandated practices and the presence of a school nurse leader (r = .165, p = .04), such that schools that indicated having a nurse leader also had higher implementation scores.
Figure 1. Pearson product moment correlation coefficients between school demographic variables and implementation scores
Note. Correlation coefficients listed first followed by p-values in parentheses. Alpha set to .05. RID [Marginalized Racial Identity], ELL [English Language Learner], EDR [Economic Disadvantage Rate], AA [African American], FTE.Nurse [Full Time Nurses], Ratio [Student-Teacher Ratio], FTE.AT [Full Time Athletic Trainers], Leg [Legislation Score], Baseline [Baseline Testing Score], Collab [Collaboration Score], EBP [Best Practices Score], Total [Total Score], CPI [Composite Performance Index]
Evaluation of systematic disparities in implementation
Results of the K-mean clustering analysis revealed two distinct groups of schools according to their total legislation score (See Figure 2). These two groups represented schools with high adherence to policy (n = 120) and low adherence to policy and best practices (n = 36). The mean legislations scores for these two groups were M = 20.6 and M = 12.19, (p<.001), respectively. Comparison between mean demographic variables across these two groups also revealed significant differences between two school demographic variables (See Table 2). That is, the low implementation group had a significantly higher percentage of students who identified as African American/Black (M = 8.88 ± 13.68, M = 5.44 ± 7.02; p = 0.045) compared to the high implementation group. Additionally, the low implementation group had nurses with significantly less experience (M = 2.19 ± 1.09) than nurses in the high implementation group (M = 2.84 ± 0.97; p = 0.001).
Figure 2. K-mean cluster plot partitioning across 2 centroids using total legislation score
Table 2. Groupwise Comparison between High and Low Implementation Schools Across Demographic Variables
Discussion
In this study we evaluated the implementation of mandated concussion management practices across public high schools in Massachusetts during the 2018–2019 academic year. We examined whether concussion policy implementation was associated with the sociodemographic makeup of schools and whether variability in implementation represents systematic disparities. Our results affirm the importance of adopting a disproportionality lens when evaluating implementation of school-based concussion policies.
School nurses in our study reported overall moderate to high rates of policy implementation, however variability across schools is evident and our results suggest that the observed variability is not random but may be systematic and related to the sociodemographic profile of the high school. To this end, results of the correlation analyses revealed relationships between nurses’ reported implementation practices and a variety of sociodemographic attributes of the school, including variables representative of race, socioeconomic status, and resource availability, consistent with previous disproportionality literature.29 In addition, our study results demonstrate a relationship between implementation practices and the sociodemographic factors of the student population. That is, schools with more marginalized students had lower concussion management implementation scores. The results of the k-means clustering analysis highlight the importance of disaggregating implementation data to gain a more nuanced understanding of the variability in policy implementation that cannot be detected when only averaged data are observed. Moreover, beyond the disparities observed between the included schools, there were also disparities observed between the schools who did (N = 156) and did not (N = 148) respond to the survey. That is, the schools that did not respond to the survey or provide a school name had significantly higher rates of students who identified as English Language Learners and Black/African American and had significantly less White students. Given that these patterns mimic the findings of the disparities in respondent schools, it is possible that the extent of disparities between schools is greater than observed, and that the rate of policy implementation may be overrepresented by schools with higher implementation. That is, it is feasible that nurses who are resourced to implement existing policy are the same nurses who have bandwidth to respond to the survey and vice versa; in schools where implementation of concussion policies is limited, the same factors that impede implementation may also impede nurse response to the disseminated survey. Additionally, if a nurse felt that implementation was suboptimal in their school, they may self-select out of the study out of fear of penalty.
Some of the policies most infrequently implemented appear to be related to communication and collaboration between stakeholders including teachers, sports staff, school counselors, school administrators, healthcare providers, and families. Breakdowns in communication and lack of coordination exacerbates challenges associated with managing students’ concussionsReference Rosen30 and may increase vulnerabilities for student-athletes who are unable to navigate the multiple systems involved in their care. Additionally, gaps appear related to communication with families whose first language is not English, pointing to specific aspects of policy that may systematically result in suboptimal care for specific groups of students. To this end, schools with more English language learners did trend toward decreased rates of implementation. Identifying specific components of policy that may result in systematic differences in care may offer a manageable starting place for improving equity in concussion management across all students and schools.
Limitations & Implications
This study offers important and novel information regarding the current implementation of concussion policy in one state, Massachusetts, and results illuminate the importance of evaluating implementation through a disproportionality framework going forward. However, there are limitations that warrant attention and can guide future research efforts. First, although all six regions of Massachusetts are represented in the current sample, the Boston region is under-represented because the Boston Public School system did not participate. Moreover, school nurses who did respond to our survey were employed at schools with a lower EDR and higher percentage of White students. It is possible that with a higher response rate our findings would have illuminated even greater disparities between schools. This possibility should be evaluated further. Moreover, future research should be conducted more broadly across states that may observe disparities across diverse sociodemographic factors that are not represented in the current Massachusetts sample. Another limitation of this study was the sole reliance on the perspective of the school nurse to report on concussion management practices. It is possible that other stakeholders that are not represented in this study may have had different perspectives and insights into schools’ concussion management policy implementation. Additionally, the nature of the surveys prompts caution in interpretation because objective measures (e.g., observing behaviors in schools) were not collected. Social desirability or fear of repercussion may have influenced nurse responses. This study was particularly vulnerable to this concern as we requested nurses provide the name of their school at the end of the survey. One potential area for future research might be the use of a case study or ethnographic design to allow for naturalistic observation as well as multi-stakeholder assessment, including student-athlete and family perspectives on policy implementation. Finally, the data presented in this study were collected several years ago, and thus the rates of adoption may have improved since that time. That said, the state of Massachusetts was an early and progressive adopter of state legislation surrounding concussion management and had implemented their legislation 10 years prior to data collection. Given ongoing disparities in implementation observed at that time, it is reasonable to believe they would persist today; however, ongoing monitoring of policy implementation is an important need identified in this study. Relatedly, as Massachusetts was an early and progressive adopter of concussion legislation, findings may not generalize to other states across the country that have taken different legislative approaches to the management of youth concussion. Future research should replicate this study in other states or geographical regions to better understand disparities in implementation across the country.
Research. In addition to the abovementioned next steps for research, the findings of this study also prompt future study. Given the documented disparities in implementation across schools, future qualitative work would be useful to further illuminate the specific barriers and catalysts to implementation across different schools. Through a qualitative lens, it is feasible to identify common barriers to effective implementation and mechanisms in place that support implementation, both of which can be used to improve implementation in other schools. Moreover, this study did not evaluate the impact of the disparate application of policy on the outcomes and experiences of student-athletes, a critical area for further inquiry.
Practice. One notable finding of this study is the role of integrated teams in schools. Policies that require communication between school, health, and family had the lowest level of implementation. The need for integrated teams to support student-athletes in schools has been called for in previous researchReference Hayden31 and is highlighted again in this study. Team science research32 can offer an important area of study to inform improved integration and communication among relevant stakeholders in schools. Another immediate need based on the communication challenges observed in this study is the development of information and support for students and families who do not speak English as their primary language. Nurses often reported not having the resources required to communicate to students and families who identify as ELL and results of the group comparisons indicated that on average, schools with higher rates of ELL students had lower rates of policy implementation. Taken together, these findings point toward a significant gap in supporting ELL students in the context of concussion management and return-to-activity.
Policy. These results also prompt consideration of changes at the policy level and specifically the “color blind” nature of current policy development and implementation. Previous researchers have highlighted the current assumption underlying policy implementation — that policies will be implemented equitably.Reference Wells33 Thus, no consideration for disparities is made when designing policies and planning for implementation.34 Given established patterns of disproportionality in policy implementation, Wells suggests the need to consider this reality proactively during the formation and implementation of policies. Finally, although policy development is often a “final” stage of research, our findings highlight the critical role of implementation science and continued monitoring after the development of a policy to evaluate the impact of that policy on the lives of student-athletes. Implementation science is an important field that can support these goals and should be integrated into the policy itself. In this way, emerging disparities can be identified early on and necessary changes to policy made to ensure growing gaps in treatment don’t emerge and persist.
Conclusion
The current study was the first to evaluate the implementation of concussion management practices in Massachusetts through a disproportionality lens. Results highlight that variability in policy implementation persists in the context of concussion management and is related to the sociodemographic makeup of the school, highlighting the relevance of the disproportionality framework. While these findings are specific to Massachusetts, future research in concussion management should consider the implementation practices of established policies and may benefit from evaluating implementation through a disproportionality lens to fully elucidate the effectiveness of established practices.
Open access
