One out of four (24 %) children in the USA aged 5 years or younger spends time in an organized care facility( Reference Laughlin 1 ). Early care and education (ECE) programmes, which are facilities (including classroom environments, staff, policies and practices) that provide nurturing care, support for development and learning experiences for children aged 5 years or younger, are a strategic setting for implementing strategies to prevent obesity( Reference Birch, Parker and Burns 2 ). Preliminary evidence suggests environmental-level strategies in ECE, such as improving policies and practices related to eating, physical activity and sedentary behaviours, appear to directly influence children enrolled in these programmes( Reference Benjamin, Cradock and Walker 3 , Reference McWilliams, Ball and Benjamin 4 ).
In 2007, Nemours Children’s Health System (Nemours) implemented an intervention in Delaware to promote healthy eating and physical activity among children and young children ranging in age from 0 to 5 years in a variety of settings, including ECE. A key part of the initiative included the establishment of ‘learning collaboratives’ and ‘train-the-trainer’ models with ECE programmes, which helped these programmes identify and implement healthy eating and physical activity practices and policies( 5 ). These practices improved significantly in 81 % of the twenty-eight participating ECE programmes( 5 ), suggesting that continued work in this area is warranted.
The original Delaware model was adapted for a multi-state implementation effort in 2012 by Nemours in collaboration with the Centers for Disease Control and Prevention. The resulting National Early Care and Education Learning Collaboratives (ECELC) Project (in its fourth year at the time of writing) aims to promote healthy environments, policies and practices with regard to the following areas: Breast-feeding & Infant Feeding, Child Nutrition, Infant & Child Physical Activity, Outdoor Play & Learning, and Screen Time. To the best of our knowledge, it is the first large-scale effort aimed at improving these types of policies and practices in ECE programmes across multiple states( Reference Benjamin, Ammerman and Sommers 6 , Reference Trost, Messner and Fitzgerald 7 ). Although data have shown promise for broad implementation of projects that promote healthy eating, physical activity and reduction in screen time in childcare settings (TM Smith, DJ Schober, J Shuell et al., unpublished results), most data collected have been self-reported (e.g. using the Nutrition and Physical Activity Self-Assessment for Child Care (NAP SACC)) and not verified through objective measures. While one study indicated that the NAP SACC assessment tool is a stable and reasonably accurate instrument for use in childcare interventions, if funds allow, a more robust, less subjective measure may be more appropriate for researchers seeking an outcome measure to assess intervention impact( Reference Benjamin, Neelon and Ball 8 ).
The current pilot study was exploratory in nature and aimed to objectively measure a small sub-sample of pre-school classrooms using the Environment and Policy Assessment and Observation (EPAO)( Reference Ward, Hales and Haverly 9 ) measurement tool to assess change from before to after participating in the ECELC with regard to nutrition and physical activity. Specifically, the present study had two aims: (i) to determine how participation in the second cohort of the ECELC influenced eating and physical activity at the ECE programme level, as measured by the EPAO for sub-sampled programmes; and (ii) to compare outcomes derived from the collected EPAO data with matched outcomes derived from the collected NAP SACC data.
Methods
The ECELC intervention consisted of five main strategies: (i) self-assessment; (ii) in-person peer learning sessions; (iii) action planning and implementation; (iv) technical assistance; and (v) reassessment. ECE programme staff participated in learning sessions that included didactic presentations on content, interactive activities, and peer sharing and support. In between learning sessions, ECE programmes received action planning tasks, which encouraged them to share what they learned with their programme staff and build staff support for implementing best practices across topic areas on-site. Finally, each ECE programme received individualized technical assistance in between learning sessions in order to support programmes during their action planning phases. Reassessment (post-assessment) occurred after the fifth and final learning session approximately 10 months later.
The methodology of the present study was based on a study published in 2015 by Benjamin Neelon and colleagues. That study employed a randomized control trial to test the effect of an intervention targeting ECE programmes serving children less than 2 years of age to improve the nutrition and physical activity environments, as outlined in the Baby NAP SACC for ECE programmes( Reference Blaine, Davison and Hesketh 10 ). While the Benjamin Neelon study differed from the current study in that it compared data collected using the EPAO among intervention and control groups (and also that it utilized Baby NAP SACC v. the regular full NAP SACC), it provided a framework for the current study on conducting pre–post analysis and the comparison of tools.
Participants
To be eligible to participate in the ECELC, programmes had to serve at least fifty children, develop a Leadership Team of at least three staff (e.g. owner or director, teacher, cook) and attend each of the five in-person learning sessions. Some data were collected via an enrolment form administered electronically and included contact information, programme characteristics (e.g. number of children served) and state characteristics (e.g. presence of a Quality Rating and Improvement System (QRIS), an approach to assess, improve and communicate the level of quality in ECE programmes)( 11 ). Seven sites (North/Central Florida, South Florida, Indiana, Missouri, New Jersey, Kansas and Arizona) participated in the second phase of the second cohort of the ECELC; however, only four (North/Central Florida, Indiana, Missouri and New Jersey) participated in the current study. Kansas and Arizona were ineligible for participation due to having an early start in the ECELC, and South Florida was ineligible because investigators were not granted access to classrooms. Because the current pilot study was exploratory in nature, and due to budget constraints, six programmes from a pool of 559 ECE programmes enrolled in the second phase of the second cohort of the ECELC were randomly selected from each of the four sites to be observed using the EPAO (n 24). Programmes were excluded from analysis if they were unable to complete an EPAO observation at both pre-assessment and post-assessment or if they did not complete all aspects of the ECELC (five programmes), resulting in a final analytical sample of nineteen programmes.
Trained observers (one observer per classroom) conducted observations using the EPAO in the same pre-school-aged classroom for two consecutive days at pre-assessment (August through October 2014, prior to the launch of the ECELC) and again at post-assessment (August and September 2015, two to four months after the completion of intervention activities). Assessments occurred two to four months after the completion of the intervention activities for several reasons, chiefly due to the ‘real world’ challenges associated with conducting the study within a set budget. One specific example was that travel and schedules of several trained observers had to be coordinated in a way where observations could be conducted on consecutive days among ECE programmes within each site. Assessments were conducted in pre-school-aged classrooms as it was the largest proportion of children served across the programmes (56 %), although it ranged across sites from 33 % (Missouri) to 69 % (New Jersey). Additionally, most NAP SACC items apply to children of pre-school age.
EPAO measurement tool and scoring
The EPAO was developed to objectively assess environments of ECE programmes( Reference Ward, Hales and Haverly 9 ). All items in the EPAO were utilized for this analysis and were divided into two subgroups (Nutrition and Physical Activity) comprised of sixteen separate domains. Domains in the Nutrition subgroup included Fruits and Vegetables (ten items), Whole Grains and Low Fat Meats (six items), High Sugar/High Fat Foods (nine items), Beverages (twelve items), Staff Behaviours Regarding Nutrition (seven items), Nutrition Environment (four items), Nutrition Training and Education (six items) and Nutrition Policy (fourteen items). Domains in the Physical Activity subgroup included Active Opportunities (five items), Sedentary Opportunities (four items), Sedentary Environment (three items), Portable Play Environment (seven items), Fixed Play Environment (eight items), Staff Behaviours Regarding Physical Activity (five items), Physical Activity Training and Education (five items) and Physical Activity Policy (six items). Per EPAO protocol, item responses were coded on a three-point scale and scored as 0 (best practice not met), 1 (close to best practice) or 2 (best practice met). Scores were totalled within a given domain, for a total of 20 possible points per domain; a higher score translated to a greater number of best practices being observed. Scores were summed for each domain to calculate a total EPAO score (0–320 points) made up of a Nutrition sub-score (0–160 points) and Physical Activity sub-score (0–160 points).
Comparing EPAO change scores with NAP SACC change scores
ECE programmes completed the NAP SACC instrument following the first learning session and post-assessment occurred during the action period prior to the last learning session. The NAP SACC consisted of four topic areas: Breast-feeding & Infant Feeding (twenty-three items), Child Nutrition (forty-four items), Infant & Child Physical Activity (twenty-two items) and Screen Time (twelve items)( Reference Benjamin, Ammerman and Ward 12 ). The evaluation crosswalk method was used to identify the domains with which each NAP SACC item most aligned (see online supplementary material, Supplemental Table 1)( Reference O’Sullivan 13 ). Two researchers independently assigned each individual NAP SACC item to an EPAO domain that fit the objective of both the NAP SACC item and the corresponding EPAO domain. In cases of disagreement, items were discussed among the research team until consensus was reached. Because the observations occurred only in pre-school classrooms, NAP SACC items were excluded if they did not apply to pre-school-aged children (e.g. the Breast-feeding & Infant Feeding assessment), resulting in eighty-nine of the 121 NAP SACC items being utilized in this analysis. Each item had four response options, ranging from non-compliance with a particular best practice to total compliance with said best practice. When the response option representing total compliance with a given best practice was selected, the best practice was considered being met (1=best practice met). All other responses were considered to mean the best practice was not being met (0=best practice not met). The raw NAP SACC composite scores for each domain were calculated as the sum of outcomes of each applicable NAP SACC item. Because the number of NAP SACC items differs from the number of EPAO items for each domain, the raw scores were then scaled to be directly comparable to their respective EPAO domain as 20 times the raw score divided by the number of items. This resulted in similar maximum scores (20 points) per domain of the EPAO and composite scores of the NAP SACC, aiding in interpretation across the findings from the two measurement tools.
Statistical analysis
The SAS statistical software package version 9.4 was used for all statistical analyses. Descriptive statistics were calculated across all EPAO domains and NAP SACC composites at pre-assessment and post-assessment. A change score for each domain was calculated by subtracting the pre-assessment score from the post-assessment score. A paired-sample t test was utilized to examine whether mean scores changed from pre-assessment to post-assessment across the sub-sampled ECE programmes. In order to test if the data collected using the EPAO resulted in similar findings as data collected using the NAP SACC, Pearson correlation coefficients were used to measure the linear correlation between the change scores for each of the sixteen EPAO domains and corresponding NAP SACC composites. To accommodate the sample size of nineteen programmes, statistical significance was set at a two-sided α level of P<0·10, which enabled the power to approach 0·8 (β=0·22)( Reference Schoenfeld 14 , Reference Moore, Carter and Nietert 15 ).
Results
As described in the ‘Methods’ section, twenty-four programmes were randomly selected to be observed using the EPAO for the current study. Five programmes were unable to complete an EPAO observation at both pre-assessment and post-assessment (e.g. at least one programme closed for business between pre-assessment and post-assessment) or did not complete all aspects of the ECELC, resulting in a final sample of nineteen programmes. The majority of participating ECE programmes in the present sub-study were non-profit (68 %), most programmes (74 %) provided only a full day of care, and about half of ECE programmes participated in the US Department of Agriculture’s Child and Adult Care Food Program (47 %; Table 1). Overall, programme accreditation was low (32 %). Of all nineteen participating programmes, only six (32 %) participated in their state’s QRIS. All programmes provided some type of meal or snack throughout the day; the majority provided both meals and snacks (90 %). Most of the programmes prepared meals or snacks on-site (84 %), while relatively few only catered (11 %), and one programme used a combination of preparation on-site and catered.
Table 1 Characteristics of early care and education (ECE) programmes participating in the environment and policy assessment and observation sub-study (n 19)
CACFP, Child and Adult Care Food Program.
Data were collected from pre-school classrooms in nineteen programmes across four US states in 2014.
The mean total EPAO change score (for both Nutrition and Physical Activity) across all programmes was 26 (sd 38) points, which was a 17 % increase from pre-assessment to post-assessment (P=0·008; Table 2). The Nutrition sub-score contributed the most to the overall score with a change score of 19 points, which was a 25 % increase from pre-assessment to post-assessment (P<0·001). Physical Activity contributed 8 points to the overall change but was not significant. At the domain level, five of the nutrition domains underwent a statistically significant change, with Nutrition Environment showing the greatest improvement of 6 points (P<0·001). One of the physical activity domains, Active Opportunities, showed a significant improvement with an increase of 2 points (P=0·001).
Table 2 Change in Environment and Policy Assessment and Observation (EPAO) score from pre-assessment to post-assessment (n 19)
Data were collected from pre-school classrooms in nineteen programmes across four US states from 2014 to 2015.
*P<0·10, **P<0·05, ***P<0·01, ****P<0·001.
Scores for the EPAO domains and their corresponding NAP SACC composites are shown in Table 3, as well as the Pearson correlation coefficients. Ten of the sixteen domain–composite pairs shared directionality in their change scores indicating both the EPAO and NAP SACC were able to detect similar changes when programmes were measured as a group. However, when comparing the EPAO change score with the NAP SACC change score for each programme, the correlation coefficients and P values demonstrate less of a relationship between the two measurement tools. Only the domains and composites of Nutrition Policy (r=−0·4, P=0·06) and Physical Activity Policy (r=−0·6, P=0·02) were found to have a statistically significant correlation. Interestingly, the correlations implied a negative relationship between the two scores, implying that as one increased the other decreased. No other relationships between EPAO domains and NAP SACC composites were significantly related.
Table 3 Mean change scores for Environment and Policy Assessment and Observation (EPAO) domains and Nutrition and Physical Activity Self-Assessment for Child Care (NAP SACC) composites and Pearson correlation coefficients
Data were collected from pre-school classrooms in nineteen programmes across four US states from 2014 to 2015.
*P<0·10, **P<0·05.
† Correlation coefficients are calculated on programme-level change scores for each domain.
Discussion
Although the present study was a pilot study with a small sample size, we did find that programmes assessed using the EPAO changed with regard to several physical activity and nutrition environment best practices( Reference Ward, Hales and Haverly 9 ). Overall, the programmes improved by about 17 % of the total EPAO score, with the majority of improvements occurring in the nutrition-focused domain. Programmes improved in the areas of Whole Grains and Low Fat Meats, High Sugar/High Fat Foods, Staff Behaviours Regarding Nutrition, the Nutrition Environment, and Nutrition Training and Education. However, programmes did not improve significantly in the areas of Fruits and Vegetables, Beverages, or Nutrition Policy. For Physical Activity, programmes improved significantly in the areas of Active Opportunities, but no other domains. Results of similar studies promoting healthy eating and physical activity in childcare settings have also shown interventions involving self-assessment and action planning enable change in programme-level practices( Reference Benjamin, Ammerman and Sommers 6 , Reference Trost, Messner and Fitzgerald 7 , Reference Davison, Falbe and Taveras 16 ).
The original intent of the NAP SACC measurement tool was to serve as an aid in self-assessment and action planning and it was not intended as an objective outcome measure( Reference Benjamin, Ammerman and Sommers 6 , Reference Benjamin, Neelon and Ball 8 , Reference Ward, Hales and Haverly 9 , Reference Benjamin, Ammerman and Ward 12 ). Accordingly, all ECE programmes participating in the ECELC developed personalized action plans based on their self-assessments and were given autonomy to focus on their most desired changes related to any of the five NAP SACC sections, including Breast-feeding & Infant Feeding. This is important to consider because domains that resulted in little or no change scored relatively higher at pre-assessment when compared with domains that resulted in significant change. For example, even though there was no significant change to the domain of Portable Play Environment, the score from the pre-assessment was 13·01, which was the second highest score at pre-assessment. It is unknown exactly how ECE programmes chose to focus their targeted action planning, although it is likely they selected areas of higher need, which may have been domains with relatively lower scores at pre-assessment and potentially why programmes did not improve significantly in several areas (i.e. fruits and vegetables, beverages, reducing sedentary opportunities, improving portable play environment or the fixed play environment, or enhancing staff behaviours regarding physical activity) as measured by the EPAO.
The EPAO generally resulted in different outcomes from the NAP SACC. Only the EPAO domains of Nutrition Policy and Physical Activity Policy correlated with their NAP SACC counterparts, although it was a negative, and therefore an unexpected, correlation. Another study that aimed to validate the NAP SACC using the EPAO as the gold standard assessed the relationship of cross-sectional scores of each measurement tool( Reference Benjamin, Neelon and Ball 8 ). Kappa statistics ranged from −0·01 to 0·79, which was considered poor to substantial agreement. While the prior study tested the relationship between the EPAO and the NAP SACC using cross-sectional scores, the current study aimed to determine if the data collected using the EPAO resulted in the same outcomes as data collected using the NAP SACC. Accordingly, change scores were used to test the relationship between the two measurement tools. Regardless, our study corroborates the findings of the previous study, suggesting the NAP SACC may tap into slightly different constructs from those from the EPAO, especially given that the former is a self-assessment and the latter is an observational measure.
Since the NAP SACC is a self-assessment, there is greater opportunity for bias due to social desirability, which is common among nutrition- and physical activity-based self-reporting( Reference Hebert, Ma and Clemow 17 – Reference Hebert, Clemow and Pbert 21 ). The EPAO is completed by an independent observer so there is far less chance for social desirability bias. Additionally, the NAP SACC pre-assessment is completed early in the ECELC, when ECE programme staff may have less knowledge about topics related to nutrition and physical activity than at post-assessment. Therefore, rather than a reflection of true change, the NAP SACC may sometimes be a reflection of increased knowledge and more accurate responses; similar interventions have promoted an increase in health-related knowledge among childcare providers( Reference Bonis, Loftin and Ward 22 ). Future research could test the NAP SACC and EPAO in a larger sample, which would increase the power of the analysis and potentially lead to more consistency between change scores of the two tools. However, the EPAO is costly and requires additional staff time, and therefore the NAP SACC may be more feasible as an assessment tool to implement on a wide scale.
Our study has some limitations to note. The small sample size did not allow for enough power to adequately detect relationships between the EPAO and the NAP SACC at the traditional level of significance (α=0·05). Additionally, since outcome data were all quantitative, we did not have the contextual information to assist in explaining and interpreting findings. Further, state-based healthy eating and physical activity initiatives outside the ECELC like the Missouri Eat Smart and MOve Smart Guidelines for Child Care( 23 ) may have influenced change. It should also be discussed that the post-assessment ideally would have occurred directly after the completion of the intervention. However, as described earlier, this evaluation required utmost coordination among the study team and participating ECE programmes, which led to post-assessment occurring two to four months after the completion of the evaluation. Nevertheless, data from a study currently in review suggested that best practices and policies related to breast-feeding support, child nutrition, physical activity, and screen time reduction were sustained one year after this intervention ended (TM Smith, C Blaser, C Geno Rasmussen et al., unpublished results), so we expect that changes made during the intervention would be unchanged two to four months after the completion of the intervention. Despite these limitations, the current study helps elucidate nuances in the implementation, practicalities and ultimately the comparison of outcome data using the NAP SACC and EPAO. These changes, while just one step in potentially reducing obesity among children aged 5 years or younger, can help inform other obesity prevention interventions in ECE programmes moving forward.
The overall significant improvements reported through the EPAO measurement tool in the present pilot study suggest that participation in the ECELC resulted in changes in some, but not all, critical nutrition- and physical activity-related practices in ECE programmes. However, considerable differences in data reported using the NAP SACC compared with objective data using the EPAO suggest NAP SACC data should be interpreted with caution and objective measurement should be used when feasible. Self-assessment and observational methods for assessing nutrition and physical activity practices in ECE programmes have strengths and limitations; factors such as cost and feasibility should be taken into account when choosing a measure. However, both methods have merit and are important to assess and advance environmental interventions intended to change nutrition and physical activity policies/practices in ECE.
Acknowledgements
Acknowledgements: The authors would like to thank all of the National Early Care and Education Learning Collaboratives Project stakeholders, including State Implementing Partners, State Project Coordinators, Trainers and Leadership Team members, for their participation in this evaluation. Also, they would like to thank Dr Dan Schober, Dr Mary Story, Dr Dianne Ward and Stephanie Mazzuca for providing expertise on study design and measurement protocol. Finally, the authors would like to thank staff at the Gretchen Swanson Center for Nutrition for their support on this evaluation. Financial support: Funding for the National Early Care and Education Learning Collaboratives Project was provided by a five-year Cooperative Agreement (1U58DP004102-01) to the Nemours Children’s Health System from the Centers for Disease Control and Prevention. The funders advised on the design and analysis, but did not have a role in the writing of this article. Conflict of interest: None. Authorship: J.S. and A.L.Y. conceived and supervised the study; C.B. completed the analysis; T.M.S. led the writing; C.P. and T.M.S. assisted with the study; C.G.R. advised on the study; all authors contributed to the writing. Ethics of human subject participation: This study was approved by the Nemours Institutional Review Board.
Supplementary material
To view supplementary material for this article, please visit https://dx.doi.org/10.1017/S1368980017000155
