Reviews have found moderate evidence that theory-based healthy diet promotion programmes for children and adolescents in schools can change dietary intakes(Reference Ammerman, Lindquist and Lohr1–Reference Van Cauwenberghe, Maes and Spittaels3). However, as soon as health promotion programmes are used in real-life settings, the most serious threat to their effectiveness is low levels of fidelity of programme implementation, also known as type 3 error(Reference Dobson and Cook4). Research in a range of settings has shown that various aspects of the implementation of health promotion interventions, such as adherence (implementation as intended by the developers) and completeness (the proportion of the activities proposed by the programme that was actually used), affect programme outcomes(Reference Wilson and Kurz5–Reference Durlak and DuPre7). Derzon et al. re-assessed the outcomes of forty-six effectiveness studies of substance abuse prevention programmes, correcting for implementation and exposure of control groups, and found beneficial programme effects for 46 % of the studies which initially reported no effects(Reference Derzon, Sale and Springer6). Two other studies found that programme outcomes were not only related to overall implementation, but also to the implementation of specific programme elements(Reference Stevens, Van Oost and De Bourdeaudhuij8, Reference Telzrow, McNamara and Hollinger9). Examining the relationship between the degree of completeness of programme implementation and the variability in programme outcomes can help generate leads for programme improvement and the improvement of implementation strategies.
Krachtvoer is a Dutch school-based healthy diet programme for 12- to 14-year-olds attending the first two years of prevocational education. Programme goals are to increase the consumption of fruit, to decrease the consumption of fats by replacing high-fat snacks and sweets (e.g. chocolates, candy bars) by non-fat or low-fat alternatives (e.g. chewing gum, acid drops) and to stimulate daily consumption of a healthy breakfast(Reference Bessems, Van Assema and Martens10). The programme consists of eight lessons, including lessons with a fixed protocol and optional activities. Some of the lessons are mainly theoretical, for example those with information on national dietary recommendations, while others are mainly practical, for example those including fruit shake preparation or tasting activities. The lessons are intended to be taught during the nutrition lessons by a school teacher of biology (subject which is focused on technical knowledge, e.g. on nutrients) or care (subject which is focused on practical knowledge and skills, e.g. cooking skills).
The programme was originally developed in 2001 and revised in 2007. An effectiveness study of the revised programme revealed both immediate and longer-term (6 months after the end of implementation) increases in students’ fruit consumption. As regards snack and sweets intake, students less often consumed sweets from the unfavourable sweets category (e.g. less often chocolates, more often chewing gum or no sweets)(11). On the other hand, the trend towards falling breakfast frequencies which is seen among Dutch youngsters between the ages of 12 and 16 years(Reference Raaijmakers, Bessems and Kremers12) did not slow down as a consequence of the programme.
The goal of the current study was to examine the relationship between changes in students’ dietary intakes and the level of completeness of implementation of the full programme and of specific programme elements. We used an outcome measure of fruit consumption and sweets consumption for which short- and longer-term effects had been found in the effectiveness study(11). Although the effectiveness study found no effects on breakfast frequency (i.e. breakfast frequency decreased in the intervention and in the control group), we included this measure to examine whether the lack of effect could be explained by a type 3 error. Specific research questions were:
1. Can the favourable changes in fruit and sweets consumption found in the effectiveness study be related to the completeness of programme implementation and to the completeness of implementation of specific programme elements?
2. Can the unfavourable changes in breakfast consumption found in the effectiveness study be related to suboptimal programme implementation and incomplete implementation of specific programme elements?
Content of the programme
The main programme elements are one lesson to make students enthusiastic about the programme (lesson 1), three lessons for raising awareness (lessons 2 to the first part of lesson 4), one practical food exposure activity (i.e. taste testing, fruit tasting or preparing a fruit shake: lesson 5), one lesson on the influence of advertisements (lesson 6) and two lessons in which students formulate, implement and evaluate a personal action plan (lessons 7 and 8). Before the start of the programme teachers can choose the optional lesson on nutrients (lesson 0). Additional optional programme elements within lesson 4 (second part) and lesson 7 (first part) are participating in a nationwide recipe contest (as homework), reading and discussing the Krachtvoer magazine, knowledge tests on the programme website, and additional food exposure activities as proposed in lesson 5.
The programme builds on the three phases of self-management theory(Reference Boekaerts, Rintrich and Zeider13), successively aiming at raising awareness of personal dietary intakes, proposing solutions in order to meet the Dutch dietary guidelines and setting personal goals for dietary improvement(Reference Bessems, Van Assema and Martens10, Reference Martens, Van Assema and Paulussen14). A programme overview is presented in Box 1. A more extensive programme description is published elsewhere(Reference Bessems, Van Assema and Martens10).
Box 1 The Krachtvoer programme
Lessons
The programme covers eight lessons and one optional pre-programme lesson on nutrients. Some slots in the lessons 4, 5, 7 and 8 can be filled by six optional activities (of which at least two should be used).
• Before the start of the programme:
○ Lesson 0: Nutrients (optional element). Students learn about nutrients (e.g. saturated fats, carbohydrates), their functions in the human body and food products containing these nutrients
• Phase 1. Students are made aware of their own dietary intake and deviations from the Dutch dietary guidelines.
○ Lesson 1: Nutrition, foods and health (fixed element). Students’ knowledge on fruit, breakfast and high-fat snacks is tested in a quiz. Students receive a lunchbox with a flyer and three healthy food items representing the three topics.
○ Lesson 2: Fruit and fruit juices (fixed element). Students learn to read fruit juice labels. Knowledge about the differences between fruit juices and other juices is tested with a fruit juice test. They are made aware of their own fruit or fruit juice consumption, and become familiar with national recommendations.
○ Lesson 3: Breakfast and snacking (fixed element). Students are made aware of their own breakfast habits and high-fat snack consumption. The disc of five and an overview of low- and high-fat snacks are discussed and compared with their own breakfast and snack intakes.
• Phase 2. Students specify reasons for deviations and are encouraged to propose solutions.
○ Lesson 4 (part 1): Barriers to healthy eating (fixed element). Students answer questions on personal reasons for (not) meeting the dietary recommendations on fruit, breakfast and snack intakes, and give each other tips on healthy eating. Students receive a take-home bag with healthy products, a newsletter with tips and recipes, and a notepad designed to involve their parents.
○ Lesson 4 (part 2) and lesson 7 (part 1) (optional element).
▪ National recipe contest. Students participate in a national recipe contest.
▪ Magazine. Students work with a magazine offering information, tests, puzzles, a horoscope, role model stories and healthy recipes.
▪ Website. Students visit the Krachtvoer website to read information, do a snack test (i.e. distinguishing between high- and low-fat snacks) and a knowledge test (e.g. practical and theoretical knowledge about fruit, breakfast and snacks), and send e-cards.
○ Lesson 5. Food exposure (teachers can choose one or more elements).
▪ Taste testing. Students judge products by tasting, smelling and looking at (unfamiliar) fruit, breakfast products and favourable snacks.
▪ Fruit tasting. Students bring fruits from home to school and taste them together.
▪ Preparing a fruit shake. Students prepare a healthy tasty fruit shake
○ Lesson 6. Advertisements (fixed element). Advertising tricks are discussed and are applied in an advertising poster produced by the students.
• Phase 3. Students implement and evaluate an action plan.
○ Lesson 7 (part 2): Personal action plans (fixed element). Students use a program on the website to generate a personal action plan (what, when, where) to improve their fruit, breakfast or snack intake during the next week.
○ Lesson 8: Evaluation of personal plans (fixed element). In-class evaluation of the action plans, followed by an optional activity.
Methods
Study design, sample and data collection
We included students and teachers from the intervention condition of the effectiveness study, i.e. students who were taught the Krachtvoer programme and the teachers who implemented the programme(11). Respondents from the control condition were not included since they were not exposed to any element of the programme, but received another nutrition education programme.
Data were collected from thirteen schools, fifty-three classes and 1117 students. Student questionnaires were filled in as a classroom activity at baseline (T0; 1–4 weeks before the lessons), at post-test (T1; 1–4 weeks after the lessons) and at the follow-up test (T2; 6 months after the lessons). Students who objected to participation were allowed to do another task during the lesson. The study was exempt from ethical review according to prevailing Dutch standards because there was no risk for participants and participation was voluntary(15). Completeness of implementation was assessed using teacher logbooks. All twenty-two teachers who implemented Krachtvoer were asked to fill in the logbook after each lesson.
Measures
The student questionnaire assessed background variables, i.e. gender, country of birth of both parents as an indicator of ethnicity (non-Dutch if one or both parents had been born abroad(Reference Keij16)) and postal code as an indicator of socio-economic position (SEP)(17). Teachers provided data on school-related variables, including the school year of their class (first or second year) and the educational sub-track within the Dutch prevocational education system (lower (sub-track 2) or higher (sub-tracks 3 and 4)).
Fruit consumption items were derived from a validated fruit and vegetable FFQ(Reference Van Assema, Brug and Ronda18), and included an item on the number of days a week on which fruit was usually consumed and an item on the number of servings of fruit consumed on these days. Examples of one serving of fruit (e.g. one apple, one bowl of strawberries) and half a serving of fruit (e.g. one mandarin) were added to the question. Frequency and quantity scores were multiplied and divided by seven to obtain the average fruit consumption per day. One item was used to measure the usual consumption of sweets. In line with the programme and the dietary guidelines provided by the Netherlands Nutrition Centre(19), a distinction was made between unfavourable types of sweets (e.g. chocolates, candy bars) and more favourable types (e.g. acid drops, chewing gum). Frequency items were formulated (e.g. ‘Which category of sweets do you usually eat?’, with options ‘I do not eat sweets’, ‘I usually eat… (followed by examples of favourable products)’, ‘I usually eat… (followed by examples of unfavourable products)’ and ‘I eat sweets from both categories equally’). Breakfast consumption was measured with one FFQ item about the number of days per week on which breakfast was usually consumed.
Completeness of implementation was measured by asking teachers to indicate which programme elements (lessons and optional activities) they had implemented. For each of thirteen programme elements as proposed in Box 1 (except for the optional recipe contest which was a homework assignment), teachers could tick off the activities they had implemented on a list of all proposed activities. A score for completeness of element implementation (range 0–1) was calculated by dividing the number of implemented activities by the number of proposed activities within that programme element. A score for completeness of programme implementation (range 0–13) was calculated as the sum of the scores for the activities of each programme element.
Statistical analysis
Student and teacher data were matched on demographics, class and school. Change scores for the continuous outcomes of fruit and breakfast intake were calculated by subtracting baseline scores from the scores at the two post-tests. The four answering categories of sweets consumption were dichotomized into two new categories: favourable consumption (‘no sweets’ and ‘favourable products’) v. unfavourable consumption (‘unfavourable products’ and ‘both favourable and unfavourable products’). Descriptive statistics were used to describe the background characteristics of participating students and teachers, and the percentages of students who had changed their fruit, sweets and breakfast intakes at the two post-tests. Mixed linear regression analyses were carried out using the statistical software packages PASW Statistics 17 for continuous outcomes and MLwiN version 2·02 for binary outcomes. We tested for selective student drop-out (due to missing post-test data or implementation data) with mixed logistic regressions using MLwiN, with drop-out at any moment as the outcome variable (inclusion = 1, dropout = 0), and with student background variables (gender, ethnicity, SEP, school year and educational sub-track) and baseline dietary intake variables (fruit, sweets, breakfast) as fixed factors, and with a random intercept of the class the student was in. Pearson's correlations were calculated to check if implementation scores for the thirteen programme elements were strongly interrelated. Based on high Pearson's correlations (>0·75) and the content of programme elements, two sum scores were calculated, one for lessons 1 to 4 (creating awareness; range 0–4) and one for lessons 7 and 8 (personal plans; range 0–2). Since there was little variation between teachers implementing the three practical activities, we calculated one sum score for the taste testing, fruit tasting and fruit shake preparation activities (range 0–3).
The relationships between changes in fruit and breakfast consumption and completeness of programme and element implementation were tested by mixed linear regression analyses for the individual post-tests, with change scores of fruit or breakfast as outcome variables, and gender, ethnicity, SEP, year (first or second) and sub-track as fixed factors. The first analysis included completeness of programme implementation as a factor, while the second analysis included completeness of the implementation of the individual elements, including lesson 0 (range 0–1), the lessons on awareness (range 0–4), the practical activities (range 0–3), the advertisement lesson (range 0–1), the tests on the website (range 0–1), the magazine lesson (range 0–1) and the lessons on action plans (range 0–2). Comparable mixed logistic regression analyses were conducted for the dichotomous outcomes on sweets consumption, correcting for baseline intake.
Results
Response rates and participant characteristics
Eighteen of the twenty-two participating teachers filled in the teacher logbook after each lesson. They had implemented the lessons in forty-five of the fifty-three participating classes, in lessons on biology (n 6), care (n 6) or a combination both subjects (n 6), over an average period of 59 (sd 26·5) d. Four teachers did not fill in the teacher logbook, which meant that 154 students in their classes were excluded from the study. Additionally, sixty-six students had to be excluded at the first post-test, and eighty-seven were excluded at the second post-test, as they had been absent from the lesson in which the questionnaire was completed or had incomplete background characteristics, making it impossible to link the data of separate measurements. Analyses at the first post-test included a total of 897 students, while those at the second post-test included 876 students. Students who dropped out of the study were significantly more often second-year students (OR = 1·67, 95 % CI 1·13, 2·47; P < 0·01) and had consumed breakfast less frequently before programme implementation (OR = 0·91, 95 % CI 0·83, 0·98; P < 0·05).
Half of the participating students were male. The majority of the students were in the second year (60·0 %), attended the higher educational sub-track (85·1 %) and were of Dutch ethnicity (79·8 %). The mean SEP score was comparable to the average Dutch SEP score (mean for participants = −0·05; Dutch mean = 0·00 (scale 4 to −4; low to high)). The students included in the study had less than 5·0 % missing values on all items, except for sweets consumption (missing values T1 = 8·0 %; missing values T2 = 10·2 %).
Changes in fruit, sweets and breakfast consumption
Mean fruit consumption at baseline was 0·97 (sd 0·78) servings/d. A total of 58·6 % of the students usually consumed sweets from the unfavourable category. At baseline, the mean number of days per week on which breakfast was consumed was 6·31 (sd 1·61).
Table 1 shows that mean fruit intake increased in the short term (+0·14 pieces/d) as well as in the longer term (+0·06 pieces/d). At the first and second post-test, 16·4 % and 18·5 % of the students, respectively, had shifted their usual sweets consumption from the unfavourable category towards the favourable category. The mean number of days on which breakfast was consumed had decreased by 0·20 and 0·17 d/week at the first and second post-tests, respectively.
Table 1 Changes in fruit, sweets and breakfast consumption among Dutch students (n 876) in thirteen prevocational schools participating in the Krachtvoer healthy diet promotion programme
n.a., no mean change score could be calculated for dichotomous outcome measures.
Completeness of implementation of the programme and of specific programme elements
The mean score for completeness of programme implementation was 8·0 (sd 1·9; range 4·8–11·0). Table 2 shows that the lowest score for completeness of implementation was found for the action plan lesson (mean = 0·47), while the fruit shake preparation activity was most fully implemented (mean = 0·90). The first fixed lessons were implemented more completely than the later lessons.
Table 2 Number of teachers who completed the lesson or activity and completeness of implementation of programme elements, assessed in terms of mean score, standard deviation and range, among Dutch teachers (n 18) in thirteen prevocational schools participating in the Krachtvoer healthy diet promotion programme
†Two lessons and three optional activities consisted of one activity only.
Changes in dietary intakes in relation to completeness of implementation of the programme and of specific programme elements
Table 3 shows the relationships between the scores for completeness of implementation of the programme and specific elements and the short-term and longer-term changes in students’ fruit, sweets and breakfast consumption. The score for completeness of programme implementation was significantly related to an increase in fruit consumption in the short term. The scores for completeness of implementation of the practical activities and the advertisement lesson were significantly related to an increase in fruit consumption in the short and longer term. No other significant relationships were found.
Table 3 Completeness of implementation of the programme and the specific programme elements as predictors of changes in fruit frequency, sweets consumption and breakfast frequency among Dutch students (n 876) in thirteen prevocational schools participating in the Krachtvoer healthy diet promotion programme
Reported changes were corrected for a random intercept of class, and the fixed factors of gender, school year, educational track, socio-economic position and ethnicity.
*P < 0·05; ***P < 0·001.
Discussion
The first aim of the present study was to investigate whether favourable changes in fruit and sweets consumption that were found in an effectiveness study could be related to the completeness of the implementation of the Krachtvoer programme and the completeness of implementation of specific programme elements. The second aim was to examine whether unfavourable changes found in breakfast consumption could be related to the incompleteness of the programme implementation and the incompleteness of implementation of specific programme elements.
We found that the completeness of implementation of the programme was related to short-term, but not longer-term increases in fruit consumption. A possible explanation is that Krachtvoer, and the instrument used to measure its completeness of programme implementation, included elements not only addressing fruit consumption, but also snack and breakfast consumption. The sensitivity of our measure may have been especially problematic in the long term when the programme effect had decreased. Furthermore, completeness of implementation of the food exposure activities and the advertisement activity was related to both short- and longer-term changes in fruit consumption, implying that these elements made a unique contribution towards changes in fruit consumption. Increased fruit consumption was not only related to programme elements addressing fruit and fruit juices (i.e. taste testing, fruit tasting and preparation of a fruit shake), but also to a programme element which focused on environmental influences on food consumption in general (advertisement activity, applying advertisement tricks). These were all very practical activities, which were implemented with high levels of completeness by the teachers, and were appreciated better by the students than all other elements(Reference Bessems, Van Assema and Martens10). This importance of practical activities is in agreement with the findings of a qualitative study that was held among students during the process of developing the programme, and that indicated the importance of tasting and recipe preparation activities (e.g. preparing a fruit shake), as well as other activities in which students were allowed ‘to doing things themselves’(Reference Martens, Wind and Van Assema20). The importance of appreciation is in agreement with studies that also found appreciation to be related to changes in dietary intakes(Reference Tak, Te Velde and Brug21, Reference Wind, Bjelland and Perez-Rodrigo22). The key element of Krachtvoer seems to be that it uses practical and highly appreciated methods, which are effectively implemented by teachers. It is worth exploring if appreciation mediates the relationship between programme outcomes and the completeness of implementation of certain activities.
We found no relationships between sweets consumption and the completeness of implementation of the programme or specific elements, possibly because our outcome measure for sweets consumption lacked sensitivity. We recommend studying this relationship using a more sensitive item combining type of sweets and frequency of consumption.
Further, our findings indicate that the lack of effect on breakfast intake found in the effectiveness study(11) was not attributable to incomplete implementation. This finding stresses the conclusion of the effectiveness study, that our school-based educational intervention was unable to change breakfast consumption.
The current study had some limitations as well as some strengths. Overall, the programme elements were implemented quite well (with completeness scores for specific programme elements ranging from 0·47 to 0·90), indicating that there was limited variation in the completeness of implementation scores between teachers. This made it more difficult to answer our research questions. Using a sensitive score for completeness of implementation (counting the number of activities that were implemented in each programme element) enabled us to include the separate programme elements in our analyses that were implemented by all teachers (i.e. lessons 1–3). Still, if our sample had shown more variation regarding implementation scores of these lessons, a relationship between the completeness of implementation of these lessons and the changes in dietary patterns might have been revealed as well. A strong point of our study was that we assessed the completeness scores for individual programme elements, which helped us distinguish the relative importance of individual programme elements. Still, the implementation of these specific elements may have been related to other aspects which were not measured, such as teachers’ enthusiasm and respect for students’ opinions(Reference Dusenbury, Brannigan and Falco23) or students’ engagement(Reference Ringwalt, Pankratz and Gottfredson24), or may have been a result of multiple testing. However, the findings on the importance of full implementation of practical programme elements are consistent over time and in line with the findings of an earlier qualitative study(Reference Martens, Wind and Van Assema20), and are therefore considered to be plausible. Another limitation was the use of logbooks to measure programme implementation. Although observational methods have been recommended to measure implementation(Reference Durlak and DuPre7), these are more time-consuming and hence were infeasible within the context of the current study. Completeness is only one element of the fidelity of programme implementation and the present study did not asses programme adherence and quality, which have been shown to be related to outcomes of interventions to prevent drug abuse(Reference Dusenbury, Brannigan and Falco23). We therefore recommend that future studies consider including assessment of completeness, quality, adherence, responsiveness and differentiation(Reference Proctor, Silmere and Raghavan25, Reference Carroll, Patterson and Wood26). However, the time investment required for this may be prohibitive for large research populations. Finally, we did not include control schools in the current study since these were irrelevant for the present research questions. Therefore, relative changes in the intervention schools as compared with controls as were obtained in the effectiveness study have not been considered here.
Conclusions
Our study has indicated the importance of stimulating teachers to implement the Krachtvoer programme to a high level of completeness, with a strong emphasis on the practical elements. To our knowledge, we are among the first to try and quantify the contributions of the implementation of particular programme elements to outcomes on dietary patterns. Although including the completeness of implementation of programme elements in a randomized controlled trial is a challenge, we would still strongly recommend this for future studies. It can help distinguish a minimum required level of completeness of implementation, as well as distinguish which programme elements are essential, thus providing directions for programme implementation strategies. Further studies should explore appreciation as a mediator of the relationship between the completeness of implementation of various types of activities and programme outcomes.
Acknowledgements
This work was supported by the Netherlands Organisation for Health Research and Development (ZonMw), grant number 63200011. The authors declare that there are no conflicts of interest. All authors contributed to the design of the study. K.M.H.H.B. and R.C. performed statistical analyses. K.M.H.H.B. and P.v.A. prepared a draft of the manuscript. All authors contributed to reviewing and improving the manuscript in terms of content and approved the final manuscript.
