In Canada, nearly a third of 6- to 17-year old children and adolescents are overweight or obese(Reference Rao, Kropac and Do1). Overweight has consequences on their health and quality of life(Reference Puhl and Luedicke2,Reference Kenney, Gortmaker and Davison3) . The school environment can influence students’ diets(Reference Driessen, Cameron and Thornton4,Reference Seliske, Pickett and Rosu5) , since they spend more than 180 d a year in school and ingest around a third of their daily energy at school(Reference Tugault-Lafleur, Black and Barr6). Several evaluations found in schools a widespread availability of hypoenergetic, unhealthy and competitive foods, especially for older students(Reference Winson7–Reference Molnar, Boninger and Wilkinson14). Following this ascertainment, in 2008, the WHO called on its members to develop and implement school policies and programmes that would promote healthy eating and increase physical activity in students(Reference Roberto, Swinburn and Hawkes15,16) . Likewise, the United States Department of Agriculture and several Canadian provinces announced guidelines governing school food and beverage offerings(Reference McIsaac, Shearer and Veugelers17–Reference Martorell21).
Following the implementation of healthy food procurement policies, studies revealed an overall improvement of the nutritional content of meals served in school(Reference Niebylski, Lu and Campbell22,Reference Watts, Mâsse and Naylor23) , thus reducing daily energy (16 %), fat (27 %), Na (18 %) and sugar (37 %) intakes(Reference Niebylski, Lu and Campbell22,Reference Williamson, Han and Johnson24) . In spite of these observations, there was a growing concern from school stakeholders that requirements for healthier food options at school were driving students to purchase food outside of school(Reference Lysyk25), thus reducing profits for school food services. It was brought to light that fast-food retailers in the schools’ environment were becoming more attractive and accessible. For example, in a Canadian province, schools averaged twenty-seven outlets, with a maximum of sixty-five fast-food restaurants within 1 km(Reference Hanning, Luan and Orava26). Consequently, some schools have been re-introducing less nutritious foods to their offerings(Reference Morel and Voyer27). Therefore, in 2016, the WHO recommended ongoing surveillance of school food environments(28). Precise and repeated measurements of children’s school environments are needed, especially in terms of food supply(Reference Chriqui, Pickel and Story29,Reference Raine Kim, Kayla and Lee30) .
In line with the Quebec Governmental Preventive Health Policy and its Cross-Ministerial Action Plan(31,32) , our team developed a longitudinal web-based data collection platform, the School Information System (SIS). The SIS aims to describe school environments and to monitor the implementation of school policies. It generates an automated and personalised report for each participating school to promote the implementation of up-to-date and efficient interventions. We first prioritised the development of the food offer domain on the SIS, but this platform is meant to evolve into a multidimensional data collection system (e.g. physical activity facilities, schools’ environment).
The SIS operates with web-based self-diagnostic questionnaires completed by school stakeholders, eliminating the need for paper-and-pencil questionnaires or field observations by research assistants. Expected benefits from this web-based platform include low-cost data collection(Reference Buchanan and Smith33), once the costs of programming and online deployment have been covered(Reference Granello and Wheaton34), and increased data reliability by eliminating biases related to data entry due to human error and to the interviewer(Reference Granello and Wheaton34). Respondents can also expect greater schedule flexibility, since they can complete questionnaires when it best suits them and take breaks as needed without the pressure of a time limit(Reference Lindhjem and Navrud35). Faster returns and higher response rates are also predicted, compared to the paper-and-pencil method(Reference Kongsved, Basnov and Holm-Christensen36).
Nonetheless, it is important to ensure that the design of the web-based self-diagnostic questionnaires remains simultaneously simple to use for all respondents, as well as scientifically valid(Reference Jean37). Therefore, the general objective of this project was to develop and validate the first SIS web-based self-diagnostic questionnaire on the school food offer.
Methods
First, we went through steps inspired by the work of Jean (2015)(Reference Jean37) and Granello and Wheaton (2004)(Reference Granello and Wheaton34) for the development and preliminary validation of the web-based material. These included a context validation of the concept of the main school food service’s offer and a content validation of the questionnaire by experts. Second, we realised an on-site validation of the web-based material. To do so, we assessed the inter-rater reliability of the questionnaire by comparing observations from two measures of the food offer: a web-based self-diagnostic questionnaire completed by food service managers (FSM) and a printed version of the same questionnaire completed by registered dietitians (RD) using on-site observations. Concurrently, we qualitatively assessed the web-based material through user observations and interviews. Finally, we surveyed participating schools on their appreciation of the automated and personalised report they were offered following their participation. Figure 1 shows the flow of the methods.
Flow of the methods of development and validation
Development and preliminary validation of the web-based material
Self-diagnostic questionnaire
The procedure began with a qualitative and quantitative assessment of the concept of the main school food service’s offer. This was done using the validation theory(38,Reference Streiner, Norman and Cairney39) , through a review of the scientific literature(Reference Nathan, Wolfenden and Morgan40–42) and of relevant healthy eating policies(43–45). We listed indicators (e.g. presence of fried food in the à la carte menu, use of a cyclic menu, removal of meat fat before cooking) and wrote operational definitions related to those indicators. A panel of experts(Reference Polit, Beck and Owen46), involving nine RD working in schools, school boards, public health regional boards and a governmental nutrition department, was invited to independently comment on each operational definition. The aim was to reach agreement on a common language. The experts were also asked to comment on each indicator and to rate their relevance on a four-point Likert scale (where 1 = ‘irrelevant’, 2 = ‘slightly relevant’, 3 = ‘somewhat relevant’ and 4 = ‘very relevant’). The aim was to assess the current reality of the main school food service’s offer adequately by modifying, discarding or adding indicators. We calculated an average relevance rating for each individual indicator, as well as a global average. We preferred the use of averages rather than medians to allow extreme values to influence the assessment’s outcomes. The content validity index (CVI)(Reference Waltz and Bausell47) was calculated by dividing the number of indicators with average relevance ratings ≥ 3 by the total number of indicators. A CVI of 0·80 was set as the acceptable threshold(Reference Krippendorff48); the list of indicators had to be adjusted at least until this threshold was reached by removing the least relevant indicators. Then, we used the resulting list of indicators to create the questionnaire. The first draft of questionnaire items was taken from the paper-and-pencil questionnaire previously developed by P. Morin et al. that was meant to be completed by RD or nutrition technicians(Reference Morin, Demers and Gray-Donald49). Some items were also inspired by recent existing questionnaires assessing the school food offer(42,Reference Ajja, Beets and Chandler50) . Items and their wording were adapted to a self-reporting format and to the expected users, which would mostly be FSM without nutrition expertise. We paid attention to formulating neutral questions to limit the social desirability bias among respondents. We aimed to develop a questionnaire of a maximal completion duration of 45 min for the targeted respondents.
The questionnaire’s content was validated by seven experts(Reference Polit, Beck and Owen46) from the former panel. They were asked to comment on the capacity of the overall questionnaire to measure the concept of the main school food service’s offer, the relevance of each question to assess indicators of the concept, missing or redundant questions, the clarity of the questions’ formulation and the completion time.
Then, ‘think-aloud’ interviews were conducted to assess readability and to ensure that respondents interpreted the questions as we expected(Reference Willis51). This was realised with five volunteers with diverse levels of nutrition literacy. Volunteers were asked to consider the questions one at a time and to comment aloud for each of them. We then finalised the improved questionnaire, which was slightly longer for secondary schools than for elementary schools due to their larger offer of à la carte foods.
Automated and personalised report
The SIS makes possible to automatically generate a report for school stakeholders from their questionnaire answers. The report had to demonstrate to what extent the offer of a school food service met the priority components of the orientations related to healthy eating in the government’s Framework policy on healthy eating and active living, ‘Going the Healthy Route at School’(45). It also had to consider the government’s ‘Healthy Eating Vision for Creating Healthy Eating Environments’(43) and current scientific evidence when recommendations from the Framework policy were outdated. To create such report, RD from our research team drafted different sections of text for different possible combinations of questionnaire answers. Then, a statistician automated the generation of the report from the answers submitted by respondents, according to the text provided for the different scenarios by the RD. The resulting document had a length of up to eight pages. It contained the first section including a visual summary of the evaluated orientations classified as fully attained, partially attained and representing challenges. The second section detailed, for each orientation, possible improvements according to heathy eating policies and the current scientific evidence. Its text acknowledged schools’ key strengths and challenges, and recommended, when needed, the addition to the food offer of specific nutritious options (e.g. products containing fruits, vegetables and whole grains) and the replacement of specific foods that are less nutritious (e.g. products that are highly processed, fried and rich in added fat and sugar). We used colour codes in a summary page and throughout the report to help the reader locating useful information quickly.
Platform
The web-based platform of the SIS was first developed starting with the script shared by a Swedish team who previously developed a platform with research objectives similar to ours(Reference Patterson and Elinder52), then questionnaires were added. The SIS included a short intake questionnaire aimed at school principals. Once completed, the self-diagnostic questionnaire on the main food service’s lunchtime offer became available and the principal could assign it to the FSM. Each questionnaire included one web page per question as well as separate pages for the introduction, instructions, conclusion and final submission. Platform functionalities were adapted by the programmer to the needs of the project (e.g. automatic email distribution of invitations to log into the SIS including unique credentials for each school, questionnaire attribution to school stakeholders by the principal and generation of automated and personalised reports).
On-site validation of the web-based material
Recruitment
Between February and June 2018, we invited a convenience sample of 195 schools to participate. We aimed to recruit a minimum of thirty elementary schools and thirty secondary schools, public or private, of different sizes and from different regions. The eligibility criteria were to: (1) offer food at lunchtime and, during a school visit by a team member; (2) have the school principal available to complete the intake questionnaire and (3) have the FSM available to complete the self-diagnostic questionnaire on food offer at lunchtime (45-min meeting). Once a school principal and a FSM both agreed to participate, the research team scheduled a visit at the school.
Inter-rater reliability test of the self-diagnostic questionnaire
We needed to ascertain the reliability of FSM’s answers to the questionnaire through an on-site inter-rater reliability test, since most of them are not nutrition experts and since the questionnaire had been adapted to a web-based format.
On the day of a visit, we sent an email invitation to the school principal, including their access to the SIS and the intake questionnaire. Through the SIS, they could then assign the questionnaire on the food offer to their FSM, who automatically received their own email invitation. Before and during the school’s lunchtime, the RD from our research team visiting the school completed the questionnaire in its printed version by direct observation and by retrieving the required information from the food service’s menu and recipe book. She gathered the observational data used to complete the questionnaire by taking notes and photographing menus and recipes to allow later revisions of her answers if needed. After lunchtime, the FSM was invited to access and complete the questionnaire in the presence of the visiting RD. Figure 2 shows the workflow during the visit.
Main steps of school visits for the on-site validation of the web-based material. *RD, registered dietitian; **FSM, food service manager
The RD’s answers were double-entered in Excel worksheets to limit data entry errors and the FSM’s answers were exported from the SIS to Excel worksheets. We conducted analyses using R version 3.3.0. We combined the data collected during the inter-rater reliability test in elementary schools and secondary schools for all identical questions. Intra-class correlation coefficient (ICC) and Cohen’s κ were used to assess the inter-rater reliability between the answers of the FSM and those of the RD. Since some questions included numerous subquestions, we grouped data from closely related questions and subquestions to increase the number of observations, which increased the stability and the reliability of the results. Since there is no specific reliability test for multiple-answer questions, we dichotomised each answer choice into a yes/no choice, then combined and treated it as a nominal variable. We carried out an ICC coefficient (one-way random effects model)(Reference Koo and Li53) for one continuous variable pertaining to the protein content. The ICC was interpreted according to Koo and Li(Reference Koo and Li53) where values < 0·5 indicate poor reliability, values ≥ 0·5 and < 0·75 indicate moderate reliability, values ≥ 0·75 and < 0·9 indicate good reliability and values ≥ 0·9 indicate excellent reliability. A κ coefficient was calculated for the nominal variables, while the weighted version of this coefficient was used for the ordinal variables. The difference between those two coefficients is that for the weighted version, the penalty for different answers that are far one from another (e.g. high v. low) is higher than the penalty for different answers that are close (e.g. high v. medium). κ and weighted κ coefficients were both interpreted according to Fleiss(Reference Fleiss54) where values < 0·4 indicate poor reliability, values ≥ 0·4 and < 0·75 indicate fair to good reliability, and values ≥ 0·75 indicate excellent reliability.
Qualitative assessment of the web-based material
When visiting schools, during the completion of the intake questionnaire by a school principal and of the self-diagnostic questionnaire on food offer by a FSM, the visiting RD encouraged each respondent to share with her any question, comment or suggestion that came to their mind as they completed their questionnaire. However, she did not provide answers to limit bias induction. She recorded these interactions on paper along with her own observations, such as hesitations of difficulties identified in the respondent, and their completion duration. Immediately following the completion of a questionnaire, each respondent (school principals and FSM alike) was interviewed (less than 10 min) on their appreciation of the SIS’s functionalities, the navigation parameters and the simplicity of the questions.
Appreciation survey on the automated and personalised report
Since the validation of the self-diagnostic questionnaire was underway, we used the RD’s answers to generate the personalised reports for the schools participating in the validation study, thus ensuring accuracy of statements and recommendations. To assess the school stakeholders’ appreciation of the report, 2 weeks after they received it, we sent them an email invitation to complete a 2-min anonymous survey.
Results
Development and preliminary validation of the web-based material
For the concept evaluation, an initial list of ninety-eight indicators was submitted to the expert panel. Among those with an average relevance rating < 3, six indicators were maintained due to their clinical importance, but were reformulated, five were merged and the remainder were removed. A list of eighty-one indicators remained, with a CVI of 0·93. They averaged an item relevance score of 3·4 out of 4. In their qualitative comments, experts expressed concerns mostly regarding the comprehensiveness of the measured concept of the main school food service’s offer, which would entail high number of questions and completion duration. Therefore, we decided to evaluate the main food service’s offer at lunchtime only, since most of the food is offered during that period of the school day (Fig. 3). Other dimensions of the school food offer (e.g. vending machines) and other school day periods (e.g. breakfast and recess) would eventually be assessed by other self-diagnostic questionnaires in the future.
Process of circumscribing the concept of school food offer for the questionnaire development
For the questionnaire content validation, experts acknowledged the coherence between the indicators and the questions. They appreciated the completeness of the answer choices and the ease of responding. However, they were pre-occupied with the important number of questions. Since we targeted a maximal completion time of 45 min, we focused on four categories (Fig. 3). The questionnaire comprised a maximum of 26 questions, mostly answered by ‘yes’, ‘no’ or ‘I don’t know’, including 266 subquestions, mostly representing food listings. For example, if the respondent checked ‘Yes’ to the question ‘Do you serve a daily menu?’, the subsequent questions referred to the types of meat and alternatives, grain products, side vegetables, appetisers, desserts and beverages offered in the daily menu. From the indicators’ definitions, we created a glossary alongside the questionnaire to promote the common understanding of technical terms such as ‘à la carte menu’ and ‘side dishes’ and to increase the internal validity of the questions. Observations from ‘think-aloud’ interviews only led to minor vocabulary changes. Once added to the web-based platform, the questionnaire comprised thirty pages for elementary schools and thirty-four pages for secondary schools.
On-site validation of the web-based material
Of the 195 schools contacted, 39 were visited between April and June 2018 (Fig. 4). Reasons for refusing to participate were the involvement of the school administration in other research projects, an upcoming change of food service company, the relocation of the school, time constraints and the period of year being unsuitable to participate in research. FSM completed the questionnaire within an average of 31 ± 11 min in elementary schools and 37 ± 12 min in secondary schools.
Recruitment flowchart for the on-site validation of the web-based material
The average percentage of agreement between FSM and RD was 82 % ± 15 and the average of all κ coefficients was 0·64 ± 0·19; therefore, the overall strength of the agreement between the results of RD and FSM was considered fair to good(Reference Fleiss54). Except for one question, all strengths of agreement measured by κ coefficients were considered fair to excellent(Reference Fleiss54) (Table 1). The average strength of agreement was excellent for questions on the salad and sandwich bars and fair to good for the questions on the daily menu and the à la carte menu (Table 1).
Reliability measures of the questionnaire from the inter-rater reliability test between FSM and RD
FSM, food service managers; RD, registered dietitians.
* For secondary schools only.
† All subquestions were combined for each question.
‡ Mean of the subquestions for each of the question’s three blocks. The weighted version of the κ coefficient was used.
§ Multiple-answer questions were transformed and combined to obtain nominal variables.
Regarding the only quantitative question measuring the amount of source of protein served in the daily menu, measured by the ICC, the agreement was poor (ICC = 0 (−0·35, 0·34)). However, two of the thirty-two pairs of observations used to evaluate this question contained extreme values. When excluding those two pairs of observations, the ICC increased to 0·71 (0·47, 0·85), indicating a moderate agreement.
During the thirty-nine school visits, RD observed and interviewed twenty-seven school principals and thirty-nine FSM. Among them, twelve school principals completed the intake questionnaire simultaneously during a school board meeting. This procedure was suggested by the school board to facilitate the participation of principals but prevented to perform the ‘think-aloud’ assessment and the individual interviews. Notes from the observations mostly referred to difficulties experienced by FSM when answering specific questions. For example, 28 % of FSM had a hard time evaluating the protein content in their daily menu and differentiating à la carte items from their daily menu. RD noticed that when questioned about the last 5 d of service, several FSM excluded the current day, whereas the RD would include it when completing their paper-and-pencil version of the questionnaire on food offer. During the interviews, FSM recommended to simplify the instructions and to shorten the questions. They mentioned that the glossary was useful, but not easy enough to use. Overall, school principals and FSM commented that the navigation through the SIS was fluid (email invitations, access, questionnaire assignment, etc.). However, they suggested additions such as a progress bar at the top of the questionnaire and a button to save answers and exit the questionnaire. After the thirty-nine school visits, RD considered that the qualitative data from their observations and interviews had reached saturation, since no new information emerged from the last visits.
In the weeks following the reception of their automated and personalised report, twenty school stakeholders from the participating schools accepted to complete the appreciation survey, including twelve principals, four vice-principals, two school service managers, one cook and one coordinator. All respondents were generally either satisfied (70 %) or very satisfied (30 %) with their personalised report and 90 % of the respondents indicated that they would use their report if they came to make changes in their food offer. Their levels of agreement with positive statements regarding six main characteristics of the report were high (Fig. 5).
Levels of agreement with statements regarding the automated and personalised report among respondents of the appreciation survey (n 20). 
, Strongly disagree; 
, disagree; 
, indifferent; 
, agree; 
, strongly agree
Discussion
This paper presents the development and validation of a self-diagnostic questionnaire for schools on the main food service’s lunchtime offer, which was integrated into a web-based platform, the SIS.
This questionnaire is based on a comprehensive review of the literature on school food offer and of instruments measuring it. The qualitative and quantitative assessment of the concept of school food offer deemed necessary(Reference Streiner, Norman and Cairney39) to ensure that our questionnaire would allow exhaustive and precise measurements. This process showed that the selected indicators, the precursors of the questions, were accurate and comprehensive. Indeed, the CVI was over the ‘acceptable’ threshold of 0·8(Reference Waltz and Bausell47) for the final list of indicators. Validating the questionnaire’s content with a panel of experts was also essential, since questions were adapted to a new format (web-based) and to new respondents (FSM, rather than RD and nutrition technicians). Most of the experts’ qualitative feedback following the concept and questionnaire content validations referred to the exhaustiveness of the concept and to a concern about the questionnaire’s duration. To address these issues, we circumscribed the concept to the foods offered in the main school food service during lunchtime, which was effective, since most FSM who participated in our study completed their questionnaire within the 45-min limit we had planned.
We focused especially on the readability of the self-diagnostic questionnaire and the ease to navigate through the SIS, since future respondents will use the instrument without direct assistance and are not nutrition experts. Readability is an important psychometric component of a self-assessed instrument(Reference Calderón, Morales and Liu55); it influences the content validity of an instrument, as validity of answers is compromised if respondent comprehension is lacking(Reference Streiner, Norman and Cairney39). Respondents may understand the concept behind a question but may choose to skip it or give invalid answers if the question’s formulation is too complicated. Each question needs to be simple and straightforward. Therefore, we assessed readability during both principal steps of this study (the development and preliminary validation of the web-based material and the on-site validation of the web-based material). Our results from the interviews on the appreciation of the SIS’s functionalities and navigation parameters showed that overall, both the questionnaire’s content and the platform were globally appreciated by the respondents. However, though a minimal level of computer literacy will likely always be necessary to use the SIS, minor changes were needed in order to improve its user-friendliness(Reference Strabac and Aalberg56,Reference Shin, Johnson and Rao57) and were implemented by our team, such as changing the order of questions and response choices, rephrasing questions and adding photos as examples. We also requested the help of a user experience (UX) expert to ensure a smoother process for the SIS users and to design tutorials.
The inter-rater reliability results indicate an average percentage of agreement of 82 %, which shows that the questionnaire accurately measures the main food service’s lunchtime offer. However, the κ coefficients’ range is very extensive (0·0–1·0). Nathan et al. (Reference Nathan, Wolfenden and Morgan40) validated a computer-assisted telephone survey measuring the nutrition environment in elementary schools using direct observation as the criterion standard. They also found limited agreement for items assessing foods sold through canteens (κ range −0·6–0·81). In our study, the κ analysis highlighted the difficulties that were observed during the data collection. For example, in the food section of the daily menu, we found a probable systematic error in the inter-rater agreement on supply during the last 5 d of operation. Observations notes showed that FSM often excluded the current day from their evaluation, whereas RD included it. This therefore decreased the percentage of agreement without informing the validity of the questions. Lower levels of inter-rater agreement observed for the section on the à la carte and daily menus can also be explained by the great variety of food items found these types of menus, especially in secondary schools. We found that FSM often forgot products as they were relying on their memory rather than checking menus and food displays as the RD did. To address this issue, following this validation study, we added clearer instructions on how to complete the questionnaire and included frequently forgotten food items among the examples cited for each food category. Moreover, we included a new feature allowing respondents to print on paper more complex questions, making it easier for them to make in-person verifications in the food service.
We found that the agreement for the continuous variable pertaining to the total amount of meat and alternatives served in one portion of the daily menu was poor (ICC: 0 (−0·35, 0·34)). FSM could reliably identify sources of protein in their menu but had difficulty estimating the amounts served in the plates. According to Ward et al. (Reference Ward, Mazzucca and McWilliams58), the low reliability estimates may reflect inherent difficulties in measuring them, rather than inconsistent or misreporting by respondents. Therefore, we decided to remove this question from the validated and final version of the questionnaire. Finally, the weighted κ was strongly penalised by occurrences where one of the two evaluators indicated a high frequency, whereas the other indicated a low frequency. In practice, we noticed that RD often identified a greater number of elements in the main and side dishes (e.g. they included the second option for the main dish and the bread rolls), which reduced κ coefficients. Thus, we added frequently observed examples for each food and beverage item in the validated questionnaire to help FSM remember frequently forgotten items.
Results from our appreciation survey seem to confirm the relevance of providing an automated and personalised report to school stakeholders. The report attempts to bridge the evidence-to-action gap that exists by proposing evidence-based orientations that might improve the school food offer(Reference Dearing and Kreuter59). It allows school stakeholders to compare the current state of their food offer to an ideal situation, while contributing to the improvement of the knowledge and skills of FSM(Reference Bosc-Miné60). As mentioned by the Swedish team(Reference Patterson, Quetel and Lilja41), an automatic tailored feedback like our personalised report goes beyond the traditional approaches to knowledge transfer since it focuses on what can and should be done in a specific school to initiate a dialogue between its administration and its FSM, thus promoting a healthier food offer. Moreover, the automated and personalised report also recognises strengths in the main food service’s offer, thus providing positive reinforcement. A similar, yet less detailed report (called ‘Score Card’) is currently offered by the Centers for Disease Control to US schools using the School Health Index(61). The report provided by the SIS could also assist public health stakeholders (e.g. dietitians in regional public health departments) who support and guide schools in their efforts to create healthy environments, in line with government guidelines and best practices.
Methodological limitations and strengths
It is important to mention some limitations in this validation study to interpret the results appropriately. First, the small number of schools visited limits the scope of our results. This was mostly due to school administrations’ lack of time to participate in our study. Indeed, school visits took place from April to June, which is a very busy period due to the end of the school year. Moreover, in addition to recruiting principals, we needed the participation of FSM to go forward with the inter-rater reliability test. Most of them were employed by private companies rather than directly by schools, which made recruitment more challenging. For these companies, the SIS could be a useful tool to support the continuous improvement of their offer, but its use could also be perceived as risky if it allows a comparison of their offer with that of competitors. This might have contributed to a smaller sample size than desired. However, we still reached saturation with our qualitative data (on-site observations and interviews) with our sample, which suggests that we would not have made other observations of major importance with additional schools. Moreover, by combining data from elementary and secondary schools for all identical questions and grouping data from closely related questions and subquestions, the number of observations for each question was sufficient to conduct statistical tests.
Three RD from the research team visited different schools to compare their answers with those of FSM to calculate the inter-rater reliability, which may have caused some variability in the quantitative data that was obtained. To minimise the possible bias introduced by the different interpretations of the three RD, they agreed on completion guidelines during a practice session. They also had regular meetings where they could discuss unexpected situations and agree on how to deal with them.
Finally, the effects of some biases must also be considered. Even though the questions were written in a neutral, non-judging way as much as possible, the social desirability bias was expected to have some influence on the answers of school service managers. The RD mitigated as much as possible such effect by discussing with respondents the purpose of the study: evaluating the quality of our platform and questionnaire, rather than evaluating their current food offer.
Several strengths of our project should also be highlighted. The validation of the concept was carried out by a panel of nine experts, whereas according to Polit et al. (Reference Polit, Beck and Owen46), the first iteration of expert content validation would ideally involve between 8 and12 experts. During visits, the RD were confronted with a wide variety of school contexts, and their findings were rich in lessons for the future of the project. The inter-rater reliability test used observational data gathered and interpreted by RD, which are competent professionals heavily trained for nutritional assessments. Moreover, RD had the opportunity to openly ask the respondents about the observations they made throughout their visit, which greatly helped with the interpretation of quantitative data and with the planning of future improvements to the platform and questionnaire. For example, the questionnaire will be used as a basis for developing a shorter tool with fewer questions targeting indicators of what we want to measure and monitor provincially.
In conclusion, our team successfully developed a user-friendly and reliable web-based platform that collects school-level data and achieves knowledge mobilisation with school stakeholders. The questionnaire we implemented on the SIS offers a new opportunity for the assessment and surveillance of school food offer, which can have a significant influence on children’s dietary choices and intakes. To our knowledge, the SIS is the first instrument of its kind developed specifically for the context of the province of Quebec’s educational system. Ultimately, this platform could further enhance our research and evaluation capacity in school environments by facilitating the collection of school-level longitudinal data on various domains, thus providing scientifically validated information to decision-makers.
Acknowledgements
Acknowledgements: We would like to thank the following individuals and organisations for their contributions to this study. Emma Patterson, PhD, shared the script of the SkolmatSverige (School Food Sweden) platform with our team, which helped to accelerate the programming of our own Web-based platform. Yan Kestens, PhD, contributed to the design of the study. Polygon Research (previously Treksoft Solutions Inc.) contributed to the design and programming of the web-based platform and questionnaire. The Ministère de l’Éducation et de l’Enseignement supérieur du Québec (Quebec Ministry of Education) supported this study and facilitated our communication with school stakeholders. Sonia Pomerleau, RD, revised the script of the automated and personalised report. Michelle Kearney, RD, contributed to the school recruitment and visits. Special thanks to the panel of experts for contributing to the validations of the questionnaire’s concept and content, and to the school stakeholders and food service managers for participating in this study. Financial support: This work was supported by the Evaluation Platform on Obesity Prevention and the Quebec Heart and Lung Institute (EPOP-QHLI) through a development grant from the Fondation Lucie et André Chagnon; the Government of Quebec (GQ, Inter-ministerial Action Plan 2017–2021 of the Government Policy on Health Prevention, measure 1.4, action 2); and the Quebec Research Funds-Health (FRQS, ref.#31022). EPOP-QHLI, GQ and FRQS had no role in the design, analysis or writing of this article. Conflict of interest: There are no conflicts of interest. Authorship: P.M., A.B. and A.L. formulated the research questions. P.M., A.B., P.G. and A.L. designed the study. P.M., A.B. and M.L. carried out the study. P.M., A.B., M.L., P.G. and A.L. participated in data analysis and in the drafting and critical revision of this article. Ethics of human subject participation: The Research Ethics Board of the Université de Sherbrooke stated (2017-05-04) that school stakeholders would be involved in this research as part of the normal course of their work and therefore would not be participants in the spirit of Article 2.1 of the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (Reference Rao, Kropac and Do1). Therefore, ethics approval was not required.
