Background
Lifestyle programs to improve diet and physical activity behaviors are increasingly being offered in team-based primary care (PC) in Canada, largely related to efforts to curb the obesity epidemic, prevent type 2 diabetes, and decrease cardiovascular risk. Such programs are costly to organize and deliver, are currently taking many forms, and vary in length, format, and content. Anecdotally, such programs often suffer from high attrition rates, although this is not well documented in the healthcare system (Miller and Brennan, Reference Miller and Brennan2015). What is known is that attrition in weight loss programs varies from 10 to 80% and is associated with poor long-term adherence to lifestyle change (Moroshko et al., Reference Moroshko, Brennan and O’Brien2011; Desroches et al., Reference Desroches, Lapointe, Ratte, Gravel, Legare and Turcotte2013). Research to date has not been able to identify optimal formats and organization of programs, although it is generally thought that longer (weeks to months), more behaviorally oriented programs are more successful (Lin et al., Reference Lin, O’Connor, Evans, Senger, Rowland and Groom2014). Lifestyle change is inherently challenging for most people, and it would be expected that the organization of program delivery may have an impact. Assessment of the patient experience at the program level could add valuable information to inform quality improvement efforts over time. Older client satisfaction questionnaires have generally been inadequate for quality improvement purposes, as they have focused mainly on interpersonal skills of one provider and/or in the context of one disease (Salisbury et al., Reference Salisbury, Wallace and Montgomery2010). Choice options or scaling have also often been inadequate to detect more minor problems, with patients being mainly satisfied with their experiences.
Lifestyle services are only one possible service within PC and it would be best if any new questionnaire could be positioned within overall development of indicators of the patient experience in PC more generally. In Canada, the Canadian Institute for Health Information (CIHI) undertook a program of work to develop a comprehensive national framework for assessing the quality of PC services (CIHI, 2006; 2012). Some indicators are best assessed by patients and considerable work was done to develop draft questionnaires relevant to the patient experience (CIHI, 2013). Wong and Haggerty reviewed previous publicly available questionnaires and identified 22 dimensions and sub-dimensions of PC services (Wong and Haggerty, Reference Wong and Haggerty2013). This approach was taken as these dimensions represented a ‘reduced’ list of commonly agreed upon aspects of PC (Haggerty et al., Reference Haggerty, Burge, Levesque, Gass, Pineault, Beaulieu and Santor2007).
Building on this work, we used the Wong and Haggerty framework to develop a set of questions addressing patient experience of lifestyle programs, which could be used on their own or with other questions developed by CIHI or others. The intent was to develop a series of questions for self-report that could be adapted to different PC practice contexts, were in line with previous work on PC quality indicators, aligned with key concerns for program improvement, and were broadly aimed at both diet and exercise. Core principles of survey design were addressed (Aday and Cornelius, Reference Aday and Cornelius2006), and we report here initial development and pilot testing of the questions based on the input of the providers delivering the programs as to the most important dimensions to assess.
The specific objectives were to: (i) develop a self-report tool to assess patients’ experiences with lifestyle services (nutrition and physical activity) offered in interdisciplinary PC settings and (ii) pilot test the tool as part of an evaluation of a lifestyle intervention study to reverse metabolic syndrome.
Methods
The development of the questionnaire consisted of three phases: determination of the high-priority concepts/variables to include, generating the first draft of the questionnaire and modifying it based on feedback, and pilot testing the modified questionnaire in a PC nutrition and physical activity intervention program. Ethics approval for the first two phases was obtained from the University of Guelph Research Ethics Board and for the third phase from the Research Ethics Boards of the pilot testing sites.
Phase 1: Determining questionnaire topics
Study design
The first task was to reduce and prioritize which variables should be considered in a self-administered questionnaire. A group-administered cross-sectional questionnaire study design was chosen, based on prior experience of the strengths and limitations of inter-professional focus groups in eliciting diverse perspectives on lifestyle services in PC (Bonilla et al., Reference Bonilla, Brauer, Royall, Keller, Hanning and Dicenso2016; Bourgeois et al., Reference Bourgeois, Brauer, Simpson, Kim and Haines2016), the complexity of the framework which required some explanation (see Figure 1), and the need to document rankings.
Dimensions of patients’ experiences in primary healthcare (Wong and Haggerty, Reference Wong and Haggerty2013). Rectangle boxes represent dimensions and spheres in the same color represent sub-dimensions. Black box indicates top-ranked dimensions, and check marks indicate next ranked dimensions.
Development of group-administered questionnaire
The plan for the group session was developed with a professional moderator who recommended use of a graphic to support review of the dimensions of the framework (see Figure 1), followed by general group discussion and review, before individual completion of a ranking exercise using a paper-based questionnaire.
Population/sampling
As the questionnaire would be mainly used by PC providers offering health education/coaching/counseling services for diet and physical activity, it was considered relevant to solicit the views of providers on the most critical dimensions to include. Family Health Teams (FHTs) were chosen to recruit providers as they include a variety of healthcare providers in addition to family physicians, and many offer lifestyle group programs [Ontario Ministry of Health and Long Term Care (MOHLTC), 2016]. All FHTs within a 2-h travel radius of the University of Guelph who had allowed the MOHLTC to post their contact information were invited by email to participate.
Data collection
Participants provided written informed consent before beginning the group session, which was conducted over the lunch period. Each session was moderated by the professional moderator, who guided participants through a discussion of the Wong and Haggerty dimensions and sub-dimensions (Wong and Haggerty, Reference Wong and Haggerty2013) and elicited participants’ ideas of aspects most important to include in a patient experience questionnaire on lifestyle services. Participants then individually rated the importance of each of the dimensions/sub-dimensions, and ranked their top five dimensions/sub-dimensions. Each session lasted about 60 min; sessions were audio-recorded and a note-taker took notes on the main concepts discussed. After analysis, results of ranking were sent back to participants by email as a form of member checking.
Analysis
Using the ranking exercise responses, each dimension/sub-dimension was given a total importance score to inform prioritization, with higher scores indicating higher prioritization for inclusion in the questionnaire. Dimensions/sub-dimensions were then given a priority rank by discussion group score and an overall priority rank by total score. We further analyzed the discussion group audiofiles using directed qualitative content analysis to identify themes that added context to ranking results (Hsieh and Shannon, Reference Hsieh and Shannon2005).
Phase 2: Creating draft for pilot testing
A sequential process of question development and revision was undertaken. Possible questions were adapted from previous questionnaires. New items were generated if suitable questions were not found. Only one to two questions were chosen to address each concept.
A convenience sample of participants in lifestyle programs reviewed the questions using cognitive interviewing. Participants were recruited from a group of 25 patients attending a lifestyle program at one FHT. Of the 25 patients, 18 expressed interest in participating in the interview and 11 were available for cognitive interviewing. A sample size of 10 is generally sufficient to identify major issues (Blair and Conrad, Reference Blair and Conrad2011). Interviews were scheduled for 60 min and digitally audio-recorded. The interviews were conducted using a combination of ‘think aloud’ techniques with scripted, spontaneous, concurrent, and retrospective probing (Willis, Reference Willis1999; Beatty and Willis, Reference Beatty and Willis2007). Participant comments and problems identified were aggregated by question and grouped by problem type using Willis’ coding classifications (Buers et al., Reference Buers, Triemstra, Bloemendal, Zwijnenberg, Hendriks and Delnoij2013). Revisions were made to the questionnaire after each set of five or six interviews. After revision, the 23 questions were ready for pilot testing (Table 1).
Concepts considered for the first questionnaire draft, by Wong and Haggerty dimension/sub-dimension
Phase 3: Pilot testing questionnaire in PC
A 12-month nutrition and physical activity before-after feasibility study for metabolic syndrome (n=293) conducted in three diverse PC sites across Canada, provided an ideal opportunity to pilot test the questionnaire (Jeejeebhoy et al., Reference Jeejeebhoy, Dhaliwal, Heyland, Leung, Day, Brauer, Royall, Tremblay, Mutch, Pliamm, Rhéaume and Klein2017). The objective of this phase was to assess multi-modal delivery feasibility, clarity, missing/redundant concepts, and floor/ceiling effects.
Adaptation of the generic questionnaire consisted of writing an introduction and having the research group (including several family physicians) review the questionnaire. No further changes were made to the questions. The questionnaire was translated into both French and Russian to meet the language needs of the participating PC sites. Participants could complete either a paper-based version or an online version (French and English) available using Qualtrics online survey software (qualtrics.com 2015 Provo, Utah). The Russian version was offered only in paper format. Questionnaire data collection was managed by research coordinators (RCs) from each of the PC sites. Each RC received the questionnaires in the languages appropriate for their site along with pre-stamped return envelopes. The paper questionnaire included a web address to the online questionnaire so that participants could complete the questionnaire in either paper or online modalities. RCs were also provided with three different reminder modes: paper postcards, a telephone script template, and an email template. RCs were instructed to remind patients to complete the questionnaire using one of the three modes, 1 to 2 weeks after the questionnaire was mailed out to the participants but this process was not monitored.
Participants were asked to complete the questionnaire after completion of the 12-month study or at the point where they had decided to withdraw from the study. There was no financial remuneration offered for completing the questionnaire. For the analysis, all paper questionnaires were transcribed into Qualtrics and all data was then downloaded and exported into Microsoft Excel (2010) for analysis. Analysis included questionnaire response rates and individual question response rate. Comments written on the questionnaires were thematically analyzed to identify issues of question redundancy or clarity.
Results
Provider ranking and discussion of dimensions/sub-dimensions
A total of 38 providers attended a discussion at one of five FHTs. These included one rural FHT, three urban non-academic FHTs, and one urban academic FHT. Most participants identified as female (92%), the average age was 40, and there were multiple provider categories in attendance including nurses/nurse practitioners (n=10), dietitians (n=7), mental health counselors (n=5), administrative staff (n=7; eg, executive directors, reception staff), and other (n=9; not disclosed but included other health professionals not listed). There were no family physicians in the discussion groups despite requests to recruit physicians as part of the mixed group of providers. On average, participants had been in their profession for 14 years and had worked in their FHT for 3.8 years. The top five provider ranked dimensions/sub-dimensions were: ‘trust,’ ‘general communication,’ ‘first-contact accessibility,’ ‘whole-person care,’ and ‘respectfulness’ (Figure 1 also shows the next five ranked dimensions/sub-dimensions). This ranking did not change when the number of top five votes was tallied for each dimension and sub-dimension. Member checking confirmed the top five ranked dimensions.
As noted in Table 1, column 3, providers understood the concepts from the framework well, but would have combined some categories, added a category or felt some concepts overlapped. Within comprehensiveness of services, some participants did not see the need to separate out the two sub-dimensions, ‘services provided’ and ‘health promotion and primary prevention’:
I would see services provided and health – I would see both of these – I’m having a hard time separating them. I’m seeing them as one… so the services I’m giving are more health promotion or preventing further complications down the line rather than kind of that immediate tertiary care…
(DG2 Participant)
The concept of equity was not obvious in the framework and was identified in one group as an important aspect to capture:
What about equity? So all patients whether they have a home or are homeless or handicapped have equitable service, the same as your typical patient. Equity you know whether it be a child or have a horse and buggy…how can we meet those needs.
(DG4 Participant)
For some participants, there were dimensions of the framework that overlapped substantially such as access and comprehensiveness of services, trust with all other dimensions, whole-person care with team functioning, and accommodation with patient safety.
So you could, you could attribute trust to every single – It ties in globally… I think trust ties into every single of these categories. …Every single one.
(DG1 Participant)
Certain dimensions were not highly ranked for a patient questionnaire, but were discussed by provider groups in detail. For example, ‘accommodation’ issues came up in each group but the specific issues differed across organizations. ‘Team functioning’ was a sub-dimension that many felt would be difficult for patients to comment on, as sometimes patients are not aware of who is part of the team and that a well functioning team might not be ‘noticed.’ Several groups discussed the need to find out if services provided were meeting patients’ needs, however thought it would be difficult for patients to know and compare different possible services to know what might meet their needs better. Comments from member checking confirmed the final ranking, with recognition that all dimensions are important to quality care.
Developing questionnaire
We used the ranking and discussion concepts that emerged from the review of transcripts as a starting place for selecting and adapting questions from previous work, especially analyses done in the development of the CIHI patient experience questionnaire (Table 1) (Haggerty et al., Reference Haggerty, Bouharaoui and Antor2011a; Reference Haggerty, Burge, Beaulieu, Pineault, Beaulieu, Levesque, Santor, Gass and Lawson2011b). Response options for Likert questions were similar to past questions. For example, within the sub-dimension of ‘first-contact accessibility,’ providers mentioned that patients may have a poor awareness of the available lifestyle services and might not find out about them for many years. ‘Services provided’ had not been ranked highly, but given that this would be foundational to patient experience of lifestyle services, core questions were added. A question on attrition was also added, given the evidence of high attrition in lifestyle programs (Moroshko et al., Reference Moroshko, Brennan and O’Brien2011).
Participants for cognitive interviewing were 11 adults (10 women, 1 man) who ranged in age from 32 to 78 with an average of 57 years. After two rounds of revision, the questions for pilot testing consisted of 23 questions: 21 multiple choice, with five that included space to add extra comments and two open-ended questions (Table 1 – Questions used for pilot testing).
Pilot testing
A total of 164 questionnaires were received (57 French, 107 English, and 0 Russian) of 293 potential recipients (56% response rate); 76% as paper questionnaires and 24% online responses. RCs used differing approaches; some did mail the questionnaires, while others completed the questions with participants as an interviewer-administered questionnaire. None of the Russian questionnaires were completed as the RC at this site translated and completed English questionnaires with participants. Question responses for closed-ended questions (Q1–Q21) ranged from 93% to 100%; responses to the two open-ended questions were 84% (Q22) and 68% (Q23). The <100% response rate for the first few questions may have been because participants missed these questions due to double-sided copying of the paper questionnaire. Comments reflecting a misunderstanding of clarity or meaning of questions were received for Q6, 7, 9, 11, 15 and 20 and revised wording is highlighted in Table 1. With respect to possible floor and ceiling effects, several questions had four or five ordinal response options, (such as Never to Very Often) where extreme values were rarely used, but only Q12 was changed at this point in development, as 100% of respondents felt they were always treated with courtesy and respect. Response options for this question were changed to two options: ‘always’/‘not always.’ Two questions were deleted or combined with another question as the responses and comments suggested redundancy (Q16 and 21). The revised questionnaire reflecting these comments consists of 20 multiple choice questions and one open-ended question which can be completed in about 5 min (Supplementary Material File). The actual results of the pilot test will be reported elsewhere along with results of a focus group study.
Discussion
This initial work to develop and pilot test a new self-administered patient experience questionnaire for lifestyle programs in team-based PC was successful in generating a set of questions based on the perceptions and insights of providers and patients as to the critical issues to be assessed for quality improvement purposes in PC. It is intended to supplement other data to assess the overall effectiveness of such programs. Lifestyle programs are increasingly being offered for diabetes prevention, cardiovascular disease prevention and/or weight management within PC. The work builds on the strong framework and key indicators already developed by CIHI over the past several years (CIHI, 2013). Thus, it is hoped these questions could be used in conjunction with other questions and methods from the framework for assessing PC services.
Review and prioritization by providers suggested some overlap in concepts from the providers’ perspectives, but there was general consensus that all concepts in the Wong and Haggerty framework were important. Their framework cataloged questions developed by previous groups on the patient experience of PC (Wong and Haggerty, Reference Wong and Haggerty2013). As the questionnaire was intended from the beginning to be used in routine practice, issues of feasibility, relevance, and acceptability to providers and patients were paramount in initial development. A self-administered questionnaire would be most feasible, but it was clear from the outset that it would be impossible to address all aspects of the framework in such a questionnaire. After discussion, the same five dimensions/sub-dimensions emerged from all discussion groups. These were: trust, first-contact accessibility, and three under interpersonal communication – general communication, whole-person care, and respectfulness.
Accessibility and interpersonal communication have been enduring foci in previous questionnaires and our results confirm their importance based on the large number of questions included in these concepts. The pilot testing revealed ceiling effects for respectfulness, with 100% reporting they had been treated with respect. Trust stands out as it has been contentious in some studies (Brauer et al., Reference Buckland, Bashir, Moore and Straus2012) and less commonly measured in older general PC questionnaires. Safran et al. (Reference Safran, Kosinski, Tarlov, Rogers, Taira, Lieberman and Ware1998) were among the first to include trust as a dimension in the Primary Care Assessment Survey (Safran et al., Reference Safran, Kosinski, Tarlov, Rogers, Taira, Lieberman and Ware1998), related to trust in the physician (see Hall et al., Reference Hall, Zheng, Dugan, Camacho, Kidd, Mishra and Balkrishnan2002 for review of the older literature) (Hall et al., Reference Hall, Zheng, Dugan, Camacho, Kidd, Mishra and Balkrishnan2002). There is renewed interest in improved measurement of trust as it has been suggested that trust is associated with intervention effectiveness (Ozawa and Sripad, Reference Ozawa and Sripad2013). Given that trust is a multi-dimensional concept that has been mainly directed to trust in physicians, additional psychometric studies will be needed to develop a brief trust scale targeted to lifestyle programs offered by teams (Muller et al., Reference Muller, Zill, Dirmaier, Harter and Scholl2014). Only two aspects of trust were included in the draft questionnaire; trust in the provider and trust in the information.
The discussion with providers revealed a nuanced understanding of the strengths and limitation of patients’ insights on their experience. For example, it was noted that patients may not be able to comment on the appropriateness of offered lifestyle services for their health and personal situation. This is particularly relevant in lifestyle services as programs can vary widely in their focus. For example, the area of obesity management remains highly controversial within healthcare and in the general community ranging from acceptance of ‘health at every size’ to a strong weight loss focus (Tylka et al., Reference Tylka, Annunziato, Burgard, Danielsdottir, Shuman, Davis and Calogero2014). It is beyond the ability of the patient experience questionnaire to assess the appropriateness of lifestyle services offered, which is better assessed in other ways (eg, clinical outcome measures).
Patient experience questionnaires for lifestyle services are frequently used in PC practice, but rarely have undergone rigorous development and validation. A strength of our approach was to build on a framework for main dimensions assessed by previous researchers, while incorporating the priorities of the front-line providers involved in delivering services in PC. Established methods for initial assessment were used, including cognitive interviewing with participants from the target population and pilot testing in both English and French.
Due to feasibility issues, the providers who participated were all working in one model of team-based PC in southern Ontario and it is not known if priorities would differ in other models or locations. Pilot testing in three diverse PC settings across Canada as part of a study of lifestyle treatment of metabolic syndrome (CHANGE program) helped to address generalizability of the tool (Jeejeebhoy et al., Reference Jeejeebhoy, Dhaliwal, Heyland, Leung, Day, Brauer, Royall, Tremblay, Mutch, Pliamm, Rhéaume and Klein2017). Physicians as a group may or may not have somewhat different priorities than other providers with respect to the most important dimensions of lifestyle programs, although the dimensions of the Wong and Haggerty framework were identified from review of questionnaires directed mainly to physician practice (Wong and Haggerty, Reference Wong and Haggerty2013). Lack of physician input in the initial discussion was partially mitigated in the pilot testing phase, as several family physicians on the research team reviewed the questionnaire and are co-authors on another paper of patient experience of the CHANGE program (Klein et al., Reference Klein, Brauer, Royall, Israeloff-Smith, Klein, Tramblay, Dhaliwal, Rheaume, Mutch and Jeejeebhoy2017).
Some might also argue that patients should have been involved in the initial priority setting exercise. As the terminology for describing the dimensions would be unfamiliar to most patients, meaningful patient engagement on setting priorities was beyond the scope of the current project. A different and targeted methodology to obtain meaningful input would be needed, based on our experience with patient engagement strategies for new obesity services (Royall et al., Reference Royall, Brauer, Atta-Konadu, Dwyer, Edwards, Hussey and Kates2017) and in practice guideline development (Buckland et al., Reference Brauer, Sergeant, Davidson, Goy and Dietrich2017). Instead, we sought patient involvement in review of the actual questionnaire. Interestingly, no comments advocating for new concepts emerged in that process. As with physician views, somewhat different priorities may emerge in future studies comparing patient and provider priorities.
Pilot testing suggested that multi-modal availability (paper and online versions) is needed to ensure the majority of participants have an opportunity to comment on their experiences, especially among middle-aged and older groups. While a self-report questionnaire was planned, in some cases the RC completed the questionnaire in-person with participants at a clinic visit. This may have undermined assessment of the clarity of the questions and for missing/redundant concepts, as RCs could have explained the concepts. This could also have led to social desirability bias and greater floor/ceiling effects, as participants may have felt constrained in expressing any problems with the program. Thus, only one question was revised due to ceiling effects.
Conclusions/next steps
A set of questions were drafted and pilot tested on the patient experience of lifestyle services in team-based PC organizations. These questions are intended to be self-administered and to identify key aspects of programs that may require improvement. Local providers from one model of team-based PC prioritized the dimensions to be assessed. Accessibility and sub-dimensions of interpersonal communication were highly ranked and their inclusion was consistent with previous patient satisfaction surveys. Trust was also highly ranked, but has been less commonly assessed in previous questionnaires (Wong and Haggerty, Reference Wong and Haggerty2013). Given the multi-dimensional nature of trust, further development of questions to assess trust within lifestyle programs is needed. Pilot testing in French and English suggested additional minor wording changes. Further development and validation of the questionnaire is needed, including assessment of test-retest reliability, item scaling and convergent and divergent validity. These questions provide a starting point for further studies of the patient experience of lifestyle programs.
Acknowledgments
The authors gratefully acknowledge the support of the Association of Family Health Teams of Ontario in advertising the availability of the study opportunity, to Margaret Alfieri, Tracy Hussey, and Linda Dietrich from Dietitians of Canada for ongoing conceptual and practical support of this program of work, and to colleagues with Metabolic Syndrome Canada for review of the questions and for access to patients for pilot testing.
Financial Support
Anneli Kaethler received financial support for her MSc thesis from the Department of Family Relations and Applied Nutrition, University of Guelph and from Metabolic Syndrome Canada for some of her time.
Conflicts of Interest
None.
Ethical Standards
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional guidelines on human experimentation from the University of Guelph and with the Helsinki Declaration of 1975, as revised in 2008.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/S1463423617000937
