Background
Strong primary care systems are associated with better patient outcomes, particularly for those with chronic conditions (Hansen et al., Reference Hansen, Groenewegen, Boerma and Kringos2015). Yet around the globe, primary care clinicians report challenges in coordinating care and delivering care to the most complex and vulnerable patients (Osborn et al., Reference Osborn, Moulds, Schneider, Doty, Squires and Sarnak2015). This points to the need for targeted efforts to innovate in the delivery of primary health care with a particular focus on strategies to effectively reach the most vulnerable patients.
Research in primary care plays a critical role in informing, evaluating, and helping improve the delivery and organization of health care services (Hutchison et al., Reference Hutchison, Levesque, Strumpf and Coyle2011; Hutchison and Glazier, Reference Hutchison and Glazier2013). The diversity of primary care activity within and across jurisdictions (eg, provinces, states, countries) presents an opportunity for learning about which innovations could be spread and scaled-up to provide better service to vulnerable populations and patients with chronic diseases. By comparing changes in primary care delivery across geographies and provincial/territorial boundaries, researchers, together with clinicians, decision-makers, and patients, can produce knowledge about what is needed to spread promising innovations across regions and which are reproducible on a larger scale.
A pan-Canadian mechanism for common measurement of the impact of primary care innovations across Canada is lacking. Currently, pan-Canadian measurement and reporting has been more focused on the acute care sector. For example, a report released on the performance of primary care in Canada in 2016 reported on only 16 of the 51 indicators that policy makers and clinicians had deemed important in 2012 [Canadian Institute for Health Information (CIHI), 2016]. This suggests that little progress has been made toward collecting and reporting primary care indicators; further emphasizing the need for developing common and shared data sources in primary care. More recently Canadian data using the Quality and Costs of Primary Care (QUALICO-PC) indicators have been reported from Canada (Wong et al., Reference Wong, Chau, Hogg, Teare, Miedema, Breton, Aubrey-Bassler, Katz, Burge, Boivin, Cooke, Francoeur and Wodchis2015; Breton et al., Reference Miedema, Easley, Thompson, Boivin, Aubrey-Bassler, Katz, Hogg, Breton, Francoeur, Wong and Wodchis2016; Anisimowicz et al., Reference Anisimowicz, Bowes, Thompson, Miedema, Hogg, Wong, Katz, Burge, Aubrey-Bassler, Yellanc and Wochis2017; Katz et al., Reference Katz, Herpai, Smith, Aubrey-Bassler, Breton, Boivin, Hogg, Miedema, Pang, Wodchis and Wong2017).
In order to speed up progress in collecting data using the same measures and data sources across research projects and facilitating knowledge exchange across jurisdictions, the Canadian Institutes of Health Research (CIHR, 2013) and its provincial funding partners required cross-jurisdictional collaboration on indicator measurement and evaluation as part of a signature research initiative on community-based primary health care (CBPHC). The first of its kind in funding structure, the initiative invested $33.4 million in funding to the 12 multi-jurisdictional five-year team grants. These 12 teams are undertaking programmatic, cross-jurisdictional, and interdisciplinary research to develop, implement, evaluate, and compare innovations in primary care. In order to facilitate collaboration across the 12 teams, a condition of this funding was that a portion of the grant money could only be spent on collaboration and cross-team data collection and analysis of common indicators. We report here on the process by which the 12 teams worked together and fostered cross-jurisdictional collaboration, including collection of common indicators with the goal of using the same measures and data sources.
Methods
Initial conditions of funding
Twelve CBPHC team grants were selected by a rigorous international peer review process and funded by CIHR (2012) and partners in 2013. The signature initiative’s key research areas included: improving access for vulnerable populations and innovations in chronic disease management and prevention. To encourage building capacity and scaling-up successful innovations, each team was required to include at least two decision-makers from different jurisdictions, at least one health professional, and a community-engagement process to provide feedback on the proposed care delivery model (CIHR, 2012).
Each team was required to allocate $50 000 CAD/year over the course of the grant (total per team=$250 000) for reviewing, agreeing, and operationalizing common measurement and assessment strategies. Although cross-collaboration was among the requirements of receiving the funding, there were no a priori decisions on how the teams would work together.
Working group formation
A working group was established with representatives from each of the 12 teams as well as the funder during the first year of funding. The primary objective of this group was to develop a common set of indicators to describe the knowledge generated by these 12 cross-jurisdictional teams and to develop a process to work across the teams (Riddell et al., Reference Riddell, Edwards, Thompson, Bernabe-Ortiz, Praveen, Johnson, Kengne, Liu, McCready, Ng, Nieuwlaat, Ovbiagele, Owolabi, Peiris, Thrift, Tobe and Yusoff2017). The group met through a series of teleconferences and face-to-face meetings, and was chaired by one of the 12 teams’ principal investigators.
Overarching research questions
The working group agreed upon collecting indicators within several core dimensions of primary care with the intention of addressing three objectives across the 12 teams:
∙ Describe the impact of improving access to CBPHC, particularly for vulnerable populations;
∙ Examine the impact of alternative models of chronic disease prevention and management in CBPHC on patient and system outcomes; and
∙ Describe the structures and context that influence the implementation, delivery, cost, and potential for scale-up of CBPHC innovations.
Procedures for consensus on indicators and measures
The working group undertook an iterative process of discussion, consideration of existing primary care measures from the literature (Starfield, Reference Starfield2001; Hogg et al., Reference Hogg, Donskov, Russell, Liddy, Johnston and Mayo-Bruinsma2008; Watson et al., Reference Watson, Broemeling and Wong2009) and stakeholder engagement. We incorporated the CIHI (2012) pan-Canadian primary health care indicators and previous work developed in the Canadian context from measurement experts in primary care (Fortin et al., Reference Fortin, Stewart, Poitras, Almirall and Maddocks2012; Haggerty et al., Reference Haggerty, Roberge, Freeman, Beaulieu and Breton2012; Wong et al., Reference Wong, Chau, Hogg, Teare, Miedema, Breton, Aubrey-Bassler, Katz, Burge, Boivin, Cooke, Francoeur and Wodchis2015). Teams were expected to incorporate the agreed upon common measures into their respective programs and to report on them.
In choosing measures and indicators, specific efforts focused on refining them based on feedback from teams’ First Nation and Inuit partners. Team members were asked to reach common ground about the priority dimensions to be included in data collection, where possible, across all of the 12 teams.
Data collection on their use of the common indicators
Teams were surveyed using self-administered questionnaires on their planned research programs, including study design, methods of data collection, data sources, and study populations, as well as their coverage of the common indicators, and whether they were collecting the exact or modified versions of the indicators.
Findings
Description of the teams, their research, and sources of data
The teams were heterogeneous in relation to their specific research questions, structure, and methodology, each with several studies or sub-studies using a variety of methods (Table 1). The teams’ projects were similar in the sense that they were written for internal validity and intervention fidelity with potential for generalizability of results. The teams employed a range of study designs including case studies, cross-sectional observational studies, pre–post intervention, randomized controlled trials, and longitudinal study designs. Taken together, the teams covered all of Canada with the exception of one province and one territory (Saskatchewan, Yukon, respectively) (Table 1). Three teams have Indigenous partners and three are collaborating internationally. Nine of the 12 teams are collecting data in English and French. Some teams are collecting data in other languages, including Cantonese, Greek, and Italian.
Table 1 Location and brief description of 12 teams projects
NL=Newfoundland; NB=New Brunswick; NS=Nova Scotia; PE=Prince Edward Island; ON=Ontario; AB=Alberta; BC=British Columbia; MB=Manitoba; QC=Quebec; PHC=primary health care; NU=Nunavut; NW=Northwest Territories.
More information about the projects can be found at http://www.cihr-irsc.gc.ca/e/45817.html
The dimensions of primary care and the agreed upon common indicators
Based on the funded teams’ proposals and contexts, feasibility, collective expertise, and primary health care indicators developed by the CIHI (2012), the working group agreed upon five primary health care dimensions, namely, access, comprehensiveness, coordination, effectiveness, and equity (see Table 2).
Table 2 Coverage of agreed upon primary care indicators across the 12 teams
PHC=primary health care; Pa=patient; Pr=clinician; Org=organization; HAD=health administrative data.
We identified 19 indicators covering the five dimensions that could be measured using publicly available items and scales (see Additional File 1). Common indicators in addition to those within the five dimensions include population/patient characteristics, multimorbidity, patient-reported outcome measures (PROMs) of health and well-being [VR-12-SF (Ware et al., Reference Ware, Kosinski and Keller1996) and EQ-5D-5L (Xie et al., Reference Xie, Pullenayegum, Gaebel, Bansback, Bryan, Ohinmaa, Poissant and Johnson2016)], health care utilization, and health care costs. All scales, including the Team Climate Inventory (Anderson and West, Reference Anderson and West1998) were selected based on having appropriate reliability and validity, prior use in primary care, user burden (where possible, items that were already being collected by some teams or could be easily added to planned data collection) and cost (with a preference for tools that are available free of charge to facilitate sustainability of this work) (Figure 1).
Figure 1 The process and criteria of development of the common indicators among the 12 teams
Data sources included patient, provider, and organizational surveys, and common algorithms in health administrative data. Validated tools for use with administrative health data include common algorithms for identifying age strata or examining multimorbidity, as examples.
Methods used to collect the data included interviews or secondary analysis of health administrative data after an approved data access request was completed (five of these teams are linking these data with other sources such as survey data) (Table 3).
Table 3 Sources and types of data collection used by the 12 teams
a Other data collected by teams include items such as community readiness surveys, volunteer satisfaction surveys, and publically available statistical data.
Coverage of indicators by the teams and sources of data
Some teams had started collecting data prior to the funding decision and formation of the working group. Where possible, the agreed upon measures were incorporated into teams’ research plans after the funding decisions were made and the working group was formed. Some teams are collecting modified versions of the agreed measures.
The most commonly collected indicators are in the dimensions of access (all teams), coordination (11/12 teams), and equity (11/12), whereas the least commonly collected indicators are related to health care costs (six teams) (Table 2). In addition, all teams are collecting measures related to patient characteristics, multimorbidity, and health care utilization.
Group process for cross-team collaboration
Following collaboration on the common indicators, the working group devised an authorship policy ratified by the principal investigators of all teams. This policy is informed by previous policies made by some of the 12 teams and their institutes as well as academic publishing guidelines [Hogg et al., Reference Hogg, Rowan, Russell, Geneau and Muldoon2014; International Committee of Medical Journal Editors (ICMJE), 2017].
The working group is now developing and implementing cross-team collaborative projects using data collected through the common indicators. Cross-team project ideas are pitched to the group in the form of one-pager summaries and smaller working groups are formed to lead the studies. Progress is shared with the larger group and other researchers can join at different stages. Cross-team projects are facilitated with the help of a cross-team research coordinator.
This work provides the largest pan-Canadian opportunity for cross-jurisdictional learning and knowledge generation on the processes, outputs, and patient-reported outcomes in primary care service delivery. For example, across the teams, we now have the largest source of data on team functioning in primary care practices and the largest source of data on PROMs in the Canadian primary care context.
Discussion
This unprecedented process of collaboration across 12 interdisciplinary CBPHC research teams operating across Canada in 13 jurisdictions is a unique opportunity for new knowledge generation that exceeds beyond the knowledge outputs of any one team. We have successfully built the infrastructure and fostered partnerships across the teams to advance primary care research and innovation. Across the teams, there are over 120 individual researchers, staff, and students who have developed ways for working together across Canada and now share common ways of measuring key dimensions of primary care.
The collected common indicators can serve as a standard set of items and scales for common primary care performance reporting across Canada; they could complement the gaps in data for existing reporting initiatives such as reporting on the CIHI (2016) pan-Canadian primary care indicator set. As an added value to such quantitative measures, our combined efforts provide a richness of multiple data sources, diverse perspectives, and contextual factors. Finally, cross-comparative analysis of data collected through validated tools such as EQ-5D-5L and Team Climate Inventory in different contexts could inform of their utility in the Canadian primary care.
In the context of research impact (Greenhalgh et al., Reference Greenhalgh, Raftery, Hanney and Glover2016), the development of common metrics, a data infrastructure, and a collaborative process across the 12 teams has the potential to make a dramatic contribution to the key aims of the signature initiative. In addition, because of the heterogeneity of the teams and the diversity of local context, the learnings of the 12 teams and the working group bring more real world relevance to measurement of primary care. The data collected through the common indicators, complemented with local context, can answer questions such as ‘what works for whom in what circumstances’ (Pawson and Tilley, Reference Pawson and Tilley2001). The working group across the teams is an innovation in how to best share and reflect on heterogeneous and common data to foster improvements in primary care delivery and organization. Moreover, we have developed capacity across these teams to collect data for future analysis and bring understanding of the context about outcomes related to spread and scale of innovations in delivery and organization of primary care.
A key factor in the success of the common indicators working group is each team’s willingness to collaborate openly, engage with each other and commitment to the process. Other key success factors include leadership of the working group that entailed active engagement of all members, a research coordinator, and regular liaising with the funder and different teams based on their needs. An additional condition needed for this work is embedded funding. By allocating funding and mandating cross-team cooperation, CIHR and its partners has created an environment for collaboration rather than competition.
This work began with broad aims and varied methodologies, which required dedicated collaboration on behalf of the research teams in order to establish common ground and move towards the cross-team objectives (CIHR, 2012). The lack of standardization in applying the measures and data collection will limit the kinds of analyses possible. Future analyses will require a thorough assessment of potential bias introduced by the heterogeneity of data collection methods. At the very least, we now have a foundation of researchers and trainees from which to build cross-team research and can incorporate important learnings from other examples of cross-team research in primary care, such as the Agency for Health care Research and Quality Patient-Centred Medical Home research wherein specific goals and objectives were defined at the beginning and projects were carried out independently, with outcomes pooled at the end (Genevro and Meyers, Reference Genevro and Meyers2013).
Reaching consensus on common indicators was challenging at times. In part, this was a normal process of group formation where we moved through the norming, storming, and forming stages (Tuckman, Reference Tuckman1965). Moreover, there has been turnover of investigators and staff in this group. Much of our disagreement was influenced by the high value placed on traditional approaches to research methods. These methods are guided by fidelity to a priori specification of hypotheses-testing, which drives sampling, randomization, and data collection processes.
An ongoing challenge is having the working group members act as liaisons between the working group and their individual teams, the latter of which include community partners and other diverse stakeholder groups. Finding ways to optimize the group process and utilize all expertise (funders, researchers, and community partners) to develop a mutually acceptable set of measures and common approach to data collection are resource and time intensive. Indeed, each team found it important to respect the values and preferences of their represented sub-populations while also collaborating with other teams and moving toward common goals.
Other challenges associated with conducting cross-team research include confidentiality related to data sharing and ensuring accurate and meaningful comparisons of results across teams with divergent study designs, target patient populations and that not all teams were using methods compatible with the entire set of measures. Ownership of data is of critical importance in relation to working with First Nations and Inuit partners; teams who were specifically committed to participatory research methods and principles of Ownership, Control, Access, Possession (First Nations Information Governance Centre, 2014) meant that decisions to collect common indicators and participate in cross-team analysis of data require additional memoranda of understanding with Indigenous community partners. In that regard, the data-sharing processes opted by the common indicators working group are experiments that can be replicated in other contexts to further facilitate cross-jurisdictional and collaborative PHC research.
Conclusions
Twelve CBPHC research teams, supported by specific funding, formed a working group and engaged in a process to agree on and collect common data indicators for the generation of knowledge beyond any one team’s ability. As a result of this signature initiative, a wealth of data stemming from the common indicators has been generated in primary health care. Collected process, outcome, and contextual data from the 12 teams will be analyzed using mixed methods to describe elements of the interventions and the contexts that will influence their implementation and scale-up. Future work should define measures a priori within request for proposals and require that teams build this into their applications. Such an approach could enable funders and researchers to synthesize findings across funded studies to be able to give a cohesive picture of findings from an initiative.
The 12 teams collaboration has laid the groundwork for measurement in future CBPHC endeavors, and while the measures may not be directly transferable to other high-priority areas of health care (eg, end-of-life care) our framework and processes can be used to inspire and fast-track similar future initiatives. Our findings can provide guidance for those at the coalface of innovation in primary care from practice-level service delivery and planning through to national performance measurement and reporting.
Acknowledgments
The authors are thankful to Kue Young, Stewart Harris, Janusz Kaczorowski, Erica Di Ruggiero, Robyn Tamblyn, Nancy Edwards, Meg McMahon, Meredith Kratzmann, Bojana Petrovic, and Dawn Mooney for their comments and contributions.
Authors’ Contribution
S.W. and J.L. wrote the initial draft of the manuscript. All teams participated in the process of developing the common indicators. All authors contributed to the interpretation, writing, and approval of the final manuscript.
Financial Support
This study has been supported by CBPHC signature initiative grants from the CIHR. The funder did not play a role in study design, data collection, analysis, and interpretations of the data.
Conflicts of Interest
None.
Open access
