Background and aims
Global commitment to out-of-hours primary care (OOH-PC) and recognition of the importance of this healthcare format is increasing (Hong et al., Reference Hong, Thind, Zaric and Sarma2020; Steeman et al., Reference Steeman, Uijen, Plat, Huibers, Smits and Giesen2020). Additionally, research has demonstrated that its implementation has the potential to improve the quality of care, optimise efficiency, and reduce the strain on emergency departments (EDs) (Mohsin et al., Reference Mohsin, Forero, Ieraci, Bauman, Young and Santiano2007; Guttmann et al., Reference Guttmann, Schull, Vermeulen and Stukel2011; Whittaker et al., Reference Whittaker, Anselmi, Kristensen, Lau, Bailey, Bower, Checkland, Elvey, Rothwell, Stokes and Hodgson2016; Hong et al., Reference Hong, Thind, Zaric and Sarma2020; Allen et al., Reference Allen, Cummings and Hockenberry2021). Although there is no consensus on the definition of ‘appropriate’ or ‘inappropriate’ use of the ED, several studies find that many medical problems presented in the ED could be managed in a primary care setting, as they do not always require specialist care (Derlet & Ledesma, Reference Derlet and Ledesma1999; Carret et al., Reference Carret, Fassa and Kawachi2007; Durand et al., Reference Durand, Gentile, Devictor, Palazzolo, Vignally, Gerbeaux and Sambuc2011; Kraaijvanger et al., Reference Kraaijvanger, Rijpsma, Van Leeuwen, Van Dijk and Edwards2016).
OOH-PC is operationally defined in terms of time frame, as primary care delivered on weekdays outside business hours, approximately from 6:00 PM to 8:00 AM, during weekends or public holidays (O’Donnell et al., Reference O’Donnell, Foster, Macdonald, Burns and Gannon2015). Such care is classified as unscheduled, meaning that no appointment or forward planning is arranged beforehand (O’Donnell et al., Reference O’Donnell, Foster, Macdonald, Burns and Gannon2015). However, there is a discussion in the literature on whether out-of-hours care should only provide urgent care or include non-urgent care as well (Keizer et al., Reference Keizer, Smits, Peters, Huibers, Giesen and Wensing2015; O’Donnell et al., Reference O’Donnell, Foster, Macdonald, Burns and Gannon2015; Barnes et al., Reference Barnes, Agostino, Ceramidas and Douglas2022). Nonetheless, for this paper, we adopted a broad and commonly accepted definition of OOH-PC based solely on the time frame, specifically excluding care delivered in the ED or other secondary or tertiary levels.
OOH-PC encompasses several models of delivery, such as practice-based services in which physicians within an individual or group practice look after their own and each other’s patients during OOH times (Berchet & Nader, Reference Berchet and Nader2016). Another model is general practice cooperatives (GPCs), which are large-scale self-organised groups of general practitioners (GPs) providing out-of-hours care in a region (Berchet & Nader, Reference Berchet and Nader2016; Colliers et al., Reference Colliers, Remmen, Streffer, Michiels, Bartholomeeusen, Monsieurs, Goris, Coenen, Verhoeven and Philips2017). Additionally, there are also retail or medical clinics located within grocery stores or pharmacies, typically staffed by nurses or other health professionals (Berchet & Nader, Reference Berchet and Nader2016).
Many countries have now adopted OOH-PC (Steeman et al., Reference Steeman, Uijen, Plat, Huibers, Smits and Giesen2020), necessitating its inclusion in the decision-making process regarding future investments. For decisions that aim to maximise welfare, the perspectives of the healthcare payer, the hospital, and society are informative, in which the latter incorporates the full range of relevant costs and effects, including patient long-term outcomes and productivity losses (Byford & Raftery, Reference Byford and Raftery1998). However, evidence on the economic evaluation of OOH-PC service delivery is scarce and limited, despite numerous suggestions to robustly assess the causal impact of improving access to primary care on the use of other services, outcomes, and costs (WHO & UNICEF, 2022). Some studies in this domain have focused only on the estimation of cost implications (Brogan et al., Reference Brogan, Pickard, Gray, Fairman and Hill1998; Scott et al., Reference Scott, Simoens, Heaney, O’Donnell, Thomson, Moffat, Ross and Drummond2004; O’Dowd, Reference O’Dowd2006; van Uden et al., Reference van Uden, Ament, Voss, Wesseling, Winkens, van Schayck and Crebolder2006; Eichler et al., Reference Eichler, Imhof, Chmiel, Zoller, Senn, Rosemann and Huber2010; Moth et al., Reference Moth, Huibers and Vedsted2013; Lin et al., Reference Lin, Loy, Boothe, Bennett, Tarbox, Prabhu and Sturgeon2021; Morreel et al., Reference Morreel, Homburg, Philips, De Graeve, Monsieurs, Meysman, Lefevere and Verhoeven2022). Others have incorporated effects by including immediate health system effects that manifest within a short time frame of patient presentation, such as admissions or practice attendance (Hansen & Munck, Reference Hansen and Munck1998; Lattimer et al., Reference Lattimer, Sassi, George, Moore, Turnbull, Mullee and Smith2000; Moore et al., Reference Moore, Young, Irving, Goodacre, Brennan and Amos2021; Flaherty et al., Reference Flaherty, Klarman, Cajusma, Schon, Exantus, Beau de Rochars, Baril, Becker and Nelson2022). Although OOH-PC is commonly considered a short-term intervention, it is also crucial to capture long-term societal outcomes (Deidda et al., Reference Deidda, Geue, Kreif, Dundas and McIntosh2019).
This study aims to first provide a comprehensive overview of the effect measures currently used in the economic evaluations of OOH-PC interventions and, second, propose an additional set of societal effect measures to capture the broader economic value of this healthcare service. The additional measures relevant to OOH-PC are identified and discussed based on an overview of economic evaluations of integrated care programmes.
Approach and development
Systematic literature search
We consulted PubMed, SCOPUS, Web of Science, EconLit, Cochrane reviews, NHSEED, and Health Technology Assessment databases for articles that performed economic evaluations of OOH-PC. See Appendix A for details on the search strategy. Eligibility criteria were agreed upon by two researchers (JP and LW) using the PICOTS framework (Population, Intervention, Comparator/Context, Outcome, Timing, and Study Design/Setting) to define inclusion and exclusion criteria (Table 1).
PICCOTS
The online search identified 2717 unique results; we identified one additional article by screening the references of seven systematic reviews on OOH-PC effectiveness. After removing 299 duplicates, we selected 101 from the title and abstract screening, and finally, we included 13 in this overview after reading the full text. See Appendix B for the PRISMA diagram.
We evaluated the quality of the 13 studies using the Consensus Health Economic Criteria (CHEC) list, which is a 19-point checklist researchers use to evaluate the reporting and methodology of published economic evaluations (Evers et al., Reference Evers, Goossens, de Vet, van Tulder and Ament2005). Of the studies, 10 received a score greater than or equal to 75%, while three received a score of between 51% and 74%. (See Appendix C for the evaluation approach and the assessment results). Note that scores cover both the quality of the study conducted and the completeness of reporting. However, if a paper does not discuss context because it seemed obvious, we flagged the corresponding item as not applicable and excluded the item from the denominator following the CHEC guidelines.
The 13 included studies comprised 11 research papers that evaluated the cost-effectiveness and efficiency of various OOH-PC healthcare interventions and modalities (Broekman et al., Reference Broekman, Van Gils-Van Rooij, Meijboom, De Bakker and Yzermans2017; Chesteen et al., Reference Chesteen, Warren and Woolley1986; Flaherty et al., Reference Flaherty, Klarman, Cajusma, Schon, Exantus, Beau de Rochars, Baril, Becker and Nelson2022; Flynn, Reference Flynn1998; Hansen & Munck, Reference Hansen and Munck1998; Lattimer et al., Reference Lattimer, Sassi, George, Moore, Turnbull, Mullee and Smith2000; Moe et al., Reference Moe, Oland and Moe2019; Moore et al., Reference Moore, Young, Irving, Goodacre, Brennan and Amos2021; Patwardhan et al., Reference Patwardhan, Davis, Murphy and Ryan2012; Poole et al., Reference Poole, Schmitt, Carruth, Peterson-Smith and Slusarski1993; Sterner et al., Reference Sterner, Coco, Monroe, King and Losek2012) and two protocol papers proposing full economic evaluations (Reuter et al., Reference Reuter, Desmettre, Guinemer, Ducros, Begey, Ricard-Hibon, Billier, Grignon, Megy-Michoux, Latouff, Sourbes, Latier, Durand-Zaleski, Lapostolle, Vicaut and Adnet2016; Wijers et al., Reference Wijers, Schoonhoven, Giesen, Vrijhoef, Van Der Burgt, Mintjes, Wensing and Laurant2012). Since our focus was on the effect measures and not on the results, these protocols are sufficient and studies that published the results were not included.
Among these studies, five compared alternative ways of providing OOH-PC (Broekman et al., Reference Broekman, Van Gils-Van Rooij, Meijboom, De Bakker and Yzermans2017; Hansen & Munck, Reference Hansen and Munck1998; Lattimer et al., Reference Lattimer, Sassi, George, Moore, Turnbull, Mullee and Smith2000; Reuter et al., Reference Reuter, Desmettre, Guinemer, Ducros, Begey, Ricard-Hibon, Billier, Grignon, Megy-Michoux, Latouff, Sourbes, Latier, Durand-Zaleski, Lapostolle, Vicaut and Adnet2016; Wijers et al., Reference Wijers, Schoonhoven, Giesen, Vrijhoef, Van Der Burgt, Mintjes, Wensing and Laurant2012), while seven compared OOH-PC with ED or with a ‘no OOH-PC’ scenario (Chesteen et al., Reference Chesteen, Warren and Woolley1986; Flaherty et al., Reference Flaherty, Klarman, Cajusma, Schon, Exantus, Beau de Rochars, Baril, Becker and Nelson2022; Flynn, Reference Flynn1998; Moe et al., Reference Moe, Oland and Moe2019; Moore et al., Reference Moore, Young, Irving, Goodacre, Brennan and Amos2021; Poole et al., Reference Poole, Schmitt, Carruth, Peterson-Smith and Slusarski1993; Sterner et al., Reference Sterner, Coco, Monroe, King and Losek2012). Additionally, one study compared OOH-PC with urgent care centres, primary care physicians, ED, and without intervention (Patwardhan et al., Reference Patwardhan, Davis, Murphy and Ryan2012). In total, these studies evaluated six types of OOH-PC: telephone support, nurse-delivered care, telemedicine, home delivery, late night/weekend/holiday clinics for alcohol intoxication, after-hours clinics/family practices or urgent care centres, and jointly operating ED and GP care. See Appendix D for a further description of the studies.
Measures previously used in the economic evaluation of OOH-PC
With the formalisation of OOH-PC implementation, numerous published studies have concentrated on its effectiveness. In the upper section of Table 2, we have compiled common effectiveness measures based on the findings of seven systematic reviews (Foster et al., Reference Foster, Moffat, Burns, Gannon, Macdonald and O’donnell2020; Fry, Reference Fry2011; Garratt et al., Reference Garratt, Danielsen and Hunskaar2007; Hong et al., Reference Hong, Thind, Zaric and Sarma2020; Huibers et al., Reference Huibers, Smits, Renaud, Giesen and Wensing2011; Leibowitz et al., Reference Leibowitz, Day and Dunt2003; O’Donnell et al., Reference O’Donnell, Foster, Macdonald, Burns and Gannon2015). In contrast, the literature on cost-effectiveness is comparatively limited and systematic reviews are absent (O’Donnell et al., Reference O’Donnell, Foster, Macdonald, Burns and Gannon2015). Therefore, we performed a systematic search of the literature and retrieved 13 studies as outlined in Section 2.1. The lower section of Table 2 compiles the effect measures based on the findings of these studies.
Overview of effect measures related to the process, patient outcome, and healthcare resource use in the effectiveness and economic evaluation of OOH-PC interventions
Table 2 shows the effect measures related to process, patient outcomes, and healthcare resource used to evaluate OOH-PC interventions’ effectiveness and economic evaluations. According to the Donabedian paradigm, we categorise the measures into three groups according to the consensus-based entity submission types outlined in the Centers for Medicare & Medicaid Services Measures (CSM) Inventory Tool (Measures Management Systems, 2023). A process measure refers to the evaluation of specific steps, procedures, and environment essential to provide quality care (Measures Management Systems, 2023). When a process measure is managed effectively, the likelihood of achieving the desired outcome increases. For example, more general access to healthcare services can contribute to reduced mortality (Measures Management Systems, 2023). A patient outcome measure focuses on assessing a patient’s health status or any changes in their welfare resulting from healthcare interventions (Measures Management Systems, 2023). Examples of outcome measures include mortality rates and gains in quality-adjusted life years (QALYs). A measure of healthcare resource use quantifies the utilisation of healthcare services expressed in terms of natural units (Measures Management Systems, 2023). This encompasses various aspects, including diagnoses, procedures, or healthcare encounters, and can be exemplified by metrics such as the number of GP visits (Measures Management Systems, 2023).
As shown in the upper part of Table 2, effectiveness studies mostly include the use of healthcare resources, a few patient outcome measures, and a limited number of process measures. Similarly, the lower part of Table 2 shows that OOH-PC economic evaluations focus on the use of healthcare resources and a few patient outcome measures, while process measures remain relatively underutilised.
The need for a broader scope for OOH-PC
OOH-PC falls within the umbrella of integrated care, and it is necessary to measure and evaluate its broader effects at various levels. Given this, OOH-PC has been shown to improve access to care for those in need, reduce ED visits, and promote efficiency (Dent, Reference Dent2010; Lowe et al., Reference Lowe, Localio, Schwarz, Williams, Tuton, Maroney, Nicklin, Goldfarb, Vojta and Feldman2005; Piehl et al., Reference Piehl, Clemens and Joines2000). Moreover, integrated care represents care that is coordinated across professionals, facilities, and support systems, is continuous over time, and is responsive to people’s needs, values, and preferences (Schneider et al., Reference Schneider, Burgers, Friedberg, Rosenthal, Leape and Schneider2011). It encompasses treatment plans, methods, and models of care that enable improvement in patient experience, promote efficient service delivery, reduce healthcare expenditures, and improve population health through enhanced coordination and continuity of care (Plochg et al., Reference Plochg, Klazinga and Starfield2009; Shaw et al., Reference Shaw, Rosen and Rumbold2011). These types of interventions impact various outcomes at various levels, necessitating the measurement and evaluation of multiple outcomes (Tsiachristas et al., Reference Tsiachristas, Stein, Evers and Rutten-van Mölken2016). Furthermore, they alter existing care processes and pathways and impact providers, patients, and communities (Baxter et al., Reference Baxter, Johnson, Chambers, Sutton, Goyder and Booth2018), henceforth the recommendation for a broader evaluation. A broad scope of evaluation has been applied to various areas of integrated care for public decision-making (Nolte & Pitchforth, Reference Nolte and Pitchforth2014).
Table 3 presents an overview of the effect measures used previously or recommended for use to measure the effects of integrated care in economic evaluations. We derived this information from two systematic reviews and several individual research studies (KPMG, 2018; Nolte & Pitchforth, Reference Nolte and Pitchforth2014; Rocks et al., Reference Rocks, Berntson, Gil-Salmerón, Kadu, Ehrenberg, Stein and Tsiachristas2020; Steuten et al., Reference Steuten, Vrijhoef, Severens, Van Merode and Spreeuwenberg2006; Tsiachristas et al., Reference Tsiachristas, Cramm, Nieboer and Rutten-Van Mölken2013). Like Table 2, the measures are categorised into process, healthcare resources, and patient outcome. Although healthcare resource use measures are important indicators of health system performance, it is also essential to consider patient-centred outcomes, including population health, patient, or community well-being. The combination of patient-centred and process-related measures allows for a more comprehensive evaluation.
Overview of effect measures related to the process, patient outcome, and healthcare resource use in the economic evaluation of integrated care interventions
Additional key measures relevant for OOH-PC economic evaluations
Comparison of Tables 2 and 3 reveals that integrated care interventions employ effect measures beyond immediate outcomes. These effects are more extensive, covering broader health system and patient effects compared to those used in OOH-PC. Consequently, we advocate for the inclusion of additional effects in the evaluation of OOH-PC to improve the comprehensive understanding of the economic value. Additionally, it will create synergy between OOH-PC and conceptually similar integrated care interventions, which can complement and/or substitute OOH-PC.
Following this mapping of the effects of integrated care initiatives and OOH-PC, we identified eight additional effect measures for integrated care initiatives not yet used in OOH-PC. These measures include two process measures, one healthcare resource use measure, and five patient measures. Specifically, two identified process measures are ‘continuity of care’ and ‘health promotion’, while the additional measure of using healthcare resources is ‘informal caregiver time’. The innovative patient outcome measures for OOH-PC studies are ‘diagnosis and disease management’, ‘frequency of complications and exacerbations’, ‘self-efficacy’, ‘patient travel and time’, and ‘productivity loss’. While all eight measures are relevant for the economic evaluation of OOH-PC, in this study, we narrow our discussion to three key measures where the out-of-hours aspect is crucial and distinguishes itself from within-hours care: ‘continuity of care’, ‘health promotion’, and ‘productivity loss’. The added value of each selected measure is discussed in the following paragraphs. Among the five outcome measures not highlighted here, such as ‘diagnosis and disease management’, ‘patient travel and time’, or ‘informal caregivers time’, we believe that the difference with regular primary care is limited. However, all identified outcome measures that are currently absent from published economic evaluations of OOH-PC offer valuable insights into clinical outcomes or well-being, while depicting costs for the healthcare payer, the patient, or society.
Table 4 shows a comprehensive summary of the effect measures for OOH-PC evaluations. It lists additional measures proposed from integrated care and those used in previous OOH-PC evaluations. The effect measures that we emphasise in the main text are coloured red. In the subsequent paragraphs, we elucidate and delve into the proposed supplementary effect measures, offering illustrative examples from related domains.
Comprehensive summary of effect measures related to process, patient outcomes, and healthcare resource use for OOH-PC evaluation
Productivity losses averted (opportunity cost of seeking healthcare)
Economic evaluations must consider the opportunity cost of seeking care, which can affect personal and workforce time, resulting in productivity loss (Ray et al., Reference Ray, Chari, Engberg, Bertolet and Mehrotra2015; Weinstein et al., Reference Weinstein, Siegel, Gold, Kamlet and Russell1996). From the patient’s perspective, productivity loss contributes to wage loss and causes undesirable experiences when seeking care (Handley & Hollander, Reference Handley and Hollander1999; NHS Primary Care Commissioning, 2012). Additionally, the difficulty of taking time off work and the lack of access to convenient care are two factors often cited as barriers to accessing regular-hour primary healthcare (Friedberg et al., Reference Friedberg, Hussey and Schneider2010; NHS Primary Care Commissioning, 2012). In contrast, these two factors are facilitators of increased use of OOH-PC (Zhou et al., Reference Zhou, Abel, Warren, Roland, Campbell and Lyratzopoulos2015). Therefore, OOH-PC offers patients a convenient avenue to access primary care outside of working hours, helping to avoid wage and time losses. This is also relevant for caregivers responsible for caring for the disabled, elderly, and young children. OOH-PC grants these individuals the ability to refrain from taking time away from work and sacrificing their caregiving obligations concerning their care recipients’ healthcare needs. This also applies to students and school-aged children. Research indicates that students who rely on public clinics often miss entire days of school per appointment (Kornguth, Reference Kornguth1990). Furthermore, research findings indicate that children frequently use OOH-PC in European settings (Huibers et al., Reference Huibers, Moth, Bondevik, Kersnik, Huber, Christensen, Leutgeb, Casado, Remmen and Wensing2011), and its use may help reduce school absenteeism (Institute of Medicine, 1997). Productivity gains in this group can be achieved by minimising absenteeism among students and school-aged children. Therefore, it is important to include the productivity loss avoided in the evaluation of OOH-PC, as it represents an important positive gain for patients, caregivers, students, and school-aged children. This gain is an important influence in this domain, and it is crucial not to underestimate this potential advantage. Additionally, it is important to consider the productivity loss caused by using OOH-PC. Furthermore, opportunity costs are important due to the increased recognition of patient-centred care (Baker, Reference Baker2001). Subsequently, this has led to an emphasis on innovative healthcare delivery options that reduce the time burden (Ray et al., Reference Ray, Chari, Engberg, Bertolet and Mehrotra2015).
Productivity gains have been included in economic evaluations of telemedicine (Patel et al., Reference Patel, Turner, Alishahi Tabriz, Gonzalez, Oswald, Nguyen, Hong, Jim, Nichols, Wang, Robinson, Naso and Spiess2023; Snoswell et al., Reference Snoswell, Taylor, Comans, Smith, Gray and Caffery2020), which is a conceptually similar intervention to OOH-PC. Although not delivered specifically outside of working hours, telemedicine also provides convenient, patient-centred care that improves access to care and reduces time burden. Additionally, it seeks to reduce the inefficient use of higher-level facilities, including EDs (Sun et al., Reference Sun, Lu and Rui2020). By providing the necessary services by electronic means, telemedicine serves individuals who have difficulties in making appointments or may not have the time, resources, or motivation to travel to traditional clinics (Zhang et al., Reference Zhang, Cheng, Zhu, Huang and Shen2021). Eliminating these logistical difficulties results in decreased patient productivity loss in the economic evaluation of telemedicine (Agha et al., Reference Agha, Schapira and Maker2002; Kubes et al., Reference Kubes, Graetz, Wiley, Franks and Kulshreshtha2021). Estimation of productivity losses or gains in economic evaluations is complex and uses various measures. These include the human capital approach, friction cost approach, or multiplier approach, which can account for productivity in natural units, opportunity costs, and replacement costs (Jiang et al., Reference Jiang, Wang, Si, Zang, Gu, Jiang, Liu and Wu2022). Because there is no universally accepted measure, the choice of approach must depend on existing national guidelines while considering available information in a specific context.
Access to health promotion services and early intervention
Health prevention involves taking action to avoid the onset of disease and associated risk factors (Radhakrishnan, Reference Radhakrishnan2017). These actions include vaccinations, prophylaxis, education of people about behavioural and medical health risks, and disease detection (Radhakrishnan, Reference Radhakrishnan2017). Similarly, health promotion involves empowering people to take control of their health and its determinants (Radhakrishnan, Reference Radhakrishnan2017). Examples of these include dietary and nutritional interventions, interventions to mitigate social ills such as domestic violence, and interventions to promote sexual and reproductive health, such as family planning services (Radhakrishnan, Reference Radhakrishnan2017).
Reliable access to health-promotive and preventive services in primary healthcare is important for improving health outcomes and reducing the financial burden of treating diseases (Hostetter et al., Reference Hostetter, Schwarz, Klug, Wynne and Basson2020). Providing these services promotes timely diagnosis and treatment, increasing the chances of success (World Health Organization, 2018). An efficient and effective primary healthcare system must provide promotive and preventive services (Van Weel & Kidd, Reference Van Weel and Kidd2018). However, despite the benefits, many settings do not have optimal access to these recommended services (Borsky et al., Reference Borsky, Zhan, Miller, Ngo-Metzger, Bierman and Meyers2018; Levine et al., Reference Levine, Malone, Lekiachvili and Briss2019).
Having access to primary healthcare is a crucial factor that determines whether people receive promotive and preventive services (Xu, Reference Xu2002; Friedberg et al., Reference Friedberg, Hussey and Schneider2010). OOH-PC is efficient in improving access to primary care (Hong et al., Reference Hong, Thind, Zaric and Sarma2020). Therefore, OOH-PC represents a suitable resource for those who would otherwise not seek care at a regular-hour health facility for the necessary services and check-ups. This is particularly so in settings or modalities where OOH-PC permits or is used for such services. Without such care and services, delayed diagnoses and treatment can lead to complicated disease management (World Health Organization, 2018). Given the profound role that OOH-PC can play in delivering promotive and preventive services, a robust economic assessment must consider incorporating these benefits. Their omission could produce a conservative estimate of the economic benefits of OOH-PC.
Improved access to preventive and promotive health services has been used in conceptually similar interventions. For example, a recent cost–benefit analysis (CBA) from Australia examined nurse-led primary healthcare facilities and explored how they impacted the provision of promotive and early interventions (KPMG, 2018). Similarly, economic evaluations of mobile and community clinics included the adoption of preventive and promotive services (Liu et al., Reference Liu, Guo, Wang and Xin.d.; Oriol et al., Reference Oriol, Cote, Vavasis, Bennet, DeLorenzo, Blanc and Kohane2009; Stillmank et al., Reference Stillmank, Bloesl, McArthur, Artz and Lancaster2019).
In the context of a societal evaluation, it has been proposed that the value of healthcare services should not be limited to their value to patients alone (Culyer et al., Reference Culyer, Chalkidou, Teerawattananon and Santatiwongchai2018). While the well-being of patients remains a focal point, it is imperative to incorporate the perspectives of healthcare providers to gain a comprehensive understanding of the practical implications of the delivery of healthcare services. This is of relevance for health promotion and preventive services, as providers often face heavy workloads (Smits et al., Reference Smits, Keizer, Huibers and Giesen2014; Royal College of General Practitioners, 2019). This can limit their capacity to offer non-urgent care on an unscheduled basis, as they must prioritise urgent care. Thus, in addition to the potential benefits to the patient, it is necessary to quantify the deterrent effects, if possible, and consider the viewpoint of service providers as relevant and appropriate measures of value regarding health-promoting and preventive services in the OOH-PC domain. However, striking a balance between the potentially opposing needs and perspectives of patients and providers is a complex but indispensable task.
Measuring the impact of an intervention on the acceptance of promotive or preventive services involves assessing whether the number of visits related to these services has changed since the implementation of that intervention (KPMG, 2018). However, establishing the long-term economic impact of preventive care and health promotion is challenging for several reasons. At the population level, it can be challenging to isolate the impact of individual interventions because the impacts are bundled or extend beyond healthcare. Additionally, some settings may encounter challenges with missing or inferior data, or practical/ethical/privacy problems when linking data from several sources or contacts (Colliers et al., Reference Colliers, Bartholomeeusen, Remmen, Coenen, Michiels, Bastiaens, Van Royen, Verhoeven, Holmgren, De Ruyck and Philips2016). In cases where data are accessible, the estimation of QALYs can be considered using clinically preventable burden scores, as has been demonstrated in previous research (Maciosek et al., Reference Maciosek, LaFrance, Dehmer, McGree, Flottemesch, Xu and Solberg2017; Stillmank et al., Reference Stillmank, Bloesl, McArthur, Artz and Lancaster2019).
Continuity of care
Relationship continuity of care is the maintenance of continuous and sustained relationships between patients and healthcare professionals (Gulliford et al., Reference Gulliford, Naithani and Morgan2006; Hill & Freeman, Reference Hill and Freeman2011). Research findings demonstrate that establishing a good, trust-based, and long-term relationship with a primary care physician of one’s choice can lead to improved health outcomes, better quality of care, and reduced healthcare expenses (Starfield et al., Reference Starfield, Shi and Macinko2005). On the other hand, management continuity refers to care systems facilitated by integration, coordination, and the sharing of information between different providers (Gulliford et al., Reference Gulliford, Naithani and Morgan2006; Hill & Freeman, Reference Hill and Freeman2011). Continuity of care in primary healthcare, both in relationship and in management, is beneficial for patients, clinicians, and health systems. It leads to increased patient satisfaction, improved care for chronic patients, increased use of preventive care, promoted adherence to medical advice, reduced dependency on hospitals, and reduced mortality (Gray et al., Reference Gray, Sidaway-Lee, White, Thorne and Evans2018; Sidaway-Lee et al., Reference Sidaway-Lee, Gray and Evans2019).
Despite its benefits, continuity of care remains low in some primary care settings and is rarely measured (Sidaway-Lee et al., Reference Sidaway-Lee, Gray and Evans2019). Measurement and comparison of continuity rates among providers can, in turn, improve continuity (Kontopantelis et al., Reference Kontopantelis, Reeves, Valderas, Campbell and Doran2013). Furthermore, ensuring patient-centredness and continuity of care are crucial attributes of a well-functioning primary care system. Therefore, it is important to establish whether an intervention demonstrates a reasonable level of continuity.
In the out-of-hours domain, relationship continuity may not be present. The feasibility of primary care physicians providing 24-hour care is influenced by several factors, including provider preferences, patient needs, existing market supply, and financial considerations (O’Malley et al., Reference O’Malley, Samuel, Bond and Carrier2012). Therefore, if OOH-PC is not provided by the usual physician, a mechanism is necessary to facilitate the sharing of health information and systematic notification procedures to maintain information continuity between providers to prevent fragmentation of care (O’Malley et al., Reference O’Malley, Samuel, Bond and Carrier2012). However, the exchange of patient information is of interest for all healthcare stakeholders and interventions, especially in the context of integrated care. Interaction is needed, for example, for a scheduled follow-up with the usual primary physician or when complications and/or exacerbations of previously treated conditions arise (O’Malley et al., Reference O’Malley, Samuel, Bond and Carrier2012). The degree to which primary care interventions and modalities provide continuity of relationship and care should be monitored, assessed, and included in economic evaluations.
Incorporating continuity of care into economic evaluations is not common. Nonetheless, an Australian study assessed continuity of care as a qualitative aspect of its economic evaluation (KPMG, 2018). The study evaluated changes in continuity of care after the implementation of a primary care nurse-led clinic with services offered by a nurse and a collaborating GP, who visited the site bi-weekly (KPMG, 2018). This study revealed improved continuity of care by allowing the community to follow up on health-related issues before seeing a specialist and by acting as a link between the community and other health service providers in the wider region (KPMG, 2018).
Several measures of continuity of care exist, including the Usual Provider of Care index, the Bice-Boxerman Continuity of Care index, the Herfindahl Index, and the Sequential Continuity of Care Index (Pollack et al., Reference Pollack, Hussey, Rudin, Fox, Lai and Schneider2016). Additionally, the measurement of coordination between professionals of different disciplines could use tools such as the relational coordination survey (Gittell, Reference Gittell2011).
Discussion and research implications
Various economic evaluation techniques can determine the value-for-money of OOH-PC while including the three proposed additional effects. Cost-effectiveness analyses (CEA), which measures effects in natural units, has the potential to individually incorporate the three proposed effect measures. Continuity of care, for instance, could be assessed using natural measures such as the change in the proportion of sequential patient visits at the same provider (Roos et al., Reference Roos, Carrière and Friesen1998). It can also be assessed by changes in interprofessional team communication and relationship scores (Hustoft et al., Reference Hustoft, Biringer, Gjesdal, Aßus and Hetlevik2018). The use of monetary CBA, which converts all health-related effects to monetary terms (Drummond et al., Reference Drummond, Sculpher, Claxton, Stoddart and Torrance2015), could incorporate all three additional effect measures. To monetise continuity of care, one can assign a value to its anticipated outcomes, such as avoided hospital use or prompt treatment initiations. However, it is important to avoid the potential issue of double counting when considering these outcomes. Moreover, CBA could potentially be more time-consuming than CEA and is criticised for assigning monetary values to health states (Tsiachristas et al., Reference Tsiachristas, Stein, Evers and Rutten-van Mölken2016). Cost–utility analysis can incorporate all additional measures. By considering health gains through health outcomes such as mortality, the inclusion of continuity of care is feasible (Tsiachristas et al., Reference Tsiachristas, Stein, Evers and Rutten-van Mölken2016). However, as with CBA, caution is needed to prevent double counting. A cost-consequence analysis (CCA) presents a range of outcomes alongside costs; therefore, it could integrate all the proposed additional effects. Due to its clarity, CCA is used to inform decision-making (Mauskopf et al., Reference Mauskopf, Paul, Grant and Stergachis1998) but is criticised for its inability to rank alternative interventions based on their effectiveness (Perkins et al., Reference Perkins, Steinbach, Tompson, Green, Johnson, Grundy, Wilkinson and Edwards2015). Multi-criteria decision analysis (MCDA) is a systematic comparison of different alternatives by considering multidimensional factors (Baran-Kooiker et al., Reference Baran-Kooiker, Czech and Kooiker2018). MCDA could potentially incorporate all proposed effect measures for the evaluation of OOH-PC. However, MCDA presents the challenge of assigning weights to effects based on the preferences of stakeholders within a specific setting (Marsh et al., Reference Marsh, Thokala, Youngkong and Chalkidou2018). It is necessary to further explore the applicability of MCDA approaches to OOH-PC.
Productivity costs often have a strong impact on cost-effectiveness outcomes (Krol & Brouwer, Reference Krol and Brouwer2014). Therefore, whether and how to include them in economic evaluation is a debate that has been ongoing for several years (Krol et al., Reference Krol, Brouwer and Rutten2013). Some argue that the inclusion of productivity costs raises equity concerns, as interventions aimed at the employed produce more favourable cost-effectiveness outputs compared to interventions aimed at the unemployed (Lensberg et al., Reference Lensberg, Drummond, Danchenko, Despiégel and François2013). However, excluding productivity costs due to equity concerns is contested because other cost types, such as medical costs, also discriminate across different population groups, such as between the young and the old (Krol et al., Reference Krol, Brouwer and Rutten2013). To accommodate equity concerns, it has been suggested to report productivity gains in non-monetary units such as days or hours gained or lost (Drummond et al., Reference Drummond, Sculpher, Claxton, Stoddart and Torrance2015). Additionally, equity concerns can be addressed by evaluating productivity gains or losses for the unemployed using shadow prices that consider the opportunity costs associated with unpaid work activities, including household work, shopping, and childcare (Drummond et al., Reference Drummond, Sculpher, Claxton, Stoddart and Torrance2015). On the other hand, there is a lack of standardisation and consensus regarding the methodology for measuring productivity costs (Jiang et al., Reference Jiang, Wang, Si, Zang, Gu, Jiang, Liu and Wu2022; Krol & Brouwer, Reference Krol and Brouwer2014). Recently, recommendations considered the use of instruments that include both paid-work productivity losses and those related to unpaid work (Krol & Brouwer, Reference Krol and Brouwer2014). These include the iMTA Productivity Cost Questionnaire (iPCQ) or the Valuation of Lost Productivity (Krol & Brouwer, Reference Krol and Brouwer2014). The omission of productivity costs in economic evaluations may partly be due to some national health economic guidelines that prescribe a health system perspective (Jiang et al., Reference Jiang, Wang, Si, Zang, Gu, Jiang, Liu and Wu2022). However, many economic evaluations taken from a societal perspective still exclude productivity costs (Jiang et al., Reference Jiang, Wang, Si, Zang, Gu, Jiang, Liu and Wu2022; Krol et al., Reference Krol, Papenburg, Tan, Brouwer and Hakkaart2016). This suggests a potential bias in the selection of cost types, and decision-makers need to be mindful of their inclusion or exclusion whenever the perspective is (partially) societal (Krol et al., Reference Krol, Brouwer and Rutten2013). However, decision-makers should also be mindful of whether health-related quality of life (HRQoL) has already been factored in, as it accounts for the effects of productivity gains or losses on an individual (Jiang et al., Reference Jiang, Wang, Si, Zang, Gu, Jiang, Liu and Wu2022). Including both HRQoL and productivity gains or losses may result in duplicate counting (Jiang et al., Reference Jiang, Wang, Si, Zang, Gu, Jiang, Liu and Wu2022). In response to the debates surrounding the inclusion of productivity gains or losses, there is a suggestion to present two scenarios of cost-effectiveness results, one with and another without productivity (Pritchard & Sculpher, Reference Pritchard and Sculpher2000).
The implementation of OOH-PC on a large scale may face challenges due to shortages in the health workforce (Velgan et al., Reference Velgan, Vanderheyde, Kalda and Michels2023). These shortages already make it difficult to recruit healthcare providers to perform regular contractual hours, let alone out-of-hours (The Scottish Government, 2015). Because of these shortages, a trade-off between regular and out-of-hours is likely. The provision of OOH-PC can, on the one hand, attract healthcare providers for higher pay (Broadway et al., Reference Broadway, Kalb, Li and Scott2017; Longden et al., Reference Longden, Hall and van Gool2018), leaving a gap in regular-hour care. On the other hand, OOH-PC may not interest all providers due to, for example, the impact on their work–life balance (The Scottish Government, 2015). Whether OOH-PC will threaten the sustainability of regular-hour primary care practices remains uncertain. Therefore, the aim of improving access to regular-hour primary care while concurrently improving out-of-hour continuity of care through OOH-PC requires careful balancing and consideration in future research.
Due to the unscheduled nature of OOH-PC, there is a great diversity of care provided compared to regular-hour primary care (NHS: Health Education England, n.d.). As a result, there is a growing need for adapted training and career development to ensure that providers have the right skills to handle the increasingly challenging and complex environment (GP Training: Urgent and Unscheduled Care (Including Out-of-Hours), 2022). These skills include the ability to handle medical, surgical, and psychiatric emergencies out-of-hours, the ability to make appropriate referrals to hospitals and other professionals, to manage personal time and stress, and to maintain personal security and awareness of environmental security risks (Royal College of General Practitioners, 2019; GP Training: Urgent and Unscheduled Care (Including Out-of-Hours), 2022). Additionally, providers need to be informed about the prevailing governance approaches due to the existing links between OOH-PC, ambulance services, and EDs (Royal College of General Practitioners, 2019). Consequently, there is a recommendation for multidisciplinary teams for out-of-hours care (Royal College of General Practitioners, 2019). Therefore, it is essential that resource planning for the primary care workforce considers the degree of training needed to deliver OOH-PC effectively. However, additional training resources may not always be available, which can negatively affect OOH-PC provision.
While OOH-PC serves as a viable alternative to ED care, it has the potential to trigger a supply-induced surge in healthcare utilisation during non-office hours (Longden et al., Reference Longden, Hall and van Gool2018). This influx of patients outside of regular office hours may result in increased demand for healthcare resources, which can lead to additional healthcare expenses from the healthcare payers’ perspective (Morreel et al., Reference Morreel, Homburg, Philips, De Graeve, Monsieurs, Meysman, Lefevere and Verhoeven2022). Moreover, the provision of OOH-PC may not always be a feasible option for healthcare providers due to factors such as higher clinical indemnity insurance costs. In the United Kingdom, for instance, providers face elevated indemnity insurance expenses for out-of-hours services compared to their regular office hours counterparts (NHS England and the Department of Health, 2016).
During the identification of innovative measures for the evaluation of OOH-PC, methodological challenges were encountered. First, given our scope on economic evaluations of OOH-PC, we did not conduct a systematic search for studies that focused only on the effects of OOH-PC, nor for studies that examined economic evaluations of integrated care. However, we made a conscious effort to prioritise the scientific relevance of the sources that we utilised and attempted to mitigate potential biases by using recent systematic reviews and by conducting a thorough snowballing of the references included in the selected contributions. The second challenge concerns the identification of additional effect measures in the current multidimensional and multi-objective framework, which often involves engagement with different stakeholders to develop standardised measures. However, the present study constitutes a tool that can effectively facilitate the establishment, development, and advancement of economic evaluation mechanisms for OOH-PC in line with integrated care initiatives.
The use of additional effect measures can present certain obstacles. Determining the suitability of potential measures for a given situation necessitates an evaluation of their significance. This empirical evaluation can be difficult and time-consuming, as it relies on a comprehensive assessment of previous context-specific evidence regarding potential impacts.
Conclusion
In this paper, we discussed effect measures for conducting broad welfare-gain-driven economic evaluations of OOH-PC by drawing on experience from integrated care programmes. A focus on resource use measures can be too limiting in the OOH-PC domain, where a wide range of outcomes are relevant from the health system and patient perspectives. In this regard, we identified three relevant effects not yet considered in previous economic evaluations of OOH-PC. These are ‘productivity loss’, ‘health promotion and early intervention’, and ‘continuity of care’. This proposal of additional effects is neither comprehensive nor exhaustive but serves to highlight how to broaden the economic evaluation of OOH-PC by considering additional processes and patient outcomes related to the out-of-hours context. Determining what to include or exclude depends on the specific context, considering the evaluation perspective and the strength of existing evidence supporting the significance of an effect measure within that context.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/S1463423624000318
Acknowledgements
None.
Author contributions
Conceptualisation: JP and LW; Methodology: JP, LW, and PB; Formal analysis and investigation: JP; Writing – original draft preparation: JP and LW; Writing – review and editing: LW, SM, DD, HP, PB, and VV; Funding acquisition: LW; Supervision: LW.
Funding statement
JP and LW are supported by the University Research Fund (BOF) of the University of Antwerp. PB acknowledges funding from the Antwerp Study Centre for Infectious Diseases (ASCID) and the Methusalem-Centre of Excellence consortium VAX-IDEA. These funders had no role in study design, data collection and analysis, decision to publish, or manuscript preparation. The other authors declare that no financial support was received for the conduct of this research or the preparation of this article.
Competing interests
All authors report that they do not have conflicts of interest.
Ethical standards
Not applicable.
Open access
