The implementation of the nationwide out-of-hours phone number 1733 in Belgium: analysis of efficiency and safety

Background: Belgium has a problem with inappropriate use of emergency services. The government installed the number 1733 for out-of-hours care. Through a dry run test, we learned that 30% of all calls were allocated to the protocol ‘unclear problem’. In only 11.9% of all cases, there was an unclear problem. Methods: The study aimed to determine whether the adjusted protocol ‘unwell for no clear reason’ led to a safer and more efficient referral and to evaluate the efficiency and safety of the primary care protocols (PCPs). The study ran in cross-sectional design involving patients, General Practitioner Cooperatives and telephone operators. A random sample of calls to 1733 and patient referrals were assessed on efficiency and safety. Results: During 6 months in 2018, 11 622 calls to 1733 were registered. Seven hundred fifty-six of them were allocated to ‘unwell for no clear reason’, and a random sample of 180 calls was audited. To evaluate the PCPs, 202 calls were audited. The efficiency and safety of the protocol ‘unwell for no clear reason’ improved, and safety levels for under- and over-triage were not exceeded. The GP’s judged that 9/10 of all patient encounters were correctly referred. Conclusion: This study demonstrated that the 1733-telephone triage system for out-of-hours care is successful if protocols, flow charts and emergency levels are well defined, monitored and operators are trained.


Introduction
In Belgium, there is problem with the inappropriate use of emergency services (Brasseur et al., 2019a;Philips et al., 2010a). In 2012, we registered 290 emergency contacts per 1000 inhabitants, which is far above the average in comparable nations (Carret et al., 2009;Philips et al., 2014;Uscher-Pines et al., 2013). Besides, 71% of the patients are self-referrers. Since 2003, 80 general practitioner cooperatives (GPCs) were established offering out-of-hours care in a primary care context. Against all expectations, the number of emergency contacts did not decrease (Philips et al., 2010b). The opening hours of a GPC are restricted, there are no formal agreements between GPC and ED (emergency department) and, above all, patients are more confident with the service of an ED.
A nationwide study demonstrated that there is a need to develop an effective triage system improving the out-of-hours referring process . From a review, we also know that the introduction of telephone triage systems is a promising intervention to reduce inappropriate ED visits (Van den Heede & . The overuse of the ED is also encouraged by free access (Carret et al., 2007). This inappropriate use of the ED undermines the functioning of GPCs for out-of-hours care (O'Kelly et al., 2010;Huibers et al., 2013;Philips et al., 2010b). The government installed the nationwide number 1733 for unplanned, non-live-threatening out-of-hours care. An operator directs the caller to the appropriate level of care: three levels of ambulance intervention, urgent or planned referral to out-of-hours primary care services or to planned care (https://www.health.belgium.be/en/ health/need-call-doctor-call-1733). These operators are trained to follow digital protocols. These protocols are the result of a collaboration between academic partners, the Federal Public Service Health, experts in general practice and in emergency medicine.
To evaluate the efficiency and safety of the use of the initial protocols, a dry run test phase was implemented (Van der Mullen et al., 2017). All patients were asked to call the 1733 emergency number in the presence of a researcher. The operator interrogated the patient following a flow chart and referred according to the chosen protocol to the appropriate care level. In 30% of all calls, the operator assigned the call to the protocol 'unclear problem'. In 11.9% of all these cases, there was indeed an unclear problem (eg, communication, multiple complaints). In all other cases, there was no appropriate protocol available or there was a problem on operator level.
The initial protocols were adjusted and completed with specific primary care protocols (PCP) (attachment 1). The operators received instructions and dry run training by GP-and ED-experts to improve the use of the new protocols. The protocol 'unclear problem' was renamed to 'unwell for no clear reason'. The Belgian Counsel for Urgent Healthcare approved this final version of the protocols.
In international literature, there is a lack of research on the implementation of a central, nationwide number for unplanned out-of-hours care. Since many health care systems are confronted with inefficient patient flows in unplanned or emergency care, an implementation research on this topic might contribute to improvement of this particular type of care (Ghazali et al., 2019;Young et al., 1996).
Therefore, the aim of this research was to determine whether the adjusted protocol 'unwell for no clear reason' led to a safer and more efficient referral of calls to the appropriate care level and to evaluate the efficiency and safety of the specific PCPs.

Design
The study was performed in a cross-sectional design. The earlier protocol 'unclear problem' was adjusted and renamed to 'unwell for no clear reason'. The operator chooses this protocol when the caller reports complaints that do not fit into any other protocol.
A protocol is considered as efficient when it leads with the appropriate care level: three levels of ambulance intervention, urgent or planned referral to out-of-hours primary care services or to planned care. A protocol is considered safe when the rate of under-triage is <15%. Both decisions on efficiency and safety were the result of a consensus between experts, the Intermediary Report Project 1733, guidelines and national research reports The 1733 operator follows a flow chart, instructed by the appropriate protocol, to question and to refer the caller. Each flow chart starts with a set of medical terms. Each protocol integrates different levels of severity leading to a particular level of care.
To determine whether the adjusted protocol 'unwell for no clear reason' leads to a safer and more efficient referral, a set of calls was audited by two researchers. A computerized random selection was made of all calls recognized as 'unwell for no clear reason' between May 4 and June 26, 2018. For the collection of data, a template inventorying time of day, patient history, protocol choice and level of care structured the assessment. The researchers individually assessed the calls, marked their protocol of choice and motivated this choice: correct protocol, missing protocol, fail to recognize keyword, fail of the operator, communication problem and impossible to refer. At this stage, the researchers were not in contact with each other.
The data were qualitatively analyzed according to the 'Grounded Theory approach'. For the qualitative part, the researchers (n = 5) discussed the results of the assessment of calls in different rounds. In case of disagreement with the operators, the call was labeled as 'discordant'. In case the researchers did not reach consensus, the call was presented to the advisory group. If the operator initially referred to a lower care level, the call was labeled as 'under-triage' (with the number of levels below). If the operator initially referred to a higher care level, the call was labeled as over-triage.
In the second research question, the efficiency and safety of the specific PCPs were studied following the same procedure as described above. For each of the 18 PCPs, 15 calls were randomly selected by randomization software from January 1 until June 30, 2018. All patients referred to these protocols were assessed by the GP on duty (Weekends July 27-30, 2018 and February 8-11, 2019). The assessment took place when the patient presented at the GPC and before the consult started. The GP asked the patients the reason for encounter and then selected the level of care. This option was compared to the option the operator picked. The outcome of this assessment did not affect the provided care.
Efficiency of referral by the operator was expressed as the level of concordance between the operators' and the GPs' referrals. Safety was measured by the occurrence and the level of under-triage.

Ethical approval
This study was approved per research question by the Medical Ethical Committee of the University Hospitals of the KU Leuven (041C55FA505A80, 04324FEAA15C80 and 047066BAC85E80).

Efficiency and safety of 'unwell for no clear reason'
For the period of January 2018 to June 30, 2018, 11 622 calls were registered. 756 (6.6%) of these calls were allocated to the protocol 'unwell for no clear reason' of which a random sample of 180 calls was audited. Sixteen calls were excluded because of technical errors. In total, 164 calls were included.
The researchers assigned the label 'fail to recognize keyword' to 51 calls (31%), the label 'correct protocol' to 44 (27%) calls and the label 'missing protocol' to 37 (23%) calls. They assigned the label 'fail of the operator' to 23 (14%) calls. Nine (5%) calls were labeled as 'communication problem'. No call was labeled as 'impossible to refer' (Table 1).
When assessing the efficiency of allocation to a protocol, 104 (64%) calls were labeled as 'unwell for no clear reason'. Eight calls (5%) were allocated to the protocol 'dizziness', six calls to the protocol 'skin problems' and five (3%) calls to 'airway problems' (Table 1).
When assessing the safety of referral, the researchers referred 111 (68%) calls to the same level of care as the operator. 36 (22% of total) of all calls were recognized as 'over triage' and 17 (10% of total) as 'under triage' (Table 1).
The allocation to an inappropriate protocol led in 25 (15%) calls to a discordance of care level between the operator and the researcher. In 12 calls, the referral was labeled as 'over-triage' and 13 calls were recognized as 'under triage' (Table 1).
The efficiency and safety of (referral to) the specific PCPs For each of the 18 PCPs, 15 calls were randomly selected for assessment (Table 2). For seven protocols, <15 calls were available. The researchers assessed 202 calls. The researchers assessed 126 (62%) calls as 'correct protocol', 13 (6%) as 'missing protocol', 48 (24%) as 'fail to recognize keyword', 12 (6%) as 'fail of the operator' and three (2%) as 'communication problem'. To determine the efficiency of the PCP, the researchers assessed the allocation to a protocol. 76 (37%) calls were allocated to another protocol: 16 (8%)
To determine the safety of referral to the PCP, the researchers assessed the allocation to the level of care (Table 2). 175 calls (87%) were allocated to 'out of hours primary care services', 19 calls (9%) to 'urgent out of hours primary care services', 7 calls (3%) to 'ambulance' and 1 (1%) call to 'planned care'. The researchers referred 46 (23%) calls to another care level: in 22 (11%) calls, there was under-triage and in 24 (12%) over-triage. The researchers disagreed about the level of care in 10 calls. These calls were audited and discussed until consensus: two calls remained undetermined due to a language issue and missing administration. The researchers labeled 100 (50%) calls as 'correct protocol and correct care level' and 27 (13%) calls as 'inappropriate protocol and correct care level'. 57 (28%) calls were labeled as 'incorrect protocol' but correct care level. In 18 (8%) cases, the researchers labeled a call as 'inappropriate protocol and inappropriate care level'. Nine (6%) calls were referred to a lower care level than appropriate and seven (3.5%) to a higher level.
To determine the safety of referral to a care level using the PCP, 325 allocated patients received an assessment by the GP (Table 3). Seven assessments were incomplete, and 68 assessments were completed after the consultation. In total, 250 assessments were included.
After assessment by the GP, 236 (94.4%) calls were correctly referred. In two (0.8%) cases, the patients needed referring to a higher level of care and in 12 (5%) cases to a lower level of care.

Discussion
The efficiency and safety of the protocol 'unwell for no clear reason' improved after adjusting the protocols and adding PCPs. The number of calls referred to this protocol decreased as compared to the first trial. The concordance rate between allocation of a call to this protocol by operator and researcher was 2/3, which was higher than in the test phase. The rate of under-triage by allocation to 'unwell for no clear reason' was 1/10. By implementing new PCPs, there was a halving of allocation to 'missing protocol'. The concordance of allocation to the new PCPs by operator and researcher was 2/3, but the safety levels for under-and over-triage were not exceeded. The GP's on duty judged that over 9/10 of all patient encounters were correctly allocated. In <1% of all referrals, the GP marked an under-triage.
In one-third of the referrals to 'unwell for no clear reason', the call was labeled as 'fail to recognize key word'. This rate was higher than that in the test phase and could be explained by the lack of experience with the new protocols. Training of the operators is therefore very important (Hansen & Hunskaar, 2011;Huibers et al., 2011).
By improving the PCPs, there was a manifest decrease of the number of calls assigned to 'missing protocol'. When assessing the allocation of calls to the protocol 'unwell for no clear reason', the researchers now lacked the protocols 'fever adult', 'breast problems' and 'hypertension'. By developing and implementing these protocols, the number of allocations to 'unwell for no clear reason' could further decrease (Smits et al., 2016;Brasseur et al., 2019b). In addition, researchers assessed half of the allocations to the PCP as correct for protocol and care level, and this rate could benefit from adding new PCP. The implementation of PCP seemed successful since operators allocated more than 9/10 of all calls to the GP guard post. Only 1% of all calls was referred to planned care. This number is low and probably due to a lack of experience with this newly introduced care level Plat et al., 2018;Morreel et al., 2019).
The main reasons of discordance between operator and researcher in allocation to the PCP were a fail of the operator and a fail to recognize the keyword. This was particularly the case when the age of the caller was important to allocate to a protocol. Operators often failed to question the context and the state of emergency. Time, experience and training were lacking present. A digitized algorithm could support the operators (Smits et al., 2016;Smits et al., 2017;Brasseur et al., 2019b). Operators were confronted with dissatisfied callers who expected to receive a medical advice (Giesen et al., 2007;Huibers et al., 2011;Huibers et al., 2012). Public information campaigns are necessary to align the objectives of the 1733-system and the public expectations Philips et al., 2019). Strikingly, the operators only once opted for the care level 'planned care', and the researchers also rarely opted this care level. Operators are not familiar with this newly introduced care level and callers not used to referral to postponed care. Language barriers and missing keywords in protocols were also reasons for incorrect referral. Probably, this is a point of attention that needs evaluation on regular base (Huibers et al., 2011;van der Straten et al., 2012).
The rate of under-and over-triage was acceptable as appeared by re-listening the calls and by pre-consultation assessment at the guard post. Over-triage was more likely than under-triage and mainly due to referral to care level 'out of hours primary care services' instead of referral to 'planned care'. Unlike over-triage, under-triage may result in an unsafe situation (Huibers et al., 2011;Plat et al., 2018;Brasseur et al., 2019b). Under-triage was caused by referral to 'out of hours primary care services' instead of 'urgent home visit'. In two cases, there was an unacceptable under-triage by referring the patient to 'out of hours primary care services' instead to 'ambulance'.
The strength of this research lies in the fact that it started in a dry run phase and, after evaluation and adjustment, continued in a real-life setting. Second, the protocols are now fully used in the 1733 operating center. The researchers (GP's) took part in the development and fine-tuning of the protocols during the first test phase of the implementation of the protocol.
An important limitation of the study is that the assessment of calls was based upon the competence of the researchers. Indeed, the researchers were GPs, while the majority of the operators do not have a medical background (Plat et al., 2018). Although a thorough reviewing and revision of the results was performed, misinterpretations cannot be ruled out.

Conclusion
This study demonstrated that the 1733-telephone triage system for out-of-hours care is successful if protocols, flow charts and emergency levels are well defined, continuously monitored and operators well trained. Further research should focus on the rate and conditions of under-and over-triage to improve the competences and skills of the operators. In practice, selection, education and training of operators are indispensable to guarantee safety and cost efficiency.