Introduction

It is well established that the use of out-of-hours telephone triage services (OOH-TT) services has large potential for managing unplanned and OOH healthcare needs presented by the general population [1, 2]. However, little is known with regards to the quality of these services in more vulnerable segments of the population such as older adults [1, 3, 4]. Specifically, it is well known that older adults (over the age of 65) are especially difficult to triage over TT services as they frequently present with atypical symptoms of common health concerns. At the same time, these segments of the population not only make up a large proportion of all unplanned care users [5, 6] but they are also known to be at the highest risks for experiencing adverse health events and safety incidents due to inappropriate care transfers [7] and in cases of inappropriate triage, mis-triage [8, 9].

As a first step towards addressing this issue, a recent publication by Wardlow et al., (2022) aimed to better understand how telehealth-based services should be delivered to appropriately meet the needs of older adults seeking care [10]. However, existing findings have largely focused on provider-focused principles and guidelines for promoting age-friendly telehealth practices [10]. Furthermore, some additional recent efforts also found that developing or adapting existing protocols to better fit population-specific criteria for triage in older adults may be effective for reducing rates of mis-triage in older adults. To date, these efforts have only been examined in specific and in-person triage tools [11,12,13] and it is not yet clear how well they may apply within the larger context of medical dispatch protocols used by OOH-TT services.

In Belgium, there is a national OOH-TT service which aims to provide medical assistance to patients during evenings, weekends and holidays. Depending on the region, patients calling this number are re-directed to the local on-call general practitioner (GP) for an appointment or to a non-medical operator specialized in triage using the Belgian Manual for Medical Regulation (BHMR) for guidance on further medical assistance (more information available at: www.1733.be (Dutch, French, German and English) [3, 14, 15]. Overall, the use of these medical dispatch protocols have been shown to be safe and effective for use in the general population [16,17,18]. However, lower quality of triage (particularly related to concerns of safety assessed via the frequency of adverse events, errors, and patient hospitalization rates) has been shown to be associated with triaging older adults over the telephone [19,20,21]. One potential problem associated with the use of existing protocols, such as the BHMR is the generic nature of the protocols. It is typically the case that the same protocols are used for triaging all segments of the population (i.e. including children and older adults who are known to be at higher risks for lower quality triage) with some limited considerations of age and medical history for vulnerable segments of the population. However, by incorporating key population-specific items within the protocols, the potential effectiveness, efficiency and overall quality of managing calls involving older adults in the population may be improved. As such, this Delphi study aims to achieve expert consensus regarding key items to be addressed by non-clinical operators using the BHMR medical dispatch protocols to manage OOH-TT services for calls involving older adults seeking non-urgent unplanned care across Belgium.

Methods

Study design

A three-part classic e-Delphi study was conducted to address the aim of our research through an iterative process which seeks to combine diverse expert opinions into group consensus based on the guidance of Keeney et al. (2011) and recommendations for Delphi study reporting guidelines in the health science sector by Spranger et al. (2022) [22,23,24].

Expert panel recruitment and selection

A purposive sample of N = 16 experts specialized in out-of-hours unplanned care and/or older persons across Belgium were recruited for participation as panelists in our study via the network of the research team. Identified experts were first contacted via the telephone and were given some brief information regarding the study [22]. Panelists who agreed to participate were then contacted electronically (via e-mail) with a formal invitation letter and a complete study information sheet. Panelists were also informed that their participation is voluntary and that they were free to withdraw from the study at any time.

Eligibility criteria

Eligibility criteria for panelists included experts with at least 2 years of prior professional experience relevant to unplanned care for older adults and/or telephone triage including emergency physicians, nurses, GPs, geriatricians, telephone triage operators, patient representatives and medical directors of OOH-TT services across Belgium.

BHMR protocols

All calls received by the 1733 service are managed by non-clinical operators who are trained to use the BHMR protocols; a set of computer-software integrated medical protocols (more info: https://www.health.belgium.be/nl/belgische-handleiding-voor-de-medische-regulatie#anchor-25165) [15, 25]. The use of the BHMR protocols by 1733 service operators is mandatory. The protocols examined in this study includes version 4.0 (published on 5 June, 2019) which consists of 56 different protocols that can be prompted by entering specific key words and descriptions of medical symptoms [25]. Although these protocols are largely generic, some considerations are made for specific cases involving children between the ages of 0–14, adults aged 75 and over and patients with chronic illnesses.

Data collection procedures

This Delphi study consisted of three parts. The first two parts of the study were held electronically and the third part was held via a face-to-face consensus meeting with panelists. All surveys were pilot tested before administered to panelists. During part one and part two, panelists were asked to fill out online surveys using Qualtrics software.

Part one (survey one)

Following consent to participate, participants were sent an e-mail link to complete a first online survey. The first part aimed to collect demographic information of panelists. The remainder of the survey was composed of open- and closed- ended information regarding panelists beliefs surrounding current protocols used by the 1733 service for triaging older adults seeking OOH unplanned non-urgent care. Specifically, panelists were presented with the list of protocols that are available at the 1733 service and were asked whether they believe the protocol topics considered in the existing list to be sufficient for triaging older adults as well as to self-report (based on their expert experiences) on any relevant protocol topic they believed to be missing from the existing list. Panelists were then presented with the 10 most frequently used protocols for triaging older adults via the 1733 OOH-TT service between January 1 to December 31, 2019. Complete algorithms, flowcharts, and frequency distributions were made available to enable a detailed analysis of each of the individual protocols (see Table 1) [25].

Table 1 Summary of protocol topics available for use within the 1733 OOH-TT service
Full size table

After review of each protocol, panelists were asked to provide their opinion regarding the frequency of use and content of protocols for triaging calls involving older adults seeking unplanned care. Panelists were asked whether each protocol was used as frequently as they would expect and whether they believed that any modifications should be made to the content (i.e. the keywords, interrogation questions, degree of urgency, flowchart, or other factors related to the algorithm) within a specific protocol to improve the management of calls involving older adults.

Part two (survey 2)

Panelists who completed part one of the survey were invited to partake in the second survey. The results of the first survey were used to form the basis of the second survey. Panelists were presented with a summary of (de-identified) responses that were submitted. Using a four-point Likert scale, they were asked to rate each of the submitted responses (considered as items) as “not relevant at all”; “somewhat relevant”; “quite relevant”; or “extremely relevant.” An open comments box was provided for panelists to further specify each of their responses (i.e. to elaborate on responses, or to add further suggestions and/or comments). Once all the responses were submitted, ratings for each item were analyzed for consensus and were re-submitted to panelists as necessary.

Part three (consensus meeting)

The consensus meeting aimed to achieve final consensus on items identified for each topic rated during part one and part two. The consensus meeting was held using a hybrid format and all panelists were invited to join the meeting in person or online based on their availability.

Predefined criteria for consensus

In line with expert recommendations, level of consensus was defined a priori to be reached when at least 70% of the expert panel members agreed (positive consensus defined as the item being generally scored as “quite relevant” or “extremely relevant”) or disagreed (negative consensus defined as the item being generally scored as “not relevant at all” or “somewhat relevant”) regarding the relevance of each item proposed by panelists within a survey [22]. If consensus was not reached following first-time analysis of responses obtained for an item, the survey was re-administered for a second round and panelists were requested to re-submit their responses based on group results obtained during the previous round. If consensus was not reached by the second round, then the item was to be further discussed for a final time during the round table consensus meeting.

Data analyses

Responses were quantitively and qualitatively analyzed over several rounds until expert consensus was found. Descriptive and thematic analyses were used to aggregate responses. Qualtrics was used for data gathering and analysis while SPSS Statistics (version 24.0.0.0) was used for analysis purposes.

Ethical considerations

This project received ethical approval from the Medical Ethics Committee UZ KU Leuven (study reference number S64473).

Results

Data was collected between March 31, 2022 until December 8, 2022. A total of N = 16 panelists were invited to complete survey one, out of which 12 responses were obtained followed by 10 responses for survey two. Eight panelists were present for the consensus meeting (see Fig. 1). The expert panel was comprised of n = 1 medical director of a call center, n = 2 GPs, n = 2 emergency physicians, n = 2 geriatricians, n = 3 telephone triage operators, n = 2 nurses, n = 1 other (telephone triage center coordinator) (panelists could select more than one category for multiple professional backgrounds and/or area of work). The majority of panelists worked in the field of academia (n = 6), or in a clinical setting (n = 7) and half (n = 6) had between 11 to 30 years of work experience. (see Table 2).

Fig. 1
figure 1

Timeline of Delphi study procedure

Full size image
Table 2 Expert panel characteristics
Full size table

Survey 1

In total, n = 8 panelists responded that they believe there to be several self-reported missing protocol topics which are not currently covered by the existing BHMR protocols but that should be added to improve the quality of triage for calls involving older adults ≥65 years of age seeking unplanned care. The following categories of missing protocol topics were reported to include: Skull Trauma, Generalized Fever In Adults, Medication, Neurological Problems Other than stroke, fever without an infection focus, Fall-Related, Neglect, Acute Confusion, Non-Traumatic Joint Pains, Reduced Functionality and Decline of General Conditions. Overall, at least half or more panelists also reported that all of the top 10 most frequently used protocols for triaging older adults via the 1733 OOH-TT service were used as frequently as they would expect (see Table 3). At the same time, at least half or more panelists felt that some modifications should be made to the content of all the 10 protocols to improve the overall management of calls involving older adults (see Table 4 and Table 5).

Table 3 Summary of 10 most frequently used protocol topics presented to panelists and frequency of use
Full size table
Table 4 Summary of panelist responses on whether content related modifications to be made for each protocol
Full size table
Table 5 Summary of panelist responses supporting content related modifications to be made for each protocol
Full size table

Survey two and consensus meeting

The top 10 most frequently used protocols for triaging older adults were assessed and discussed by panelists during survey two and the consensus meeting (see Table 6 below as an example with complete details available in Supplementary Tables 1 (survey two results) and 2 (consensus meeting results)). In total, consensus (positive or negative) was found for all items on two of the protocols (namely “Trauma” and “Cardiac arrest- deceased”) during survey two. Consensus was not found for the remaining eight protocols.

Table 6 Example of findings for protocol on “Breathing Difficulties”
Full size table

The remaining eight protocols were further discussed during the consensus meeting. One protocol (namely, the protocol on “Non-traumatic back pain”) was excluded from discussion because several panelists commented that the items addressed in this protocol were already addressed in the updated version of the BHMR protocols (version 5.0) which was released for use by the 1733 service during the data collection period of this study [26]. Full consensus was found for all items on the remaining protocols except the protocol “Unwell for no apparent reason.” Overall, panelists stressed that a combination of patient related factors must be addressed to provide high quality triage for adults seeking non-urgent unplanned care over the telephone (in addition to age) including patient profiles, medical habits and history, level of support from informal caregivers, known comorbidities and frailty status. Additionally, panelists also highlighted that further elements such as appropriate operator training and feasibility of integrating revised protocols (i.e. more population-specific protocols) must also be considered before implementation of protocols within the call center. The following specific outcomes were obtained for each protocol item (see details in Supplementary Tables 1 and 2):

Protocol on “Trauma”

A total of four items were addressed for the protocol on “Trauma”. Positive consensus was found for all items including fall in older adults, location of injury, reason for fall, and level of urgency.

Protocol on “Cardiac arrest- deceased”

A total of two items were addressed for the protocol on “Cardiac arrest – deceased”. Positive consensus was found for all items including nature of death and practical questions to better evaluate dispatch to the Mobile Emergency Group.

Protocol on “Breathing Difficulties”

A total of five items were addressed for the protocol on “Breathing Difficulties”. Two items including fever and symptoms related to heart failure were excluded from the round table discussion as these suggestions were already included in the new version of the BHMR protocols [26]. Positive consensus was found for the remaining items including symptoms specific for older adults/geriatric patients, coughing and coloured sputa.

Protocol on “Non-Traumatic Abdominal Pain”

A total of six items were addressed for the protocol on “Non-Traumatic Abdominal Pain”. Four items including abnormal aorta, aneurism alertness, diarrhea and pain severity were excluded from the round table discussion as these suggestions were already included in the new version of the BHMR protocols [26]. Positive consensus was found for the item hydration level and negative consensus was found for the item history about previous aneurysm.

Protocol on “Urogenital Problems”

A total of five items were addressed for the protocol on “Urogenital Problems”. Positive consensus was found for the items on problems with use of other medical devices in older adults ≥ 65, urinary incontinence, urinary overflow and possibility to send patient a GP for home visit. Negative consensus was found for the item removal of questions related to temperature and urinary retention from this protocol and added to “confusion” protocol instead.

Protocol on “Cardiac Problem Other Than Thoracic Pain”

A total of nine items were addressed for the protocol on “Cardiac Problem Other Than Thoracic Pain”. Two items including addition of swollen legs complaint to protocol for hot/cold limbs and possibility to push dimple into swollen leg (possibly indicating heart failure) were excluded from the round table discussion as these suggestions were already included in the new version of the BHMR protocols [26]. Of the seven remaining items, positive consensus was found for the items adapting current 1733 protocol for “syncope” to broader guidelines for “transit loss of consciousness and possibility to push dimple into leg given shortness of breath”. Negative consensus was found for the items on swollen leg in older adults ≥ 65, shortness of breath in older adults ≥ 65, irregular heart palpitations in older adults ≥ 65, respiratory problems in older adults ≥ 65, and orthostatic hypotension.

Protocol on “Nose-Throat-Ear-Tooth”

A total of four items were addressed for the protocol “Nose-Throat-Ear-Tooth.” Positive consensus was found for the items problem with swallowing foods and clarification of GP’s role for patients with tooth problem. Negative consensus was found for the items tooth problems for all ages (separate from nose-throat-ear protocol) and location of pain.

Protocol on “Hot or Cold Limb”

In total, three items were addressed for the protocol “Hot or cold limb.” Two items including swollen leg joints for all ages and integration of swollen legs or joints complaint intro current protocol for hot or cold limb were excluded from the round table discussion as these suggestions were already included in the new version of the BHMR protocols [26]. Positive consensus was found for the remaining item on pain severity and changes in pain severity when lifting leg up and down.

Protocol on “Unwell for No Apparent Reason”

In total, eight items were addressed for the protocol “Unwell for no apparent reason.” Positive consensus was found for the items on unwellness (for no apparent reason) in older adults ≥ 65, fever without focus in older adults ≥ 65, acute confusion in older adults ≥ 65, pain and mobility in older adults ≥ 65 and temperature and urinary retention, and for the item atypical symptoms in older adults ≥ 65 and dysregulated blood pressure. No consensus was found by panelists for the item on voluntary stopping of eating and drinking.

Discussion

This study presents one of the first scientific efforts to better understand and find expert consensus regarding key items to be addressed by non-clinical operators using computer-software integrated medical dispatch protocols to manage OOH-TT services for older adults seeking unplanned care. Our interdisciplinary expert panel found strong consensus for the majority of the 10 most frequently used BHMR protocols for triaging older adults within the larger 1733 service in Belgium. Findings showed that the majority of experts felt that most protocols used for managing calls involving older adults were used as frequently as they would expect. However, panelists also agreed that some content related modifications should be made for keywords, interrogation questions, degree of urgency and/or flowcharts used for the algorithms of the top 10 most frequently used protocols to improve the overall quality of triage for older adults.

Overall implications of our research suggest that in addition to age of patients, a combination of patient related factors must be addressed to provide high quality triage for adults seeking non-urgent unplanned care over the telephone. Additionally, some further elements such as appropriate operator training and feasibility of implementing more population-specific protocols must also be considered within the call center setting. Such considerations must be thoroughly evaluated before implementation of any new or revised protocols within practice.

Specifically, our findings shows that overall, a combination of patient related factors must be addressed to provide high quality triage for adults seeking non-urgent unplanned care over the telephone (in addition to age). However, further elements such as appropriate operator training and feasibility of implementing more population-specific protocols must also be considered. This study presents a useful step towards identifying key items which must be targeted within the larger scope of providing non-urgent out-of-hours telephone triage services for older adults seeking non-urgent unplanned care.

Despite known concerns of triaging older adults via OOH-TT services, currently existing methods are still largely catered for providing remote medical assistance to less vulnerable segments of the population. It is not yet clearly known how protocols used across various services may be adapted and/or optimized for use in older adults [3, 17, 18, 22, 23, 25, 27]. One plausible explanation for this concern as discussed during the round table consensus meeting is the understanding that it is especially difficult (and perhaps not possible) to create age specific protocols for older adults in a way that is similar to children in which specific ages are often lined with specific pathologies [28]. The importance of addressing triage for older adults by using methods in which operators prioritize assessing a combination of factors (including patient profiles, medical habits and history, level of support from informal caregivers, known comorbidities and frailty status) rather than solely evaluating differences in patient age was embedded throughout the discussion.

Another key point that was addressed by panelists is the feasibility of implementing new/revised protocols within real world settings. Panelists were largely opposed to the creation of new protocols for providing OOH-TT services to older adults due to time and clinical restrictions of the telephone operators. While the importance of providing adequate training for operators was largely agreed upon, the key focus mentioned by panelists was to consider the addition and integration of specific questions that may capture different dimensions of triage-related questions for calls involving older adults within currently existing protocols. Despite a lack of existing literature, [29] discussions on evaluating quality of training in OOH-TT operators were also highlighted to be important and in line with the surrounding literature base which has shown that operator related characteristics (such as clinical background, length of experience, type of previous experience and flexibility of decision-making regarding level of urgency) may be key factors which influence triage outcome [3, 4, 30,31,32,33]. It is therefore crucial to ensure that operators are given adequate training which can in turn, ensure consistency and high-quality dispatch of patients using OOH-TT services. Supplementing the right questions alongside well-trained operators who are able to gather the right information during a call was stressed by panelists to be a key combination for enabling the delivery of high-quality OOH-TT services for older adults.

Several strengths and limitations must be considered in interpreting the overall study results. First, it is important to note that our conclusions provide insight into discussions and consensus drawn by one particular expert panel and do not necessarily reflect objectively correct answers which must be used for delivering OOH-TT services to older adults. It is also important to note that different reflections and ideas regarding quality of triage may be produced by panelists consisting of other experts [27, 34]. As such, the results of this study must be interpreted with caution. Furthermore, level of consensus in this study was defined to be reached when at least 70% of the expert panel members agreed or disagree regarding the value of a specific item. Although this is within the standard range of measuring consensus [22], it must be noted that an alternative definition of level of consensus may yield different results. A key strength of this Delphi study is that the process inherently allows flexibility and enables panelists to assess their judgements through several rounds of the research process [30]. Furthermore, it also plays a vital role for linking together existing knowledge and areas of agreement and disagreement within this field of research through a structured process that can allow us to ultimately have an impact on improving the quality of OOH-TT protocols for use in older adults [31, 35]. A limitation that must also be considered is that no views of patient representatives were included. Despite several attempts to reach out to various eligible contacts, we were not able to include any panelists who could represent the views of older adults who have experience with the use of OOH-TT services. Finally, it is important to mention that although we intended for a national representative Delphi panel (and has therefore reached out to panelists from three regions in Belgium – Flanders, Wallonia, and Brussels), we were presented with a high dropout rate of French speaking panelists leaving us with a small expert panel of 8 participants for the final consensus round. Furthermore, the study team moderating the development of the questionnaires and consensus meeting were all based in a university and/or clinical settings located in Flanders. As such, the results are therefore biased towards expert views of panelists from the Flemish region (i.e. Flanders) of Belgium. However, given the rather flexible nature of a Delphi study to determine sample size and heterogeneity based on the larger study aim and possible range of opinions, we believe that that concern of mathematical/statistical bias is limited.

Conclusion

This Delphi study presents one of the first scientific efforts targeting the improvement of TT protocols used for triaging older adults seeking non-urgent out-of-hours unplanned care. Overall, this research is a first step towards addressing current gaps in the existing topics of the BHMR and for identifying important content related benchmarks which must be targeted within the larger scope of OOH-TT services. Our findings showed that overall, a combination of patient related factors must be addressed to provide high quality triage for adults seeking non-urgent unplanned care over the telephone (in addition to age). However, further elements such as appropriate operator training and feasibility of implementing more population-specific protocols must also be considered. This is a useful step towards identifying key items which must be targeted within the larger scope of providing non-urgent out-of-hours telephone triage services for older adults seeking non-urgent unplanned care. We hope that these findings will eventually be used for guiding quality improvements within the BHMR protocols used within the context of the 1733 service but also for similar OOH-TT settings internationally.