Introduction

Telementoring provides a platform for professionals with specific content knowledge and expertise within a particular organisation/context to collaborate, share and empower colleagues internally and externally (Agley et al., 2021; Bachynsky, 2020; Barbera et al., 2017; Christian & Andreas, 2019; Gegenfurtner & Ebner, 2019; Hauer & Quill, 2011). The virtual nature of telementoring overcomes traditional boundaries of geography, sector, professional discipline and seniority, and other siloes to integrate the way workforces can learn, seek advice and support, and consequently use newly acquired knowledge to enhance their service provision capabilities (Arora et al., 2020; De Witt Jansen et al., 2018; Gleason et al., 2020; Joshi et al., 2020; Katzman et al., 2020, 2021a, 2021b; Lewiecki et al., 2017; McPhillips et al., 2021; Nhung et al., 2021; Tosi et al., 2020). While telementoring is not a new concept, the contemporary implementation of telementoring models such as Project ECHO is a disruptive innovation which effects change within the organisation adopting the model for the first time. Organisations must be able to measure the change achieved by their implementation of telementoring innovations to benchmark for quality improvement and investment decision-making.

The COVID-19 pandemic has necessitated for many organisations to quickly adopt telementoring models to provide virtual learning and workforce development solutions in response to the rapidly changing landscapes across many sectors (Katzman et al., 2021a, 2021b). Telementoring emerged in the healthcare sector, evolving from telemedicine—or delivering direct patient care remotely via videoconferencing and teleconferencing (Singh et al., 2016), and the concept has since been replicated in a variety of non-healthcare sectors.

Project ECHO is a licensed and trademarked telementoring model which can be used to create virtual knowledge networks, or communities of practice, and incorporates case-based learning strategies from medical education and theoretical frameworks including Social Cognitive Theory, Situated Learning Theory, and Community of Practice Theory (Socolovsky et al., 2013; Wenger et al., 2002). The ECHO model™ was developed in 2003 by Professor Sanjeev Arora at the University of New Mexico (UNM) in the USA, as a platform for both improving healthcare service delivery and patient outcomes in treating Hepatitis C (Arora, Kalishman, et al., 2011). In 2011, UNM demonstrated that Project ECHO supported primary care providers to achieve equitable health outcomes in managing patients with Hepatitis C as those treated exclusively in tertiary hospital settings (Arora, Thornton, et al., 2011). It was highlighted that where geography prevented equitable access to high-quality care, in particular specialist care, the ECHO model overcame this barrier by connecting rural and remote providers with metropolitan-based experts (Arora, Thornton, et al., 2011). Thus, Project ECHO achieved positive health outcomes for patients by facilitating access to enhanced healthcare services conveniently in their local communities (Arora, Kalishman, et al., 2011).

Since the ECHO model’s initial uptake within the healthcare sector, it has grown to over 700 licensed organisations globally across 55 countries and beyond the original Hepatitis C focus to include an increasing variety of health, education and civic applications (ECHO Institute, 2020a). Diffusion of the ECHO model commences with organisational teams completing formal Immersion training, which is facilitated by one of several organisations designated as ECHO Superhubs under license by the UNM (ECHO Institute, 2019, 2020a). Immersion is a once-off compulsory training for prospective hub organisations to complete prior to becoming licensed to use the ECHO model. ECHO Superhubs are established and successful ECHO hub organisations which are designated by UNM to provide new organisational teams with Immersion training. Organisations who complete Immersion training are subsequently designated as ECHO hub sites and are licensed to replicate the ECHO model for the delivery of one or more ECHO networks as their local telementoring intervention. Superhubs are established ECHO hub organisations, which have an additional licensed function of providing Immersion training locally to actively support and mentor emerging teams from other organisations to understand how to implement the ECHO model. Superhubs support other organisational teams to understand how to implement the ECHO model with fidelity and integrate the telementoring intervention within business-as-usual processes to disrupt, innovate and improve outcomes regardless of their sectoral orientation.

The versatility of the ECHO model is evidenced through its growth and adoption across sectors other than healthcare including; business, corrections, domestic violence, education, ethics, law enforcement, quality improvement, social welfare, and more recently climate change and COVID-19 (ECHO Institute, 2020a). Fidelity to the ECHO model is universal (ECHO Institute, 2020b, 2020c), however there is no consolidated list of indicators by which teams can assess or benchmark their implementation across various sectors. This has prevented organisations from using the ECHO model to measure, compare and contrast implementation outcomes and experiences in a consistent way using universal indicators (Dearing et al., 2019). Since the emergence of COVID-19, rapid growth in organisations beyond healthcare adopting the ECHO model has occurred across a variety of sectors (ECHO Institute, 2021). This has created a gap in knowledge about what constitutes universal implementation success for organisational teams using the ECHO model globally in any sector (Dearing et al., 2019; Moss et al., 2021).

The strategies and tactics used by organisational teams implementing Project ECHO to provide telementoring solutions within their organisations may vary by sector (Moss et al., 2021). Despite strong evidence of the ECHO model being used in the healthcare sector, there is no published evidence of what universal indicators can assess the successful implementation of the telementoring model beyond healthcare (Arora, Thornton, et al., 2011; Zhou et al., 2016). It is important for organisations in any sector to be able to understand the level of success of their implementation and benchmark their ongoing use of the ECHO model for quality improvement and investment purposes. This research aimed to identify the indicators of implementation success and to develop a framework which could be scalable to any sector in which the ECHO model is used. The framework provides teams with a universal guide to measure their implementation, planning, and ongoing quality improvements of their local use of the ECHO model as a telementoring innovation.

Methods

Overview

The Delphi technique originated at the RAND Corporation in the 1950s and has proven to be a popular method to identify and prioritise issues for decision-making, with well-documented and stepwise frameworks to investigate research questions (Okoli & Pawlowski, 2004; Schmidt, 1997). The method has been described as a structured group communication process which facilitates answering a complex question through structured communication (Adler & Ziglio, 1996; Linstone & Turoff, 1975). The Delphi process includes the following key components: an opening question or prompt is offered; feedback of individual contributions of information and knowledge; assessment of the group judgement or view; opportunity for individuals to revise views; and anonymity for the individual responses (Linstone & Turoff, 1975). The Delphi technique was thus selected to obtain the most reliable consensus of implementation success indicators from across an international group of Project ECHO experts (Choi et al., 2019; Davies et al., 2021; Diamond et al., 2014; Fleuren et al., 2004; Garbutt et al., 2019).

Based on the established methodological approach for conducting Delphi studies, and the global distribution of ECHO hubs, a Delphi panel was established consisting of ECHO practitioners from international hub organisations and a five-round modified e-Delphi survey was conducted between May and December 2021 (Davies et al., 2021; Schmidt, 1997). This approach was pursued to achieve expert consensus to identify the indicators included in the framework that could assess and measure the successful implementation of the ECHO model within any organisational setting.

The established methodological criteria for reporting Delphi study results were used to ensure the quality of this research (Adler & Ziglio, 1996; Egfjord & Sund, 2020; Fleuren et al., 2004; Haggar, 2018; Schmidt, 1997; Silva et al., 2018; Wild & Torgersen, 2000). Ethical approval was provided by both The University of Queensland and Children’s Health Queensland Hospital and Health Service’s Human Research Ethics Committees (LNR/21/QCHQ/75147 and SSA/2021/QCHQ/75147). Figure 1 below provides an outline of the e-Delphi study phases.

Fig. 1
figure 1

Phases of the e-Delphi study

Full size image

Survey Development

Delphi Panel

An international panel of Project ECHO experts was established for the Delphi study to reach a consensus on the indicators of success of a Project ECHO implementation. A knowledge resource nomination worksheet (KRNW) was developed by the research team to identify an initial list of 140 individual ECHO practitioners from organisations with established ECHO hub and Superhub operations. The KRNW was informed by the academic, research, and ECHO network track records of these individuals gathered from a literature search and various open access Project ECHO websites (ECHO Institute, 2020a; Okoli & Pawlowski, 2004). An invitation to participate was sent to the 140 potential panel participants via email by the Principal Investigator (PM). A snowball method was used by inviting potential panellists to forward the invitation to their colleagues and broader network whom they believed would also be eligible. An open invitation to participate in the Delphi study was also included in the UNM’s ECHO Institute fortnightly newsletter which is distributed to a global readership of ECHO practitioners that exceeded 17,400. Inclusion criteria was defined as the Delphi participants must have completed Project ECHO Immersion training and/or have participated in the planning, coordination, and delivery of at least one ECHO Network. Interested prospective panellists completed a screening question at the beginning of the Round 1 e-Delphi survey to ensure they met the eligibility criteria. In subsequent rounds of the e-Delphi study, only those individuals who had completed the previous Delphi round were emailed the next iteration of the survey.

Ethical Considerations

Within this Delphi study, participants were able to present and react to ideas in a quasi-anonymous setting—while they were known to the Principal Investigator and perceivably to one another, their ideas, opinions, and rankings remained strictly anonymous throughout the study (Hasson et al., 2000).

e-Delphi Survey

The survey was administered via Qualtrics, an online survey platform, over five rounds. Each round was open for two-four weeks to maximise response rates, with weekly reminder emails sent during that time to non-responders to encourage them to complete the surveys. Each survey round took approximately 20–30 min to complete.

Round 1

In Round 1, panellists were asked how they would measure success of a Project ECHO implementation, by listing at least ten indicators they would consider useful. Panellists were also able to provide additional comments or recommendations if they wished. Prior to submitting their response to Round 1, informed consent was obtained from all individual panellists included in the study. The items contributed by panellists were then consolidated, with duplicates removed.

Round 2

In Round 2, panellists were provided with individualised reports mapping their Round 1 submissions to the consolidated list of indicators developed from the entire panel population. In this round, panellists were asked to verify if they agreed that their initial submissions had been fairly and accurately represented in the consolidated list (Schmidt, 1997).

Round 3

In Round 3, panellists were requested to rate each of the specific indicators in the consolidated list on a three-point scale of ‘unimportant’, ‘important’, or ‘essential’ for measuring the success of a Project ECHO implementation (Davies et al., 2021). The rationale for the three-point scale was to maximise panellist retention and to commence the rating process for the next round. Individual indicators which reached consensus amongst the panel were retained for further consideration in Round 4. In this round, consensus was defined as 75% of the panel agreeing that the indicator was either ‘important’ or ‘essential’ (Diamond et al., 2014).

Round 4

In Round 4, panellists were informed of indicators from Round 3 that had not reached consensus and were asked to rate those retained on an eleven-point numerical scale (with terminal anchors of 0 = strongly disagree and 10 = strongly agree) (Davies et al., 2021). The rationale for the eleven-point numerical scale was to elicit more specific ratings on the importance of each indicator. Indicators that achieved a consensus of at least 75% of the panel rating greater than 6 were retained for Round 5.

Round 5

In Round 5, panellists were asked to reconsider and re-rate the indicators from Round 4 using the same rating scale. To facilitate this process, each panellist was presented with their individual rating for each item in Round 4, with the overall panel mean rating for that item. Indicators that achieved a consensus of at least 75% of the panel rating greater than 7 were retained for inclusion in the final list of implementation success indicators.

Final Phase

In the final phase of this study, the entire research team (n = 4) reviewed and consolidated the final list of indicators retained from Round 5 during three one-hour consensus meetings. This involved the amalgamation of similar or overlapping indicators, removal of unmeasurable items, and slight rewording to optimise clarity. At this point, the entire research team collectively agreed that the thematic domains had emerged from the data and were used to group the final consolidated list of indicators as the final step. The final list approved was by the research team and distributed to all Delphi panellists to acknowledge the study’s data collection had concluded. These findings could be generalisable to any organisation/sector implementing Project ECHO to indicate the success of their implementation.

Results

Panellist Characteristics

The Delphi panellists were individuals working at ECHO hub organisations, who had completed Project ECHO Immersion training, and/or had participated in the planning, coordination, and delivery of at least one ECHO Network as a minimum. Table 1 describes the demographic characteristics of the Delphi panellists in this study. In Round 1, 73 participants from 36 ECHO hub organisations in 6 countries participated, with a significant proportion of females coming from healthcare and university organisations in Australia and the USA. In Round 2 there were 67 panellists retained, followed by 53 in Round 3, 43 in Round 4 and 41 in Round 5, representing a 92%, 79%, 81%, and 95% retention rate per round, respectively. The panellists that were not retained over the five rounds were also largely female, from healthcare and university organisations in Australia and the USA.

Table 1 Delphi Panel Demographics
Full size table

Delphi Rounds 1–5

Figure 1 provides a summary of each round in the e-Delphi study that resulted in the development of the final framework with domains for the indicators of success (Appendix 1). In the first round, 733 individual indicators were submitted by 73 participants. The 733 indicators were then consolidated into a list of 121 randomly ordered indicators developed by the research team to remove duplicates, synthesise language and terminology (Appendix 2). In the second round, 67 retained panellists (92%) verified that they agreed that their initial submissions had been fairly and accurately represented in the consolidated list (Schmidt, 1997). The remaining 6 panellists (8%) did not respond to the Round 2 invitation. In the third round, 53 panellists (79%) and 109 indicators were retained, with 12 indicators (10%) not reaching consensus and were subsequently removed from the list (Appendix 3). In the fourth round, 43 panellists (81%) and 72 indicators were retained, with 37 indicators (34%) not reaching consensus that were subsequently removed from the list ahead of the final round (Appendix 4). In the fifth round, 41 panellists (95%) were retained, with all 72 indicators (100%) reaching consensus and were retained into the final phase. Data from Round 4 and 5 were analysed with SPSS software to determine the mean scores for each indicator and where consensus was achieved. Please note: all Appendices for this study are included in the supplementary files.

Final Phase

When the final list of indicators from the Delphi panellists were reviewed by the research team, seven indicators were identified has having similar meanings, and as a result were amalgamated with other indicators (Appendix 5). Revised wording edits were made to 46 indicators for optimal clarity (Appendix 6). Eleven indicators were agreed to be redundant, with remaining indicators adequately capturing the same data (Appendix 7). Eight indicators were retained with no revised wording (Appendix 8).

The final list comprised 54 indicators of success that were aligned in the framework under four thematic domains illustrated in Table 2: (1) spoke participant engagement, (2) ECHO Hub/ECHO Network design and operation, (3) ECHO Hub team engagement and (4) Local Impact. Indicators of spoke participant engagement have been defined as those which measure the number, interactivity and participation experience of individuals who join ECHO Networks from a variety of spoke locations to connect and learn with panel teams centrally coordinated by the hub. Indicators of ECHO Hub/Network design and operation have been defined as those which measure the design and operation of an organisation’s ECHO Hub, and/or individual ECHO Networks. Indicators of ECHO Hub team engagement have been defined has those which measure the number, interactivity and participation experience of individuals who facilitate and manage ECHO Hub functions. Indicators of Local Impact have been defined as those which measure the increase or improvement in workforce development, capacity, system integration and efficiency. The four thematic domains were identified through an inductive thematic approach to group individual indicators to target specific areas of implementation success identified by the panellists.

Table 2 Thematic domains and indicators of implementation success for project ECHO—framework (N = 54)
Full size table

The framework provides recommendations for when each indicator be measured (Pre-Launch, Launch, Growth/Continuous Improvement) which is consistent with the three phases of replicating the ECHO model in any organisation (Children's Health Queensland Hospital and Health Service, 2021). Pre-Launch is the preparatory phase of implementation following the completion of Immersion training prior to the launch of a hub’s first ECHO Network. The Launch phase is the period encompassing the actual launch and first iteration of a hub’s pilot ECHO Network. The Growth/Continuous Improvement phase refers to the continuous period beyond the hub’s first ECHO Network, where the hub team undertake growth and continuous improvement activities within their hub.

The indicators are also aligned to which stakeholder group they relate to (Individual Spoke Participant/Panellist, ECHO Network, ECHO Hub, Organisation, System). To provide organisational teams implementing the ECHO model with consistent methods by which to measure each of the indicators with fidelity, recommended approaches have also been included. Using the framework, teams can analyse changes that occur over the lifecycle of their implementation with some indicators varying between being dynamic, static, and iterative—meaning that there are indicators which can be measured at random, as a once-off, or routinely. The variety of indicators highlights that implementation teams can capture learnings about their journey’s successes and/or areas for improvement based on what is of most interest/relevance. Indicators in this framework lend themselves to being measured or analysed at random, once-off, or routine milestones which will support ECHO practitioners to report on their successes at milestones throughout their implementation journey for quality improvement and ongoing investment opportunities. The detailed version of this final framework is included in Table 2.

Discussion

The aim of this study was to identify indicators of implementation success using an e-Delphi approach, and compile them in an internationally relevant framework to measure and assess the implementation success of the ECHO model within any organisation. The purpose and intention of this framework is to provide structure and a frame of reference to support planning and decision-making (Cash-Gibson et al., 2019; Davies et al., 2021). The final framework comprised 54 specific indicators that were mapped across four thematic domains following the e-Delphi study. This framework provides a clearly articulated list of indicators required by ECHO practitioners to evaluate if their local implementation of the ECHO model is successful. The intention of this framework was to compile the indicators identified during the e-Delphi study, by global ECHO practitioners with expertise in implementing the ECHO model, under key thematic domains accompanied with recommendations for measurement. This provides ECHO practitioners with a consistent approach by which to measure each of the indicators in any sector with fidelity when assessing the implementation milestones within their organisation.

Globally, the rapid uptake of virtual telementoring models like Project ECHO has coincided with the onset of the COVID-19 pandemic (ECHO Institute, 2021). In 2019, there was approximately 232,000 ECHO session attendances by spoke participants, which grew to 1,162,000 in 2020. This represented a five-fold upswing in the utilisation of virtual solutions for workforce development and mentorship (ECHO Institute, 2021). The rapid increase in utilisation justified a need to swiftly develop this framework to support emerging and existing ECHO practitioners and implementation teams to understand what indicators could measure implementation success in any organisation or sector. Given there is a vacuum in the literature for guidance on what constitutes implementation success for Project ECHO, this study consolidated expertise garnered from international ECHO practitioners with cumulative experience and understanding of the ECHO model across a variety of contexts to identify the indicators in this framework for wider use.

To date, no previous studies have examined what ECHO practitioners and implementation teams consider to be universal indicators of implementation success for telementoring innovations like Project ECHO, particularly beyond the healthcare sector. These findings complement and build on other previously developed organisational readiness and implementation process tools/frameworks for ECHO practitioners at the beginning of their implementation phase (Serhal et al., 2018). One key point of difference with this study’s findings is that these domains and indicators have been elicited by ECHO practitioners across a variety of sectors (healthcare, education, university, public and private) where the ECHO model remains in use as a telementoring solution. This suggests that the framework may be relevant beyond the traditional context of many ECHO telementoring solutions operating within the healthcare sector.

This framework presents the most important indicators that need to be considered when implementing the ECHO model within organisations as determined by seasoned ECHO practitioners and implementation teams from across the globe. Further research investigating the application of this framework within ECHO hubs globally may also provide valuable insights to inform ongoing quality improvement efforts of hub organisations. This would provide ECHO Superhub training teams with a better understanding and response to the training and support needs of new ECHO hub teams. This may lead to more successful implementations globally. While this framework has been developed with the ECHO model at front of mind, it could be easily adopted for use for other innovative telementoring-like solutions being implemented by organisational teams. It has global relevance for adoption by a range of stakeholders, including ECHO practitioners, implementation teams, executive decision-makers, and other system managers across multiple sectors.

The framework domains captured in Table 2 of Spoke participant engagement, ECHO Hub/ECHO Network design and operation, ECHO Hub team engagement and Local Impact, provides a foundation for ECHO practitioners and implementation teams with a universal framework to consistently assess and benchmark their local successes across any organisational, sectoral, or geographic context. The framework provides organisational teams with consistent methods to measure each of the 54 indicators over the duration of their implementation. Indicators in this framework lend themselves to being measured or analysed at random, once-off, or routine milestones which will support ECHO practitioners to report on their successes at milestones throughout their implementation journey and benchmark against other global hubs. This framework might also be used by executive decision-makers and investors to inform how they organisationally commit to and financially invest in the ECHO model (Moss et al., 2020).

Since uptake of the ECHO model has expanded beyond the healthcare sector, this framework offers guidance and support for new and existing ECHO hubs as the model’s global diffusion continues. This framework also offers support for designated ECHO Superhubs to enhance their Immersion training curricula as it integrates with the universal phases of replicating the ECHO model (Pre-Launch, Launch, Growth/Continuous Improvement) (Children's Health Queensland Hospital and Health Service, 2021; ECHO Institute, 2019). Implementation teams could benefit from adopting this framework at Immersion training to enhance their pre-launch phase understanding of what indicates implementation success of the ECHO model in their local organisational context. This framework could facilitate enhanced consistency in reporting across ECHO hub teams’ implementation and evaluation planning activities by using these indicators and recommended measures. The framework highlights which indicators relate to specific stakeholder groups so implementation teams can collect and report the most meaningful data that indicates success for that audience (Individual Spoke Participant/Panellist, ECHO Network, ECHO Hub, Organisation, System) to measure their impact. This framework could be harnessed by ECHO hub teams and other telementoring researchers globally to plan, measure and showcase the changes achieved by their implementation of telementoring innovations using these indicators as consistent benchmarks for quality improvement and investment decision-making.

A strength of this study is the inclusion of a large e-Delphi panel with acceptable rates of panellist retention. The panel included 73 experts and retained 41 (56%) throughout the five survey rounds. Despite there not being a recommended or ideal number of panellists in a Delphi study, it is typical for panels to range in size between 10 and 100 (Diamond et al., 2014; Hasson et al., 2000; Kidholm et al., 2018; Snyder-Halpern, 2001). Given there were 36 ECHO hubs represented in this study out of approximately 700 active global hubs, this Delphi panel might be considered large. A second strength was that this Delphi had a global reach and demographic diversity, comprising experts from across six countries and five broad organisational types. This breadth in representation will ensure that the framework is internationally relevant.

Due to the high number of participants from the healthcare sector, the authors acknowledge there may be some residual desirability bias. Some indicators of success that can more easily be measured and reported on by teams using the ECHO model within the healthcare sector may have been ranked consistently higher than others (Ecken et al., 2011). The authors acknowledge that this study did not explore the correlation between the measures of implementation success that were identified during the e-Delphi with actual measurement of outcomes or impact. A subsequent research study is in train to investigate the application of the framework in a variety of organisational and sectoral contexts to test the framework.

Conclusion

The results of this international e-Delphi study have been presented as a framework of indicators to support organisational teams to measure the success of their implementation of Project ECHO. The framework consolidated 54 distinct indicators which could be generalisable to any organisation/sector implementing Project ECHO to illustrate and showcase the successes or areas for improvement of local implementations of the ECHO model or other new telementoring innovations globally. By framing the indicators under four key domains, ECHO practitioners and implementation teams can consistently assess and benchmark their local successes or failures regardless of organisational, sectoral, or geographic context. The findings of this study are translational for audiences from other countries and sectors to indicate successful implementations and benchmark for ongoing quality improvement and investment in the organisation’s use of the ECHO model or other telementoring innovations.