It is estimated that half of all lifetime mental illnesses begin before age 14 years, 75% occur by age 24 [1], and only half of children who meet criteria for major depressive disorder (MDD) will be identified before they reach adulthood [2]. Of the youth identified to have MDD, only 1 out of 5 will have exposure to an evidence-based treatment [3]. Current research demonstrates a rising trend in the prevalence of youth diagnosed with MDD, going from 8.7% in 2005 to 11.3% in 2014 [2]. This trend has continued throughout the COVID-19 pandemic, with data from a recent study indicating that depression rates among adolescents doubled to 25% in 2020 [4].

Children and adolescents who experience depression are at higher risk of other comorbidities, such as substance abuse [5] and chronic medical conditions [6, 7]. Youth with depression are more likely to struggle with school [8] and interpersonal relationships [9]. These unintended consequences of untreated depression can lead to lifelong difficulties, as adults who experienced depression during adolescence have been shown to be more likely to smoke, abuse alcohol, suffer from migraines, and report low levels of social support [10]. Depression can also be fatal, as youth diagnosed with MDD are at a higher risk to die by suicide. Suicide is the second leading cause of death for teens [11], with depression demonstrating a potentially modifiable risk factor. Despite the possible negative outcomes, youth with mental health concerns experience significant barriers to accessing appropriate care.

The average delay between onset of symptoms and biopsychosocial intervention is between 8 and 10 years [12, 13]. Most US counties do not have a single practicing child psychiatrist, with rural areas being the hardest hit by the shortage [14]. Primary care providers (PCPs) are the best positioned to assist with the task of identifying mental health conditions early and increasing access to appropriate treatments [15]. However, pediatric PCPs are ill equipped to appropriately treat youth with mental illness [16, 17]. They have had little formal training in child mental health beyond attention-deficit/hyperactivity disorder [18]. Pediatric PCPs also are limited by barriers associated with billing, reimbursement for providing mental health care, clinical knowledge related to appropriate treatments, and lack of family follow-through on referrals [19].

A midwestern academic institution attained grant funding for its Center for Telehealth to support select specialty divisions to utilize a novel learning platform to create a course targeting designated service areas based on community provider requests and hospital prioritizations. The selected platform is called the Extension of Community Healthcare Outcomes (Project ECHO®) model. This conceptual model originated at the University of New Mexico designed to improve non-specialist capacity to treat complex medical conditions where access to a specialist is challenging. Knowledge was de-monopolized and specialty care amplified, providing access to the right care for the right patient at the right time. The four principal components of the Project ECHO® platform include (1) videoconference sessions; (2) sharing best practices to reduce variability in care; (3) using case-based learning to maximize learning efficiency and increase non-specialist expertise; and (4) monitoring outcomes through a Web-based database. Project ECHO® course sessions follow a similar format: brief, high-yield, didactic with time for questions and reflections, followed by facilitated case discussions presented by cohort participants [20, 21]. Videoconferencing technology respected busy professionals’ time and directly connected providers, who may be located remotely, to specialists. The virtual technology created an approachable, community atmosphere allowing the ability to see the faces of the speakers, participants, and presentations simultaneously. This key component of the format encourages active, bidirectional communication between and among the participants and specialists.

This study will focus on the outcomes of the first course created within the mental health series of the academic institution’s initiative to utilize the Project ECHO® model with fidelity. The mental health specialty subject expert (a triple board trained child psychiatrist) selected pediatric depression as its pilot course. This decision was supported by the funding institution’s adoption of the US Preventive Services Task Force for Mental Health recommendation of universal depression screening annually starting at age 12 years, as well as the growing demand in the emergency department (ED) for primary mental health concerns. The core didactic curriculum was modeled after the Guidelines for Adolescent Depression – for Primary Care (GLAD-PC), released in 2010 and revised in 2018. Important differences in the treatment algorithm included updated guidance on screening, practice preparation, and changing “watchful waiting” to “active monitoring” for depression with mild severity [22]. This paper will analyze the effects of psychiatric specialists utilizing the Project ECHO model on improving the mental health capacity of PCPs focusing on a single disorder, adolescent depression.

Methods

Course Content and Development

The content for the best practice for clinical decision making was taken from the GLAD-PC toolkit to assist PCPs to identify, assess, manage safety, provide interventions in the office and upon referral to a specialist, and ensure ongoing follow-up for the management of youth with depression. The framework positioned PCPs to act as the medical monitors for youth with MDD, and to take a step-wise approach to initiating evidence-based treatments.

Session content and course evaluation tools were developed by subject matter experts (triple-board psychiatrists), and the information was organized and validated by continuing medical education specialists. Orientation to the format, the development timeline, and tasks to develop and launch the course were organized and led by telehealth program management support. Utilizing a content matrix, each session’s content, topics, objectives, test questions, and self-efficacy statements were aligned. The didactic content was presented by child and adolescent psychiatrists and psychologists. Along with the presented didactic material, a panel consisting of various interdisciplinary specialists represented by pharmacists, advanced practice nurses, psychiatric therapists and counselors, and a family advocate representative were enlisted to participate in the didactic discussions. Application of information from the curriculum was reinforced through case discussions presented by the community learners.

Each session followed the recommended structure of the Project ECHO format. The sessions started with introductions of the hub team, and learners then transitioned into a brief, high-yield didactic with pragmatic content, which was followed by interactive discussion, a case presentation provided by a learner, and a facilitated discussion. All sessions closed with thoughts and recommendations from the panel of experts and closing statements from the hub team.

Recruitment

Participants were recruited from the primary service area of a large, midwestern academic children’s hospital, where wait times to see a child and adolescent psychiatrist were often more than 3 to 6 months. The maximum number of participants in a cohort was 25, and because of presumed high demand for the course, the decision was made to limit the number of learners to one per practice. Participants were from various-sized pediatric primary care offices, with differing internal resource availability assessed through registration questions. Participants agreed to attend 80% of sessions, present a case that may be discussed during the sessions, and complete the pre- and post-cohort clinical knowledge and self-efficacy assessments.

Evaluation and Measurement

Demographic data, practice-level resources, baseline clinical knowledge, and approximated practice patterns through self-report of confidence were captured through registration. The cognitive knowledge test questions were developed and reviewed by adult learning specialists within the continuing education department. These specialists read through each question to ensure that each question was reflective of stated objectives and phrased in an easily understood manner. After registration and associated pre-test completion, participants were provided with a link to the videoconferencing meeting platform. This novel teaching platform allowed participants flexibility with location, while preserving face-to-face interactions with specialists and each other, creating a virtual community of learners. After each session, learners were invited to respond to a brief survey, allowing them to provide specific feedback on the didactics and overall effectiveness of the session.

Participant Demographics

Practice details and demographic information were collected at the point of registration, including provider credentials, practice type, size, and location. Participants were also surveyed on their intended goal for choosing the Project ECHO: Common 2 Complex Adolescent Depression Course.

Clinical Knowledge Assessment

Evaluation of participant capacity to identify and manage youth with MDD was measured through computer-based administration of multiple-choice questions before and again at the end of the course. Composite knowledge scores were calculated on the number of correct answers out of the total number of questions asked. Clinical knowledge questions for cohort 1 were created to reinforce learning objectives based on an outline from the traditional 1-h didactic model of teaching. Topics were covered in six sessions and included the GLAD-PC algorithm overview, identifying depression, assessment of safety/suicidality, initiating medications, adjusting medications, referring to evidence-based psychotherapy, and documentation and billing. The Project ECHO immersion training was provided 2.5 months before the mental health series pilot course was planned to launch and after the questions were created. The Project ECHO session format stressed the importance of the high-yield objectives in a brief didactic, and the depression course curriculum objectives were streamlined to align with the fidelity of the model.

Following cohort 1, the hub team reviewed the test questions to validate that question content was covered in the revised curriculum objectives (following the immersion training). As a result, 4 of the 16 original knowledge questions in cohort 1 were removed due to being irrelevant to the course. Based on post-session feedback collected from the learners and didactic presenters, the curriculum content objectives from cohort 1 were reorganized slightly to facilitate ease of delivery when presenting the same material in cohort 2 (Table 1). Additional knowledge questions were created for cohort 2 to better capture the material actually presented during the session. Composite knowledge scores were calculated on the number of correct answers out of the total number of questions asked.

Table 1 Course curriculum and learning objectives
Full size table

Self-efficacy Assessment

Participants were asked to rate their confidence regarding their ability to identify and manage depression using a developmentally informed Likert scale (1 = not confident at all; 2 = slightly confident: I am familiar with this action but am not confident to perform independently; 3 = moderately confident: I feel confident to perform initial action, knowing resources are available for further coaching/consultation; 4 = confident: I feel confident to perform this action; 5 = extremely confident and able to coach others). Composite test and single-question self-efficacy scores were calculated by summing the numerical representation of each answer (i.e., 1–5) and identifying the median for comparison between pre- and post-tests. Themes addressed by these questions included PCPs’ confidence with the GLAD-PC assessment and management algorithms, non-pharmacologic techniques to use in office, family depression education, medication management, assessing safety in the office, and coordinating care with a mental health team.

Secondary Measures

Seeking psychiatric evaluation and treatment through the main academic local pediatric hospital ED was selected as a secondary measure to approximate the influence of the mental health education on provider referrals and patients seeking treatment for psychiatric services. Frequency of visits to the ED by patients who identified an ECHO depression cohort participant as their PCP was collected for both 12 months before and 12 months after completing the course. The primary diagnosis for the ED encounter had to be a psychiatric-related diagnosis by the ED provider at the time of discharge. The diagnosis codes included in the data were mental, behavioral, and neurodevelopmental disorders, F01–F99. Inpatient admission rates from referred patients from cohort 1 participants were calculated. No inpatient admission rate was calculated for cohort 2. No comparison group was selected to provide a control group for either cohort 1 or 2 participants.

Following each session in both cohorts, participants were asked whether they found the time invested in attending worthwhile. Their responses were captured on a Likert scale from 1 to 7 (1 = strongly disagree; 7 = strongly agree).

Data Analysis

In addition to descriptive statistics, pre- and post-course assessment differences in single-question self-efficacy and single-question knowledge scores were analyzed using Minitab software. Mathematical and statistical methods were utilized to analyze the difference between pre- and post-testing to determine the effect of the intervention. Because the data collected are not normally distributed, non-parametric statistical assessments were employed. McNemar’s test was used to evaluate single-question knowledge scores and analyze whether the proportional change at two points in time for a population is significant. Wilcoxson signed-rank test served as the testing instrument for single-question self-efficacy results and composite results of both domains. Wilcoxson’s test examines if the population mean of two samples ranks differently. The statistical instruments utilized consider the calculated mean and standard deviation for each dataset to determine the significance of the variation between pre- and post-scores.

Results

Cohort Participation

At the time of this analysis, two participant cohorts have completed the course. Cohort 1 was completed in 2018 and had 18 registered participants (representing 18 different practices) within the primary service area of a major academic children’s hospital. The attendance rate for participants was 85% overall and remained stable throughout the course, except for two classes with a 72% attendance rate. Cohort 2 was completed in 2019 and did not overlap with cohort 1. Participants for cohort 2 were recruited from the primary service area and across the state, resulting in 25 registered participants. Attendees dropped to 23 after two participants withdrew. The average rate of attendance for cohort 2 was 75%.

Demographic Data

Provider and practice attributes are referenced in Table 2. Most participants were medical doctors (83.8%) practicing in a private group (70.2%) within the primary and secondary service areas of a large midwestern academic children’s hospital (89.2%).

Table 2 Demographic data for cohorts 1 and 2
Full size table

Knowledge Pre- and Post-testing

Overall clinical knowledge test scores for both cohorts achieved statistically significant improvement from pre to post. Sixteen of the 18 registered participants from cohort 1 completed the post-test assessments. Twenty-one of the 25 registered participants who attended cohort 2 completed the post-test assessment.

Cohort 1 demonstrated a median composite score change of 15.6% (median 56.3 to 71.9%, p 0.002). Post-test responses from cohort 2 resulting in the median composite score change of 16.0% (median 58.0 to 74%, p < 0.001).

Single-question analysis for cohort 1 resulted in zero knowledge questions with a statistically significant score change pre- versus post-course. Single-question knowledge assessment questions from cohort 2 resulted in five questions reaching significant change in the topic areas of medication management and non-pharmacological interventions.

Self-efficacy Assessment

Cohort 1 pretest self-efficacy demonstrated a median composite score of 3 (i.e., moderately confident) rising to 4 (i.e., confident), self-reported on post-course assessment (p = 0.002). Cohort 2 self-efficacy reported an initial median composite score of 2 (i.e., slightly confident) and following the course as 3 (i.e., moderately confident) (p = 0.001).

Practice Behavior

After course completion, surveys resulted in 32 out of 37 (78%) total participants responding yes, indicating that they changed their clinical practice following the course. Free-text responses of how they have modified their practice because of the course were obtained. The major themes of their responses included treating more difficult depression cases than they had in the past, changing their initial medication choice to an evidence-based, first-line selective serotonin reuptake inhibitor (US Food and Drug Administration–approved fluoxetine or escitalopram), providing more frequent follow-up for their depressed patients, utilizing standardized tools such as the Patient Health Questionnaire-9 (PHQ-9) and Columbia-Suicide Severity Rating Scale (C-SSRS) for screening and ongoing monitoring, and engaging in more collaborative communications with therapists and child psychiatrists.

Secondary Measures

Eighty percent of participants from cohorts 1 and 2 attended 80% of the sessions offered, 12 total. Of the 119 post-session surveys received, 86.6% (n = 103) strongly agreed or agreed that taking time out of their schedule to attend was a worthwhile investment.

Prior to the course, 103 unique patients with a PCP who later participated in cohort 1 resulted in 119 visits to the ED for psychiatric evaluation. For the 12 months following cohort 1 completion, 68 unique patients identified a cohort 1 participant as their PCP, demonstrating an individual average reduction in patients presenting to the ED of 34%. Those identified patients resulted in 71 ED visit encounters, generating a reduction in ED visits by 41%.

Cohort 2 participants were identified as PCPs for 140 unique patients who presented to the ED for a total of 204 ED visits and discharged with a primary psychiatric diagnosis. Twelve months following the course, 116 unique patients with a PCP from cohort 2 (decrease of 17%) resulted in 137 ED visits (decrease of 33%) and were discharged with a primary psychiatric diagnosis (Fig. 1). The rate of these ED visits from cohort 1 resulting in inpatient psychiatric admission was calculated as 50% before the course and 80% following for patients of cohort 1 participants.

Fig. 1
figure 1

Patients and encounters from mental health referrals to the emergency department by cohort participants: pre- vs. post-course completion

Full size image

Discussion

This study was conducted as a single-center, proof-of-concept pilot program. The results indicate the Project ECHO model is an effective way to improve pediatric PCP confidence in and knowledge of providing evidence-based treatment for depression in the primary care setting. Participants in two cohorts demonstrated statistically significant improvements in their self-efficacy scores in treating depression with utilizing the core concepts of the course: GLAD-PC assessment and management algorithms, non-pharmacologic techniques, family education, medication management, safety assessments, and care coordination with mental health teams. Participants also demonstrated a significant change in clinical knowledge of the same core concepts. Cohort 1 improved their correct test answers by 15.6% (p = 0.002), and cohort 2 improved by 14% (p = 0.000).

The specialist goal for creating this specific course was to improve pediatric PCPs’ self-efficacy and knowledge around treating adolescent depression in the primary care setting. The potential burden of measuring further outcomes was evident from the beginning. The system of care in pediatric primary care is not standardized, nor is the clinical documentation or electronic health record systems. To require all participants to collect data in the same manner accurately was judged to be difficult for pediatric PCPs and be a barrier for participation in the initial pilot project for depression.

A secondary measure was needed to approximate whether improved clinical knowledge and self-efficacy resulted in a change in provider behavior. ED referral for a primary mental health concern was selected because this was a single-site data that did not rely on participants to capture on their own.

It would not be accurate to interpret the ED referral data as solely reflecting outcomes from the ECHO course. How this information was attained and recorded within the electronic record of the ED was variable. The pediatrician was not always listed for patients, which would impact the data. Additionally, there were several multidisciplinary interventions occurring concurrently that were impacting the primary service area psychiatric mesosystem and possibly provider referrals to the ED for psychiatric evaluation. These interventions included a large group of primary care practices focusing on the implementation of PHQ-9 screenings, a new program by which licensed mental health clinicians taught the C-SSRS to pediatricians and their practices, and a new mental health rapid access clinic through the academic center open to community referrals. To minimize the isolation of data collected by each intervention, efforts were made to share these interventions during the depression ECHO course to increase awareness and enhance PCP utilization of these resources.

For the ED referral data, there was no control provider from the practice selected, nor was it clear if the participating provider shared the information with the other providers in the practice. In the future, developments of a plan to obtain more baseline practice data from participants to better understand subtleties of changes in the office culture are being considered. Future directions can include using before and after ECHO participation practice-level data of a small number of patients identified by the provider to document variables such as rate and frequency of depression screening, registered nurse triage of patients in crisis, medications prescribed, and the time between a positive screening score and subsequent follow-up, as well as evidence and efficiency of a care coordinator tracking patient adherence and outcomes. Additionally, modifications will be made to the self-report mechanism of practice change. Currently, it is in a free-text format, which limits how these data can be analyzed. For future cohorts, there will be standard answers from which to choose to better capture and understand the data.

After participation, the volume of patients who listed a participant from cohort 1 as their PCP and discharged from the ED with a primary psychiatric diagnosis was reduced by 40%. After cohort 2, the volume was reduced by 33%. These data suggest that pediatric PCPs who have completed the ECHO course may be more comfortable managing higher risk patients and have more ability to assess risk levels independently, resulting in higher percentages of those patients being referred requiring inpatient psychiatric admission. These findings may also demonstrate that pediatric PCPs could improve patient access to the appropriate level of specialty care after receiving direct education and support from pediatric psychiatric specialists. Data on admission rates was not collected for cohort 2 due to the COVID-19 pandemic affecting all aspects of pediatric care.

The curriculum created in this series focused solely on evidence-based treatment guidelines for depression. During the intensive review of case presentations, participants did not focus on the treatment of depression alone. Cases presented by providers often led to discussions about primary or comorbid anxiety disorders and resistant patients and families more often than depression concerns. After analyzing the content of case discussions, the team has moved toward next steps to developing an ECHO course focused on anxiety disorders.

Over a decade of data on the Project ECHO model have shown it to be an effective tool to help build subspecialty capacity within primary care across many different medical disciplines. Past pediatric Project ECHO behavioral and mental health programs have taken a broader approach to educating PCPs on mental health topics, choosing to have one course for multiple diseases [23] or teaching about a single diagnosis like autism spectrum disorder with the focus on identification and managing comorbidities [24]. The Project ECHO pediatric course presented takes a different approach by focusing on an in-depth teaching of depression, which has not occurred in this way previously. The evidence suggests that this deeper dive approach is also effective for building community provider capacity to treat pediatric mental health concerns. Future Project ECHO courses can be developed taking this same approach to provide comprehensive understanding of single diagnoses or clusters of diagnoses like anxiety disorders and disorders associated with disruptive behaviors.

In summary, pediatric PCPs are increasingly expected to identify and manage mental health problems and have little to no experience or psychiatric training. Pediatric PCPs who follow best practices utilizing evidence-based guidelines within their scope of practice facilitate their ability to initiate appropriate interventions. This pediatric depression-based ECHO project supported a practice shift for cohort participants by offering bi-directional, rapid learning for multiple PCPs simultaneously. This project demonstrates improvement in medical provider confidence, knowledge, and likely referral practices, allowing patients to gain more access to care at an appropriate level of intervention. A disease-focused, provider-based Project ECHO curriculum can affect real change for PCPs and patients.