Introduction

COVID-19 has created an unprecedented crisis worldwide, highlighting that a well-trained health workforce who can continuously update their skills in response to new pathogens and emergencies is imperative to a population’s health [1,2,3,4]. However, health worker education in sub-Saharan Africa still faces numerous challenges that limit progress toward provision of quality care and attainment of Sustainable Development Goal-3 (Good Health and Well-being). These include shortage of qualified faculty and teaching resources, outdated curricula, and lack of institutional capacity [5,6,7]. In Ghana, these challenges exist in the larger context of health worker shortage, despite significant gains in the last decade, and the unequitable distribution of this workforce across the country’s different regions [8]. Nurses and midwives remain the primary, and, in underserved areas, sometimes the only health workforce. Therefore, investing in their continuing professional development is crucial, so that they can update and enhance their professional skills and keep themselves and their communities safe [9,10,11].

Since the confirmation of the first COVID-19 case in Ghana in March 2020 [12], Ghanaian health workers have reported low perceived preparedness to respond to COVID-19. Although several factors contribute to their level of perceived preparedness (e.g., access to personal protective equipment and supportive management), training was found to be the strongest predictor [13]. Health workers who had not received COVID-19-related training were at highest risk for stress, anxiety, and burnout [13, 14]. One of the initiatives that responded to the gap in COVID-19 training to address health worker preparedness was SickKids-Ghana Paediatric Nursing Education Partnership COVID-19 Response Project, hereafter referred to as “the project.” It set out to train 10 000 health workers in Ghana by designing and implementing four relevant continuing professional development courses, provided at no cost to course participants.

The implementing partners for this project include Ghana College of Nurses and Midwives (GCNM), Ghana Health Service (GHS), Ministry of Health (MoH) Ghana, and the Centre for Global Child Health at The Hospital for Sick Children (SickKids), Canada. The project, funded by the Government of Canada through Global Affairs Canada, is an extension of the SickKids-Ghana Paediatric Nursing Education Partnership (PNEP) and builds on 10 years of collaboration between SickKids and its Ghanaian partners [15, 16]. Courses were designed by the project’s content experts in Ghana and Canada, and one of the project’s primary stakeholders, the Nursing and Midwifery Council of Ghana (N&MC) accredited and promoted the courses.

Courses were delivered through a "two-pronged strategy": e-learning and in-person. The courses were also adapted for and made available to health workers outside of Ghana. The project, which continues until April 2023, at the time of writing (July 2022) has surpassed its targets. A total of 18 855 health workers in Ghana and outside of Ghana (primarily from East and West Africa) have completed at least one of the four courses. This manuscript presents the results of evaluating the project's implementation and outcomes using data for a subset of Ghanaian health workers who have taken the courses (n = 9966).

Key evaluation questions

This evaluation answers two questions:

  1. (1)

    How successful was the project in implementing the two-pronged strategy (e-learning and in-person) for the four professional development courses?

  2. (2)

    To what extent did the courses contribute to strengthening health workers' capacity regarding COVID-19 prevention and health promotion?

Implementation success [17,18,19] was defined by stakeholders as (i) reaching a broad range of frontline health workers across Ghana, including in underserved areas; (ii) delivering courses that are relevant to Ghanaian health workers' learning needs; and (iii) efficient resource use (time, human resources, technology, and financial) to deliver the courses. Health workers' capacity is defined in "Measures" section.

Existing evaluations: the larger context

Currently, there is a gap in the evaluation of continuing professional development efforts in low-resource settings, especially e-learning interventions [6, 20, 21]. Well-designed e-learning interventions—those delivered through information and communication technology using a variety of instructional designs [6, 20]—can potentially alleviate the burden of faculty shortages and provide access to affordable education irrespective of geography. They offer a convenient way to gain knowledge and skills and are easier to update than curricula delivered by other means [22]. Financially, they cost less than traditional face-to-face training because of reduced costs associated with human resources, travel, and institutional infrastructure, and enable scaling up the reach of training at a comparatively low cost [23, 24]. Many health workers in Ghana already use their mobile phones for learning purposes, have a positive attitude toward e-learning, and exhibit high levels of preparedness for using e-learning interventions [25, 26]. Challenges with internet connectivity and the cost of data are recognized barriers to maximizing the benefits of e-learning interventions, particularly in rural areas [6, 25, 26].

Evaluations comparing the quality of e-learning to traditional in-person learning have produced mixed results. Some argue that e-learning can produce comparable or better gains in knowledge and practice [27]; others suggest that in-person training is superior, because it offers opportunities for hands-on exercises [22]. A meaningful comparison, however, depends on assessing the specific context surrounding the training (course duration, topic complexity, technology used, learners' baseline knowledge, trainers' level of expertise) and its evaluation (design, sample size, types of variables). For example, training on specialized topics, training combined with supportive supervision, and training whose evaluations are conducted right afterward generally have a larger effect size compared to general topic courses, training without supportive supervision, and training with evaluations undertaken months afterward [28, 29]. Finally, effect sizes are generally larger when the training intervention has a targeted sample specifically selected based on the assumption that participants will benefit, rather than when the sample is more diverse [29, 30]. Unfortunately, much of the existing evaluation literature lacks this important contextual information [23, 28, 31, 32].

Curriculum

From May to August 2021, the Ghana College of Nurses and Midwives and the SickKids Centre for Global Child Health developed the following courses, working in collaboration with experts from the Ghana Health Service and Ministry of Health:

  • Course One: COVID-19 in Ghana: Prevention and Health Promotion (e-learning),

  • Course Two: COVID-19 Vaccines in Ghana: Communication and Behaviour Change (e-learning),

  • Course Three: COVID-19 in Ghana: Child and Adolescent Health (e-learning), and

  • Course Four: COVID-19 in Ghana: Promoting Physical and Mental Health of Children, Families, and Health Workers (in-person).

The courses were designed to target a wide range of health workers including nurses, midwives, and physicians. The first three courses were designed for delivery via the World Continuing Education Alliance (WCEA) e-learning platform, while Course Four was designed for in-person delivery. The e-learning courses included narrative descriptions, interactive activities, and case studies where learners applied their knowledge. The participants could take as many of the e-learning courses as they wanted. The in-person course allowed for additional multi-modal teaching and learning strategies, including skills stations, role play, and case presentations. The participants of the in-person course were also provided with technical support to access the e-learning platform and were encouraged to take the other three courses. All four courses included a pre-and post-knowledge test and a commitment to change action plan. In January 2022, courses were updated to reflect new information about COVID-19 and its management.

Implementation

Approach 1: e-learning

The e-learning courses were launched in September 2021. The World Continuing Education Alliance (WCEA) e-learning platform allows users to download course content to their smartphones or other portable devices and continue the course offline. However, users must be connected to the internet to verify course completion and submit answers to multiple-choice exams and evaluation questions and to obtain their certificates. To recruit participants, the Ghana College of Nurses and Midwives and the Nursing and Midwifery Council (Ghana) promoted the courses through their websites and social media platforms. WCEA also profiled the courses on their platform to encourage participation.

Approach 2: in-person

In-person training started in November 2021 while adhering to COVID-19 risk mitigation safety protocols. A targeted approach was used to recruit health workers from identified rural areas with poor internet connectivity and who do not have frequent training opportunities. The Ghana Health Service Director-General, regional and district managers, facility managers, and the project team collaborated to select participants. Similarly, the head offices of the Christian Health Association of Ghana (CHAG) and the Ahmadiyya Muslim Mission nominated participants. During the in-person course, participants not only developed competencies around COVID-19, but they also received support to navigate the WCEA platform and were encouraged to take additional e-learning courses to support their ongoing professional development.

Methodology

Ethics approval was obtained from the Ghana Health Service Ethics Review Committee in October 2021. A set of evaluation questionnaires were designed by curriculum development team members and the project's evaluation team. A group of graduates from the Ghana College of Nurses and Midwives piloted the evaluation tools to ensure clarity of language. For in-person sessions, questionnaires identical to those used in the e-learning courses were filled out using paper surveys.

Data included and missing data

The results presented here focus on 9966 health workers who either took the e-learning courses between October 26, 2021, and February 22, 2022 (n = 9250), or attended one of 19 two-day in-person courses offered between December 29, 2021, and April 2, 2022 (n = 716). The missing data consists of (i) individuals who chose not to answer certain evaluation questions, and (ii) those who did not click on the last button on the e-learning platform and did not go to the page containing the evaluation questions and their certificate of completion. A push notification was sent out to remind and encourage the learners to complete the process, but it did not improve the rates of missing data.

Measures

Implementation: Reach was captured using a demographic survey completed after each course. A satisfaction survey (5-point scale) with an open-ended text question measured the relevance of the strategy. The strategy’s efficiencies were measured qualitatively by analyzing ongoing partner discussions throughout the project. Although comprehensive economic analysis was outside the scope of the evaluation, the financial cost of developing and delivering the courses was monitored to inform implementation. Cost categories included Human Resources (HR) and meeting costs for curriculum development, HR and platform costs for e-learning, and HR and travel costs for in-person training.

Outcomes: Health workers' capacity measurement was informed by Finn and Colleague's conceptual framework [33] and used five variables:

  • Confidence in addressing misinformation about COVID-19 with patients, colleagues, and community members (5-point scale),

  • Ability to communicate effectively to support individuals experiencing negative impact of COVID-19 (5-point scale),

  • Recommending the COVID-19 vaccine to patients and colleagues, if available (5-point scale),

  • Knowledge about COVID-19 using the score of a multiple-choice exam covering key aspects of the curriculum in each course (zero to 100), and

  • Comfort level with e-learning (5-point scale).

Responses were captured using pre–post-surveys.

Analytical approach

Statistical analysis was performed using IBM SPSS Statistics, version 25. Descriptive statistics were conducted to summarize the characteristics of the participants. The distribution of variables was analyzed and, subsequently, Wilcoxon Signed Ranks test for paired samples was used to measure the capacity of the participants regarding COVID-19 prevention and health promotion before and after each course. The only exception was the multiple-choice exam score of Course Four (in-person): due to logistical challenges, it was not possible to link the pre- and post-knowledge scores for this group; therefore, the Mann–Whitney U test was used. The significance level was set at 0.01. Effect sizes associated with Wilcoxon Signed Ranks tests and Cohen's criteria of 0.1 small effect, 0.3 medium effect, and 0.5 large effect were used to report findings [35, 36]. Since these thresholds are arbitrary and fail to consider important differences in characteristics of educational intervention, the practical significance of the results was determined by ongoing team and stakeholder discussions and situating results within the relevant literature [29, 30, 34]. A total of 4 034 qualitative comments for the open-ended survey question were analyzed using a thematic content analysis method [35], identifying the top five categories of comments. Percentages in tables are rounded and might not add up to exactly 100.

Results

Reach

As shown in Tables 1 and 2, 75% of participants were under the age of 34. Female health workers accounted for 71% of learners. Most of the participants were general nurses, enrolled nurses, or community health nurses. The top three facility types were district hospitals, health centres, and Community-based Health Planning and Services (CHPS) compounds. Participants in e-learning courses followed these overall demographic trends. Participants in the in-person course, however, were 52% female, consisted of 19% midwives (third highest frequency), and the top facility type in which they worked was CHPS compounds.

Table 1 Characteristics of 9966 Ghanaian health workers who took the four courses
Full size table
Table 2 Geographic distribution of 9966 Ghanaian health workers who took the courses
Full size table

Table 2 shows that 25% of all participants came from Central and Greater Accra Regions. Less than 4% were from North East and Ahafo regions. While the participants of e-learning courses followed these geographic trends, the participants in the in-person study mainly came from Volta, Western North, Bono, and Oti regions, reflecting the recruitment strategy described in "Approach 2: in-person" section.

Relevance

Mean overall satisfaction for Courses One (health promotion), Two (vaccine), Three (child and adolescent), and Four (physical and mental health) was 4.49 ± 0.59 (n = 6381), 4.49 ± 0.59 (n = 3385), 4.47 ± 0.57 (n = 2553), and 4.73 ± 0.48 (n = 714), respectively. For e-learning courses, participants described the learning as "relevant to my work" and "well-structured" (Table 3). They described the e-learning environment as "logically laid out" and "a faster, safe, and convenient way of learning." Some felt "encouraged to take other courses on the WCEA app" and "regret[ted] not joining the platform earlier." Interactive photos and videos made learning easier, and participants wanted more of them. However, some participants, especially those working in rural areas, reported that limited network coverage and unstable internet created difficulties for fully downloading the course content, especially photos and videos. Some participants commented about the cost of data associated with downloading and uploading content. A suggestion for improvement was to create a space on the courses' platform, where learners could interact.

Table 3 Top five categories of comments by health workers who took the courses
Full size table

Participants who completed Course Four (in-person) described it as "helpful" and "the best." They appreciated the opportunity to participate in this course, and comments were made about the need for this type of training to be conducted in person "due to poor network service in rural settings." A frequent comment was the wish for training to be repeated for other health workers "to also build their capacity." Case-based learning was viewed positively, leading to "better understanding of the material." The facilitators and the organization of the course were described as "superb." Areas for improvement included dedicating more time to training, given the content volume (Table 3).

Efficiency

Decisions regarding resource efficiency (time, human resources, technology, and financial) took place throughout the project, during both curriculum development and implementation. Curriculum development meetings between experts in Ghana and Canada took place virtually. The content team made efforts to avoid duplication of content and to focus on specific priority areas, such as child health and vaccine hesitancy, for which limited open access content was available.

Core curriculum development cost approximately 35% of the total investment for the project between May 2021 and April 2022. E-learning implementation cost 10% of the total investment for the project in the same period and reached 9250 health workers across Ghana. The World Continuing Education Alliance (WCEA) platform was selected, because over 80 000 nurses, midwives, and students in Ghana were already registered on it, representing a broad geographical reach. WCEA’s integration with Nursing and Midwifery Council, Ghana's system for continuing professional development, facilitated course accreditation.

Implementation of the in-person strategy (Course Four) cost 55% of the total investment, reaching 716 health workers living in underserved regions. Regional trainers were chosen to deliver the course, cascading training nationwide and strengthening regional training capacity.

Participant's capacity

Course one: prevention and health promotion

A total of 7243 individuals completed Course One (Table 4). The statistically significant shift in levels of agreement (strongly agree or agree) with the statements ‘I am able to effectively support individuals experiencing negative impact of COVID-19’ increased from 74.7% to 84.9% between baseline and post-training, with a medium effect size. Similarly, the statistically significant shift in levels of agreement in relation to the statement ‘I feel comfortable taking online courses’ increased from 72.2 to 84% with a medium effect size. The shifts in levels of agreement with the statements ‘I feel confident addressing misinformation’ and ‘I would recommend the vaccine’ were statistically significant with small effect sizes. The knowledge score increased by 14% from baseline to after course completion with a medium effect size.

Table 4 Participants' capacity before and after taking Course One: Prevention and Health Promotion (N = 7243)
Full size table

Course two: vaccine

A total of 4023 individuals completed Course Two (Table 5). The statistically significant shifts in levels of agreement (strongly agree or agree) with the statements ‘I feel confident addressing misinformation’ and ‘I am able to effectively support individuals experiencing negative impact of COVID-19’ increased between baseline and post-training, from 71.7% to 81.1% and from 70.9% to 80.9%, respectively. Both effect sizes were medium. Similarly, the statistically significant shift in levels of agreement in relation to the statement ‘I feel comfortable taking online courses’ increased from 71.6% to 80.6% with a medium effect size. The shifts in levels of agreement with the statement ‘I would recommend the vaccine’ was statistically significant with small effect size. The knowledge score increased by 32% from baseline to after course completion with a medium effect size. The 'clear language' used in this course was considered helpful in transmitting knowledge about COVID-19 to the larger community. Some learners said the course had 'eradicated misconceptions' and 'cleared doubts about the vaccine' (Table 3).

Table 5 Participants' capacity before and after taking Course Two: Vaccine (N = 4023)
Full size table

Course three: child and adolescent health

A total of 2892 individuals completed Course Three (Table 6). The statistically significant shift in levels of agreement (strongly agree or agree) with the statements ‘I am able to effectively support individuals experiencing negative impact of COVID-19’ increased between baseline and post-training, from 76.6% to 85%. Similarly, the statistically significant shift in levels of agreement in relation to the statement ‘I feel comfortable taking online courses’ increased from 75.1% to 84.5%. Both effect sizes were medium. The shifts in levels of agreement with the statement ‘I feel confident addressing misinformation’ and ‘I would recommend the vaccine’ were statistically significant with small effect sizes. The knowledge score increased by 25% from baseline to after course completion with a medium effect size.

Table 6 Participants' capacity before and after taking Course Three: Child and Adolescent Health (N = 2892)
Full size table

Course four: mental and physical health

A total of 716 individuals completed Course Four (Table 7). The statistically significant shifts in levels of agreement (strongly agree or agree) with the statements ‘I feel confident addressing misinformation’ and ‘I feel comfortable taking online courses’’ increased between baseline and post-training, from 79.1% to 99.3% and from 66.6% to 98.2%, respectively. Both effect sizes were large. The shifts in levels of agreement with the statements ‘I am able to effectively support individuals experiencing negative impact of COVID-19’ and ‘I would recommend the vaccine’ were statistically significant with medium effect sizes. The knowledge score increased by 19% from baseline to after course completion with a medium effect size. Participants reported being encouraged to take online courses (Table 3).

Table 7 Participants' capacity before and after taking Course Four: Mental and Physical Health (n = 716)
Full size table

Discussion

Judged against the evaluation criteria (broad reach, efficiency, and relevance) the implementation of the strategy was found to be successful. Rapid curriculum development using virtual meetings was made possible by a well-established partnership, and good working relationships between Ghanaian and Canadian partners. Investment was maximized by delivering training to a broad range of health workers across Ghana using a two-pronged strategy of e-learning and in-person training, each offering benefits and challenges in a Ghanaian, and pandemic, context.

The speed at which health workforce training can be scaled up is particularly important during a pandemic, when timely information is essential. Despite some initial concerns about the feasibility of this approach, particularly around the reach and uptake of e-learning courses, this strategy was successfully implemented, reaching a large number of health workers (n = 9250) across Ghana in a 6-month period. Awareness building of the courses and choosing a learning platform that was already known to health workers in Ghana improved efficiency. However, as has been observed by others [3, 21, 26], the evaluation identified challenges with e-learning, including unstable internet and the cost of data for downloading and uploading content.

As others have observed [25], the in-person training took considerably more resources than e-learning, especially given the need for pandemic-related safety and logistics. Compared to e-learning, fewer individuals (n = 716) were able to be trained via this training modality. However, the in-person course offered hands-on learning to health workers who needed COVID-19-related training and were more likely to experience barriers to accessing e-learning.

Learners found the courses relevant to their needs and their feedback identified several implication for future design of such courses. For the e-learning courses, interactive content, visual aid (photos, videos), and small file sizes to help with the download of course content were important. For the in-person course, ensuring the optimal content-to-delivery time ratio and using plain language learning material proved important.

All four courses contributed to strengthening health workers' capacity regarding COVID-19 prevention and health promotion but the effect size (i.e., the amount of change) varied from small to large depending on the course and variable measured (Tables 4, 5, 6 and 7). For e-learning courses, the effect sizes varied from small to medium, and for the in-person course, the effect sizes varied from medium to large. While these effect sizes are useful in painting an overall picture, with education evaluation, a 'small' effect size on a difficult-to-change variable (e.g., attitude toward recommended the vaccine) could be as valuable as a larger effect size on something easier to change (e.g., knowledge) [29,30,31]. In some situations, the initial levels of confidence were already high at baseline, thus a large change (large effect size) would not be expected or possible.

Although the in-person course is not directly comparable to the e-learning courses due to several key differences (length, teaching methods, and participants' characteristics as described in "Approach 2: in-person" section), the observed large effect sizes for the in-person course were encouraging, albeit expected. As mentioned earlier, the project made concerted efforts to target participants with less access to training opportunities for the in-person course ("Approach 2: in-person" and "Reach" section); targeted samples typically result in larger effect sizes [29]. Furthermore, the in-person course allowed the participants to ask the trainers questions in real-time, work through a hands-on exercise about addressing misinformation, and receive in-person support for accessing the e-platform, which are likely to have contributed to the observed larger effect sizes for these variables (Table 7).

Limitations and strength

This evaluation lacks a longitudinal perspective as no follow-up evaluations were conducted to determine whether participants were applying what they learned in their practices [36]. It was beyond the scope of the evaluation to conduct a comprehensive economic analysis, and as such, only costs to the program were calculated. Other costs such as the cost to the health system when health workers leave work to participate in training, or the cost of data for e-learners were not analyzed. Similarly, benefits of training master trainers for the in-person course were not part of this cost analysis. Evaluation was done internally by the research teams at SickKids and the Ghana College of Nurses and Midwives, with the potentially biases associated with internal evaluation. Strengths associated with this evaluation include the use of both quantitative and qualitative data [37]. In addition, a broad range of stakeholders were involved in the interpretation of results, making them valid and useful [38]. Finally, this evaluation contributes to bridging a gap in the assessment of large-scale continuing professional development efforts in low-resource settings, especially e-learning interventions.

Conclusions

The success of the strategy was due to well-established partnerships, the quality and relevance of the curriculum, and the two-pronged delivery approach which maximized reach while reducing barriers to accessing education. Health workforce training efforts need to be accompanied by other investments in health systems, notably facility infrastructure, faculty development, and good quality data.