Background

Under-five mortality remains persistently high in Pakistan at 74/1000 live births and the country has one of the world’s highest neonatal mortality rates (42/1000 live births) in the world [1]. Diarrhea and pneumonia have a high prevalence among Pakistani children—incidence of diarrhea and acute respiratory infections is 19% and 14% according to Pakistan Demographic Health Survey (PDHS) [1]—and the two diseases are responsible for deaths of 90,000 under-five children every year [2]. A large proportion of these deaths is preventable if effective and low-cost interventions such as oral rehydration therapy (ORT), breastfeeding, continued feeding during illness, zinc supplementation, and antibiotics are available and appropriately used [3]. However, only 37% of children under five with diarrhea in Pakistan receive ORS, 8% get zinc supplements and only 46% children with acute respiratory infections or pneumonia receive antibiotics [1]. Improving access and utilization of these lifesaving interventions is crucial if Pakistan is to achieve its sustainable development goal (SDG) 3.2 targets to reduce neonatal mortality to at least 12 per 1000 live births and under-5 mortality to at least 25 per 1000 live births [4, 5].

Access to prompt, appropriate and affordable treatment is critical to reduce childhood mortality due to diarrhea and pneumonia. Community-based interventions involving community health workers (CHWs) have been introduced in many countries to increase the access to services for deprived population for management of these illnesses [6, 7]. World Health Organization (WHO) and UNICEF recommend that integrated community case management (iCCM) of these diseases through CHWs is an effective way of increasing access to quality and affordable services.

CHWs are lay health workers, selected from their communities, and imparted with short-term training [8]. Evidence shows that CHWs can effectively assess, treat, and refer diarrhea and pneumonia cases to health facilities [9]. In Pakistan, National Program for Family Planning and Primary Health Care commonly known as Lady Health Worker’s Program (LHW-P) was launched in 1994 to improve access to community health services in rural and peri-urban areas. Community health workers—called lady health workers (LHWs)—are selected from their local communities based on pre-defined criteria. They are given 15 months of classroom and field-based training. Each LHW is responsible for serving 100–150 households or 1000 population by visiting five to seven households daily. LHWs deliver multiple promotive, preventive and basic curative maternal and child health services including community-based management of diarrhea and pneumonia [10].

An external evaluation of the Program carried out in 2019 found that the program has not been effective in improving health outcomes for infant and child health and that lack of programmatic support and supervision is one of the underlying reasons [11]. An earlier external evaluation reported that 61% of LHWs received no feedback from their supervisors, hampering development of skills in diagnosing and managing ill children in their community [12]. Global evidence also shows that the most common reasons for suboptimal knowledge and skills among CHWs are poor quality training and lack of quality supervision [13]. There is need for more effective and efficient measures of supervision of CHWs for implementation of iCCM. Evidence suggested that supportive supervision improves worker motivation, performance, and quality of care [14,15,16,17,18]. It strengthens the trust relationship between CHWs and supervisors, two-way communication, and teamwork [14, 18]. It promotes problem-solving, cross-learning and skill sharing [18].

In order to address the supervisory gaps in LHWP, trials Nigraan and Nigraan Plus were designed and implemented as sequential clustered randomized controlled trials (RCTs) in Badin and Mirpur Khas (MPK) districts, respectively. Both trials assessed the impact of enhancing structured supportive supervision by LHSs to LHWs’ on CCM of childhood diarrhea and pneumonia knowledge and skills for management of childhood diarrhea and pneumonia in the community. This study compares the findings of the two trials and proposes recommendations for scaling-up this intervention.

Methods

Study design and setting

Nigraan and Nigraan Plus trials were cluster randomized control trials conducted in districts Badin (2015) and MPK (2019), in Sindh province.

Badin district is located at a distance of 212 km from the provincial capital, Karachi. It has a population of about 1.35 million, of which 84% is rural, and 24.63% are literate [19, 20]. MPK is an adjoining District 220 km from Karachi, has a total population of 1.5 million, 72% live in a rural setup, and 36.8% are literate [20, 21]. Badin has 11 rural health centers, 34 basic health units and 5 secondary level facilities, 1 district headquarter hospital and 4 taluka headquarter hospitals. A network of about 1100 LHWs, supervised by 36 lady health supervisors (LHSs) cater to maternal and child health needs at community level.

In MPK, there are 6 rural health centers and 38 basic health units, 2 taluka and 1 district headquarter hospital [22]. A total of 983 LHWs, supervised by 39 lady health supervisors (LHSs) provide services in the community [23]. A large number of private healthcare providers deliver care to people in both districts. Incidence of diarrhea and pneumonia in MPK is 30.3% and 10.4% whereas in Badin these are 38.7% and 9.7%, respectively [24].

Supervision mechanisms in the Lady Health Worker Program

LHWP’s supervision system comprises a network of LHSs each of whom oversees and mentors 25–30 LHWs. An LHS provides hands-on supervision by visiting each LHW reporting to her at least twice every month.

During these visits, the LHS accompanies the LHW during household visits to the community and provides verbal feedback to the LHW on her performance. All LHWs provide data from the patients they attended during the month to LHSs in monthly meetings; LHSs collate this data and report it to the district program office every month.

Sampling and data collection

Sample size

Based on incidence of pneumonia in children < 5, in Badin, the sample size was estimated at 34 LHSs to achieve 80% power for estimating the effect of the intervention at 95% confidence interval. Thirty-four functional LHSs were allocated randomly to either intervention or control arm with a ratio of 1:1. Of 25–30 LHWs working under each LHS, five were randomly recruited, to a total of 85 LHWs in each arm. Similarly using an assumption of 15% increase in ORS and antibiotic use, estimated sample size was 16 LHSs in each arm in MPK. Thirty-two functional LHSs were randomized: 16 each in intervention and control groups. Of 25–30 LHWs working under each LHS, five were randomly recruited, to a total of 80 LHWs in each group.

Intervention design

The intervention in the trials consisted of an enhanced supervision training for the LHSs in the intervention arm. After the training the LHSs regularly visited LHWs in the intervention arm in the field and provided community-based enhanced supervision through written feedback. To facilitate communication and to allow surveillance and reporting of diarrhea and pneumonia cases, all LHSs were given mobile phones and given a monthly communication allowance so that they could communicate with LHWs.

Development of enhanced supervisory training curriculum and LHS training in both districts

A detailed training manual was developed which provides training objectives, pedagogy and training content for the enhanced supervision intervention. Multiple instructional methodologies were used including face-to-face interactive lectures supplemented with role-play, audio-visual aids, simulated exercises, and hands-on training in a health facility.

Since workers in the LHWP had not had an iCCM training in about a decade, a 2-day refresher training based on existing LHWP guidelines was provided to all study LHSs (intervention and control).

LHSs in the intervention arm were then provided an additional 4-day training based on the enhanced supervisory training curriculum to improve their clinical and supervisory skills, including skills on effectively providing feedback.

Improved communication and case surveillance

A short message service (SMS) communication system was added to the existing management information system of LHWP to track diarrhea and pneumonia cases in both trials. LHWs identified cases in the community and informed LHSs after which LHS conducted a supervisory visit within 24 h. During the visit, LHS reviewed the LHWs case management practices and provided written feedback. The SMS surveillance system tracked cases reported by LHWs to LHSs. A log was maintained of all reported cases and from this list, some cases were randomly selected for skill assessment of LHWs and LHSs periodically.

Enhanced supportive supervision through written feedback cards

Besides supervision skills training for LHSs, use of modified or additional supervision tools, provision of written feedback to LHWs, active and prompt case reporting by LHWs were part of the intervention package [25]. Intervention arm LHSs provided written feedback to LHWs using a structured written feedback card. This activity added approximately 10 min to each visit’s duration. LHSs in both arms provided verbal feedback as part of the existing program protocol.

LHS and LHW skill assessments

Diarrhea and pneumonia management skills of all study LHWs and LHSs were studied periodically. In MPK, the skills of all LHWs were assessed during midline and endline assessments, while for the baseline assessment, 51 LHWs (27 intervention and 24 control) were selected randomly for assessment of skills related to case management of diarrhea and 58 LHWs (28 Intervention and 30 Control) for assessment of skills pertaining to pneumonia. Similarly, in Badin 25 LHWs were chosen randomly for assessing CCM skills for diarrhea and pneumonia at baseline midline and post-intervention.

Skill assessment scorecards

In both trials, clinical and supervision skills were assessed by independent evaluators using structured clinical and supervision skill assessments scorecards. The iCCM clinical skill assessment scorecard was developed based on LHWP curriculum. This tool was used to assess history taking, physical examination, diagnosis, treatment and counselling skills of LHWs and LHSs. A structured scorecard was used to assess LHSs’ supervision and mentoring skills. The clinical mentoring section assessed LHSs’ ability to evaluate LHWs’ clinical skills whereas the supervision component assessed LHSs’ provision of verbal and written feedback and her demeanor.

Data analysis

For both trials data were analyzed using SPSS version 21 [26]. Descriptive statistics were computed for socio-demographic characteristics and skill levels of LHWs/LHSs. Mann–Whitney U, independent-sample t, Pearson’s Chi-square tests were used to assess socio-demographic difference between intervention and comparison arm. A p-value < 0.05 was considered to be significant.

Ethical consideration

Ethical approval for both trials was obtained from the Aga Khan University Ethical Review Committee. An informed written consent was obtained from each LHS and LHW for their involvement at the beginning of each trial. Both trials are registered with the ‘Australian New Zealand Clinical Trials Registry’. Registration numbers: Nigraan Trial: ACTRN1261300126170; Nigraan Plus: ACTRN12617000309381.

Results

Socio-demographic characteristics

In MPK, the intervention arm LHSs and LHWs’ median age, education, median years of experience with LHWP, lives in house with under 5 years child, and median time to reach health facility were comparable with comparison arm LHSs and LHWs, respectively. However, compared to study arm LHWs, comparison arm LHWs take more time to reach health facility. Similarly, in Badin, LHSs and LHWs in both arms were comparable in most socio-demographic characteristics except for a higher median age in comparison arm LHWs and LHSs, and higher median years of experience in comparison arm LHSs (Table 1).

Table 1 Socio-demographic characteristics
Full size table

Lady health worker performance for iCCM of diarrhea

Improvement was observed in intervention the arm LHWs’ performance for iCCM of diarrhea between first (pre-intervention) and third assessments in both trials. For instance, enquiring about danger signs (60% to 96% in Badin and 89% to 100% in MPK), correctly assessing for dehydration (28% to 92% in Badin and 74% to 95% in MPK), correctly classifying the illness (24% to 68% in Badin and 85% to 96% in MPK), advising correct treatment (12% to 28% in Badin and 63% to 84% in MPK). Comparison arm showed little improvement for most of the indicators in Badin and few indicators in MPK. In comparison arm, the proportion of LHWs correctly assessing for dehydration increased in Badin trial (28% to 64%), but decreased in MPK (71% to 26%). Similarly, comparison arm LHWs’ performance for counselling the caregivers to seek care immediately if child has danger signs of diarrhea improved in Badin (40% to 96%) while it deteriorated in MPK (63% to 19%). Furthermore, comparison arm LHWs performance indicators such as classifying diarrhea correctly (28% to 40% in Badin and 75% to 83%) and referring the child to doctor for correct indication (36% to 92% in Badin and 54% to 66% in MPK) improved in both trials (Table 2).

Table 2 LHWs’ skills for diarrhea
Full size table

Lady health worker performance for iCCM of pneumonia

Improvement in intervention arm LHW skills for iCCM of pneumonia was observed for some indicators between pre-intervention and post-intervention assessments in Badin and MPK. LHWs performance for measuring the respiratory rate correctly increased considerably between first and third assessments in intervention arm of both trials (12% to 44% in Badin and 46% to 76% in MPK). Similarly, the proportion of LHWs counselling caregivers to seek care immediately if child has danger signs increased greatly between first and third assessment in intervention arms of Badin and MPK (36% to 100% in Badin and 50% to 83%). Likewise, other indicators of performance such as classifying illness correctly and advising correct treatment also improved in intervention arm of both trials (Table 3).

Table 3 LHWs’ skills for pneumonia
Full size table

Lady health supervisor performance for iCCM of pneumonia

The proportion of LHSs correctly counting the respiratory rate increased in intervention arm of both trials between first and third surveys (18% [n = 16] to 80% [n = 15] in Badin and 42% [n = 12] to 88% [n = 16] in MPK). Similarly, intervention arm LHSs’ performance of correctly classifying illness enhanced between pre- and post-intervention assessments in Badin trial (0% to 100%), but remained unchanged in MPK (was already 83% at baseline).

Lady health supervisor performance for iCCM of diarrhea

Comparing baseline to endline survey, LHSs’ practice of assessing the dehydration correctly improved in intervention arm of both trials (6% to 100% in Badin and 92% to 100% in MPK). Likewise, LHSs’ skill of classifying illness correctly was relatively high in intervention arm of Badin (62% to 73%) while exhibiting a slight decline in MPK (92% to 88%) between pre- and post-intervention assessments.

Lady health supervisors’ supervisory performance

In Badin and MPK trials, most of the indicators of intervention arm LHSs’ supervisory performance showed improvement between pre- and post-intervention assessments. In both trials, LHSs practice of giving written feedback improved between baseline and second assessments (0 to 88% in Badin and 25% to 75% in MPK). Furthermore, in intervention arm, the practice of LHSs giving verbal feedback improved between pre- and post-intervention assessments in both Badin and MPK trials. In terms of clinical supervision, the proportion of LHS reviewing treatment suggested by LHWs increased between first and third assessments in both trials (50% to 94% in Badin and 38% to 94% in MPK) (Table 4).

Table 4 Supervisory performance of lady health supervisors
Full size table

Discussion

The results of Badin and MPK trials depict that overtime in the study arm there was improvement in LHSs and LHW performance for correctly assessing dehydration, counting respiratory rate, classifying the illness, advising correct treatment, counselling caregivers to continue feeding and seek care immediately if child has danger signs of diarrhea or pneumonia. There was also improvement in clinical mentoring and supervision by the LHSs. There was high uptake of feedback provision by LHSs and this was largely maintained throughout the study especially for the verbal component. Moreover, LHSs practice of providing written feedback improved as a result of the supportive supervision intervention.

These findings are similar to results of a study conducted in Kenya, which showed that the training along with clinical coaching and continuous supportive supervision improved and maintained CHWs’ skills related to iCCM of diarrhea and pneumonia [27]. Additionally, a study from South Sudan showed an improvement and retention of correct disease identification and treatment skills of CHWs following a supportive supervision intervention [28].

Skills-based training loses its impact if regular supervision and support are not provided [29, 30]. A study conducted in Guinea-Bissau reported that CHWs’ skills of diagnosis and management of childhood diarrhea improved following training but 3 months later the effectiveness of training decreased [29]. This can also be reflected from the results of both trials, poor skills at baseline and subsequent improvements in skills of LHSs and LHWs. This implies that CHWs can improve skills mastery and retention following uninterrupted supportive supervision and monitoring.

Frequency of providing feedback during the supervisory visits improved in both trials. Feedback is an essential component of supportive supervision, and it influences the motivation and performance of workers. A study conducted in Uganda showed that CHWs’ performance in managing diarrhea, pneumonia, and malaria increased five times when workers received feedback from their supervisor [31].

Some improvements were also noticed in comparison arm of both trials. This is not surprising. Refresher training was provided to comparison arm which could have brought improvements in the outcome indicators. Additionally, monitoring of comparison arm LHWs was also carried out, which could have motivated comparison arm LHSs and LHWs to update their knowledge through available local resources. Furthermore, all LHSs of the district (both from study and comparison arm) assemble every month at the same office for a collective appraisal meeting as per LHWPs administrative setup. Moreover, it is much harder to limit the spread of intervention if the participants of intervention and comparison are from the same geographical locality [32].

The improvements in the intervention arm could have been more substantial. However, a number of contextual factors could have hindered the study implementation or intervention effectiveness such as lack of transportation facilities, inadequate supply of medicines, excessive burden of ancillary activities on LHWs and LHSs, and delays in salary disbursement [11]. Lack of transportation could also have prevented LHSs from making frequent supervision visits.

Our study has some potential limitations. Both intervention and control arms were located in the same districts so there may have been some transfer of knowledge between intervention and control LHWs and LHSs. This may have led to some underestimation of the improvement in the intervention arm. Due to the Hawthorne effect, health workers in both intervention and control arms may have performed in a different manner than they normally would leading to some differences in our assessment and real-world behavior.

The findings of both trials have essential implications for scaling-up the intervention. Since 2010 provincial governments have been given more responsibility for gradually taking over the funding and managing of the LHW programs. Their plans for reforms and upscaling of the program can benefit from these findings. Recent LHWP evaluations also suggest that improvement is needed [11]. Findings of both trials therefore suggest that enhancing the skills of LHSs to provide frequent, quality supportive supervision to LHWs can be one way of improving performance of the community-based services of LHWs for childhood illnesses. Future research could focus on effective approaches to sustainably embed these interventions in existing health systems and assess effectiveness in terms of population level impact such as on morbidity and mortality indicators.

Conclusion

The study found improvements in LHSs and LHWs performance for correctly assessing dehydration, measuring respiratory rate, classifying the illness, advising correct treatment, counselling caregivers to continue feeding and seek care immediately if child has danger signs of diarrhea or pneumonia. There was also progress in LHSs clinical mentoring and supervision. The findings of this study advocate potential benefits of implementing a supportive supervision intervention through LHSs with the potential to improve iCCM of childhood diarrhea and pneumonia. This can serve as a reference for scaling-up the intervention at provincial level. Training supervisors to provide detailed and frequent feedback in a supportive manner can improve the quality of care provided by the LHWs in large scale programs. Planners of programs at scale must consider the budgetary and operational aspects of such investment in human resources for sustainable delivery of quality care at community level.