Background

The majority of developing countries are not on target to achieve Millennium Development Goals four and five: to reduce the mortality rate of children under five by two-thirds and the maternal mortality ratio by three-quarters between 1990 and 2015 [1]. Malaria is a key focus for achieving these goals, with the latest World Health Organisation (WHO) guidelines promoting access to improved case management, including diagnostic testing for all suspected cases [2]. Failure to reach health targets has often been blamed on 'health system bottlenecks’ [1, 3], typically cited as inadequate numbers, quality and distribution of health workers, equipment, supplies, and infrastructure [4]. In addition, some have argued that the way health services and programs are enacted in practice is a social as well as a structural issue; a function of interactions between clients, communities, health workers, and systems [5, 6].

In spite of significant investments in Uganda since the early 1990s into programmes intended to improve health and access to quality healthcare, health outcomes remain poor across the country [7]. This poor progress has been attributed by some to a lack of alignment between externally defined programme priorities, and the priorities of local populations [8–12]. A more comprehensive approach to healthcare is called for to improve management of malaria and other febrile illnesses, attract patients to seek care, and produce health benefits at the population level.

Reviews of empirical research suggest that simple interventions to improve access to quality malaria care such as basic training or health education have had limited effect on changing healthcare provider behaviour [13, 14] or the behaviour of populations [15]. Achieving a change in behaviour may require complex interventions that address the multiple factors involved with delivery of appropriate treatment [16, 17]. After undertaking formative research in Tororo, eastern Uganda, to understand local realities and aspirations for quality of care, an intervention package was designed to improve access to good quality healthcare for the local population [18]. The PRIME intervention consists of four components: workshops in Health Centre Management for health workers in-charge of facilities; training and supervision visits in Fever Case Management for all health workers; workshops in Patient Centered Services for all health workers; and support of supply of artemether-lumefantrine (AL) and rapid diagnostic tests (RDTs) when stocks run low. Table 1 shows the topics for each module covered in the PRIME intervention. The manuals for delivering the intervention are available online at http://www.actconsortium.org.

Table 1 PRIME intervention components, modules and topics
Full size table

The primary and secondary outcomes of the PRIME intervention are being evaluated in a cluster-randomised trial comparing health centres that receive the intervention with 'standard care’ health centres that do not receive the intervention. The PRIME trial is evaluating outcomes on three levels: an annual cross-sectional community survey will assess the impact of the intervention on key population-based health indicators in children under fifteen, with the primary outcome as prevalence of anaemia in children under five; a cohort study will assess the impact of the intervention on key longitudinal indicators in children under five, with antimalarial treatment incidence density as the primary outcome; and patient exit interviews will assess the impact of the intervention on key indicators of case management for malaria and other illnesses in children under five treated at health centres, with the primary outcome as inappropriate treatment of malaria. The timing of the PRIME evaluation activities can be seen in Figure 1 and further details of the trial protocol can be found elsewhere [19].

Figure 1
figure 1

Timeline of activities for the ACT PROCESS study.

Full size image

Many researchers are now arguing for more comprehensive evaluations of complex interventions that attend to implementation, mechanisms of change, and context [20]. Such evaluations have been uncommon, and many of those that have existed alongside randomised controlled trials have been critiqued for poor integration with quantitative findings and methodological limitations [21], prompting the challenge for more carefully planned evaluations. In the PROCESS study, we adopt a theory-driven approach to our evaluation, aiming to understand what the PRIME intervention was and what it did, and aiming to contribute to broader discussions of how quality of care may be changed in similar contexts. It has been argued that carrying out theory-driven evaluations, including realist evaluation and the theory-of-change approach, is a way forward for opening the 'black box’ and exploring the modes of effects of complex interventions [22, 23], including within randomised controlled trials [24]. There is much variation in the concerns and methods used by those adopting a theory-driven evaluation approach [25, 26], although most propose to explicate a theory or model of the programme/intervention and use this to guide and strengthen evaluation questions and analysis [27]. We set out to understand the PRIME intervention and its actions by mapping out the intended intervention programme and contrasting this with the realities of implementation in practice and local interpretations of intervention effects, as well as interpreting contextual influences and attempting to assess impact within and outside of the intended consequences of the intervention.

Methods and analysis

Aim

The PROCESS study aims to evaluate the implementation, mechanisms of change and context of the PRIME intervention at health centres in rural Uganda to inform interpretation of outcomes in the main cluster-randomised trial.

Objectives

  1. 1.

    To develop a comprehensive logic model of the PRIME intervention, mapping components of the intervention through to their intended effects and outcomes.

  2. 2.

    To evaluate the implementation of the intervention, including health worker training, health centre management tools, and the support of supply of AL and RDTs for malaria when stocks run low.

  3. 3.

    To understand mechanisms of change of the intervention components, including testing hypotheses and interpreting realities of intervention components in context.

  4. 4.

    To develop a contextual record over time of factors that may have affected intervention implementation, mechanisms of change and trial outcomes, including factors at community, health centre and district levels.

  5. 5.

    To capture broader expected and unexpected impacts of the intervention and trial activities in communities, public health centres, and at private providers.

Study setting

The PRIME trial is taking place in seven sub-counties of Tororo District, eastern Uganda. Detailed descriptions of the health centres, population profile relating to health, and a history of interventions in the area can be found elsewhere [18, 28]. All lower-level public health centres in the study area (n = 22) were eligible to participate, but due to overlapping catchment areas for two pairs of health centres, one from each pair was randomly excluded. A total of 10 health centres were then randomly allocated to receive the intervention and 10 to continue with standard care (n = 10). Further information about the randomisation process and other details of the main trial can be found in the trial protocol [19]. The area is highly endemic for malaria, with an estimated 562 infective bites per person per year [29]. The clinical focus of the intervention was therefore on the diagnosis and management of malaria.

Study design

The PROCESS study is a mixed-method evaluation. Figure 2 depicts our framework for the evaluation activities. Our focus in this study is on documenting the PRIME intervention, understanding the mechanisms involved in the PRIME intervention in practice, and describing the context of the PRIME intervention and evaluation. The majority of impact evaluation activities are being carried out under the PRIME trial [19].

Figure 2
figure 2

Framework for ACT PROCESS study.

Full size image

Objective one: development of a logic model

The articulation of an intervention’s intended pathway of change is recommended as a starting point for evaluation [30]. The output of this articulation is variously referred to as a 'change model’ [31], 'logic model’ [32], and 'theory of change’ [33]. Here, we use the term 'logic model’ because we see it as a display of the logic specific to this intervention trial rather than a theory that has broader application. The logic model for the PRIME intervention trial intends to articulate a set of hypotheses and assumptions upon which the trial’s outcomes and activities are based. The logic model is intended to set out the pathway of change from the PRIME intervention inputs through to the outcomes measured in the PRIME evaluation, incorporating proximal mechanisms and the conditions assumed to be required in order to support change. The development of the logic model is intended to be a process that can lead to refinement of intervention design as well as guide the design of evaluation activities [34]. Its development is therefore planned to be a dynamic process, occurring alongside intervention development, and involving team members involved in both the PRIME intervention and PRIME evaluation activities.

Objective two: implementation evaluation

The implementation evaluation aims to document how the intervention is delivered and received, and to compare this with intended implementation. This should equip the research team to know whether trial outcomes can be attributed to the intended intervention or if there was implementation failure [22]. The intended implementation, consisting of inputs, process, and outputs—elsewhere termed 'implementation theory’ [23], 'process theory’ [35], or 'action model’ [31]—will be described elsewhere, including the rationale for the design. The implementation will be assessed in terms of the aspects detailed in Table 2, following Saunders et al.[36], for each of the PRIME training intervention components (Health Centre Management, Fever Case Management, and Patient- Centered Services).

Table 2 Implementation evaluation assessment domains, questions and methods
Full size table

Self-filled questionnaires

Data will be collected through self-filled questionnaires completed by all trainers (n = 5) and participants (n = 40 – 50) after each workshop, for which written consent will be obtained at the first workshop. Questions will elicit yes/no responses, four-point Likert scale responses to statements from 'strongly agree’ to 'strongly disagree’ and open-ended responses. Participants will be encouraged to complete questionnaires honestly, with no repercussions for involvement with the project and with confidentiality assured. Observational data will also be collected in a non-structured format by the implementation evaluation team during trainings, to record any aspects relevant to fidelity of the implementation, dose delivered and received, recruitment, and contextual factors. Finally, in-depth interviews with implementers, stakeholders and health workers will be carried out 9 to 12 months after the training, with topic guides including questions relating to the implementation of activities. Analysis will involve descriptive statistics and content summaries of the open text fields in the questionnaires. In-depth interviews will be analysed thematically, using NVivo software to code the transcripts.

Objective three: mechanisms evaluation

The mechanisms evaluation aims to describe the 'actual’ pathways of change from intervention inputs to measured outcomes, enabling comparison with the intended pathways of change in the trial’s logic model. This will involve two lines of questioning: first, testing hypotheses from the logic model, evaluating whether intended mechanisms 'work’ as planned; and second, understanding the way that the intervention ideas, principles, and materials are taken up, adapted, contested, rejected, and recast in context.

Hypothesis testing

The ability to test hypotheses within the intended pathways of change of an intervention trial is important for the trials such as PRIME where a randomised approach can be undermined by a long and complex pathway from the intervention inputs delivered to health workers and the trial outcomes at the population level [37]. Hypothesis testing enables the research team to establish plausibility that outcomes are attributable to the action of the intervention, rather than to other factors or a different mechanism triggered by the intervention [38, 39]. Hypothesis testing also enables consideration that outcomes are attributable to particular aspects of the intervention.

Semi-structured questionnaires

For each of the intervention components, questionnaires will be designed for self-completion by all health workers (n ≅ 60) and all health workers in-charge (n ≅ 20) in both arms of the trial between 9 and 12 months after the intervention started. The questionnaires will attempt to assess change in line with each intended output of the intervention through the comparison between arms of a series of responses to statements with four-point Likert scale response options, from 'strongly agree’ to 'strongly disagree.’ Respondents will be asked to provide written consent at the start of the questionnaire. Open-ended questions will also be included on each topic to encourage expansion by respondents. Questions to assess the Health Centre Management component will ask for health worker in-charge experiences and opinions on financial management, stock management, and health management information systems. Questions to assess the proximal impact of Patient- Centred Services, will be designed to assess 'patient-centeredness,’ adapting a tool tested in several Northern countries [40]. An assessment of health worker motivation and feelings towards work will be undertaken within the questionnaire by adapting a tool piloted in Tanzania [41], generating scores for 'internal motivation’ and 'external motivation.’ Questions about malaria case management will cover knowledge, confidence, trust in tests, and other factors that may hinder or support quality case management. For each domain, aggregate scores will be created and will be compared between arms using one-way analysis of variance. Clustering of responses by health centre will be assessed through the intra-class correlation coefficient and will be adjusted for in the analysis if significant (ρ >0.1) using a random effects linear regression model in STATA (Statacorp, Texas). Finally, health workers in the intervention arm only will be asked to reflect on the usefulness of the training they received from PRIME and the frequency they used the information and skills learned. Descriptive statistics will be used to report on responses to each question.

Communication assessments

A key hypothesis in the PRIME intended pathway of change is the improvement of health worker communication due to participation in the Patient-Centered Services workshops, and the impact of this on patient or caregiver satisfaction with the consultation, specifically with the interpersonal skills of the health worker. We will attempt to assess both the patient-centeredness of health worker communication in both arms, and the satisfaction of a subset of caregivers of children under five in both arms. This will first involve the audio recording of consultations, which will be rated for patient-centeredness following domains and rating methods developed by the Patient-Doctor Communication Group in Canada [42, 43]. Second, caregivers will be interviewed on exit from the consultation and asked a series of questions following the same domains as the audio recording rating, with questions adapted from an existing questionnaire also developed in Canada [44]. The questionnaire will be translated and pretested. Responses will be allocated an aggregate score for each domain, following the format of the recorded consultations. The target sample size is 100 consultations and exit interviews, spread equally across the 10 intervention and 10 standard care health centres. Both health workers and caregivers will be informed about the study and asked to give written consent before participating. We will repeat this communication assessment three times: at baseline, immediately after the intervention training, and between 9 and 12 months after the intervention. At each time point, the same health workers will be sought out to participate to enable within-subject consistency. Comparisons of scores for the consultations and exit interviews will be drawn between arms using the same methods as for the semi-structured questionnaires described above.

Enactment of the intervention

The ability to understand how the intervention was interpreted and enacted by different actors is especially important in cluster trials with relatively few units randomised to each arm; 10 in each for the PRIME trial. Here there is potential for large between-cluster differences in outcomes and receptiveness to interventions, which limits interpretation of results of cluster randomised trials [37]. An in-depth understanding of the context into which the intervention is introduced, and of the interpretations of the intervention by actors expected to make changes is important to learn what was useful about the intervention, for whom, and why. This should enable more meaningful interpretation of the generalisability of the intervention to different settings [24].

In our endeavour to understand how the intervention is enacted, we are concerned with the meanings it presents to different actors, drawing on an interpretativist perspective [45] as well as with the socio-material network of resources, forms, people, and groups that enact the intervention, drawing on a relational perspective [46]. While we are inspired by the premise of realist evaluation, with an aim to understanding 'what works, for whom, in what circumstances, and why?’ [47], we do not follow closely the methodology of interrogating many potential Context-Mechanism-Outcome triads, which assume each are bounded entities. Rather, we see the intervention as a script that is produced or reproduced in different activities and interactions, generating meaning in an ongoing sense for actors engaging with it. Our focus in this part of the PROCESS evaluation is therefore to understand the socio-material elements that constitute change, including but not limited to the methods, contents, and materials of the PRIME intervention.

In-depth interviews

We will carry out in-depth interviews with a range of actors to elicit narratives of the intervention and its objectives in context, including its effects and factors shaping interpretation and integration of its ideas, resources, and processes. We will invite one health worker per intervention health centre to be interviewed (n = 10), as well as a range of sub-county and district representatives from across the study area (n = 10). We will also interview implementers of the PRIME intervention including trainers and those involved in delivering supplies (n = 6). Interviews will be tape-recorded, transcribed, and translated where necessary. Transcripts will be imported into NVivo version 8 (QSR International) and coded iteratively: ideas emerging will be labelled and grouped into themes as patterns emerge. Particular attention will be paid to understanding what and how changes are perceived to have occurred as well as analysis of the way each intervention was taken up, or not, in practice.

Focus group discussions

To understand interpretations by community members of the PRIME intervention, which intended to impact health outcomes through enhancing health centres, we will carry out a series of focus group discussions (FGDs). Two target groups from populations in the study area will be invited: primary caregivers, representing those most frequently visiting public health centres; and household heads, representing those with influence over family resources as well as those privy to local political discourses. Three further sub-groups will be included according to location: FGDs will be held with groups of participants living in close proximity (within a two-kilometre radius) of intervention health centres; in close proximity to standard care health centres; or outside of a two-kilometre radius of either intervention or standard care health centres. We aim to carry out 12 FGDs following the matrix shown in Table 3. Participants will be invited in advance, selected with the help of local leaders, and will be asked to give written consent prior to the start of the FGD. Discussions will be tape-recorded and field notes will be taken. Transcription, translation, coding, and analysis will take place as for the in-depth interviews.

Table 3 Sampling matrix for number of community focus group discussions
Full size table

Objective four: context record

The context record aims to document information relating to the context of the trial that may have affected the intervention’s implementation, mechanisms of change, and the outcomes under measurement. A data collection exercise will be undertaken every three months, starting before the intervention and ending after one year (time points = 5), to record information sources at three levels: district health officials (n = 5 – 10); health centre staff (n = 20); and community representatives (n = 10 – 15, including health assistants and lay key informants from each sub-county). At each time point, respondents will be asked a series of structured and open questions to elicit information about the past three months in terms of any (non-trial) activities, events, policies, infrastructure, human resources, media stories, environmental or other changes that have occurred that might have impacted: health workers’ ability to engage with the PRIME intervention and to provide quality care; patient and caregivers’ ability or willingness to access care at public health centres or elsewhere; and health outcomes at the population level. The data will be typed into Microsoft Word and reviewed to extract relevant information into a timeline in Microsoft Excel. Activities/events will be colour-coded into the following categories: resources; information/education/communication; organisation/policy; and other. The timeline will be available to consult when results from other sources (both quantitative and qualitative) begin to emerge, in order to understand patterns appearing in those data over time and between health centres and catchment areas.

Objective five: impact evaluation

The impact evaluation component of this PROCESS study will be a small-scale undertaking, with the aim to understand some of the unintended impacts of the intervention [48]. The main PRIME trial will evaluate the impact of the intervention on community health outcomes. This PROCESS protocol aims to widen the lens to evaluate the impact of the trial on health centres, health workers, other providers, and community members. We will add questions to existing data collection activities listed above as well as conducting additional data collection with private providers.

Most significant change method

The semi-structured questionnaires, in-depth interviews, and FGDs will begin with open questions about the most significant change participants observed in the way they worked/sought treatment, why this was significant, and what difference it has made to them. This method aims to collect and analyse systematically significant changes from the perspectives of those involved in a programme [49]. We will use this method to elicit broader responses from participants, particularly stories in the in-depth interviews and FGDs, before prompting for intended changes and pre-defined indicators in the methods outlined above.

Private provider questionnaires

The landscape of treatment seeking in this context includes availability of antimalarials at private providers—registered and unregistered shops and clinics. An understanding of the role these providers play and the drugs and diagnostic abilities they have available is proposed to assist our understanding of the pathways of change and outcomes, as well as unintended impacts of the trial. We will therefore carry out a mapping exercise and interview private providers using a semi-structured questionnaire initially with 10 providers 9 to 12 months after the intervention and then with all providers at drug shops identified a year later, once subsidised ACTs become available through the Affordable Medicines Facility malaria (AMFm) mechanism in Uganda.

Quality assurance

The study team will be trained in the project objectives, collection of high quality data [50], and good clinical practice (GCP) guidelines. Study personnel will be trained and mentored to maintain principles for good quality research practice throughout the research process, from the design of tools, to field work engagements, to data management and analysis [51]. The team will work together to devise, pilot and revise standard operating procedures (SOPs) for all study activities. These SOPs will be adhered to or adapted throughout the research process. Meetings will be held at least weekly between the field team and investigators to identify, discuss, and resolve any issues arising from the evaluation practice and study findings. Clear line management will be established within the study team, and frequent performance feedback will be given by the study investigators to the team leader and on to the members of the study team.

Ethics

The study protocol and information sheets have been approved by the Ugandan National Council for Science and Technology (UNCST Ref HS 864), the Makerere University School of Medicine Research & Ethics Committee (SOMREC Ref 2011–103), and the London School of Hygiene and Tropical Medicine Ethics Committee (LSHTM Ref 5831). All potential participants will be informed of the purpose and nature of the study before being invited to participate and sign written consent forms. The discussion, information sheets and consent forms will be in the most appropriate language for the participant—whether English, Luganda, Japadhola, or Swahili—and a copy of the forms will be left with the participant. Risks and benefits of participation will be discussed and any questions relating to the research answered. Participants will be informed that all records will be kept as confidential as possible, with participants being recorded and quoted by number rather than name. However, participants will be made aware that any participation in the research study may involve a loss of privacy. Participants will be given the option of not being quoted at all, anonymously or otherwise, or included in any of the analyses. If a potential participant is unable to read or write, their fingerprint will substitute for a signature, together with a signature from a witness to the informed consent procedures.

Trial status

The PRIME trial field work completed in July 2013. Data cleaning and analysis of the final community survey, including the primary outcome for the survey and the overall trial, has not yet begun. The PROCESS study field work completed in July 2013. Data analysis for the PROCESS study has been ongoing, in order to inform the development of lines of enquiry throughout the trial period.

Discussion

The findings of this PROCESS study will be interpreted alongside the PRIME study. We plan to use the MATRICS (Method for Aggregating The Reporting of Interventions in Complex Studies) approach to bring together complex data from multiple sources to evaluate a complex intervention [52]. This evaluation represents a significant undertaking in addition to the main trial. The investigators will attempt to assess which methods are most informative in such evaluations of complex interventions in low-resource settings.