Background

Considerable progress has been made in increasing women’s access to modern contraception [35]. Nevertheless, there is continuing unmet need, particularly for those with less education, lower incomes, and younger age, and high rates of discontinuation due to poor quality of care and negative patient experience [10, 35, 39]. In recent years, there have been renewed efforts to understand better and address issues surrounding the quality of care and clients’ experiences of contraceptive services. Studies have identified a range of quality care issues that negatively affect clients’ experiences of interacting with contraceptive services [3, 17, 25].

Community engagement and more patient-centred care have come to the forefront as essential mechanisms to address issues around quality of care and poor client experience [24, 40]. Evidence increasingly indicates the positive effects of social accountability on improving the quality of care and experience in other health sectors and concerning contraception specifically [5, 13, 38]. We define social accountability as “citizens’ efforts at ongoing meaningful collective engagement with public institutions for accountability in the provision of public goods.” ([18], pg 161). It is often best recognized through the tools used to facilitate the process, such as community scorecards, social audits, and participatory budget. As a potentially transformative change process, social accountability transforms the norms, values, and attitudes of those seeking services and those providing them, which together can bring about changes in the health system delivery and, in the longer term, population health outcomes. However, the all-important changes in service users and providers’ norms, values, and attitudes are often not measured, possibly because they are not observable but rather latent and multi-dimensional.

Though few validated measures aim to capture these variables, a notable exception is CARE’s Women’s Voice tool that provides an important starting point for researchers in this area [34]. As part of a more extensive complex intervention study [36], we adapted and validated measures of service users’ attitudes and behaviors in a social accountability process to improve family planning services. In this paper, we describe the process of adapting the measures and assessing their validity and reliability in Ghana and Tanzania.

Methods

The larger study’s theory of change about how service users’ attitudes changed during the social accountability process determined the dimensions of the measures. To develop the survey items, we adapted existing items. Trained data collectors used a survey to collect data from women attending contraceptive services in Tanzania and Ghana. We used reliability analysis, exploratory, and confirmatory factor analysis to assess the validity and reliability of these measures in each country.

The development of these measures is part of a more extensive complex intervention study, Community and Provider Social Accountability Intervention (CaPSAI), undertaken in Tanzania and Ghana [36]. These countries were selected based on the following criteria: (1) existence of a national civil society organization (CSO) partner with experience in social accountability, (2) low modern contraceptive prevalence rate, (3) availability of contraceptive services at the point of contact where cost is not a barrier to access, (4) an enabling environment in terms of the potential for the health system to respond to the social accountability, and (5) the existence of formal structures linking the community with the health system (e.g. health committees).

The study took place in Mbeya City and Chunya districts in Tanzania, and Abura-Asebu-Kwamankese, Komenda-Edna-Eguafo-Abirem, and Mfantsiman districts in Ghana. The sites were selected based on (1) the provision of contraceptive services; (2) availability of the following methods: a barrier method, a short and long-acting method, emergency contraception, and at least referral for permanent methods in districts; and (3) no social accountability interventions in FP/C currently underway [36] Table 1

Table 1 Characteristics of the study settings [36]
Full size table

.

Dimensions of the measure

The theory of change drew on existing empirical and theoretical work on social accountability more broadly and specifically related to sexual and reproductive health (see [7, 8, 36]). This informed the dimensions in the measure. The theory of change, Fig. 1, details the inputs of the social accountability process (across the top of the diagram), how these correspond to the cumulative intermediate outcomes at three levels: (1) service users, (2) social networks and (3) service providers, which in turn, effect intended reforms in the quality of care that contribute to contraceptive choice, including increased uptake and use of modern contraceptive methods. As detailed in elsewhere, social accountability engages community members and health services actors in dialogues to identify shared challenges and develop action plans that can lead to improvements in service quality in the health system and in at the individual level, the service user or potential user knowledge and engagement with the health system, both in terms of their own health seeking behaviour and their participation in dialogues with authorities [36].

Fig. 1
figure 1

CaPSAI Theory of Change

Full size image

Item adaptation and development

Following the development of the dimensions, we identified existing validated measures for each domain. We drew heavily on CARE’s Women VOICES tool, a validated measure the aimed to capture similar intermediate outcomes concerning maternal health in Malawi [34]. We added three domains to those used in the VOICES tool to represent the CaPSAI theory of change fully. First was the ‘Knowledge and awareness of rights’ domain that aimed to capture the service user’s perception of rights were also included [42]. For service users’ efficacy with health care providers, we used the National Health Service (NHS) measure of patient activation [28]. To capture changes in service user’s awareness of how to bring about changes and improve their local services, we created items based on theoretical work on political capabilities [22, 41]. We adapted five VOICES validated scales with acceptable reliability to the contraceptive services and local context; for example, family planning services have a charge in Ghana but not in Tanzania. A total of 14 domains were included with 75 items (see Tables 2 and 3). A five-point Likert scale was used for all the item with the exception of two sets of items had different ranges in their original format. A 6 point scale was used for Self-efficacy with health care providers (set A) ‘Women’s participation in household decision- making’ and a dichotomous scale was used for ‘Self-efficacy with health care providers (set A)’ as originally used [28, 34].

Table 2 Domains identified based on Theory of Change
Full size table
Table 3 List of items included, per domain
Full size table

We ascertained content validity of the overall items through consultation with experts in social accountability and family planning, and the Principal Investigators who reviewed the questionnaire. Also, the World Health Organization (WHO) Forms Committee, which was composed of technical experts in contraception, social scientists, biostatisticians, and data managers, reviewed the instruments.

Survey administration

English surveys were available in both countries and translated in Akan in Ghana and into Kiswahili in Tanzania. Back translation was within a normal range, and pretesting the questionnaire was satisfactory for use in the study populations.

The same eligibility criteria for participants were used in both sites (see [36]). A sample of over 750 women aged 15 to 49 years accessing contraceptive services was interviewed prior to the start of the intervention in each country. Sampling was calculated using a priori sample size calculation with the ratio of 10 responses per item ratio and guidance of more than 500, which equals a very good sample for validation [4, 34]. Our sample calculation was based on 75 items of the full survey of items, including post test items. The same items were administered as part of a client exit interview survey in Tanzania and Ghana. The item related to cost of service varied between the countries as there is a nominal fee charged in Ghana. In Tanzania, the item was ‘The government ensures that family planning services are free of cost’ and in Ghana it was ‘The government ensures that family planning methods are free of cost.’

A total of 118 questions were asked of respondents upon leaving a facility, and only 58 scale items and 9 domains were included in the following analysis. Five domains and 17 items were excluded from this analysis because they were not scales or were items that applied after the intervention had been implemented.

The other survey items included questions about demographics, reproductive and family planning history, relationship status, income, occupation, and religion.

Client exit interviews were conducted on the day of recruitment at the facility in a private location. In Ghana, a total of 15 data collectors (5 females and 10 males) were trained in the survey over 3-day training in April 2018. In Tanzania, a total of 14 data collectors (8 females and 6 males) were trained over in the survey over 5 days in March 2018. Data collection was conducted using a tablet-based questionnaire to capture real-time data using OpenClinica and was later uploaded onto a secure server. Data collection in Tanzania started on 26 March and was completed on 25 May 2018, and all respondents choose to be interviewed in Kiswahili. In Ghana, data collection started on 9 April 2018 and was completed on 4 June 2018, and 46.4% choose to be interviewed in English while 53.6% choose to be interviewed in Akan.

Having being assessed for eligibility, respondents completed the informed consent process. There were no incentives given to women and girls to participate in the study. However, study participants who agreed to participate were reimbursed for their travel cost, where it was permitted by country-specific ethical requirements. In Ghana, the research team supported the travel cost to the facility with five Ghana cedis (~ 1 US dollar) given after the interview. In Tanzania, no reimbursements were given.

Psychometric analysis

We assessed the item and scale reliability followed by exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) for each country. We assessed the reliability of items and scales to test the internal consistency. The EFA aimed to identify the relationships among items and then group the items as part of a factor. CFA was conducted to confirm the theory behind the grouping of items.

We started with assessing reliability using the Cronbach’s alpha to determine item -to- item correlation (or homogeneity) of all 58 observed items and determined the overall alpha for each scale in each country. A Cronbach’s alpha of 0.60 was considered acceptable reliability and 0.70 or higher to be good reliability [16]. Items were removed, according to standard procedures, if the overall alpha improved substantially if an item was removed [23]. Scales with Cronbach’s alpha of ≥.60 were retained [34].

We conducted exploratory factor analysis (EFA) to determine how all 58 observed items clustered together and explore the underlying factor structure in each country. We computed the communality for each item, defined as the proportion of variance in the item attributable to common factors; and used a Kaiser-Meyer-Olkin (KMO) of Sampling Adequacy to assess the suitability of items for the factor analysis. Overall, and factors with KMO value > 0.5, for factor analysis were considered suitable for factor analysis [19, 21]. To determine the factors, we used eigenvalues in accordance with the Kaiser Criterion [20]. We examined the eigenvalues and the scree plot of eigenvalues, and factors with Eigenvalues greater than 1.0 were retained [20].

We used a rotated factor analysis using the maximum likelihood estimation (MLE) with oblique oblimin rotation to determine the factor loadings and variance. Factor loadings assess how items are weighted for each factor and the correlation between the variables and the factor. We used the proportion of variance in the item explained by the factors jointly to assess the reliability of the item in the context of all the factors. Items with factor loadings with values less than 0.40 were excluded. A minimum of three items per factor is recommended, and factors with two items or less were excluded [26].

Confirmatory Factor Analysis (CFA) was done to confirm whether the constructs identified in EFA had a good fit statistically. We applied three recommended models to test for goodness-of-fit [32]. The Standardized Root Mean Square Residual (SRMR) is a measure of the mean absolute correlation residual with a threshold of ≤0.08; the Root Mean Square Error of Approximation (RMSEA) measures the estimated discrepancy between the population and model-implied population covariance matrices per degree of freedom, and a score of ≤0.06 is acceptable, and Comparative Fit Index (CFI) measures the relative improvement in the fit of a researcher’s model over that of a baseline model, and a CFI ≥ 0.95 considered an acceptable fit [6]. The CFA structural model was is presented for each country.

Results

Demographic characteristics

In total, 750 in Ghana, and 752 women in Tanzania completed the survey. This sample is based on 10 respondents per item to produce a reliable estimate [6]. Table 4 shows the demographic characteristics of respondents.

Table 4 Demographic Characteristics of Ghana and Tanzania Sample
Full size table

The mean age of the women that participated in the survey was 28.4 in Ghana and 27.8 in Tanzania. In Tanzania, 84.8% of women were currently married compared to 63.5% in Ghana. A higher percentage of women in Tanzania were never-married (30.0%) than Ghana (10.3%). In both countries, injectable and implants were the predominant current contraceptive method.

Analysis

To assess the scales in Tanzania and Ghana, we conducted three sets of analyses to (1) assess the reliability of the subscales, (2) determine how many factors to retain and reduce the items, and (3) verify our proposed groupings separately per country.

Tanzania

We started with a reliability analysis using the Cronbach’s alpha to determine item -to -item correlation of all 58 observed items and determine the overall alpha for each subscales in Tanzania (see Table 5). The standardized alpha was greater than 0.60 (the acceptable reliability in social science research) for 6 of the 11 subscales and was retained. When the alpha was lesser than 0.70, items were removed if their removal improved the overall alpha scale. A total of 6 items were removed; this included questions 38, 65, 70, 82, 83, and 84. Subscales with Cronbach’s alpha of ≥.60 were retained.

Table 5 Reliability Analysis in Tanzania
Full size table

We undertook a KMO measure of sampling adequacy, and all items had a KMO of 0.5. The overall KMO score was 0.85, suggesting that there is a sufficient correlation between the variables to conduct exploratory factor analysis.

The EFA used principal factors with oblique oblimin rotation for all 58 items that yielded 11 factors. All factors had eigenvalues greater than 1.0; see Table 6 (which explained 100% of the variance).

Table 6 Eigenvalues in Tanzania
Full size table

In total, five items loaded on factor 1, five items loaded on factor 2, four items loaded on to factor 3, four items loaded on to factor 4, two items loaded on to factor 5, three items loaded onto factor 6, three items loaded onto factor 7, three items loaded onto factor 8, three items loaded onto factor 9, three items loaded onto factor 10, and two items loaded onto factor 11 Table 7. We discarded items with factor loadings of less than 0.4, and 12 items were removed, see Table 7. Factors with less than two items were excluded as this is not a sufficient number for factor analysis. On this basis, two factors were removed, and 10 factors were retained. A total of 40 items were retained.

Table 7 Factor loadings for Tanzania
Full size table

To name the factors, we assessed what items were retained from the reliability analysis and the exploratory factor analysis and the original domain groupings. Factor 1 continues to reflect the domain items related to a client’s knowledge of their health rights and is named ‘Knowledge of health rights.’ The items included in factor 2 reflected knowledge of household decision-making in relation to finances and sexual and reproductive health and retained the name ‘Women’s participation in household decision-making’. The items included in factor 4 relate to clients’ perceived mistreatment by health care providers and is named ‘Mistreatment by health workers’. Items in factor 5 relate to the clients’ ability to attend a community meeting and are named ‘Ability to attend a community meeting.’ Factor 6 included items related to the ability to actively participate in a community meeting and is named ‘Ability to participate in a community meeting.’ Items in factor 7 relate to the client’s ‘Perception of quality’ and retain the name, ‘Perception of quality services’. The items in factor 8 relate to the understanding of how to bring about change in contraceptive clinics and are named ‘Awareness of accountability mechanisms.’ Factor 9 includes items that relate to the sense of social capital and cohesion and retains the name ‘Collective efficacy’. Factor 10 included items related to a sense of support from others during a crisis and retains the name ‘Community support in the time of crisis.’ For factor 11, the items related to how clients thought change could be achieved and retains the name ‘Mutual responsibility for and support of services’ (Fig. 2).

Fig. 2
figure 2

CFA structural model for Tanzania

Full size image

Ghana

We started with a reliability analysis using the Cronbach’s alpha to determine item -to- item correlation of all 58 observed items and determined the overall alpha for each subscales in Ghana (see Table 8). The standardized alpha was greater than 0.60 for 8 of the 11 subscales and was retained. When the alpha was lesser than 0.60, items were removed if their removal improved the overall alpha scale. A total of 8 items were removed; this included questions 38, 39, 41, 65, 70, 82, 83, and 84. Scales with Cronbach’s alpha of ≥.60 were retained.

Table 8 Reliability Analysis in Ghana
Full size table

In the KMO measure of sampling adequacy, all items had a KMO of 0.5, and the overall KMO score was 0.80, suggesting that there is sufficient correlation between the variables to conduct exploratory factor analysis.

The EFA used principal factors with oblique oblimin rotation yielded 13 factors with eigenvalues greater than or equal to 1.0 (see Table 9).

Table 9 Eigenvalues in Ghana
Full size table

The revised item data was used in the confirmatory factor analysis to analyze a good fit. The SRMR score of 0.05, the CFI score of 0.94, and the RMSEA score of 0.03 are good. All three score suggests acceptable goodness-of-fit.

In total, seven items loaded on factor 1, five items loaded on factor 2, three items loaded on to factor 3, four items loaded on to factor 4, four items loaded on to factor 5, five items loaded onto factor 6, three items loaded onto factor 7, three items loaded onto factor 8, three items loaded onto factor 9, four items loaded onto factor 10, four items loaded onto factor 11, two items on factor 12 and one item on factor 13 Table 10. Factors with less than two items were excluded as this is not sufficient for factor analysis, this reduced the number of factors from 13 to 10. We discarded items with factor loadings less than 0.4, and two items were removed, see Table 11. In total, 10 items were removed, and 48 items were retained.

Table 10 Factor loadings for Ghana
Full size table
Table 11 Final items per country
Full size table

Similar to Tanzania, to name the factors, we assessed what items were retained and the original domains. These corresponded with those found in Tanzania, with the exception of one factor that was dropped in Tanzania but retained in Ghana. The items included in this factor related to a clients’ sense of activation and are named ‘patient activation.’ There was also a difference in what items were retained in relation to household decision making; those pertaining to contraception were excluded, whereas those related to decisions over finances, over seeking health care, and visiting others were included (Fig. 3).

Fig. 3
figure 3

CFA structural model for Ghana

Full size image

The revised item data was used in the confirmatory factor analysis to assess the goodness-of-fit. The SRMR score of 0.057, the CFI score of 0.88, and the RMSEA score of 0.04, all suggest an acceptable goodness-of-fit.

Discussion

We adapted and validated the measures of service users’ attitudes and behaviors in a social accountability process to improve contraceptive services in Ghana and Tanzania. The measure has high construct validity and reliability in both countries. We identified several subscales in both countries: 10 subscales in Tanzania and 11 subscales in Ghana. Many of the scales and items were shared across both settings, as shown in Table 11. There were some differences in how the scales performed in the different contexts; however, there was convergence in the analysis to suggest that this measure may be relevant to other settings.

The first group of scales related to clients’ awareness about their health and contraceptive entitlements. This includes the following scales: knowledge of health rights, mistreatment by health workers, and perception of quality. While these scales are conceptually distinct constructs, when combined, they measure a critical aspect of clients’ knowledge of their rights and entitlements. In the social accountability canon, being aware of one’s health rights and entitlements is a critical precursor to generating critical consciousness and people’s ability to participate in collective action [9, 12, 14, 29]. As George [12] reminds us, “Not only is access to information essential for improving health awareness and access, it is impossible to mobilise for change without it. People cannot demand services and accountability if they do not know what they need and what they are entitled to.”

Several of the scales related to capturing an individual’s perception of their ability to affect change are included: ‘Women’s participation in household decision-making’, ‘Self-efficacy with health care providers’, ‘Ability to attend community meetings’, and ‘Ability to participate in community meetings’. There was also a group of scales related to if clients knew how to improve their existing circumstance and included: ‘Mutual responsibility for and support of services’ and ‘Awareness of accountability mechanisms’. Many have scholars working in social accountability have argued that it is important the people see that change is possible, and they themselves are agents of change [9, 33].

The final group of scales were related to collective identity and action, and included: ‘Collective efficacy’ and ‘Community support in the time of crisis’. In collective action, social cohesion and social capital are central to the change process. This starts with groups of people initially identifying commonalities with each other moves towards a belief that the group can work together to bring about changes [11, 30, 31]. Such solidarities provided people with a sense of agency and collective identity necessary to confront unequal power [12, 14, 29].

The measures also speak to another gap in understanding the poor quality of care still reported in contraceptive services [1, 15, 37]. Harris et al. [15] argue that current tools do not adequately determine the prevalence or impact of negative client experiences in contraceptive programs and that current measures can de-emphasise and misdirect attention from client experiences of coercion, corruption, and disrespect and abuse when they come for family planning. The scale, ‘Mistreatment by health workers’, responds to this gap and by better capturing all dimensions of patients’ experience, we can learn what is working, or not, in terms of quality of care [2].

The study benefited from the rigorous methodology for the validation of psychometric scales [6, 27]. A limitation at this stage is that the test-retest was not conducted to examine if the measurement tools reliably replicate the result in the same situation and population. A test-retest was added to the end line data collection.

Conclusion

In this paper, we share the findings from testing a 58-item scale to measure intermediate changes among health service users during a social accountability process to improve contraceptive services. The study suggests that the multi-dimensional scales have high construct validity and reliability in both countries. Though there were differences between contexts and in some of the items and scales, there was convergence in the analysis that suggests that this measure may be relevant to multiple settings and needs to be validated in new settings.

The refined tool resulting from the CaPSAI Project has both research and programmatic utility. It will be useful for research to understand the monitoring and evaluation of social accountability processes and could help develop and target interventions. The validated scales allow for a more robust measurement of the intermediate outcomes. This scale will facilitate measurement to improve community engagement in contraceptive programs.