Background

South Africa has one of the highest burdens of HIV infections in the world with an estimated prevalence in 2009 of 17.8% in the adult population [1]. Since the start of antiretroviral treatment (ART) roll out in the public sector in 2003 almost one million people were estimated to be on ART by the end of 2009 [2, 3]. This number represents approximately 37% of the population eligible for ART initiation based on WHO guidelines [3].

While several studies have shown ART to be effective in reducing HIV related deaths in South Africa [4–10], there is only limited research on the quality of ART service delivery in the country. Research in clinics in the Free State found high patient satisfaction with ART services, notwithstanding discontent with human resource shortages, overburdened staff and long waiting times [5]. In Gauteng, a comprehensive evaluation of two community health centres (CHC) and two hospital-based chronic care management and treatment (CCMT) sites providing ART found that high quality chronic care was being delivered in the public health system [11], although there were problems with both space and staff shortages. This resulted in long waiting lists, some patients being turned away without drugs, inadequate follow-up testing and failure to trace patients lost to follow-up. The study identified staff burnout and dissatisfaction as the major threat to quality of care. More recently, a study in Cape Town found that the service package for pre-ART care was not being fully implemented, resulting in gaps in the quality of care and missed opportunities for integrated care and positive prevention [12].

Guidelines and performance standards to define quality ART service provision have been developed at national and international level [13, 14]. They set out what is expected from health care workers at each of the respective service areas (reception, front station, clinician’s consultation, HIV counselling, nutritional counselling, social worker’s assessment and pharmacy) on the various occasions that patients use their services. It has been argued that patient volume, limited funding, the shortage of health care workers, a generally overburdened public health care system and other factors make these standards difficult to meet in resource-limited settings like South Africa [5, 15–20].

The objective of this study was to use performance standards to evaluate the quality of ART services provided at ART clinics in and around Pretoria in order to define service areas that required improvement and to assist clinic management to target interventions for quality improvement.

Methods

Seventeen ART clinics in Tshwane district, Gauteng Province, South Africa and neighbouring districts participated in the study between June and November 2009. The ART clinics were selected based on their geographical location in the greater Pretoria area and that they could be accessed using public transport. Eleven patients were recruited from a NGO run clinic as participant observers. They carried out the assessment of ART services at the remaining 16 government clinic sites. To ensure confidentiality and anonymity, the clinics were randomly given a number between 1 and 16.

The clinics in the study did not differ from one another in terms of the way they organized and executed the services they provided. They all used the National Department of Health’s Performance Standards for Antiretroviral Therapy[13] a tool that outlines the procedural requirements for the ART initiation and subsequent Follow-up visits.

According to this guideline, a patient who qualifies for ART (based on the patient’s immune status or clinical condition) first needs to be assessed for treatment readiness ( Additional file 1 Textbox 1). The second visit is the ART initiation visit when the patient is supposed to start ART treatment.

All staff members were trained on and were qualified to meet these performance standards prior to being appointed at their respective.

As an investigation of services offered on the treatment readiness visit, the study assessed the quality of services provided at the reception area and front station as well as during the clinician’s consultation, HIV counselling, nutrition counselling, social worker’s assessment and pharmacy (optional). Data were collected using a checklist based on performance standards for each service area. Table1 sets out the performance standards that were assessed. In addition, data were collected on how quickly an appointment at the facility could be made by the participant observers on telephonic request, the time spent at the facility overall and the time each participant observer spent at each of the service areas during his or her appointment. Basic frequency analysis was done using Microsoft® EXCEL.

Table 1 Percent of performance standards (PS) fulfilled across all assessed facilities
Full size table

The criteria used for selecting participant observers were that they had previous exposure to a public ART clinic, had sufficient understanding of the intention of the evaluation and had similar socio-demographic characteristics as the patient population served by public ART clinics. Also they had to be mobile and available to make multiple clinic visits. Participant observer training lasted approximately three hours and was conducted by the research team in one on one or small group sessions. It included a briefing on the purpose of the study, the provision of fictitious blood results that would qualify a participant observer to start ART, an explanation of the checklist used to screen adherence to performance standards and an instruction on how to document the findings. Participant observers were instructed to act as patients and not to disclose their research role. They also were encouraged to fill out the checklist as soon as possible after completing a service in order to minimize recall bias. After conducting each evaluation, participant observers were interviewed to determine the validity of their experiences. They were asked about their experience at the clinic and the answers in the checklist were reviewed. To ensure that the checklist worked and that the participant observers recorded meaningful information the method was tested twice prior to carrying out the assessment.

Each study clinic agreed to be assessed by participant observers on their adherence to the ART performance standards with the understanding that they would not be made aware of the timing of the evaluation nor would they be told who the actual participant observers were. The plan was to generate 80 assessments, with each clinic being assessed on five different days by different participant observers. Due to the limited availability of some participant observers, in practice it was only possible to complete a total of 70 evaluations.

Data about the total number of patients that visited the clinic in the months of assessment and the workload per staff was collected from clinic registers. Workload per staff was calculated as the total number of patients that visited the clinic in the month of the assessment divided by the full time equivalent (FTE) per staff in that month. It was not possible to disaggregate the volume of patients by the reason for or type of visit as this information was not reliably collected at the assessed clinics.

The results were analyzed for each facility in the following way. The services that were available at the clinics and which services participant observers were able to access were determined. Then an overall clinic performance score (CPS) of between 0 and 100 was calculated for each facility. The CPS was defined as the percentage of performance standards met by all services provided in the facility. To estimate the overall quality of service provided in the 16 assessed facilities the median of the CPS was calculated. Similarly a service performance score (SPS) was calculated for each service area at each facility, defined as the percentage of performance standards met by a particular service area per facility as assessed by the participant observers. To estimate the overall quality of a service area across all clinics, the median of the SPS was calculated.

In order to define service performance gaps across all the clinics, each performance standard was looked at individually and the percentage value of the particular performance standard being met across all sites was calculated. According to the percent of performance standards met, performance was ranked as excellent (≥ 90%), very good (80% - 89%), good (70% - 79%), acceptable (60% - 69%) and unacceptable (< 60%).

The overall time spent at each facility was calculated as the median of the times spent per visit at the clinic.

The possibility that the high workload of clinicians or counsellors affected their adherence to performance standards, inter alia, reduced the time they spent with patients and affected the quality of the services they provided, was also analysed. We therefore sought to determine if the workload of clinicians and counsellors during the month of the visit was a predictive factor for the duration of the visit, and if the duration of the visit was a predictive factor for the quality of the service. This was derived from a linear regression analysis (Microsoft® Excel, version 2007) of the information available for each visit where the correlation coefficient R 2 was used to describe the association. Accordingly, the closer R 2 values were to zero the lesser the association, the closer the values were to one the higher the correlation.

This research was approved by the Faculty of Health Sciences Research Ethics Committee of the University of Pretoria, South Africa (Protocol number 75/2011) and conducted in compliance with the Helsinki Declaration.

Results

Clinics

Of the 16 clinics assessed, two were situated in the city centre of Pretoria, six were in former townships and eight clinics were in rural areas. Table2 shows the total number of patients per facility per months in which the assessments took place and the workload per clinician and HIV counsellor.

Table 2 Total number of patients per month per facility and workload per staff (clinician and counsellor)
Full size table

Bookings

In nine clinics (56.3%) no booking was required and the patients could access the facility on any day. The remaining seven clinics required bookings, with a median of 14 days until appointment (interquartile range [IQR]: 11.5 days) (Figure1).

Figure 1
figure 1

Median duration until appointment per facility in days. The left end of the light grey bar indicates the minimum, the right end of the light grey bar indicates the median and the right end of the dark grey bar the maximum of days until appointment at a particular facility.

Full size image

Services offered

Almost all clinics provided core services (reception and front station, clinician’s consultation, HIV counselling and pharmacy). Adjunct services, such as social worker assessment and nutrition counselling, were offered in 75% and 68.8% of the facilities, respectively (Figure2).

Figure 2
figure 2

Service availability at the facilities and services accessed by participant observers.

Full size image

Services accessed

Access to services was influenced by their general availability at the clinic and their actual availability on the day of visit. On the 70 clinic visits, participant observers were able to access the reception and front station on all occasions (100%), the clinician on 59 (84.3%) visits, the HIV counsellor on 49 (70%) visits, the pharmacy on 37 visits (52.9%), the social worker on 10 visits (14.3%) and the dietician on 10 visits (14.3%) (Figure2).

Quality of service

The median of the clinic performance scores (CPS) for the 16 assessed facilities was 68.5 (IQR: 18.7). However, one quarter of the clinics did not meet minimum standards (CPS < 60). Across all the facilities, the services with the lowest performance scores were clinician’s consultation (SPS 67.3) and HIV counselling (SPS 70.7) (Table2).

Quality gaps

Quality gaps refer to the extent to which standards internal to the service were met during a visit. A detailed analysis was made of the quality gaps in the clinician’s consultation and HIV counselling, the two least performing services that are also the two most essential services. It revealed that clinicians did a physical examination in only 41.1% of the visits. As part of their screening for TB they asked questions about difficulties in breathing and the presence of night sweats or a cough in 52.5%, 57.6% and 66.7% of the consultations, respectively. Counsellors addressed PMTCT with female patients in only 50% of consultations and the prevention of HIV infection to others in only 64.6% (Table3).

Table 3 Service performance scores (SPS) and clinic performance score (CPS) per facility
Full size table

Time

Overall, the median time participant observers spent at the clinics for a first visit appointment was 4.6 hours (IQR: 3.15 hours). The median time they spent with the clinician was 20 minutes (IQR: 15 minutes) and with the counsellor, 25 minutes (IQR: 28 minutes) (Figure3).

Figure 3
figure 3

Median total time participant observers spent at the facility and contact time participant observers spent with the clinician and HIV counsellor.

Full size image

Factors predicting service performance

A comparison of the workload of the clinician or counsellor with the time they spent in contact with participant observers revealed that a lower number of patients seen per month per staff did not necessarily translate into longer consultation times (Figure4a and 4b). Further, a correlation of consultation contact times with the quality of the clinician or counsellor care revealed that longer consultation times did not necessarily mean better performance or care (Figure4cB and 4).

Figure 4
figure 4

Correlation of the workload and the duration of the visits per staff (Figure 4a [clinicians] and 4b [counsellors]), and correlation of the duration of the visits and performance per staff (Figure 4c [clinicians] and 4d [counsellors]). Each dot (♦) represents a particular visit; R2: correlation coefficient; PS: performance standards.

Full size image

Discussion

This cross-sectional study assessed access to and quality of services in 16 ART clinics in greater Pretoria between June and November 2009. The CPS, as the overall measure for service quality at the facilities, was 68.5%. This means that, on average, about two thirds of the expected performance standards were fulfilled at the clinics assessed. While this average is suggestive of an acceptable general result, it conceals the fact that the CPS varied markedly among the facilities ranging from 39.7% to 91.7%. Four clinics did not meet minimum standards (60% of the expected performance standards), while only one clinic performed excellently (CPS > 90).

Deeper analysis of the results on service quality paints an even less satisfactory picture. A comparison of the various service areas across the clinics reveals that the most essential services, the clinician’s consultation and HIV counselling had the weakest performance. With a median SPS of 67.3 and 70.7, respectively, they scored markedly lower than social worker assessment, pharmacy and nutrition counselling. Especially, clinician consultations varied markedly in their adherence to performance standards at the assessed clinics. In five clinics, clinicians failed to adhere to half of the performance standards, while in four others clinicians adhered to more than 80% of the expected performance standards. One reason for the overall poor performance of clinicians and HIV counsellors, when compared to the other services at the clinic, is the higher number and more sophisticated nature of the expected performance standards of clinicians and counsellors. Arguably, these could make their services more sensitive to pressures of time and patient volume at the clinics. The findings here suggest, however, that the problem is deeper. An examination of each service area’s expected performance revealed the clinician consultation to have a high number of shortfalls in core performance areas. The most worrisome of these were the low rates of physical examination (41.1%) and failure to do comprehensive TB screening. Although clinicians almost always asked about previous TB treatment, they underperformed in assessing actual TB signs and symptoms. Particularly, assessment of difficulty in breathing and questions about the presence of night sweats, cough and weight loss was done in only half of the consultations. Also, while most clinicians took a good drug history, which included questions about previous medication, the concomitant use of herbal/traditional medication and the use of any prophylaxis, they often failed to ask about previous exposure to antiretroviral drugs. Further, a psychosocial history was taken in only half of the consultations. Not even half of the female patients were asked if they were pregnant, and even fewer were referred for cervical cancer screening. Thus, many clinicians failed to get all the information they would need to decide upon the further clinical management of their patients and eventually to determine the choice of the ART regimen. The observations also reveal that apparently important opportunities for integration of care (TB and cervix cancer screening) were missed.

As with clinician performance standards there were also many shortcomings in HIV counselling. Counsellors infrequently assessed patient knowledge about HIV. They mostly did not provide basic information about ART, such as showing pill samples, explaining how antiretroviral drugs work and discussing the importance of treatment adherence. Counsellors also did not consistently address the practical issues of taking ART, such as advising patients not to share medication with other patients and to not start or stop ART without consulting a clinician. Similarly, preventive measures, such as PMTCT and how to prevent infection to others as well as advice on the importance of physical activity, proper nutrition and other life style issues were infrequently addressed. These findings suggest that many patients may not be well prepared for ART, especially key aspects of treatment adherence. Also, the opportunity to address HIV transmission was frequently missed.

By contrast, the front station satisfactorily met its performance standards, except that nurses rarely measured height, inconsistently took temperatures and often didn’t greet patients. Although greeting a patient may appear to be of less importance as a performance standard compared to clinical standards of care, it may impact on the acceptance of the services provided by patients and the community.

Almost all performance standards for the reception, the pharmacy and nutritional counselling were met at a good rate (> 70%), as were the performance standards for the social worker consultations, except with regard to interpersonal, gender based violence, which was addressed in only half of the consultations.

These results point in the same direction as the findings of a recently published retrospective study in Cape Town, which assessed the quality of care during the pre-ART period (the time between HIV counselling and testing until initiation of ART). Although different in approach, the study found similarly low rates of screening for TB symptoms or cervical cancer as well as missed opportunities to integrate care and prevention [12].

Time spent in getting the services is also an important indicator of quality of service at ART clinics as it impacts on the acceptance of the services. Time here is understood as both overall time spent at the clinic and time spent in actual service contact. In this study, the median overall time of 4.6 hours spent by participant observers at a treatment readiness assessment visit was substantially longer than the NDOH’s target of 3.8 hours [21]. However, the median contact time of 20 minutes with the clinician and 25 minutes with the HIV counsellor, the two key services of an ART clinic, were not even half the NDOH targeted time of 45 and 60 minutes, respectively. This finding goes a long way to explaining the less than satisfactory performance of clinician and counsellor standards, as no matter their intentions, they did not have enough actual time to complete the tasks expected of them.

The longer overall time spent at clinics and the shorter time spent with the clinician or counsellor suggests an overburdened system. This finding is supported by the fact that in almost all clinics the actual number of patients per month per clinician and counsellor exceeds the NDOH benchmark of 500 patients per month per clinician and 100 patients per month per counsellor [21].

At the same time, a lower number of patients seen per month per staff at a clinic did not necessarily translate into longer consultation times. In other words, some facilities managed clinician and counsellor contact times in a way that allowed them adequate time to consult, despite very high patient loads, while others with lower patient loads did not. This result highlights the importance of organization and management in service quality.

This said, the finding that longer consultation times did not necessarily result in better performance also points to another critical factor in determining performance standards, namely, that of individual professional performance. Less than optimum quality of care in the clinician’s consultation and HIV counselling was not just a matter of time constraints. It was also determined by the individual health care worker’s skills and performance. Given available training, this problem is likely to be less a matter of knowledge and more a matter of attitude and motivation. Limited available research in Gauteng suggests that burn out and dissatisfaction have saturated staff capacities at clinics [11].

The quality of ART services in South Africa is not only determined by staff performance at the clinics, as measured by the adherence to performance standards and the time spent in getting the services. It is also gauged by the general accessibility and availability of such clinics. While waiting times of up to several months to get an appointment at an ART clinic were common in South Africa only a few years ago, this problem has been unequivocally overcome in greater Pretoria and probably in many other urban areas as well [17, 22, 23]. This study found ART services to be generally easily accessible in the study area. It was greatest at nine clinics where patients could walk in and be attended to without prior booking while most patients got an appointment within 14 days in the seven clinics where bookings were required. This finding is in line with the countrywide trend towards greater access to ART services.

At the same time, this study suggests that the ability of patients to access the full spectrum of ART services to assess treatment readiness was also a factor of clinic functioning. Services considered essential to the determination of ART initiation, especially consultation by a clinician and HIV counselling, were not always accessed. Those that might be considered more optional, at least on the first visit, such as pharmacy services, social worker assessments and nutritional counselling were even less available (52.9%, 14.3% and 14.3%). The reasons for the non-availability of especially core, but also adjunct services would require further investigation. This notwithstanding, incomplete and/or multiple ‘initial’ assessment visits is likely to have direct negative consequences for patients, including delayed ART initiation. It also carries negative implications for the clinics, potentially contributing to patient dissatisfaction with their services.

The design of this study, using participant observers as assessors, allowed for an evaluation of the facilities in their day-to-day operations. However, some limitations have to be considered when interpreting the results. The relatively small number of observations (n = 70) limits the ability to generalise the findings beyond the sites investigated. Also, small clinics with low patient volumes were over-represented in analyses across all clinics. And there is the possibility that the assessments of the participant observers were influenced by recall and interpretation bias. This said, we believe that these results, based on participant observer assessments against defined performance standards, provided a deeper insight into the reality of patient care in public ART clinics than patient satisfaction surveys that assess the service quality.

Conclusions

In conclusion, this study of first appointment (treatment readiness) performance standards of ART clinics in and around Pretoria presented a mixed picture of the quality of services. ART clinics were easily accessed and were of an overall acceptable quality. However, they showed long waiting times and variability within services, especially in the key areas of clinician’s consultation and HIV counselling. These findings suggest that there is a need to improve service performance. In this regard, particular attention should be paid to the time patients spend at facilities, and especially the time they spend in clinical consultations and counselling. Patients would benefit from a review by facility management of their approach to several areas of clinic functioning, especially general access (for example, by having a booking system, and by differentiating between first and follow-up appointments), operational flow (in order to ensure adequate time for patient contact with key service stations) and staff training, practice and development. The vexing problem of ensuring quality ART services is likely to continue to dog a system where ART services are widely available and easily accessible (at least in most urban [23], although not in most rural [22] areas of South Africa) but capacities are saturated. Lastly, participant observation measured against performance standards provided novel insights into the issues of quality of ART care in a primary care setting that previously have not been obtained.