Introduction

In 2019, an estimated 24 million people were living with HIV (PLHIV) in Sub-Saharan Africa, of whom approximately 73% were adults 15–49 years of age, approximately 75% of whom were on antiretroviral treatment (ART) [1]. Maintaining high rates of adherence to ART is vital to maintain viral suppression and reduce morbidity, disease progression and mortality among PLHIV in a sustained manner [2,3,4,5]. The level of adherence to ART required to prevent virological failure and the emergence of antiretroviral drug resistance was previously considered to be at least 95% [6]. However, more recent studies suggest that current regimens are more forgiving of missing doses, with levels of 80% adherence potentially being sufficient [3]. Maintaining consistently high adherence levels over long time periods has been challenging for PLHIV. This is due, among others, to medication fatigue and dissatisfaction with HIV consultations provided at the clinic [7, 8]. Commonly used adherence assessment methods applied by clinicians in low-resource settings are self-reported adherence, pill counts and pharmacy refill counts [9, 10]. These methods are often used in standard clinical practice to support meaningful discussion about adherence between PLHIV and health care providers [10]. However, due to recall and social desirability bias, self-report methods tend to overestimate PLHIV’s actual adherence levels, whereas pill counts and pharmacy refill counts can easily be manipulated and may be too cumbersome to perform in routine clinical practice [11].

Several alternative adherence monitoring tools have been recommended to overcome these drawbacks including digital adherence tools (DAT) [12], which make use of mobile phone communication. With the widespread use of mobile technology in Sub-Sahara African countries, adherence strategies deployed by mobile phones have the chance to reach a large user audience [13]. An example of such a strategy is the Wisepill® device for real-time medication monitoring (RTMM). RTMM records the date and time of each opening of a medication box and is thus less susceptible to overestimating medication adherence than self-report and pill counts [14]. However, RTMM has its own technical challenges due to the fact that it relies on a battery in the device and on network availability in order to send a signal about an opening of the box to a server as a reflection of actual medication intake [2, 15]. As a result, inconsistent capturing of actual doses missed was reported in several studies investigating the use of RTMM [16,17,18,19]. Moreover, RTMM may underestimate actual adherence levels if participants do not ingest the pills directly from the device, but for example put retrieved pills in their pockets, so-called pocket dosing [20].

Previous studies have investigated which of the adherence assessment methods self-report, pharmacy refill count, or RTMM, may be the best predictor of virological suppression [21,22,23,24]. However, to our knowledge, there is limited evidence of the ability of those adherence methods to predict viral suppression when combined in a low income setting. Therefore, as part of our REMIND-HIV trial [25], the objective of this study was to examine whether any of the following adherence measures is a better predictor of PLHIVs’ viral suppression: (1) self-report, (2) pharmacy refill counts, (3) RTMM, (4) a combination of self-report and pharmacy refill count or (5) all three adherence assessment methods combined.

Methods

We conducted a post-hoc analysis of part of the data from the randomized controlled REMIND-HIV trial, in which PLHIV had been randomly allocated to (1) RTMM, (2) Short Message Service (SMS) reminder texts or (3) standard of care and followed for 48 weeks. Details of the trial have been described elsewhere [25]. The study was approved by the College Research and Ethical Review Committee (CRERC) of Kilimanjaro Christian Medical University College (KCMUCo) and the National Health Research Ethics Sub-Committee (NatHREC) of the National Medical Research Institute (NIMR) of Tanzania. The trial was registered at the Pan African Clinical Trials Registry under PACTR201712002844286.

Study population

Participants were recruited from two sites, which were Kilimanjaro Christian Medical Centre (KCMC) and Majengo Health Centre, both located in Moshi, Tanzania. PLHIV were approached by study nurses during a common clinic visit. Informed consent was obtained from all participants followed by screening for eligibility. The inclusion requirements were: (1) 18–65 years of age, (2) receiving antiretroviral treatment for at least 6 months, (3) subjectively judged by a nurse counsellor to be poorly adherent to medication, based on missed clinic visits, returning excess leftover medication, and/or having continuously high viral loads, (4) able to read and write and (5) able and willing to provide consent to study participation. We excluded participants if they (1) were admitted to the hospital and/or (2) participated in similar studies investigating digital adherence tools.

Study Procedures

After obtaining informed consent from participants, study nurses interviewed participants and completed a screening form, containing inclusion and exclusion criteria, demographics, medical history, HIV history and times of usual ART intake. A secured web-based electronic data capture software system (REDcap) was used to collect and manage data. RedCap supports data validation, has an auditing trail and allows for data verification [26]. After completion of screening, the data manager performed randomization in REDcap using block randomization, stratified by gender and study site. Participants were randomized in one of three arms, RTMM, SMS or control arm, at a 1:1:1 ratio [25]. Participants were expected to attend the clinic every 2 months, according to standard care [27]. At each clinic visit, adherence was recorded through self-report, pharmacy refill count and, in the RTMM arm, additionally through RTMM. Participants were followed for 48 weeks. Viral load was measured at baseline and at the last week 48 study visit. For the present study, adherence and viral load data obtained at the week 48 study visit are used. Adherence measures considered the period since the last study visit preceding week 48. We did not include adherence data from the full study follow-up due to incompleteness of the data during earlier visits, though we considered leftover medication from the before-last visit.

Adherence measures.

Self-reported adherence

Self-reported (SR) adherence was measured using a questionnaire that was administered during a face-to-face interview by study nurses at each study visit. The questionnaire included two adherence questions: (1) ‘How many pills do you take per day?’ and (2) ‘How many pills did you miss in the past month?’ We calculated adherence taking the number of swallowed pills divided by the number of prescribed pills using the following formula as described previously [28]. We calculated adherence as follows:

$${\text{Self - reported adherence in the past month}} = \left\{ {\frac{{\left[ {\left( {{\text{Number of days in the past month }} \times {\text{ pills to take per day}}} \right) - \left( {\text{missed pills}} \right)} \right]}}{{\left( {{\text{number of days in the past month }} \times {\text{ pills to take per day}}} \right)}}} \right\} \, \times { 1}00\%$$

Pharmacy refill counts (PR)

A case report form was administered face to face to record pharmacy refill data at each study visit. The study pharmacist recorded the number of pills dispensed during the previous visit by asking ‘How many pills were given to you at the previous visit?’ while checking the medical file for the same information. In addition, the left-over pills returned during the previous and current visit were counted. For the participants who did not return pills, we asked to recall the number of pills that were left at home. In case leftover pills were unknown, our assumption was that all pills had been taken as prescribed in during the previous visit. Adherence was calculated as follows:

$${\text{Pharmacy - refill adherence}}: \, \frac{{\left( {\left( {{\text{pills dispensed at previous visit }} + {\text{ returned pills at previous visit}}} \right) \, - {\text{ returned pills at current visit}}} \right)}}{{\left( {{\text{number of days between visits}}*{\text{number of pills to take per day}}} \right)}}\;*\;{1}00\% .$$

Assuming that levels higher than 100% were representing 100%, we truncated maximum adherence at 100%.

RTMM adherence

Participants in the RTMM arm were given a Wisepill® RTMM device to monitor their medication intake in real time. When the device is opened, information including the time stamp is wirelessly sent using General Packet Radio Services (GPRS) to a secured web-based central database. Each opening was recorded, which was taken as a sign that the participant ingested the dose. If the box was not opened on time (agreed time between participant and healthcare provider), the participant received a short message service (SMS) text on his/her mobile phone which acted as a reminder to take medication.

Adherence levels were calculated at the 48-week follow-up visit of the study.

$${\text{Adherence}} = \, \left( {\frac{{\text{Number of openings over a given period}}}{{{\text{number of expected openings }}\left( {\text{based on prescription of number of dosing moments per day}} \right)}}} \right)\,*\,{1}00\%$$

Virological failure

HIV viral load data was obtained at 48 weeks of follow-up. The Tanzanian HIV guidelines direct health care workers to act once someone has 1000 copies/mL i.e. to provide enhanced adherence counselling or switch treatment [27, 29]. However, laboratory equipment in our study sites can determine viral load as low as 20 copies/ml. Therefore, plasma HIV RNA < 20 copies/ml was defined as virologically suppressed, while plasma HIV RNA < 1000 copies/ml was categorized as stable and plasma HIV RNA > 1000 copies/mL was categorized as unstable. As the trend in analyses of both cut-off values were the same, to answer our objective, we only described a viral load level > 20copies/mL as representative of virological failure.

Statistical analysis

Statistical analyses were conducted with Stata v.15. In the analyses, we included all participants who had a viral load measurement at week 48. The analyses that included RTMM-based adherence were only based on participants who were in the RTMM arm as RTMM was not used in the other arms.

To evaluate the ability of the various adherence measures to predict a detectable viral load, we conducted analyses using a cut-off of > 20 copies/mL. We calculated the sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) for different adherence cut-off values for each adherence assessment method separately. As previously studies suggested that current regimens are more forgiving for poor adherence than older regimens and that viral suppression can be achieved with 80% adherence, we classified participants as having poor adherence or good adherence using adherence cut-off values of 80%, 85%, 90%, 95% and 100% whereby the percentage stands for the percentage of doses taken. We used these to determine at which cut-off the prediction of a viral load ≥ 20 copies/mL was strongest.

For each of the adherence measures, its sensitivity was defined as the percentage of participants with a viral load ≥ 20 copies/ml who were identified by the methods as being poorly adherent at a certain adherence cut-off. Its specificity was defined as the percentage of participants with viral load < 20 copies/mL who were identified as being adherent at a certain adherence cut-off. The positive predictive value (PPV) was defined as the percentage of non-adherent participants with a detectable viral load, and the negative predictive value (NPV) as the percentage of adherent participants with an undetectable viral load.

The adherence measures were also combined to determine how two or three of them might impact sensitivity, specificity, PPV and NPV. To create composite adherence measures, participants were identified as non-adherent if they were below the adherence cut off in any of the combined measures under consideration. For example, when self-report and pharmacy refill counts were combined at a certain same cut-off and adherence was below 95% in either self-report or pharmacy refill count, the combined variable also was considered being below 95%.

For each of the adherence measures, sensitivity and (1-specificity) at the various adherence cut-off values were plotted in receiver operating characteristic (ROC) curves based on all the adherence data to determine the accuracy of an adherence measure to predict viral load. An Area under the ROC curve (AUC) value of 0.5 indicates that a test has no discriminatory capacity and an AUC of 1.0 indicates perfect discriminatory capacity. For screening purposes an AUC of 0.7 or higher is usually considered sufficient [30].

Besides the ROC curves analysis, logistic regression was used to identify which adherence measure predicted detectable a viral load ≥ 20 copies/mL while adjusting for gender and type of ART regimen. Analysis of baseline data of the parent trial had shown that TLE (the combination of tenofovir, lamivudine, efavirenz) was a significant predictor of viral load < 20copies/ml at study entry and therefore type of ARV regimen was categorized as TLE or another regimen. Two-sided p-values of < 0.05 were considered statistically significant in all analyses.

Results

A total of 249 participants were enrolled and randomized and 233 (93.6%) completed the 48 weeks of the study with an available HIV viral load result at week 48. Of those, 161 participants (69%) had a viral load of < 20 copies/ml and 31% of the 233 had a viral load of ≥ 20 copies/ml at week 48. Of those with VL < 20copies/ml and with VL ≥ 20copies/ml, the majority (71.4–65%) were female, mean age were 43 years and 40 years, the median time since first known positive HIV test was 6.8 years and 8.5 years respectively. Furthermore, participants had used their current ART regimen for a median of 4.3 years and 4.7 years respectively. Most participants (76%) were using a first-line regimen which included efavirenz, nevirapine or dolutegravir and for eight participants the regimen was not recorded at week 48 (Table 1).

Table 1 Demographic and treatment characteristics of participants (N = 233)
Full size table

Predictive value of individual adherence measures

In terms of the ability to predict a detectable viral load of ≥ 20 copies/ml, Table 1 shows that the sensitivity was lowest for self-reported adherence, higher for adherence by pharmacy refill counts and highest for adherence by RTMM at all adherence cut-off levels. Conversely, specificity was highest for self-reported adherence, lower for adherence by pharmacy refill counts and lowest for adherence by RTMM. The PPV ranged from 100 to 35% for self-reported adherence depending on the cut off value, while it was below 42% for adherence by pharmacy refill counts and RTMM. The NPV was consistently high (above 70%) for each of the measures at all cut-offs (see Appendix. Table 2).

Overall, the AUCs of sensitivity versus 1-specificity of the individual adherence measures was lower than 0.7. Pharmacy refill and RTMM had AUC values of 0.56 and 0.54 respectively, while self-report had an AUC value of 0.52. The optimal adherence cut-off points, closest to the upper left part of the figure were 89% for RMM, 96.2% for pharmacy refill and 100% for self-report (see dots in Fig. 1).

Fig. 1
figure 1

Receiver Operating Characteristic (ROC) curves for each assessment method (VL ≥ 20copies/ml) based on continuous adherence data

Full size image

Of the three measures, only self-reported adherence was significantly predicting viral load > 20 copies/ml at cut-offs of 85%, 90% and 95% in logistic regression analyses, after adjustment for gender and type of ART regimen. A regimen consisting of efavirenz, tenofovir and lamivudine, i.e., TLE, was a significant predictor of virological failure (p < 0.03). However, confidence intervals were wide, indicating low precision (see Appendix. Table 3).

Predictive value of combinations of adherence measures

When we combined self-reported adherence and adherence by pharmacy refill count, there were no major difference in sensitivity and specificity compared to the individual measures (See Appendix Table 4). However, when all the three measures were combined, we observed a higher value of sensitivity and NPV at high cut-offs ranging from 95 to 100% (see Appendix Table 5: Combined measures at VL ≥ 20 copies/ml). The AUC for the combined measures slightly increased (0.60) as compared to the AUC values recorded among single-adherence measures (0.56, see Fig. 2). The optimal adherence cut-off point for the combined measures, closest to the upper left part of the figure, was 92% (see dots in Fig. 2).

Fig. 2
figure 2

Receiver Operating Characteristic (ROC) curves for combined assessment method (VL ≥ 0copies/ml) based continuous adherence data

Full size image

In logistic regression models, the combined adherence measures did not significantly predict adherence at any adherence cut-off level (See Appendix Table 6). Furthermore, regardless of the cut-off used, being on a TLE regimen was the only independent predictor of a viral load < 20 copies/ml (see Appendix, Table 7).

Discussion

This paper describes the accuracy by which three adherence measures, individually or in combination can predict virological failure, at different adherence cut-off levels. Overall, we found that adherence assessed exclusively by self-report, pharmacy refill count or RTMM were insufficiently sensitive to predict virological failure. Sensitivity markedly improved by combining all three measures, but the practical feasibility of such an approach would need to be studied.

We found that self-reported adherence had the lowest sensitivity, but the highest specificity to predict virological failure as compared to adherence assessed by pharmacy refill count and RTMM.

This implies that virological failure occurred in many participants despite a high level of self-reported adherence. This finding is in line with previous studies, which have shown that participants tend to overrate their adherence level. This likely reflects recall and/or social desirability bias and fear of being judged negatively by health care workers [22, 28]. Our finding that self-report had the highest specificity is in line with previous studies showing that reports of poor adherence can be trusted. Other advantages of self-reports are that they are relatively cheap and easy to implement in clinical practice [31,32,33,34,35].

All adherence measures investigated in the present study had areas under the ROC curves that were below 0.70, the minimal value for screening purposes, indicating insufficient ability to distinguish between patients with and without virological failure. Still, we observed that adherence by pharmacy refill counts and RTMM had showed a more promising performance compared with self-reported adherence. Similar findings of pharmacy refill adherence and RTMM having higher AUC values than self-report methods were observed in studies conducted in Tanzania and Botswana [32, 36, 37]. Other studies also found that an electronic monitoring device had a higher sensitivity in predicting virological failure compared to self-report for participants with > 80% adherence [10, 38].

Therefore, in the context of routine clinical practice, where RTMM is not yet available, or is quite expensive, our findings demonstrate that pharmacy refill counts could provide a better prediction of virological failure given its higher sensitivity compared to self-reported adherence. In addition, RTMM could be cost-effective in a context of differentiated service delivery, i.e. if prioritized for use in poorly adherent participants. This could be particularly relevant in settings were viral load monitoring is not available [39].

When we combined the three measures, we observed the measures performed better compared to individual measures as the highest sensitivity and PPV were recorded at a higher range of cut-offs (95–100%) for viral load of > 20 copies/mL. Our finding that the optimal adherence cut-off to predict VL failure was around 90% or more is consistent with the WHO’s guidance that achieving a degree of adherence to ART of 95% reduces both the emergence of antiretroviral drug resistance and the risk of HIV disease progression.[40]. The ROC curve of combined adherence measures indicated a slightly higher AUC value compared to single-adherence measures.

Our results imply that where possible, existing adherence methods need to be combined to obtain a more comprehensive assessment of adherence as each adherence assessment method may capture a different aspect of medication taking behaviour and will give a better prediction of virological failure [38, 41]. Our findings also imply that self-reports of poor adherence should be taken seriously, and that patients reporting poor adherence might benefit from adherence counselling and intervention.

This study has some limitations. First, each adherence measure has its own specific limitations that may have affected the results. For RTMM, we assumed that all the openings of the device indicated intake of the medication by participants. We are aware that medication may not have been taken, but rather shared, or dumped [42]. Moreover, the Wisepill® device occasionally lost connectivity with the server and failed to record intake data on time, as participants had forgotten to charge the device. Second, the sample size was small, particularly concerning the number of participants using the RTMM device (one-third of the total trial population) which has likely limited our ability to predict virological failure. Hence, our results should be considered exploratory. Third, the study was conducted in only two clinics from the urban Kilimanjaro region, limiting the generalizability of the study outcomes, e.g. to rural populations of PLHIV. Fourth, the results of self-report and pill counts may have been influenced by recall bias, other errors (e.g., miscalculation) and/or self-interpretation by the participants. The use of laboratory methods to detect plasma drug levels might have resulted in an improved AUC to predict detectable viral load [43]. Finally, whereas WHO endorses the use of a linear visual analogue scale (VAS) to potentially reduce bias in assessing self-reported adherence [44], we chose to use the self-reported adherence measure in the manner it is currently most commonly used as part of standard of care in Tanzania.

The strength of this paper is that, to our knowledge this is the first study that compared the performance of three adherence measures in the context of a randomized clinical trial. Also, our findings included both manually recorded data (self-report, pharmacy refill) during clinic visits and electronically (automated real time data from the Wisepill box). This allowed us to compare and identify potential discrepancies between the data sources as described previously [45].

Conclusion

Our results show that adherence assessed by either self-report, pharmacy refill count or RTMM on its own did not perform well in predicting virologic failure. This could potentially be improved by combining all three measures, but the practical feasibility of such an approach would need to be studied. Given the fact that we had a small sample size, particularly considering the number of participants using the RTMM device, we would encourage researchers to investigate the same in bigger studies in order to be able to have adequate power for conclusions. In a context where RTMM is not available, our data show that pharmacy refill adherence could provide a better prediction of virological failure than self-report, but that reports of poor adherence should be taken seriously.