Evaluation of the Sustainability of an Intervention to Increase HIV Testing
Sustainability—the routinization and institutionalization of processes that improve the quality of healthcare—is difficult to achieve and not often studied.
To evaluate the sustainability of increased rates of HIV testing after implementation of a multi-component intervention in two Veterans Health Administration healthcare systems.
Quasi-experimental implementation study in which the effect of transferring responsibility to conduct the provider education component of the intervention from research to operational staff was assessed.
Persons receiving healthcare between 2005 and 2006 (intervention year) and 2006 and 2007 (sustainability year).
Monthly HIV testing rate, stratified by frequency of clinic visits.
The monthly adjusted testing rate increased from 2% at baseline to 6% at the end intervention year and then declined reaching 4% at the end of the sustainability year. However, the stratified, visit-specific testing rate for persons newly exposed to the intervention (i.e., having their first through third visits during the study period) increased throughout the intervention and sustainability years. Increases in the proportion of visits by patients who remained untested despite multiple, prior exposures to the intervention accounted for the aggregate attenuation of testing during the sustainability year. Overall, the percentage of patients who received an HIV test in the sustainability year was 11.6%, in the intervention year 11.1%, and in the pre-intervention year 5.0%
Provider education combined with informatics and organizational support had a sustainable effect on HIV testing rates. The effect was most pronounced during patients’ early contacts with the healthcare system.
KEY WORDSHIV testing provider education sustainability VA hospitals
There is a growing literature on the types of interventions required to improve healthcare quality1. To reap long-term benefits, the gains brought about by such programs must be sustained beyond the initial interventional period. However, achieving sustainability (i.e., the routinization and institutionalization of improved processes), is difficult and may be dependent on characteristics of the intervention that are not examined during the trial that demonstrates effectiveness. Sustainability is not often studied, and when it is, the results are often disappointing2, 3, 4.
Herein we report on the sustainability of a successful intervention to increase HIV testing. The clinical benefits of identifying and treating asymptomatic human immunodeficiency virus (HIV) infected individuals are firmly established and more cost-effective than many other general population preventive services5, 6, 7, 8, 9, 10, 11, 12, 13, 14. However, 21% of the 1.1 million HIV-infected persons in the United States remain undiagnosed15. Similarly, only 30% to 50% of Veterans Administration (VA) patients with known, documented risk factors for HIV infection have been tested16,17. Therefore, we previously implemented a multi-modal intervention based upon computerized decision support, provider education and feedback, and organizational changes that significantly increased HIV testing rates in at-risk individuals who receive care at VA medical facilities18. Over a one-year period, implementation of this program increased the cumulative rate of ever being tested for HIV from 20.1% to 53.7% (p < 0.001). In contrast, there was no change in three control facilities.
Once the interventional year was over, we turned project responsibility over to preexisting primary care clinical leadership. This leadership chose to dramatically reduce the labor-intensive provider education campaign and merged what little that remained into routine clinical management (e.g. weekly staff meetings). They did, however, continue the largely “fixed” changes in the systems infrastructure for HIV testing, which required substantially less support to maintain (i.e. the computerized decision support, feedback reports, and maintenance of organizational changes). We now report on the intervention’s sustainability in the second, sustainability year of this project.
As previously described18, the intervention program was put in place for one year in two of the five geographically separate VA regional healthcare systems (HCS) in southern Nevada and California. HCS A and B were comprised of 12 and five sub-facilities, respectively, in which primary care were provided by mixtures of academic and non-academic staff physicians, postgraduate medical trainees and mid-level providers. This study was approved by the appropriate institutional review boards.
A continuously updated, electronic clinical reminder that identifies patients at increased risk for HIV infection and encourages providers to offer HIV testing to such individuals. This reminder is triggered by HIV risk factors available in the VA electronic medical record. These include evidence of Hepatitis B or C infection, illicit drug use, sexually transmitted diseases, homelessness, and Hepatitis C risk factors18. Once triggered, the reminder was resolved by ordering an HIV test, recording the result of an HIV test performed elsewhere, or indicating that the patient was not competent to consent to testing or refused HIV testing. Once resolved, the reminder was no longer triggered.
An audit-feedback system: Providers were given quarterly reports of clinic-level HIV testing performance19.
The reduction of organizational barriers: Under federal laws specific to the VA, written, informed consent and pre-test HIV counseling have been required for all HIV tests20. To expedite this process we encouraged nurse-based rather than physician-based pre-test counseling, use of streamlined HIV counseling, and both telephone notification and brief post-test counseling after negative HIV test results18,21.
A provider education (activation) program: This included academic detailing, social marketing, and educational materials22,23. The academic detailing component involved regular informal discussions by project staff to encourage providers to prioritize the performance of HIV testing24,25. Social marketing involved having physician and nursing clinical opinion leaders encourage HIV testing by primary care healthcare providers26. Finally, we developed and distributed educational hand-outs, pocket cards and posters to promote HIV testing and increase provider comfort and abilities to provide pre- and post-test HIV counseling.
All aspects of the program were implemented in the first month of the intervention year at HCS A and HCS B and maintained during the subsequent 11 months. In support of the provider education program, members of the study team made frequent visits to the clinics to informally promote HIV testing in one-on-one ad hoc meetings with primary care providers. In addition, senior members of the study team regularly attended clinic and facility-wide meetings of primary care physicians, nurses and clinic leadership to promote HIV testing.
The study team did not participate in provider education activities during the second (sustainability) year of the study and instead fully transferred responsibility for this activity to clinic leadership. Qualitative evaluation indicated that provider education activities were much reduced and merged into routine clinical management activities such as staff meetings. Leadership did maintain other aspects of the intervention, including quarterly feedback reports of the rate of HIV testing, and the electronic clinical reminder. Organizational changes that had eased the documentation requirements for HIV testing and broadened the number of people authorized to initiate testing and counseling persisted. Distribution of educational activities, pocket cards and handouts continued at a reduced rate
Our primary analytical goal was to assess the trajectory of the monthly rate of HIV testing during the intervention and sustainability years. In addition, we assessed changes in the proportion of patients who agreed to be tested.
We obtained administrative and clinical data, including patient demographics, laboratory tests, diagnostic codes and health factors of the inpatient and outpatient encounters from August 2004 to July 2007 from a pre-existing regional VA database18. The medical records were linked across the data files by encrypted identifiers.
We evaluated outcomes during clinical visits of patients who were identified as being at-risk for HIV infection but had not been offered HIV testing (i.e., the HIV Testing Clinical Reminder had been previously been resolved). Visits by eligible patients were removed from the database subsequent to the month during which the reminder was resolved.
To assess the adjusted rates of HIV testing and refusal, we performed logistic regression analyses in which the unit of analysis was the patient who was seen at the VHA facilities in each month, had HIV risk factors, but the HIV Testing Clinical Reminder had not previously been resolved. The dependent variables were performance of HIV testing and documentation of patient refusal to be tested. The independent variables included patient demographic and clinical factors such as age, race and ethnicity, marital status, lack of housing, co-payment status, being at-risk for hepatitis C, hepatitis C infection, hepatitis B infection, illicit substance use and sexually transmitted diseases18. The two VHA healthcare systems comprised of 17 facilities where the patients were seen. To adjust for any systemic effects on patient likelihood of accepting or refusing HIV testing, we included facility-level annual patient loads and baseline HIV testing rates in the pre-intervention period as independent variables. Finally, we adjusted the covariance of the regression model for patient clustering within facilities using the Generalized Estimating Equation method. The data analysis was generated using SAS v9.1 proc genmod (SAS version 9.1. SAS Institute, Cary, NC, USA).
Patient Demographic and Clinical Characteristics
• African American
• Asian, Native American
Marital status (%)
Low income (%)
Risk factors (%)
• Hepatitis C infection
• Hepatitis B infections
• Prior sexually transmitted disease
• History of substance abuse
• History of homelessness
• Presence of risk factors for HCV infection
As discussed in METHODS, the HIV Testing Clinical Reminder can be resolved by performing an HIV test or by documenting that the patient refused to be tested. While allowing for patient choice with respect to HIV testing, minimization of the refusal rate is an important goal; once “refused” is selected, the HIV Testing Clinical Reminder did not prompt providers to re-offer HIV testing during future visits. However, we hypothesized that some “refusals” might actually reflect provider discomfort offering an HIV test33,34, and therefore that the refusal rate might decrease as providers gained more HIV testing experience.
We previously demonstrated that implementation of an integrated package of quality improvement interventions that utilizes decision support, a provider education (activation) campaign, feedback reports and organizational changes more than doubled HIV testing rates for at-risk individuals18. These results were robust with dramatic increases in the likelihood of being tested for HIV being observed across patient-level, provider-level and subfacility-level factors. Furthermore, the fraction of HIV test results that were positive remained constant (0.45%) and well within the range at which HIV testing costs less than $50,000 per quality-adjusted life year when societal benefits of testing are considered6.
We now report on the sustainability of this program during the twelve-month period after overall responsibility for the interventional program was transferred to preexisting clinical management, who chose to greatly deintensify the provider education campaign and other labor and time-intensive aspects of the intervention18,35. Remarkably, we found that the rate of HIV testing continued to increase for patients making their first, second or third visits during the sustainability period. These results indicate that despite the de-emphasis of the provider education campaign, when the frequency of medical contact is considered, the program’s impact on HIV testing rates was fully sustainable. The observation that overall testing rates declined was related to the changing make-up of the study population as patients with their first through third visits accounted for 100% of the study population in month 1, 54% of the population in month 12 and 41% of the population in month 24.
We also found that the rate at which patients refused HIV testing decreased over time. Correspondingly, the likelihood of having the HIV Testing Clinical Reminder being resolved by HIV testing increased. These results suggest that providers became more proficient at offering and discussing HIV tests and may have integrated HIV testing into their normal practice. Others have observed that normalization of HIV testing is associated with increased patient acceptance of testing36,37.
The importance of reporting the sustainability of health care interventions and of choosing appropriate measurement metrics is receiving increasing attention32. Our results indicate that assessments of the sustainability of the outcome of an intervention are critically dependent on the mode of analysis. We found that when applied to homogeneous patient population (as defined by prior use of VA healthcare), increased HIV testing rates were sustained after de-emphasis of the provider education campaign and continued to increase among patients newly exposed to the intervention (Fig. 2a). This suggests that our intervention has become part of the institutional culture of our facility, does not overburden providers and fits the implementing culture and variations of the patient population32.
Stratified analysis by the number of visits during each year reveals that our intervention was least sustained among established patients who had not previously been offered testing. We conclude that interventions that aim to maximize sustainability should consider a “tail” of provider education or other components focused on patients who do not receive recommended services on the first exposure. Also, further work needs to be done to determine the determinants of repeated non-performance. We believe that such failures are likely due to systemic barriers or a lack of provider agreement/knowledge. Notably, although theoretical38, 39, 40 and empirical observations22,23,41, 42, 43 demonstrate that the use of provider education (or activation) campaigns are necessary to transform group norms and maximize quality improvement, there is far less literature regarding the importance of maintaining these activities to sustain whatever gains are achieved during their use32.
The strengths of our sustainability analysis include, as recommended, use of a time-series analysis of monthly rates of HIV testing which allowed us to better assess the trajectory of HIV testing rates32,44. Furthermore, we examined the effectiveness of the intervention in an unselected population of at-risk veterans receiving care in a routine, real-world clinical setting.
Limitations include the fact that the sustainability analysis was done immediately after the withdrawal of study personnel from active maintenance of the intervention. It is therefore difficult to distinguish between lingering improvements from the implementation and true persistence of effects from institutionalization45. Moreover, this study was undertaken within the quality improvement infrastructure in the VA, which includes an electronic medical record, clinical reminder software and familiarity with performance measurements. Although such tools are increasingly common, this intervention might not be generalizable to other healthcare systems. Another limitation is that while sustainability can be defined as continued use of the core elements of the interventions, and persistence of improved performance32, we did not formally evaluate the continued use of the core elements of the interventions or their individual contributions to the successful sustenance of the intervention. However, surveys of the two HCSs involved in this project indicate that the organizational changes that favor HIV testing and the HIV Testing Clinical Software package have been maintained. Another limitation is that there was still room for improvement and it is unknown whether the rates of HIV testing would have increased further had the provider activation campaign been continued. Furthermore, while guidelines now recommend that all patients be offered HIV testing and that yearly testing be offered to persons who continue to engage in high risk activities14,46, 47, 48, this intervention was targeted to ensure one-time testing in patients with known risk factors. This strategy was purposely undertaken to prioritize testing for patients at the highest known risk for HIV infection and in deference to concerns that a program to promote HIV testing in all patients would be impractical in the VA as long as written informed consent was required for testing. Finally, the achieved rate of HIV testing remained less than desired. It will be important to determine the effect of removal of the written informed consent requirement for VA HIV testing in August 2009 on the rates of HIV testing49.
In conclusion, we found that when assessed in homogeneous patient populations, the impact of implementation of the coordinated use of a computerized clinical reminder, feedback reports, provider education and organizational change is sustainable after cessation of external support of the provider education component. Maintenance of the gains after withdrawal of support by the research team suggests that the organizational and behavioral changes that led to the enhanced performance of HIV testing were successfully institutionalized. These findings have substantial implications for the assessment and sustenance of quality improvements programs for clinical preventive services and beyond.
Support: This project was supported by a research grant to Drs. Goetz and Asch by the Health Services Research & Development Service (SDP 06–001)
Conflicts of Interest
Matthew Bidwell Goetz: consultancy with Monogram Biosciences; grants in the last 3 years (Gilead Pharmaceuticals), Henry D. Anaya: stock ownership in Trinity Biotechnology, which develops biomarker devices, one of which is a test for the HIV virus, and educational support in the form of unrestricted grants from both Trinity Biotechnology and OraSure Technologies. Allen Gifford: royalties for authorship of Living Well With HIV And AIDS, Ball Publishing Company. Steven Asch: unrestricted travel grant from Trinity Pharmaceuticals. The other authors have no conflicts of interest.
- 1.Committee on Quality of Health Care in America IoM. Crossing the Quality Chasm. A New Health System for the 21st Century. National Academy Press. Washington, DC; 2001:1–337.Google Scholar
- 5.DHHS Panel on Antiretroviral Guidelines for Adults and Adolescents. Guidelines for the Use of Antiretroviral Agents in HIV-infected adults and adolescents. Available at http://aidsinfo.nih.gov/ContentFiles/AdultandAdolescentGL.pdf. Accessed September 2009.
- 9.Goetz MB, Morreale A, Berman S, et al. Effect of highly active anti-retroviral therapy (HAART) on outcomes in Veterans Affairs Medical Centers (VAMC). Clin Infect Dis. 1997;25:396.Google Scholar
- 15.Campsmith M, Rhodes P, Hall I. Estimated Prevalence of Undiagnosed HIV Infection: US, End of 2006. 16th Conference on Retroviruses and Opportunistic Infections. 2–11–2009; Montreal, Canada. Abstract 1036.Google Scholar
- 16.Perlin JB. Need for routine Human Immunodeficiency Virus (HIV) risk assessment and testing. Under Secretary for Health’s Information Letter. IL 10–2005–017. Available at http://vaww.vhaco.va.gov/pubarchives/docs/10200517.pdf. Accessed September 2, 2009.
- 20.VA Policy on Confidential HIV Testing and Counseling. Available at http://www.hiv.va.gov/vahiv?page=prtop02-ov-02#S2X. Accessed September 2009.
- 22.Thomson O’Brien MA, Oxman AD, Haynes RB, Davis DA, Freemantle N, Harvey EL. Local opinion leaders: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2005: CD000125. doi:10.1002/14651858.CD000125.
- 23.Jamtvedt G, Young J, Kristoffersen D, O’brien M, Oxman A. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2006;19:CD000259.Google Scholar
- 26.Lefebvre RC, Rochlin L. Social marketing. In: Lewis FM, Rimer BK, eds. Health Behavior and Health Education: Theory, Research, and Practice. 2nd ed. Jossey-Bass; 1988:Google Scholar
- 27.Heron MP, Smith BL. Deaths: leading causes for 2003. National Vital Statistics Report. 2007;55:1–93.Google Scholar
- 28.Bray RM, Hourani LL, Olmsted KLR, Witt M, Brown JM, Pemberton MR, Marsden ME, Faulkner DL, Marriott B, Scheffler S, Vandermaas-Peeler R, Weimer B, Calvin S, Bradshaw M, Close K, Hayden D. 2005 Department of Defense Survey of Health Related Behaviors Among Military Personnel. Available at http://www.ha.osd.mil/special_reports/2005_Health_Behaviors_Survey_1-07.pdf. Accessed September 2009.
- 49.Anonymous. Elimination of Requirement for Prior Signature Consent and Pre- and Post-Test Counseling for HIV Testing. Federal Register. 2009;74:34500–3.Google Scholar