1 Background: Monitoring and Evaluation of HIV/AIDS Programs

Continuous monitoring and evaluation (M&E) is an important component of any health system. Setting meaningful targets is a first step toward improving the effectiveness of an HIV/AIDS response, including the health system itself, as well as new interventions and programs. A crucial second step is establishing indicators, also sometimes called metrics or benchmarks, for the purpose of assessing progress (or lack thereof) toward the selected targets. Best practice is to construct these indicators within the so-called “SMART” format (Doran 1981), ensuring the indicators are:

  • Specific: detailed and explicit, avoiding overly general aspects, and logically linked with the given objective

  • Measurable: capable of objective assessment in a manner that is consistently repeatable, comparable, and auditable

  • Attainable: achievable in the sense that the target can be met, but difficult enough such that the indicator is meaningful

  • Relevant: logically linked to the objective as well as resulting in information that is actionable

  • Time-bound: having predictable, repeated timelines for assessment

Health system indicators may assess broad, and sometimes overlapping, program aspects including availability, coverage, quality, and impact (World Health Organization et al. 2009). Availability indicators primarily examine health service or intervention delivery or offering to target populations and must be tailored to the unique context of each delivery modality. For example, availability of condoms in vending machines, at pharmacies, at community-based organizations, or from outreach workers must be measured differently. Coverage indicators measure the extent to which a target population is actually provided with a particular health service or intervention. These kinds of indicators can prove challenging, since an accurate measure requires a carefully estimated total target population for its denominator. Furthermore, frequency of contact—either successfully reaching a target individual or a target individual successfully accessing a service—is an important factor in assessing coverage for some types of health services. Quality indicators cover a broad range of measures including the scope and completeness of interventions or programs, their effectiveness and efficiency, and their safety profile. Although often challenging to measure, best practice for quality indicators is measurement of performance in these categories against standards set by independent, international organizations. Impact indicators primarily include disease incidence and prevalence, measurable by traditional epidemiological tools and mathematical modelling, but may also include prevalence of disease risk behaviors (e.g., sharing of drug-injecting equipment, consistent condom use; World Health Organization et al. 2009).

There has been growing recognition that monitoring the success of programs should be performance-based rather than resource-based (i.e., measures of success should consider the differing levels of resources in different settings). Increasingly, international donors have been monitoring indicators for performance and requiring evidence of progress as a part of the application process for extended or renewed funding. For example, the Global Fund to Fight AIDS, Tuberculosis and Malaria (hereafter the Global Fund) began requiring information on progress toward achieving performance-based indicators in order to drive accountability and transparency by grant recipients and to inform future funding decisions.

In September 2003, at the International Conference on AIDS and STIs in Africa, representatives from Joint United Nations Programme on HIV/AIDS (UNAIDS), the Global Fund, attending African nations, and a variety of NGOs, funding agencies, and other organizations came to an agreement on three core principles for country-level HIV/AIDS responses. Called the “Three Ones,” these principles were one, agreed-upon framework for coordinating all partners, one national coordinating authority with support from multiple sectors, and one comprehensive, country-level monitoring and evaluation system (Joint United Nations Programme on HIV/AIDS 2004). Achievement of the third of the “Three Ones” principles was expected, at the time, to be a significant challenge—countries’ HIV M&E systems needed to meet the needs of the global community engaged in fighting HIV/AIDS, national and subnational stakeholders from multiple sectors, public health workers coordinating efforts “on the ground,” and funding agencies, both domestic and international, looking to examine the returns on their investments. Furthermore, these M&E systems needed to be built, tested, and launched quickly, yet be robust enough to achieve their purpose, and at the same time be flexible, scalable, and adjustable so that they could meet the needs of the future as well. It was in this context that China embarked on its journey toward developing an M&E system for its HIV response.

2 China Begins to Develop an M&E System for Its HIV Response

In November 2003, China sent a contingent from the Chinese Center for Disease Control and Prevention (China CDC) to Bangkok, Thailand, to attend a workshop entitled “Strengthening Monitoring and Evaluation of National AIDS Programs in Asia,” presented by UNAIDS, the US Agency for International Development (USAID), the World Health Organization (WHO), the World Bank, the Global Fund, the US Centers for Disease Control and Prevention (US CDC), and others. One of the primary objectives of the workshop was to get country representatives together with multiple international funding agencies to discuss coordination of the HIV response across the Asia region and to grow M&E capacity through the sharing of experiences, challenges, and plans. Although these workshops had already been held several times in Africa, leaders stressed that M&E systems needed to be tailored to the local context in order to successfully achieve their purpose and that there was no one-size-fits-all set of targets and indicators.

Although already halfway through two major initiatives, the China’s Medium- and Long-Term Plan for AIDS Prevention and Control (1998–2010; Ministry of Health et al. 1998) and the First Five-Year Action Plan for the Containment and Control of HIV/AIDS (2001–2005; State Council 2001), China had announced its new “Four Frees and One Care” policy just a few months before this meeting, in September 2003 (State Council 2004) (see Chap. 18 for more information).

Furthermore, several international aid projects were underway already, and more were already on the horizon. Hence, while it was recognized that there was a need for detailed and high-quality information on the HIV response, China’s HIV response itself was still in its infancy. As noted in a 2003 assessment of prevention, treatment, and care for HIV/AIDS in China jointly prepared by the Ministry of Health and the United Nations Theme Group on HIV/AIDS in China, there was yet “no overall plan for the gathering, storage and usage of HIV/AIDS information” (Ministry of Health and the United Nations Theme Group on HIV/AIDS in China 2003). Thus, a strategy for the development of an HIV M&E system that would standardize data collection, storage, audit, analysis, and exchange and that would simultaneously be tailored to the unique needs of China as well as meet the needs of the international community had become a high priority (see Chap. 20 for more information).

At the Bangkok workshop, China CDC representatives sought to operationalize an M&E system based on five core principles: (1) the system needed to be practical, realistic, and simple; (2) each province was responsible for quality assurance with higher-level, centralized verification; (3) focus needed to be placed on prioritization of indicators; (4) epidemic areas needed to be identified; and, finally, (5) multiple sectors needed to be involved. At the start of the workshop, attendees from China CDC had a draft of 10 core, and 14 additional, indicators to share with the group, seek feedback on, and further evaluate based on learning from the workshop (Fig. 23.1).

Fig. 23.1
figure 1

Draft core and additional indicators presented by the China CDC at the “Strengthening Monitoring and Evaluation of National AIDS Programs in Asia” workshop in Bangkok, Thailand, in November 2003

Full size image

At the conclusion of the Bangkok workshop, attendees from the China CDC had created a 12-month action plan that included six high-level activities: (1) formation of M&E teams to do the work of M&E plan development and coordination, (2) development of an M&E information management system, (3) strengthening of case reporting activities and sentinel surveillance systems, (4) initiation of behavioral surveillance among key populations, (5) implementation of M&E training and capacity building, and (6) encouragement of community participation as well as participation by PLWH.

By the end of 2003, the M&E plan was not completely centralized—program cites still developed their own plans with assistance from expert groups, monthly and quarterly reports were generated, and progress toward targets was generally self-assessed. Furthermore, the importance of tying indicators to performance was not mentioned among official China CDC documents, perhaps because China’s efforts toward curbing the HIV epidemic were too new to detect change. However, there were subtle hints that China’s HIV programs would be judged by their impact and that stakeholders would be held accountable for generating success. For example, in a directive on HIV/AIDS in 2003, Premier Wen Jiabao stated (Shen and Yu 2005):

A responsibility system, strategic planning, monitoring and supervision, capacity building of professional teams, technical measures and management by laws must be strengthened. Meanwhile, successful experiences should be summarized. Only in this way can we realize the true objective of HIV prevention and control, which is relevant to the fundamental interests of the people.

Furthermore, State Council documents that described national HIV response policy at that time explicitly stated that those leading HIV response efforts would be held accountable for their actions through annual evaluations of their job performance and that those who engaged in irresponsible conduct would be punished (Shen and Yu 2005) (see Chap. 18 for more information).

3 International Donors Influence Performance-Based M&E in China

In 2004, several international agencies launched efforts aimed at increasing the capacity of the Chinese Government’s HIV response. Principal among these was the Global Fund, and the China CDC was the largest recipient of grant moneys worldwide in the 2000s. Over a 10-year engagement, from 2004 to 2014, the Global Fund appropriated over 320 million USD to HIV response activities in China. Although the Global Fund was ready to begin deploying funds as early as 2002, China was not ready, both politically and ideologically, to meet the Global Fund’s requirements for transparency, inclusiveness, destigmatization, and community (i.e., civil society) participation (Huang and Ping 2014). Before 2003, China was largely in denial of its HIV epidemic, and societal stigma and discrimination against PLWH and their families was common and severe (Huang 2005). Participation of civil society in the fight against HIV/AIDS was minimal at this time, and Chinese Government power was substantial and focused on suppressing behaviors deemed undesirable. Not surprisingly, harm reduction was still very controversial in China during this time—giving condoms to sex workers or offering clean needles to drug users was viewed as supporting illegal activity. Thus, China CDC’s first application to the Global Fund in 2002, for grants to support HIV prevention programs for drug users, was rejected partly because it lacked a harm reduction component (Huang and Ping 2014).

However, beginning in late 2003, sweeping political changes occurred at the highest levels of China’s government, and national HIV response policy took a major turn for the better with the announcement in December of that year of the “Four Frees and One Care” policy. Additionally, China tackled the outbreak of severe acute respiratory syndrome (SARS) virus quickly and effectively in 2003, which gave the international community new confidence in China’s public health response capabilities. And finally, China’s eagerness to compete for, and win, Global Fund grants to help with the costs associated with implementing and scaling up a range of initiatives also contributed to major changes in the way it was willing to conduct its HIV response efforts (Minghui et al. 2015). China adapted to global norms and included harm reduction objectives when applying for the fourth round of Global Fund grants in 2004. Hence, the Global Fund’s involvement in HIV/AIDS programs in China began in September 2004, with a first phase that boosted the China CARES program, which was just getting off the ground at that time (see Chap. 25 for more information).

This first award of 11 million USD from the Global Fund to finance the China CARES program helped to shape many of China’s HIV/AIDS program indicators. Pre-Global Fund China CARES targets and indicators differed from Global Fund targets and indicators (Table 23.1). As an example of how targets and indicators changed with Global Fund involvement, in the area of prevention and treatment, a pre-Global Fund target of 90% of PLWH receiving free antiretroviral therapy (ART) measured by an indicator of the rate of ART coverage for eligible AIDS patients evolved into a post-Global Fund target of universal access measured by an indicator of the proportion of adults and children with HIV still alive 12 months after ART initiation.

Table 23.1 Comparison of targets and indicators used by the China CARES program beginning in 2003 and targets and indicators decided on for the Global Fund grants supporting the China CARES program beginning in 2004
Full size table

Minghui and colleagues described the important, yet difficult-to-quantify, effect of the Global Fund’s involvement in China’s HIV response (Minghui et al. 2015):

The way China’s leaders think about development has progressed, making the public health system more transparent, open, and accountable. This quantum advance in governance derived partly from procedural and reporting requirements that came with China’s alliance to the Global Fund.

However, the influence of international donors on target setting and monitoring of performance against those targets did have some drawbacks. Projects often collected similar information, but the information could not be shared because of different definitions and coding used by different funding agencies. Staff were inundated with paperwork. And, with an increasingly fragmented system attempting to meet the needs of so many different M&E requirements, it was difficult to grasp the true, overall performance of China’s HIV programs. In this way, the national HIV response was weakened and slowed by the involvement of so many international donors (Wu et al. 2011).

4 China’s Second Five-Year Action Plan Helps Launch HIV M&E

In early 2006, China launched its Second Five-Year Action Plan for the Containment and Control of HIV/AIDS (2006–2010; State Council 2006). The plan, as described by Rou and colleagues, “called for multi-sectorial cooperation in the HIV response, including all of society, and emphasized the importance of prevention programmes and their monitoring and evaluation” and included “specific work goals and indicators according to practical measurements” (Rou et al. 2010) (see Chap. 18 for more information).

To assess the impact of this increased government support, specific targets were developed. The aim was to create a performance-based approach to monitor the progress of large-scale, government-driven HIV programs. Setting targets and determining indicators to measure progress against those targets was viewed as means to make national HIV/AIDS program spending more efficient on two levels. First, on a national level, it would provide evidence for key policy and budgetary decision-making processes. Second, on a local level, it would facilitate the internal performance management of health agencies. Initially, indicators served as a tool for assessing the progress of program implementation. Performance assessment against these targets was also especially helpful in setting goals and determining indicators for the following year.

4.1 Piloting M&E

The next step toward M&E for China’s HIV response was to conduct a pilot. Therefore, in 2006, the National Center for AIDS/STD Control and Prevention (NCAIDS) of the China CDC selected two indicators—open 300 community-based MMT clinics and enroll a cumulative total of at least 30,000 PLWH in the National Free ART Program by the end of 2006 (Liu et al. 2010). Both these targets were met. This quick, early success demonstrated to authorities that setting targets and collecting data to measure performance against those targets could effectively promote the implementation of HIV/AIDS programs. They were convinced. And, since the beginning of 2007, NCAIDS has annually set incremental, realistic, quantitative targets for performance and used those targets to hold implementers of China’s national HIV/AIDS programs accountable for their work (Liu et al. 2010).

4.2 Accountability for M&E

NCAIDS, China CDC, was leading the way for HIV M&E, and thus, it made sense that accountability was assigned to leaders within this organization. So, for ease and convenience, the different indicators were quickly categorized and assigned based on the organizational structure of NCAIDS—the directors of each technical division were held accountable for indicators that fell within their department. Early each year, specific indicators and targets for each division were written into a contract, which was then signed by the director of NCAIDS and the directors of each of the technical divisions. At the end of year, an external review panel, consisting of NCAIDS retirees (emeritus professors), China CDC experts, and academics from the School of Public Health at Beijing University, and clinicians from Peking Union Hospital, was invited to judge the performance of each division against their agreed-upon targets (Liu et al. 2010).

Accountability for achieving each target was also assigned to each province. At the beginning of each year, representatives from all provincial-level administrative regions attended a national annual working meeting to review the prior year’s performance and to set new targets for the current year. A ranking system was used—the top five provinces were awarded a National AIDS Program Quality Performance Medal, while the bottom several provinces were required to improve their performance for the next year (Liu et al. 2010). For the following year, provincial and local leaders aimed to implement the best practices of top-performing provinces. This innovative approach helped enabled the staff responsible for program implementation to carry out their daily missions in a way that made their targets more achievable.

Additionally, NCAIDS carries out regular field visits and supervision activities, which serve to remind local CDC staff of their accountability for the year’s targets. These visits and activities are usually led by the NCAIDS technical directors, but occasionally the responsible chief of HIV/AIDS Division at the National Health Commission is invited to facilitate coordination for HIV/AIDS programs across different sectors. In August and September of each year, NCAIDS also organizes monitoring activities to supervise the data quality assurance activities of at least three provinces. This monitoring aims to guarantee the authenticity of field work and reliability of the data it generates.

4.3 Data Collection to Support M&E

Early in this process of building an M&E system for China’s HIV response, data collection, storage, extraction, and use were a very laborious process. Among a myriad of issues, systems were fragmented, data collection forms had both gaps and overlaps, and data fields were not standardized, making it difficult to use existing data with confidence. However, in 2008, China’s National HIV/AIDS Comprehensive Response Information Management System (CRIMS) was launched. This system integrated eight previously existing systems and one new system; standardized data collection; moved it to a real-time, web-based platform; and provided a user-friendly interface where data entry and extraction were simple and fast. This new tool dramatically changed the information landscape for China’s HIV response and helped the HIV epidemic and response M&E effort to take a huge leap forward (Mao et al. 2010) (see Chap. 24 for more information).

5 Scale-Up and Continuous Improvement of China’s HIV M&E System

Scaling up M&E after successfully piloting the targets and indicators for MMT and ART programs in 2006 meant selecting additional targets and indicators to include for 2007. However, the selection of targets and indicators for 2007 and onward actually took place over a several-year time frame based on using a continuous improvement approach. National guidelines such as the Framework for Monitoring and Evaluation of China’s National AIDS Program (State Council AIDS Working Committee 2007) as well as international guidelines and indicators developed by the United Nations General Assembly Special Session on HIV/AIDS (UNGASS; United Nations General Assembly Special Session on HIV/AIDS 2005; Warner-Smith et al. 2009), the Global Fund (the Global Fund to Fight AIDS, Tuberculosis and Malaria 2009), and WHO (World Health Organization et al. 2009) were considered. The selection of indicators also took into account input from NCAIDS technical directors and experts from other technical agencies, as well as comments made during presentations and discussions of the prior year’s performance during the external panel’s review of performance. Furthermore, indicators and targets were directly linked to the strategic objectives assigned to different levels of health agencies.

5.1 M&E from 2007 Through 2009

Old core indicators were carried forward into 2007, and new core indicators were introduced (Table 23.2; Liu et al. 2010). All targets and indicators fell into categories of availability, coverage, quality, and impact.

Table 23.2 Core indicators used from 2007 through 2009
Full size table

5.2 M&E from 2010 Onward

Over time, China’s indicators became more consistent, sensitive to the changing epidemic, and more mature. The data required to measure performance against them also improved over time as CRIMS, which was launched in 2008, began to contain more and better-quality data covering a more comprehensive expanse of information required (Mao et al. 2010). Targets also became more ambitious. As China’s national HIV response program matured, technical expertise grew, and consistent and adequate funding continued to be applied, NCAIDS leaders and technical directors became more focused and confident in what the health system, the public health system, the CDC network, and partnering NGOs and community-based organizations (CBOs) could achieve. Thus, annual achievements against those ever-more-ambitious targets were able to keep pace.

One example is the scale-up of the National Free ART Program. Annual indicators related to improvement of ART coverage included, for example, the cumulative number of PLWH on ART, the total number of PLWH currently on ART, and the proportion of PLWH on ART who remained on ART and survived 12 months. As ART coverage (i.e., the proportion of ART-eligible PLWH who were on ART) grew, mortality among PLWH fell (Fig. 23.2; Pisani and Wu 2017).

Fig. 23.2
figure 2

ART coverage compared with mortality rates among PLWH in China, 2005–2014. Figure adapted from Pisani and Wu (2017)

Full size image

Another example of how indicators have changed over time and of how targets have become more ambitious is the UNAIDS 90-90-90 targets, which were set out in 2014. These targets were meant to cause a refocusing of efforts and a swell of momentum for the further expansion of treatment on a global scale. The targets are 90% of all PLWH know their status, 90% of all diagnosed PLWH are on treatment, and 90% of all PLWH on treatment achieve viral suppression by 2020 (Joint United Nations Programme on HIV/AIDS 2014). China retrospectively examined its performance against these targets (Fig. 23.3; Ma et al. 2018) and decided to incorporate the 90-90-90 targets, as a national strategy, into its 13th Five-Year Action Plan for the Containment and Control of HIV/AIDS (2016–2020; State Council 2017).

Fig. 23.3
figure 3

Biennial performance of China’s HIV care cascade, 2005–2014. Bars denote the number of PLWH infected (estimated), diagnosed, on ART, and virally suppressed. Percentages were calculated as the number achieving the step (numerator) divided by the number in the previous step (denominator). Figure adapted from Ma et al. (2018)

Full size image

6 Conclusion

Developing meaningful indicators and setting annual targets for performance against those indicators has been crucial for motivating implementers to do the work of scaling up massive HIV response programs at local levels nationwide. The monitoring and accountability framework built around this M&E system has rigorously and successfully tracked HIV response program inputs, processes, outputs, outcomes, and impacts. Monitoring progress against priorities has emphasized the need to focus tactical work at the local level on national strategic objectives. More importantly, annual activities around HIV response M&E have become institutionalized—it is now routine work accepted as the norm in China’s public health institutions. This supports China’s goal of eventual elimination of HIV/AIDS and sets the nation up for success into the future, when the next major public health challenge appears.