Objectives-Based Evaluation for Accountability and Learning

Abstract

This chapter features objectives-based evaluation (OBE) as a means to assessing the performance of international development finance, which remains key to achieving sustainable development priorities in many parts of the world. It lays out key evaluation criteria included in OBE and illustrates its multi-level approach that not only focuses on project-level outcomes but also on aggregate-level outcomes such as sectoral, country, or thematic level. It emphasizes the critical role of OBE in providing a big picture of how well development resources are being utilized. Again, it pays special attention to the use of the method with respect to issues of sustainable development.

Commitments of development finance naturally bring an emphasis on the delivery of agreed objectives. Where there is a stress on accountability in delivering on agreed goals, objectives-based evaluation (OBE) assumes significance. Equally, the lessons from such assessments inform what works and what does not.

Complementing inputs from impact evaluation (IE) and cost-benefit analysis (CBA), the OBE approach is applied to conduct individual project or program assessments and also aggregate sector- or country-level assessments. The assessment of the Sustainable Development Goals (SDGs) can occur at the micro project levels and also at the more macro and aggregative levels, as OBEs can ask about the extent to which the different interventions taken together address overall directions, for example, the pursuit of the SDGs.

Objectives-Based Evaluation

Overseas development assistance (ODA) is one of the core resources developing countries can use to work toward achieving the SDGs. ODA recipients are low- to middle-income countries who use the funds to invest in a wide range of sectors. The amount of ODA has increased over the past few years, exceeding US$150 billion in 2015 (at 2015 constant prices, according to OECD.Stat). At the same time, the size of ODA has been eclipsed by the size of the development financing labeled as coming from the private sector.

As development finance is used in more diversified ways, there is increasing recognition that it should be provided based on harmonized objectives and accountability. Harmonizing objectives can help improve efficiency in the use of resources, while accountability of both donors and recipients sheds light on how results are being achieved (or not) by development finance. OBE can facilitate both harmonization and accountability by establishing a set of objectives and examining achievements against these stated objectives. Assessing the effectiveness of development finance generates information for accountability and learning concerning the outcomes.

OBE is a class of evaluation methods that considers the extent to which objectives are achieved. Its strong focus on measuring outcomes against initially stated objectives encourages the formulation of clear objectives from the outset. It is guided by a theory of change that connects the inputs and activities to the outcomes. As opposed to output targets, which are set in terms of the tangible goods and services that project activities produce, outcomes measure the results likely to be achieved once the beneficiary population starts using the project outputs. Measures of outcomes look at the final results achieved, which indicate whether project goals are met (Gertler et al. 2016).

Output is something that can be controlled by a project-implementing agency. In a water project, for example, output can be measured by the number of pipes constructed. In contrast, outcomes can take longer to manifest and be observed long after the project has been completed. For instance, improved child health, measured using height-for-age (stunting) z-scores, is a potential outcome of piped water connections.

Different outcomes may be realized on different time horizons: the construction of pipes may save time and make collecting water more convenient for beneficiary households in the short-to-medium time frame, while in the longer term it may lead to improved health.

OBE Criteria

The standard criteria used in OBE are relevance, effectiveness, efficiency, sustainability, and considerations of development impact. While specified separately, these criteria also have a great deal of connectivity among them. More broadly, assessments of individual projects or aggregate programs are related to considerations of social inclusion, environmental protection, and governance, but at present these development goals are not directly used as evaluative lenses. Ratings for the standard criteria commonly use a four- or six-point scale, with categories that differ by rating institution. As an example, ratings used by the World Bank and Asian Development Bank (ADB) are summarized in Table 4.1 (IED 2016; IEG 2017a).

Table 4.1 Criteria and ratings of OBE: example of the World Bank and ADB

Relevance assesses the extent to which the financed activity is suited to a country’s development priorities. It asks whether the objectives of the project are valid and whether the activities, outputs, and project design are consistent with the objectives. Relevance considers analyses and lessons on project design, including the financing instrument and modality, as they relate to the objectives. It is important to consider both what is included in the project and what ought to be included. This criterion also looks into how the project takes into account the work of development partners and other organizations. All this means that a project, by the fact that it is approved, does not automatically qualify as being relevant; the evaluator must assess if it is.

Effectiveness measures the extent to which the project’s intended outcomes are achieved, based on a baseline and a target. For a project to be effective, outcomes should have been achieved or be likely to be achieved, and output targets should also have been substantially achieved. It also looks at factors influencing the achievement of the objectives. This is important for assessing the extent to which the outcomes achieved were a contribution of the project’s interventions. Achieved outcomes need to be plausibly attributable to the project or intervention as established through either IE or a theory-based approach and consideration of the counterfactual. If outcome targets were achieved or are likely to be achieved but output targets were not substantially met, the project will not be considered effective. This is because some outcomes at the national level, such as a higher literacy rate, may be achieved by other interventions than the one being evaluated.

Where projects have multiple objectives and outcomes, the evaluation can assign relative weights to the effectiveness of various objectives and outcomes. For example, a project may be implemented in two distinct geographical areas with one having a higher incidence of poverty. If intervention in the poorer area is seen as a higher priority, the poverty outcome in that area might be given more weight.

Efficiency measures how well resources are used to achieve outcomes. It assesses the project’s economic benefits against economic costs using CBA. In assessing efficiency, CBA, in principle, looks at costs and benefits, including unintended and indirect ones, from a societal viewpoint. Project economic-performance indicators—the economic internal rate of return, net present value, and the cost-benefit ratio—are used to determine whether net gains from investing in a project will be enjoyed by the society following project completion.

In many instances, it is instructive to compare alternative approaches to achieving the same results using a least-cost analysis. The cost estimates should be based on the economic costs incurred to implement the project, as well as provisions for the operation and maintenance of the assets over the expected economic life of the project. Project externalities—which are spillover effects and often unintended consequences such as environmental impacts—should be quantified and valued to the extent possible and incorporated into the calculations.

Sustainability here focuses on the more limited aspect (compared to the broad goal of sustainable development) of the likelihood that an activity will be maintained after donor funding has been withdrawn. It considers the major factors influencing the achievement or non-achievement of sustainability of the project’s results. Since an evaluation is typically carried out during the first few years of a project’s operational life, evaluators must make assumptions about the likely sustainability of operational arrangements, many of which are new, and about probable future operations and maintenance (O&M) arrangements. Even within this narrower confine of just one of the evaluation criteria, however, the environmental effects of a project must also be considered such as the effects on natural resource management, pollution, biodiversity, and greenhouse gas emissions.

This emphasis on environmental protection and climate change mitigation or adaptation is a relatively new requirement in evaluating projects. Climate mitigation is crucial, as the cost of adaptation is high. The less done to mitigate climate change, the more severe and expensive are the consequences. Assessments of sustainability should consider political, economic, institutional, technical, social, environmental, and financial risks. The assessment should also consider the adequacy of risk-mitigation measures. Although environmental- and social-risk avoidance may be part of the effectiveness assessment at some institutions, grave social or environmental consequences of failed mitigation may also affect the sustainability assessment.

Development impact is the positive and negative change generated by a development intervention, directly or indirectly, intended or unintended, and it mirrors the degree to which sustainable development is being achieved. It assesses the overall changes attributable to the intervention, the difference made to the beneficiaries, and the number of people affected. Development impacts to which the project contributes tend to be outside the project’s direct control, and their achievement is often not solely attributable to the project outcomes. Typically, they are dependent on other development efforts. The focus of analysis should be on the contribution of project outcomes to the achievement of the project impacts. Moreover, development impacts can also be due to unforeseen events and positive developments in areas that are outside the project scope. Such impacts should not be attributed to the project.

Multilateral development agencies have been using these criteria since the 1990s. In 1991, the Organisation for Economic Co-operation and Development-Development Assistance Committee (OECD-DAC) issued principles for evaluation of development assistance provided to public sector projects. The principles provided general guidance on evaluating projects financed using development assistance, stating that the aim of evaluation was “to determine the relevance and fulfilment of objectives, developmental efficiency, effectiveness, impact and sustainability” (OECD-DAC 1991). Also emphasized was the importance of formulating the objectives to be achieved at the baseline for high-quality OBEs.

Aggregative evaluations also incorporate these criteria. In response to the growing importance of harmonized evaluations, a workshop was held by the Working Group on Aid of the OECD-DAC on country-program evaluation methodology in 1999, and it was agreed that the criteria for evaluating development finance at the country level could draw on these criteria. Since the 1999 workshop, a common core of good practices for country evaluations has evolved, and gradually these common criteria have been adopted and endorsed by the independent evaluation offices of various Multilateral Development Banks (MDBs).

Project Evaluations

OBE assesses the outcome of development finance not only at the individual project level but also at aggregative levels such as county, sector, or theme. Lessons across projects from such aggregative assessments can bring good value.

That said, project evaluation is not only an in-depth assessment of what occurred in the investment but also a vital basis for what then can be surmised at the thematic or country levels. This means there is more to be learned from clusters of projects. Project-level evaluations are required as important building blocks to higher-level evaluations. Individual projects, for example, in transport or health, are a primary focus of OBE providing evidence on how development financing channeled through them are achieving the envisaged objectives.

As illustrated in Fig. 4.1, these project results are also building blocks for aggregative assessments spanning sectors like education or energy; themes like urban development or the environment; or nations, as in country assessments for China or Sri Lanka. They can also be inputs for corporate studies of how an organization is doing in different aspects of its development work.

Fig. 4.1

OBE of development finance. (Source: Authors’ illustration)

Public Sector Projects

Much of the OBE work is concerned with the performance of individual projects designed and implemented by governments. Applying the criteria discussed earlier, assessments arrive at conclusions on project success or failure in varying degrees. These success rates also vary a great deal across sectors, across countries, and over time.

The main value of delineating success rates (in addition to providing a judgment on the success in using financing) is in deriving lessons for improving performance going forward. Across thousands of projects financed by government and external financial agencies, some common lessons emerge (as seen in various annual evaluation reports from MDBs).

Both design and implementation feed into project success. The quality at entry—judged by indicators of project preparation, economic analysis, and due diligence—matters to the project’s eventual success. Annual evaluation reports of MDBs provide examples that taken together signal that projects with better quality at entry are seen to have a better chance of succeeding on completion.

Several aspects of implementation stand out in making a key difference to project success. Adequate allocation of budget for implementation and allowing for adequate spending on operations and maintenance of infrastructure investments are obvious considerations. The importance of pricing and regulatory policy frameworks is clear in the examples of water pricing or energy regulation. Institutional capacities of implementing countries are also important for project success.

Historically, project success in low-income countries has been seen to be lower than in middle-income countries, but this pattern is by no means universal. Furthermore, this gap has narrowed over time, as seen in the annual evaluation reports of Asian Development Bank and the World Bank. East Asia has been a perennial front-runner when it comes to project outcomes, a result especially driven by China’s strong project performance. Historically, infrastructure (energy in particular) has done well, though social sectors, environment, and multi-sectoral investments have strengthened in many settings over time.

The case of a water supply and sanitation project in Sri Lanka illustrates how these criteria are applied (IEG 2017b). There was uneven access to water and sanitation between urban and rural areas. Therefore, a water supply and sanitation program was implemented in 2004 with the objectives of increasing coverage and achieving effective and sustained use of water and sanitation services in rural communities. The project’s aim would correspond directly with what SDG 6 set out in 2015 for clean water and sanitation. The project was approved in 2003 and closed in 2010. The total costs were estimated to be US$69.1 million.

To prepare the project performance assessment report (PPAR), seven visits to project sites were made. In-depth discussions among focus groups were conducted to gather information to assess the project outcomes. The development outcome of the project was rated moderately satisfactory based on an assessment of relevance, efficacy, and efficiency (Table 4.2).

Table 4.2 Assessment of community water supply and sanitation project in Sri Lanka

Relevance of the development objective was rated substantial as it was aligned with both the government’s priorities and with the World Bank’s country assistance strategies with respect to expanding water and sanitation services to the rural population. Project design relevance was also rated substantial. The main components of the project design and its results framework were clear and logically linked to the project’s objectives of increasing service coverage and achieving effective and sustained use of water and sanitation services in rural communities.

The evaluation assessed efficacy, or the achievement of project objectives of service delivery and management, as well as demand and sustainable water use. Specifically it considered (1) increasing service coverage and (2) achieving effective and sustained use of water and sanitation services in rural communities in Sri Lanka.

The outcome of the first objective under efficacy was rated modest. The original target was not achieved. The number of people provided with access to improved water sources under the project increased by 384,100, but that was only 31 percent of the original target and 48.4 percent of the revised target at completion. The report points to two major reasons: decreased funds owing to the tsunami in 2004 and inflation. The target was not revised sufficiently despite the reduction in funds.

The second efficacy objective, which aimed to achieve effective and sustained water use, was assessed using demand-side outcomes: satisfaction, adequacy, reliability, convenience, and time saving, water quality, affordability, and sustainability of services. The achievement of this second objective was rated substantial. Satisfaction was generally high, with 88 percent of the beneficiaries of the completed water subprojects indicating they were satisfied with their access and that water adequacy had improved. A survey indicated that household connections led to sizable time savings.

The household survey at completion indicated that about 46 percent of the subprojects provided continuous water supply and 78 percent of households received piped water every day. The Ministry of Health’s sampling tests showed that only 44 percent of the subprojects provided water of satisfactory quality. The project achieved relatively high sustainability, but there were cases of water resource depletion and issues of frequent repair needs due to poor technical design in the initial phase.

Efficiency was rated substantial. The project provided access to improved water sources to fewer people compared to the original target, with slightly higher costs than estimated at appraisal. But the economic rate of return computed was still relatively high.

Community-based organizations were the main providers of water supply schemes in rural areas and responsible for operation and maintenance of water supply facilities in villages. Therefore, beyond the core criteria, risk to development outcome was rated significant because many community-based organizations faced technical, financial, and organizational challenges in sustaining the project results. Technical challenges included repair of pumps and water contamination. Furthermore, only a few community-based organizations were seen to be financially sustainable. Finally, some of them suffered from a shortage of volunteers, and the institutional arrangement for supporting them was unclear.

Private Sector Projects

Analogous to public sector projects, private sector projects can be evaluated using OBE. Though some OBE criteria are used for both public and private sector projects, there are important differences in their evaluation. Evaluations of private sector projects focus on business performance, which assesses the effect of the project on its financiers, and economic contribution, which assesses its economic effects more broadly.

Private sector project evaluation, while paralleling that of public sector projects, also has distinct features to account for the public support through MDBs to private businesses (see EBRD 2013; MDBs 2018). Since support to private sector projects is part of the broader activities of MDBs, the project assessment works within the framework provided by the MDBs’ overall mission to promote sustainable private sector investment in developing countries while also achieving social outcomes such as inclusive growth and poverty reduction.

Some MDBs have shifted away from methodologies largely driven by business performance. New approaches are based on market benchmarks and apply OBE criteria such as relevance, effectiveness, efficiency, and sustainability to private sector operations. Relevance and effectiveness in this context focus on both development and business outcomes. Efficiency looks at the financial performance (including achievement of business objectives) and economic performance.

An OBE considers a private sector project against its development outcomes, including business performance. The development impact rating is a synthesis of the impact of the project on the country’s economic and social development. Development impacts are evaluated using a with-versus-without-project comparison, that is, considering (1) what happened with the project and (2), counterfactually, what would have happened without it. When a with-versus-without assessment cannot be done, the costs and benefits to the country can be done on a before-versus-after basis.

At the World Bank Group, a private project’s development outcome is measured across four indicators: project business performance; economic sustainability; environmental, social, health, and safety effects; and contribution to private sector development. Each of these measures a distinct aspect of the project’s performance. The development outcome rating is a bottom-line assessment of the project’s results on the ground, and not an “average” of these four indicators. Each of the categories can be rated on a four- or six-point scale indicating the degree of success.

A project’s business performance measures the project’s impact on profitability and viability, the project’s contribution to other business goals, and the project company’s prospects for sustainability and growth. Sufficient financial returns are necessary to attract and reward private investment, but the assessment should also take into consideration the sustainability of the results. Projects can be structured as either loan or equity and can include institution-building components.

Interestingly, previous empirical analysis has found a strong connection between project development outcomes and an organization’s financial profitability (IEG 2009). For example, in a cohort of projects financed by the International Finance Corporation (IFC), high/high outcomes (high development outcome and high IFC investment return) were achieved in 66 percent of projects (by the number of projects, not volumes), and another 17 percent had low/low outcomes. This is suggestive of the possibility that focusing on development outcomes is also good business (IEG 2009).

Economic sustainability reflects the project’s contribution to economic growth. Projects with high economic returns clearly contribute to a country’s economic growth, and this contribution is quantifiable to some extent. Harder to achieve, but vital, is the effort to reduce poverty and improve people’s lives. It is important to address to what extent, as a result of a project, resources are being allocated more efficiently and the project portfolio is providing a net economic benefit, including the broader attributes of well-being.

It is important also to assess a project’s impact on people other than the investors (or adopt a stakeholder perspective) such as client companies and their customers, employees, government, competitors, and local residents. Examples of economic benefits and costs accruing to them are ease of access to markets and services, greater market efficiency, contribution to government revenues, contribution to poverty alleviation, social or gender equality, and employment generation.

Social and environmental protection are important components of the development outcome of private projects. That operations are carried out in an environmentally and socially responsible manner is not only sound business practice, but it is also a necessary condition for sustainable development. IFC’s Expanded Project Supervision Report assesses the project’s environmental performance in meeting regulatory requirements as well as the project’s environmental impacts through its subprojects, including pollution loads; conservation of biodiversity and natural resources; and social, cultural, and community health aspects, as well as labor and working conditions and workers’ health and safety.

Compliance with specific environmental requirements should be clearly stated in the OBE of private projects. Environmental requirements help enhance environmental management capacity and produce sound development outcomes and can be considered as proxy for acceptable environmental standards. But the effects on the ground should count most in evaluating development outcomes.

Supporting private sector development or encouraging the growth of private enterprises is a principal goal. Projects need to help create conditions conducive to the flow of private capital into productive investment. It is crucial that the benefits of growth of productive private enterprise accrue to the entire society. Projects can contribute to this purpose by contributing to the growth of sustainable and viable institutions, contributing to the development of the markets in which they operate, and by financing sustainable and viable private enterprises in the real sector.

Aggregative Evaluations

While OBE of individual projects generates information needed to assess the outcomes of each project, aggregation of project findings into an assessment of a country, sector, or theme can be of great value in seeing the combined effects. In moving from individual project assessment to aggregative evaluations, a key aspect is that projects are not usually independent of each other, but they complement, and sometimes substitute, each other. Projects are, or ought to be, connected to each other through a strategic framework that involves considerations of sequencing, prioritizing, and spillover effects.

When aggregative assessment is attempted, the core criteria are sometimes not weighted, and sometimes weighted according to policy priorities. Each rating uses a point scale of four or six categories going from high to low (e.g., gradations going from highly relevant to irrelevant). The core assessments are usually complementary and interrelated, for example, aspects of efficiency, such as good financial management, complementing effectiveness. At the same time, to avoid double counting, the same factor, say the lack of pricing for a service, may not be used the same way as the contributor to rating two separate criteria, efficiency and effectiveness.

Sector Evaluations

When individual project evaluations are added up to the sector level, as in transport or education, or at the thematic level, such as for the environment or urbanization, they bring out the aggregative impact of various projects. These also help uncover the effect of policies and investments that cut across the sector or theme. For example, a new decentralization policy might affect the collective success rates of public sector projects. Aggregative evaluations also often account for the influence of factors outside of the theme and the sector. For instance, an increase in infrastructure investments can affect the collective performance of education or health projects.

Sector evaluation is a useful tool to assess impact that goes beyond individual projects. Various organizations have conducted a range of sector evaluations, for example, in agriculture (IOE 2017; OVE 2015b), finance (OED 2018), and energy (IDEV 2015).

An example is the evaluation of World Bank-funded projects in education (IEG 2006; see also IFC 2014). This evaluation was based on a synthesis of outcomes of projects that focused on education access and learning, complemented by findings on policies and investments across the sector. The evaluation of individual projects conducted over several years was summarized and then used, along with sector-wide findings, to assess impact at the sectoral level.

The sector evaluation was done on the five criteria discussed earlier. The ratings in Table 4.3 comprise three criteria: development outcome (which is a composite rating of relevance, efficiency, and effectiveness), sustainability, and development impact.

Table 4.3 IEG ratings of completed primary-education projects by year of approval

Given the rising need to shift focus away from merely providing access to schools and toward improving the quality of education and accelerating learning, outcomes were evaluated by objectives related to access as well as learning and quality of education. Table 4.4 suggests that support for access to primary education in some respects did well, with serious gaps in improved access for girls.

Table 4.4 Outcomes by enrollment objective for complete primary-education projects

Further, Table 4.5 indicates gaps in improving learning outcomes and improved educational quality.

Table 4.5 Outcomes by objective for complete primary-education projects

Another example can be drawn from an evaluation of World Bank support for transport from 1995 to 2005 (IEG 2007). During this period, there were 642 projects with transport components, carrying a total financial commitment of US$32 billion. This was the first evaluation of the transport sector operation of the World Bank, and it provided insights that could not be obtained by evaluating individual projects given the significant diversity within the sector.

This sector-level evaluation also adopted the five criteria as project-level evaluation: outcome, which is a composite rating of relevance, effectiveness, and efficiency; institutional development impact; and sustainability. Within the evaluation period, performance of the projects improved over time across all indicators (Table 4.6). Ratings were lower when large borrowers were excluded because bigger countries in this sample had better institutional capacities.

Table 4.6 IEG ratings of projects by exit year, fiscal 1992–2006 (transport sector projects versus all others)

To gain further insight, sector evaluations can be disaggregated by region. This helps evaluate which region for a particular sector might be lagging and perhaps require further financing and technical assistance. For example, Fig. 4.2 shows that in the transport sector, South Asia is lagging in terms of outcome, while the Europe and Central Asia region and the Middle East and North Africa region may require support in institutional development.

Fig. 4.2

IEG ratings for outcome, sustainability, and institutional development (approval year fiscal 1995–2006). (Source: IEG 2007)

Country Evaluations

Country-level evaluation looks at the nationwide impacts of projects and associated activities of strategy and policy exchanges in a country. Sometimes the success of individual projects does not translate into better country performance. And despite project-level weaknesses, country-level dialogue and impact on sustainable development can be disproportionately high. The role of country expertise, quality of policy discussions, interactions with other partners, and the role played by knowledge are all factors determining country-level results.

For large lenders such as the World Bank and the IMF, the country-level program considerations are especially important, whereas for smaller bilateral financiers, country-level impact may not be that large. A variety of evaluations of country and global macrocosmic work is carried out at the IMF (see, e.g., IEO of the IMF 2016) and the United Nations (for instance, IEO of the UNDP 2018).

A country evaluation of India done in 2017 (IED 2017) assessed the performance of ADB strategy and programs for India from 2007 to 2015. ADB provided US$22.1 billion in development loans to India over these eight years, mostly for transport, energy, finance, and water and other infrastructure and services.

A full-fledged OBE approach for a country evaluation would assess the strategic objectives set out at the beginning of a country strategy and to what extent the objectives were achieved. India’s strategic directions present a very broad canvas, and ADB’s presence in the financing in terms of size and scope for this large economy is relatively small. Accordingly, the framework for the country evaluation was modest.

The OBE considered the overall performance of development finance to India to be successful. It also examined the country’s strategic agendas and special priorities, rating them individually. In accordance with India’s recent development plan, ADB country strategies for India shifted toward supporting faster, sustainable, and more inclusive growth. It evaluated ADB support by sector as well as support for strategic agendas.

Table 4.7 provides the assessment for the main sectors. The transport program was the largest component in ADB’s sovereign portfolio (35 percent) with a value of US$6 billion. Most of the loan went to construction, upgrading, and rehabilitation of rural roads mostly in lagging states in northern and eastern India, which is consistent with the sector strategy to promote connectivity and increase access of rural areas to towns and cities. Development impacts of the road program were rated satisfactory despite some delays that occurred due to limited capacity of executing agencies and contractors, resulting in efficiency being rated less than efficient.

Table 4.7 Sector-wise rating in country evaluation

In energy, the second-largest component of ADB’s sovereign portfolio (29 percent), 33 projects were approved with US$4.9 billion in value. Supporting the government’s strategy to develop a strong national grid and expand and optimize transmission and distribution systems, ADB’s support focused heavily on electricity transmission and distribution (81 percent of the energy sector investment). The energy programs also experienced delays for which efficiency was rated less than efficient. Development impacts, however, were rated satisfactory since all completed projects met their objectives of strengthening the transmission system, reducing losses, improving evacuation capacity and system reliability, and increasing access to electricity.

Water and urban infrastructure and services, finance, and public sector management were considered to have achieved satisfactory development impacts. Accordingly, the evaluation considered the overall achievement of these sectors to be successful.

Table 4.8 shows that the report also assessed ADB’s support for three primary strategic agendas: inclusive economic growth, environmentally sustainable growth, and regional cooperation and integration. The report also assessed special priorities: knowledge solutions and capacity development, gender equality, and catalyzing infrastructure investment and public-private partnerships.

Table 4.8 Assessment of the country’s strategic agendas and special priorities

Among the three strategic agendas, inclusive growth was most prominent in ADB’s lending portfolio to India, with some 90 percent of projects intended to contribute directly or indirectly to this objective. This was accomplished most notably through the financing that was made available to economically lagging states. In addition, ADB also supported inclusive growth through incorporating inclusive elements into project design (for instance, slum improvement in urban projects) and ensuring the implementation of projects in sectors with high inclusion impact, such as education, water supply, and rural electrification. About half of ADB’s support program was tagged for environmentally sustainable growth, effected mainly through projects in energy and water and urban infrastructure and services.

A similar country evaluation of development assistance to Brazil was done by the World Bank (IEG 2015; see also OVE 2015a). The support aimed at achieving greater equity, sustainability, and competitiveness, underpinned by strong economic management and governance. The evaluation recommended that the World Bank Group value benefits beyond individual interventions and make expected catalytic impact a major criterion in the design of its future strategy in Brazil.

Evaluations by Theme

Thematic evaluations can shed light on the overriding question of how development finance affects issues of inclusion, environmental protection, or governance, which form the core of the SDGs. Taken together, evidence emerging from OBE of individual projects or sectors can feed into the major themes that drive sustainable development in the aggregate. MDBs have done thematic evaluation on a number of themes: the environment and climate change (see, e.g., IDEV 2018; OVE 2014), gender and diversity (e.g., IEO of Global Environment Facility 2017; OVE 2018), and urban development (Cities Alliance 2017).

Consider a case where distributional impact is a concern and three projects—water access improvement, road construction, and deforestation activity control—were financed by an MDB. The water project may benefit the poor much more than the non-poor. Roads, on the other hand, may not connect the remote areas where the poor mostly reside, thus limiting the benefits that the poor can gain from them. It is also possible that controlling deforestation activity may on balance help the poor who depend on the sustainability of forest resources for their livelihood. Thus, different projects and investments in different sectors may affect the poor simultaneously but in different ways. Overall distributional impact can thus be assessed through a thematic evaluation.

Together, the various strands of OBE can be brought to bear on how much inclusion is taking place in the process of economic growth. This holds true also for sustainability and governance. Sustainability concerns projects that directly deal with environmental protection as well as projects in other sectors which might have an environmental impact. An evaluation of one project, or of a specific sector, is not capable of assessing the impact on environmental protection. Likewise, governance is related to projects in various sectors.

Inclusion

An ADB thematic evaluation on inclusive growth demonstrates the value thematic evaluations add to sectoral or country evaluations. ADB considers inclusive growth as a means to achieve poverty reduction, which was ADB’s objective in its Strategy 2020. This thematic evaluation defined inclusive growth as growth with social equity, that is, a growth process in which all segments of the population can participate in and benefit from, particularly the poor (IED 2014b).

The evaluation highlighted inequality in terms of income as well as access to opportunities as a serious issue in the region. Although the region’s gross domestic product (GDP) increased by 9 percent annually in the 1990s and by 8.2 percent in the 2000s, income inequality increased by about 1 percent annually in these two decades. Fast economic growth did not result in adequate access to opportunities either. Access to health care, water and sanitation services, and electricity varies widely across countries in the region and shows substantial gaps in many, for example, Bangladesh, Bhutan, Cambodia, Laos, Pakistan, and Vietnam.

The evaluation examined the promotion of inclusive growth through ADB’s development finance between 2000 and 2012, which totaled US$137 billion (IED 2014a, b). It looked at the three pillars identified in ADB’s strategy: (1) high sustainable growth to create and expand economic opportunities; (2) broader access to these opportunities to ensure that members of society can participate in and benefit from growth; and (3) safety nets to prevent extreme deprivation. As ADB’s inclusive growth framework was broad, most of its projects were categorized under the three pillars, and ADB’s support was heavily skewed toward pillar 1 (Fig. 4.3).

Fig. 4.3

Share of yearly project amounts supporting inclusive growth by pillar, 2000–2012. (Source: IED 2014b)

Given the rising inequality in the region, the evaluation suggested that the skewed focus on pillar 1 be reconsidered. The evaluation also emphasized the importance of project design and implementation in order to make projects inclusive. Considering urban-rural disparities are a major facet of unequal income distribution in the region, targeting rural areas, where the majority of the poor are found, could be an efficient way to accord special attention to benefiting lower-income groups. However, only 14.1 percent of infrastructure interventions targeted rural areas in the 13-year period. The study also suggested that the impact of infrastructure investments on inclusive growth be scaled up, for example, by linking rural infrastructures to schools, health centers, markets, and other services and opportunities.

The study assessed ADB’s support toward the inclusive growth agenda at the country level in six countries. It found that the opportunities for and obstacles to inclusive growth vary by country, and it proposed that ADB’s support toward inclusive growth should take into account the particular needs of each country and be designed in a way that maximizes the benefit to those who are poorer and with less opportunities.

Sustainability

The World Bank Group conducted a thematic evaluation of its support toward forest resource management for sustainable development in 2013 (IEG 2013). Matters of forest strategy would correspond to what was labeled in 2015 as SDG 15 on Life on Land. When the World Bank Group’s 1991 Forest Strategy was implemented, international concern was directed toward protecting tropical forests. The Strategy, therefore, focused on protecting tropical moist forests by not financing commercial logging in primary tropical moist forests. However, a review by the IFC concluded that this did not contribute to combating global loss of forest (IFC 2000).

The review argued that this was because (a) the low stumpage paid for government-owned forests was lower than the real cost of sustainably managing the forests and did not provide adequate financial incentive for private operators to engage in sustainable reforestation responsibilities, (b) private operators were not given control over the forests on which their operations rely, and (c) many governments around the world desired to retain ownership and control of their forestlands (IFC 2000).

The 2002 Forest Strategy incorporated the findings and proposals of the review and reshaped its strategies based on three pillars: (1) protecting vital local and global environmental services and values, (2) harnessing the potential of forests to reduce poverty, and (3) integrating forests into sustainable economic development.

The changes made to the 2002 Forest Strategy are summarized in Table 4.9.

Table 4.9 Differences between the 1991 and 2002 World Bank Group Forest Strategies

The changes highlight a geographical shift as well as the emphasis on sustainability. While the 1991 Strategy focused on tropical moist forests, the 2002 Strategy included all forest types. The 2002 Strategy also aimed to improve local livelihoods in Sub-Saharan Africa by protecting vast dryland forests and woodland areas, as the resources in these forests had commercial potential. As a consequence, there was a shift in emphasis toward Sub-Saharan Africa (Fig. 4.4).

Fig. 4.4

World Bank forest activities before and after the 2002 Forest Strategy by country. (Source: IEG 2013)

IEG (2013) evaluated forest sector-related projects, supported by the World Bank Group member countries and private sectors between 2002 and 2011, based on whether they balanced competing demands on forest resources and at the same time were sustainably managed. It included 289 projects approved by the World Bank in 75 countries. The evaluation was conducted based on a review of the projects and portfolio as well as extensive interviews with stakeholders and site visits.

The evaluation concluded that the World Bank Group’s forest interventions contributed positively to environmental outcomes. However, poverty reduction, for the most part, was not satisfactorily addressed. Among the interventions evaluated, the study found participatory forest management to have delivered livelihood enhancing benefits as well as positive environmental outcomes by linking forest products to markets. The study highlighted that participatory forest management should be promoted further. However, to make this intervention effective and sustainable, authority needed to be devolved to communities, and regulations needed to be improved to integrate small-scale informal forestry activities.

Governance

ADB evaluated its support toward enhancing governance in its public sector operations through a thematic evaluation in 2014 (IED 2014a). Good governance is crucial for achieving inclusive and sustainable growth. SDGs that were agreed upon in 2015 featured governance, with SDG 16 devoted to peace, justice, and strong institutions. Despite the rapid economic growth achieved in many countries in Asia, governance has not improved substantially. South Asia ranks especially low on control of corruption, regulatory quality, and political stability and absence of violence/terrorism, as per the 2017 World Governance Indicators (World Governance Indicators 2017).

Public sector management (PSM) is a sector-crosscutting practice that is often channeled through multi-sector programs and projects. Loans, grants, and technical assistance represent the main channels through which ADB supports the enhancement of governance. ADB financed US$11 billion (of which US$2.3 billion came from the Asian Development Fund) from 1999 to 2011, which makes PSM operations ADB’s fourth largest sector program during this period. A review of the success rate in 1990–1999 and 2000–2010 showed an improvement in the latter period. However, compared to other sectors, performance of PSM financing is still weak.

The analysis of failed PSM projects underscored three common factors: lack of institutional capacity and/or resources in government to undertake projects or reforms, overly ambitious or complex designs, and weak government ownership and commitment (IED 2014a). The thematic evaluation recommended more rigorous diagnostics to be conducted at the project design stage, which required sufficient understanding of institutional capacity and political commitment and incentives.

Thematic evaluations thus help MDBs identify successes as well as gaps in their development financing. Through these evaluations, MDBs can enhance their impact on SDGs that form the core of their mission.

Conclusion

In Chap. 2 we discussed the IE tools that enable evaluators to assess the extent and nature of the difference a development project makes. CBA, as discussed in Chap. 3, is a complementary approach, providing indications of the net gains derived from a project. OBE can draw on both these tools in assessing the degree to which the goals of a project are being met. Its primary emphasis is not so much the causal effect of a project, but more on the contribution the project makes in meeting stated objectives of sustainable development. Even with a counterfactual analysis built in, the results are associations, as they do not adequately control for the simultaneous effects of other projects and interventions.

OBE thus provides an overarching understanding of the effectiveness of development projects. Both external agencies and country governments can get an overview of the objectives and how they are being met under a set of common criteria, albeit with differences across different financing agencies. Evaluations of projects can be aggregated into results for sectors, themes, or countries. Such evaluations can help inform how much of the original or revised objectives are being achieved and what might be done to improve performance. Thus, OBE is a good check on accountability with respect to sustainable development that also provides lessons for future development financing.