A Primer on Public Sector Evaluations

Abstract

This chapter provides a brief survey of the program evaluation methods, their objectives, strengths, and weaknesses. The chapter first presents methods like cost-benefit analysis, cost-effectiveness analysis, the social marginal cost of funds analysis, and data envelopment analysis that are appropriate for guiding efficient allocation and utilization of resources. This is followed by a discussion on multiple-criteria evaluation (MCE), which is much more holistic. Apart from efficiency, it addresses issues like the relevance of a program, effectiveness of a program in achieving its objectives, and sustainability of the program benefits. Newer MCE approaches such as the Iron Triangle and the Results-Oriented Management Evaluation are briefly sketched. Finally, theory-based evaluation is outlined, where the focus is not just on whether a program succeeds or fails but also on how and why a program succeeds or fails.

Annex: An Example of a Multi-Criteria Evaluation Approach—The Practice by the World Bank Operations Evaluation Department/the Independent Evaluation Group

The World Bank has been a premier institution using MCE in evaluating its programs and projects. The approach used by the World Bank Evaluation Department (earlier the so-called Operations Evaluation Department, OED, and now the Independent Evaluation Group, IEG) has evolved over time. The OED/IEG approach used the following criteria.

Relevance of Objectives

Definition: The extent to which the project’s objectives are consistent with the country’s current development priorities and with current Bank country and sectoral assistance strategies and corporate goals as expressed in Poverty Reduction Strategy Papers, Country Assistance Strategies, Sectoral Strategy Papers and Operations Policy papers.

The IEG considers the following factors in overall relevance: government ownership and commitment; explicit Bank strategy; results framework; analytical underpinning; flexibility; strategic focus; appropriateness of instrument mix, Bank capacity; Bank and IFC coordination; and Bank and other development partners’ collaboration.

Rating of relevance by OED/IEG:

• High/Mostly Relevant: Most of the major objectives were highly relevant.

• Substantial/Partially Relevant: Most of the major objectives are at least substantially relevant.

• Modest/Partially Relevant: Most of the major objectives were not highly or substantially relevant.

• Negligible/Not Relevant: Most of the major objectives were irrelevant or negligibly relevant.

Efficacy

Definition: The extent to which the project’s objectives were achieved, or expected to be achieved, taking into account their relative importance.

Rating of Efficacy by OED/IEG:

• High/Achieved: Major objectives were fully met, or expected to be fully met, with no shortcomings.

• Substantial/Mostly Achieved: Major objectives were met, or expected to be met, with only minor shortcomings.

• Modest/Partially Achieved: Major objectives were met, or expected to be met, but with significant shortcomings.

• Negligible/Not Achieved: Most objectives were not met, or expected not to be met, due to major shortcomings.

Efficiency (by OED; the IEG Dropped This Criterion)

Definition: The extent to which the project achieved, or is expected to achieve, a return higher than the opportunity cost of capital and benefits at least cost compared to alternatives.

Ratings by OED (Note: IEG discontinued this criterion)

• High: Project represents sector/industry best practice in terms of cost-effectiveness, and economic returns (if estimates are available) greatly exceed the opportunity cost of capital.

• Substantial: Project meets sector/industry standards in terms of cost-effectiveness, and economic returns (if estimates are available) exceed the opportunity cost of capital.

• Modest: Project fails to meet sector/industry standards in terms of cost-effectiveness, and economic returns (if estimates are available) are near the opportunity cost of capital.

• Negligible: Project is well below sector/industry standards in terms of cost-effectiveness, and economic returns (if estimates are available) are significantly below the opportunity cost of capital.

Sustainability (by OED; the IEG Dropped This Criterion)

Definition: The resilience to risk of net benefits flows over time.

Assessments of sustainability take into account nine factors:

• Technical resilience

• Financial resilience (including policies on cost recovery)

• Economic resilience

• Social support (including conditions subject to Safeguard Policies)

• Environmental resilience

• Government ownership (including by central governments and agencies, and availability of O&M funds)

• Other stakeholder ownership (including local participation, beneficiary incentives, civil society/NGOs, private sector)

• Institutional support (including supportive legal/regulatory framework, and organizational and management effectiveness)

• Resilience to exogenous influences (including terms of trade, economic shocks, regional political, and security situations)

OED Ratings:

• Highly Likely: Project net benefits flow meets most of the relevant factors determining overall resilience at the “high level,” with all others rated at the “substantial” level

• Likely: Project net benefits flow meets all relevant factors determining overall resilience at the “substantial” level

• Unlikely: Project net benefits flow meets some but not all relevant factors determining overall resilience at the “substantial” level

• Highly Unlikely: Project net benefits flow meets few of the relevant factors determining overall resilience at the “substantial” level

• Not Evaluable: Insufficient information available to make a judgment

Result (New Criterion by the IEG)

Definition: To what extent specified output targets were met.

Ratings by IEG:

• Met: Specified output targets were fully met.

• Mostly met: Major output targets were met.

• Partially met: Some output targets were met.

• Not Met: Most output target were not met.

Outcome/Effectiveness

Definition: The extent to which the project’s major relevant objectives were achieved, or are expected to be achieved, efficiently.

Ratings by OED/IEG (Note that the IEG has consolidated the ratings into four as follows).

• Highly Satisfactory/Achieved: Project achieved or exceeded, or is expected to achieve or exceed, all its major relevant objectives efficiently without major shortcomings.

• Satisfactory/Achieved: Project achieved, or is expected to achieve, most of its major relevant objectives efficiently with only minor shortcomings.

• Moderately Satisfactory/Mostly Achieved: Project achieved, or is expected to achieve, most of its major relevant objectives efficiently but with either significant shortcomings or modest overall relevance.

• Moderately Unsatisfactory/Partially Achieved: Project is expected to achieve its major relevant objectives with major shortcomings or is expected to achieve only some of its major relevant objectives, yet achieve positive efficiency.

• Unsatisfactory/Not Achieved: Project has failed to achieve, and is not expected to achieve, most of its major relevant objectives with only minor development benefits.

• Highly Unsatisfactory/Not Achieved: Project has failed to achieve, and is not expected to achieve, any of its major relevant objectives with no worthwhile development benefits.

An important limitation of the OED approach to the assessment of outcome is that the outcome is considered independent of sustainability. A project may be judged “Highly Satisfactory” while it may not have been sustained.

Institutional Development Impact (IDI; by the OED Only)

Definition: The extent to which a project improves the ability of a country or region to make more efficient, equitable, and sustainable use of its human, financial, and natural resources through: (a) better definition, stability, transparency, enforceability, and predictability of institutional arrangements, and/or (b) better alignment of the mission and capacity of an organization with its mandate, which derives from these institutional arrangements. IDI considers that the project is expected to make a critical contribution to the country’s/region’s ability to effectively use human, financial, and natural resources, either through the achievement of the project’s stated ID objectives or through unintended effects.

OED Ratings:

• Substantial: Project as a whole made, or is expected to make, a significant contribution to the country’s/region’s ability to effectively use human, financial, and natural resources, either through the achievement of the project’s stated ID objectives or through unintended effects.

• Modest: Project as a whole increased, or is expected to increase, to a limited extent the country’s/region’s ability to effectively use human, financial, and natural resources, either through the achievement of the project’s stated ID objectives or through unintended effects.

• Negligible: Project as a whole made, or is expected to make, little or no contribution to the country’s/region’s ability to effectively use human, financial, and natural resources, either through the achievement of the project’s stated ID objectives or through unintended effects.

The IEG no longer uses this criterion.

The Bank Performance

The OED in addition also rated the Bank performance based upon “Quality at Entry” and “Supervision” as follows.

• The Quality of Entry ratings took into consideration: project consistency with Bank strategy for the country; grounding in economic and sector work; development objective statement; approach and design appropriateness; government ownership; involvement of stakeholders/beneficiaries; adequacy of technical analysis; economic and financial impact analysis; environmental assessment; impact on poverty reduction and social issues; institutional analysis; adequacy of financial management arrangements; readiness for implementation; and assessment of risk and sustainability.

• The Supervision ratings took into account two major factors: focus on development impact and adequacy of supervision inputs and processes. The focus on development impact includes: timely identification/assessment of implementation and development impact; appropriateness of proposed solutions and follow-up; effectiveness of Bank actions. The Supervision ratings took into account: adequacy of Bank supervision resources; supervision reporting quality; attention to fiduciary aspects, and attention to monitoring and evaluation.

OED Ratings on Bank Performance

• Highly Satisfactory: Bank performance was rated as Highly Satisfactory on both quality at entry and supervision, or Highly Satisfactory on the one dimension with significantly higher impact on project performance and at least Satisfactory on the other.

• Satisfactory: Bank performance was rated at least Satisfactory on both quality at entry and supervision, or Satisfactory on the one dimension with significantly higher impact on project performance and no less than Unsatisfactory on the other.

• Unsatisfactory: Bank performance was not rated at least Satisfactory on both quality at entry and supervision, or Unsatisfactory on the one dimension with significantly higher impact on project performance and no higher than Satisfactory on the other.

• Highly Unsatisfactory: Bank performance was rated as Highly Unsatisfactory on both quality at entry and supervision, or Highly Unsatisfactory on the one dimension with significantly higher impact on project performance and no higher than Unsatisfactory on the other.

The IEG instead rates Bank performance based upon (a) strategic relevance at country level and (b) effectiveness of Bank interventions. The effectiveness is assessed by relevance, result, efficacy, and overall effectiveness criteria.

The Borrower Performance (by the OED Only)

Definition: The extent to which the borrower assumed ownership and responsibility to ensure quality of preparation and implementation, and complied with covenants and agreements, towards the achievement of development objectives and sustainability.

OED rated borrower performance on there counts: (a) preparation; (b) implementation; and (c) compliance. The preparation took into consideration institutional and financial constraints. The implementation considered macro and sectoral policies/conditions; government commitment; appointment of key staff; counterpart funding; and administrative procedures. The implementing agency performance was also considered. The compliance considered all major covenants and commitments undertaken by the borrower.

Ratings

• Highly Satisfactory: Borrower performance was rated Highly Satisfactory on at least two of the three performance factors.

• Satisfactory: Borrower performance was rated at least Satisfactory on two of the three factors.

• Unsatisfactory: Borrower performance was not rated at least Satisfactory on two of the three factors.

• Highly Unsatisfactory: Borrower performance was rated Highly Unsatisfactory on at least two of the three factors.

The IEG no longer rates the borrower performance as indicated by the above criterion.

Source: World Bank (2002, 2020).