Winds of Change in Development and Response

Development is a dynamic field with new demands placed on development actors to mainstream and incorporate every few years. The Brundtland Commission report (Brundtland, 1987) brought the importance of environment to the front and center of the development debate. Similarly, the Intergovernmental Panel on Climate Change was established in 1988 to draw voices from across the globe onto a single platform to tackle climate change. However, only later did environment and climate change become mainstreamed into the programming of development partners. Environment found higher recognition first in the aftermath of the Rio Earth Summit in 1992 and the formation of financing mechanisms such as the Global Environment Facility (GEF) in 1991. Climate change also started to be reflected more explicitly in development programming in the late 2000s, in light of the global food price crisis.

Evaluation as a field has to account for these evolving trends in development. Evaluation has to reflect the emerging priorities of the development field and measure progress on their achievement. At the same time, evaluation also has to incorporate new demands emerging from within the field, such as making evaluations more equity focused, gender mainstreamed, and human rights-centric.

This chapter reflects on how the Independent Office of Evaluation (IOE) of the International Fund for Agricultural Development (IFAD) addressed the emerging demands on evaluation function in terms of incorporating concerns on environment and climate within the existing methodological framework(s), directly or indirectly, while adapting its methodology to meet internal and external demands on IFAD evaluation. This chapter also illustrates how evolving methodologies, methods, and tools have helped IFAD overcome these issues.

Evaluation Methodology of IOE

The Independent Office of Evaluation of IFAD has had three iterations of evaluation methodology, codified in the Methodological Framework for Evaluation 2003 (MFE, 2003), Evaluation Manual 2009 (first edition), and Evaluation Manual 2015 (second edition). Each of these has built on the work of the previous methodologies. The starting point for all three are the evaluation criteria of the Development Assistance Committee of the Organisation for Economic Co-operation and Development (OECD DAC) first laid out in the Principles for Evaluation of Development Assistance (OECD, 1991) and later defined in the 2002 Glossary of Key Terms in Evaluation and Results Based Management (OECD, 2002). For the purpose of this chapter, IOE’s evaluation criteria is divided into three categories: core criteria, impact criteria, and other criteria. Over the years, each of these categories has evolved to encompass different facets of sustainable livelihoods and resilience to climactic shocks. Table 1 presents IOE’s evaluation methodology over time.

Table 1 Evolution of IOE evaluation methodologies
Full size table

In terms of methodology, this chapter explores three defining features of IOE’s methodology and their role in evaluating environment, sustainability, and resilience to climate change. These link to a sustainable livelihood approach, constant evolution of methodology, and accumulated methodological experience through various products.

Sustainable Livelihood Approach and Evaluation Methodology

Conceptual Linkage Among Livelihoods, Environment, Resilience, and Agriculture

IFAD has the mandate to work toward enhancing the livelihood systems of rural populations through agricultural and nonagricultural livelihood options.

When discussing sustainable livelihoods, the definition from Carney (1998) reveals the various layers therein:

A livelihood comprises the capabilities, assets (including both material and social resources) and activities required for a means of living. A livelihood is sustainable when it can cope with and recover from stresses and shocks and maintain or enhance its capabilities and assets both now and in the future, while not undermining the natural resource base. (p. 4)

This definition interweaves the ideas of livelihoods and resilience. It also lays out another important aspect of livelihood enhancement and the resilience to shocks: interaction between human and natural systems.

Central to both the Carney (1998) definition and determining the resilience of households to vulnerabilities is the idea of livelihood assets. These are the means of production available to a given individual, household, or group that can be used in their livelihood activities and have the potential to produce something that is economically desirable (Goodwin, 2003, p. 3). Natural capital, social capital, human capital, physical capital, and financial capital are the five types of assets discussed in the literature (United Nations Development Programme, 2017), and may be tangible or intangible in nature.

According to Chambers and Conway (1991), tangible assets include food stocks; stores of value such as gold, jewelry, and woven textiles; and cash savings in banks of thrift and credit programs. This category also includes land, water, trees, livestock and farm equipment, tools, and domestic utensils. Nontangible assets include claims and access. Claims are often made in times of shocks or stress or when contingencies arise. They are made on individuals or agencies; on relatives, neighbors, patrons, chiefs, social groups, or communities; or on nongovernmental organizations (NGOs), the international community, or governments, including programs pertaining to drought relief or poverty alleviation. Access is the opportunity to use a resource, store, or service, or to obtain information, material, technology, employment, food, or income. Figure 1 illustrates the components and flows in a livelihood.

Fig. 1
figure 1

Components and flows in a livelihood

Source: Chambers and Conway (1991)

Full size image

The definition of sustainable livelihoods brings to the fore the importance of withstanding shocks and uncertainties for ensuring sustainability of livelihoods, and the role that the various kinds of assets play in doing so. Some of the major shocks that the poor face include political, climactic, and economic shocks. In this context, even before climate change and environmental sustainability became more mainstreamed into development parlance and expressed more explicitly in development theory, there was an implicit recognition of the various climactic shocks and a more explicit recognition of the broader strategies to cope with them.

Over the years, evaluation criteria have evolved to encompass different facets of sustainable livelihoods and resilience to climactic shocks. The core criteria of relevance, effectiveness, and efficiency were influenced by the Principles for Evaluation of Development Assistance published by OECD DAC in 1991 and have been reflected in every iteration of IOE’s methodology.

The second part of the methodology links directly to the sustainable livelihood approach through the criteria and subcriteria under the impact domain in Table 1. These criteria/subcriteria mirror the various kinds of livelihood assets—natural capital, social capital, human capital, physical capital, and financial capital—to varying degrees. Given the criteria’s intricate links to the sustainable livelihood approach, IOE has been able to determine whether an intervention of a portfolio of projects is able to provide or enable its smallholder target groups with assets that help them adapt to various kinds of shocks, including climactic shocks. This goes beyond looking at physical infrastructure to considering intangible assets such as individual agency and social capital, skills that can help smallholders adapt to all kinds of shocks. However, a crucial gap in using this rubric of criteria in evaluating adaptation to climate change is that it only looks at the presence or absence of certain preconditions with a causal assumption that these will help in adaptation efforts without always placing them in the context of the climactic risks that exist in a given area.

The combination of OECD DAC criteria and those derived from a sustainable livelihoods approach form the normative framework on which IOE has built accountability and learning in rural development. This normative framework is reflected in the various iterations of IOE evaluation methodology over the years (see Table 1) and has formed the basis on which IOE has built its methodology for evaluating environment and climate adaptation.

Evolution of the IOE Evaluation Methodology

As Tables 1 and 2 illustrate, each subsequent iteration of IOE evaluation methodology has undergone significant changes in terms of criteria. Livelihood assets, such as human and social assets, natural resource assets, physical assets, and financial assets, were most explicitly discussed in the MFE 2003 and are reflected similarly in the evaluation manuals of 2009 and 2015. Three broad observations illuminate the evolution of the evaluation criteria and methodology.

Table 2 Evolution of IOE evaluation questions
Full size table

First, evaluation criteria have evolved from reflecting livelihood issues to a broader focus on themes and thematic thrusts. The MFE 2003 criteria focused much more on prevalence and strengthening different kinds of livelihood assets that enhance the resilience of smallholders. The strengthening and presence of these was seen as necessary for livelihoods to be resilient and sustainable. As shown in Table 1, evaluation criteria in 2009 and particularly in 2015 reflected thematic priorities such as environment and natural resources, gender, food security, and, most prominently, climate change. This change in methodology can be attributed to the evolving nature of IFAD operations with a move away from integrated rural development programs to those with a theme focus, such as rural finance, value chains, market access, and integrated natural resource management. Much more important, the reflection of natural resource management as a separate criterion reflected an increasing recognition of and focus on environmental conservation and management in IFAD programming in the context of donor demands and supplementary financial resources from funding institutions such as the GEF.

Second, IFAD’s evaluation focus has evolved from looking at conservation status of the natural resource base toward recognizing the importance of the natural resource base for livelihoods of rural populations and their sustainable use and management. This is most prominently reflected in the evaluation questions framed under each iteration of the evaluation methodology and the number of questions under the environment and natural resource management criteria (see Table 2). In terms of questions, the 2003 MFE had two questions pertaining to the state of the natural resource base and vulnerability of rural poor to environmental risks. The subsequent iterations of methodology in 2009 and 2015 contained more expansive coverage in evaluation questions on environment and natural resource management. The later iterations of IOE evaluation methodology have essentially focused on the sustainable interactions between human and natural systems. The questions elaborated in Table 2 are by no means exhaustive and provide only initial guidance; actual questions asked under the criteria in each evaluation may differ.

Third, climate change adaptation was mentioned as a separate evaluation criterion for the first time in 2015. As Table 2 illustrates, climate change was covered more implicitly under the criterion of environment and natural resource management in 2003 and 2009, as shown in the evaluation questions that elaborate on environmental risks. Such implicit inclusion has often been meant to also cover climate change-induced risks. However, in the 2015 iteration of IOE methodology, the assessment of climate change is more explicit, with questions on current and future risks. Such a change took place as a result of an evolving and increasing emphasis on climate change adaptation in IFAD’s strategic emphasis, corporate policies, and programmatic thrusts.

Evaluation criteria have undergone change in two ways. First, newer evaluation criteria such as climate change adaptation have been added over time to reflect the evolving criteria. Second, even when evaluation criteria have remained similar, the scope of evaluation criteria has expanded in terms of the suggested evaluation questions under those criteria. A consequence of the increase in the scope of questions is that crosscutting issues are better incorporated into evaluations. Such questions also better account for complexity. For example, the 2015 iteration of evaluation methodology recognizes the dependence of target groups on the natural resource base for livelihoods and thus looks at human and natural system interaction. Similarly, the questions under the 2015 methodology also go beyond simple conservation to incorporate governance of the natural resource base.

Accumulated Methodological Experience Through Various Products

IOE produces a wide variety of products with differing scope, focus, and purposes. In the early 2000s, most of IOE’s focus was on country portfolio evaluations and project evaluations. However, IOE has increasingly moved toward a more thematic focus over the years. This started with undertaking higher plane evaluations such as corporate-level evaluations, which look at corporate and thematic priorities beyond evaluation of a country portfolio or project. This was further reinforced when, in 2011, IOE began undertaking evaluation synthesis reports on specific topics, an exercise that consolidates evaluation findings on a specific topic over a period of time.

Thus, IOE has built its experience in evaluating environment and climate change over a period of time by undertaking incrementally different kinds of analysis on the topics. For example, IOE produced evaluation synthesis reports in the past few years on topics such as fisheries, water management, and environment and natural resource management. This accumulated experience on evaluation synthesis also pointed to the need for a higher plane evaluation specifically focused on climate change adaptation. As of the time of writing this chapter, IOE was undertaking a thematic evaluation on this topic.

IOE has also been able to consolidate the findings of its numerous evaluations over a period of time to provide trends of performance on evaluation criteria in its Annual Report on Results and Impact (ARRI). Such trends are depicted on a 3-year moving average. These assessments have provided IOE with useful and contemporary insights and helped in planning evaluations that probe the underlying factors influencing performance. Figure 2 depicts the trends in IOE’s rating on environment and natural resource management criteria as shown in the 2020 ARRI (IFAD, 2020).

Fig. 2
figure 2

Trend in ratings on natural resource management

Full size image

The figure illustrates how performance saw some decline in the period of 2007–2013 before picking up again. IOE undertook the evaluation synthesis report on environment and natural resources management in 2015 to consolidate the lessons that IOE evaluations have generated on the topic and explanatory factors for performance in the area of environment and natural resource management. Thus, IOE’s products have progressively built on one another to inform the debate on thematic areas of priority for IFAD. IOE has introduced newer products from time to time to meet the accountability and learning requirements of the organization. Such evolution in its products is operationalized through the evaluation methodology, which has been revised as the need arose for tackling newer challenges in IFAD operations.

Methods for Assessment Using Normative Frameworks

IOE typically has used a wide variety of methods to collect data on evaluation questions and criteria. The evolution of IOE’s methods mirror the diverse elements of the organization’s evaluation methodology vis-à-vis environment and climate change adaptation. The sustainable livelihood element of the normative framework mostly employs qualitative methods.

Sustainable Livelihood Approach and Qualitative Methods

Livelihood assets such as natural, social, human, physical, and financial capital are instrumental in ensuring the resilience of rural populations. IOE methodology has historically focused on ascertaining the impact of IFAD interventions through creation of these assets. Various dimensions under these asset classes help in adaptation through different pathways and questions are asked under each of these to understand their role in helping smallholders adapt.

Physical Capital

Typical physical assets include tangible assets such as land, housing, and vehicles that are repositories of economic value. In terms of primary data, evaluators generally assess physical assets through individual interviews and direct observations of field interventions. For example, evaluators in the field enquire with beneficiaries on their state of current asset matrix, the composition of assets, and the utility of such assets in times of shocks. In terms of secondary data, IFAD’s measurement and evaluation (M&E) system typically captures the state of housing, size of landholding, and household items as a part of outcome and impact surveys. Such data is used by evaluators after due diligence on the methodology in the outcome surveys. The data is also validated during field visits through interviews and direct observations.

Financial Capital

For IFAD’s target groups, this category typically includes access to financial assets such as credit, insurance, savings, etc. Community-based savings groups and remittances also hold an important place in IFAD’s operations in ensuring access to financial resources in the event of facing shocks, especially financial shocks. Because discussions around financial resources can be sensitive, evaluators usually deploy focus group discussions only to understand the nature of community-based and other informal and formal sources of financial services that remain at the disposal of IFAD’s target groups and these sources’ role in helping communities withstand shocks. For example, in many community-based savings groups that IFAD creates and strengthens, financial products exist for restarting economic activities such as restocking of livestock herds and reviving crop agriculture. Evaluators conduct interviews with individuals to understand their access to alternative avenues of income such as remittances and nonfarm income, and the resilience of those sources.

Social Capital

Social capital is defined by the OECD as “networks together with shared norms, values, and understandings that facilitate cooperation within or among groups” (2007, p. 103). Thus, assessing social capital involves looking at the social dynamics between people, the functioning of community institutions, and formal and informal bonds. This requires evaluators to explore the individual and collective social capital to understand the level of trust, solidarity, and coherence that exists in a community. This is usually done through in-depth focus group discussions with individuals who share social and geographic spaces. Such focus group discussions also attempt to understand the ability of target groups to lean on their social networks in times of shocks and the willingness of the social networks to provide avenues for relief and recovery. Interviews with individuals allow evaluators to confirm the extent to which individuals are able to rely or have relied on project- or program-facilitated social networks. In terms of secondary data, IFAD uses its M&E system from time to time to capture the state of community institutions through a composite rating or a qualitative assessment of their functioning.

Human Capital

Human capital refers to skills, knowledge, and ability to work. Building human capital usually involves training in farm and off-farm livelihood activities. To gather secondary data, IFAD’s M&E system captures output data on the trainings and skills imparted. IOE also interviews participants on the nature of the skills learned and their use in the livelihood matrix. IOE pays special attention to target groups’ ability to use the skills to undertake livelihood activities that promote resilience. For example, IFAD programs in some countries help build off-farm vocational skills for professions and jobs that enable people to diversify their livelihoods or migrate seasonally for income-generating activities.

Natural Capital

Historically, IFAD has relied on interviews and focus group discussions to understand the state of the common and communal natural base and the ability of target groups to access these assets. This refers not just to access as a whole but also to equity of the access for various groups. Hence, in-depth individual interviews and focus group discussions put special emphasis on interacting with a sample that is as representative as possible when assessing access to natural capital. Direct observations have also provided a general view of the state of natural resource management. However, in recent times, IOE has employed newer methods to assess interventions on environment and natural resource management, described below.

Methods for an Evolving Methodology

As mentioned previously, IOE’s methodology has evolved in the past few iterations from assessing the presence of livelihood assets to a more thematic focus on climate change adaptation and environment and natural resource management. More specifically, successive evaluation methodologies have moved away from simply looking at target groups’ ability to access natural resources. Now, IOE considers the state of the natural resource base, interaction between human and natural systems, and climate change-induced risks to livelihoods and to the natural resource base. IOE has felt an increasing need to adopt newer methods because conventional qualitative methods do not sufficiently assess on these dimensions. For example, conventional methods cannot holistically assess the state of natural resources in a project/program area. A method such as direct observation might provide a general view of the state of the natural resource base at a certain point in time, but natural resources systems are highly dynamic and require a longer term view to understand program effects and natural resource systems at large. IOE assessments also need a quantitative evidence base on the evolution of natural resource systems. Similarly, anchoring the adaptation efforts of a program/project in local climactic risks requires a temporal and scientific analysis.

With this recognition, IOE has adopted geographic information systems (GIS) analysis as one method for data collection and analysis. GIS provides IOE with two advantages over existing qualitative methods. First, GIS delivers a temporal view of IOE’s area of interest. IOE can analyze the state of natural resources and other topographic features of interest across years, irrespective of whether an IFAD program was active in that area. This allows IOE to create a baseline situation in a given area even if program M&E does not have a baseline survey. It could also allow IOE to create a counterfactual for the project intervention, although this has yet to be piloted in IOE. Figure 3 illustrates an IOE recreation of a baseline situation in a given area. Second, GIS gives IOE spatial capability to evaluate beyond the mobility limitations of evaluators in the course of field visits. In simpler terms, GIS allows evaluators to assess interventions in areas they may not be able to visit.

Fig. 3
figure 3

Sample difference in vegetation in Nepal between 2015 and 2019

Full size image

IOE first piloted GIS in its work in 2016 as a part of an impact evaluation in Georgia, using satellite imagery to analyze changes in landscape and cropping patterns in response to irrigation system improvements. IOE retrieved satellite imagery captured prior to the program to establish a baseline, and then from the midpoint of the program and the time of evaluation. IOE used this imagery to calculate the normalized difference vegetation indexFootnote 1 and construct models to estimate crop yields before and after irrigation improvements, taking into account other variables from the program area. This analysis triangulated the findings emerging from the survey data collected as part of the impact evaluation itself.

Similarly, IOE used GIS in the context of a country-level evaluation in Nepal. Satellite imagery (such as in Fig. 3) allowed IOE to understand changes in the landscape over a period of time through visual observation of imagery. The idea was to understand the broader nature of evolution of the natural base in the general project area without constructing models, which would require more primary data on specific variables to reach any robust conclusions. Figure 3 shows that the leasehold forestry training provided in Karnali province of Nepal did result in improved density of tree cover in one of the sites. This was triangulated through qualitative questions on the value of nontimber forest produce of the forestry groups that depend on these leasehold forests. This provided a proxy for increase in forest cover and its impact on the livelihoods of target groups.

Constraints

One of the limiting factors for using GIS in IOE evaluations is the lack of systematic streamlining geotagging of project/program interventions. This prevents IOE from capitalizing on one of the major advantages of using GIS in evaluation work: the spatial capability to evaluate beyond the mobility limitations of evaluators during field visits. Usually, one of the ways to circumvent this constraint is by considering administrative boundaries of a project site location. However, IFAD’s interventions mostly target individual households at the community level and rarely work at sub-watershed/watershed level or landscape level. Thus, considering administrative boundaries would lead to weak attribution and even contribution of the program to enhancing the natural resource base. In the absence of systematic tagging of interventions as part of program/project M&E, evaluators are limited to analyzing only those sites that they have physically visited and geotagged themselves. This reduces the scope for ensuring external validity of any conclusions that evaluators may draw based on GIS analysis. Another limitation is the lack of capacity within IOE to undertake GIS-based collection and analysis of data. So far, IOE has relied on external expertise to undertake GIS analysis in evaluations.

GIS for the Future

IFAD has begun more systematic geotagging of its project/program sites and has made plans to introduce geotagging as a part of operational procedures for projects in some regional divisions. Such data is also being organized in a format that will be consistent across the organization and available on one web-based platform, IFAD Geonode. This should help IOE better use GIS to evaluate interventions moving forward. To capitalize on the opportunities provided by these new methods, IOE is exploring various avenues to build GIS-related capacity in house.

Future Perspectives on Methodology for Evaluating Climate Adaptation and Natural Resource Management

As mentioned earlier, IOE started evaluating climate adaptation and natural resource management through the framework of sustainable livelihood approach and the five kinds of capital. IOE’s initial methodological approach to resilience (including climactic risks) looked at the presence of the five kinds of capital, and their presence was seen as enabling resilience of IFAD’s target communities. However, resilience needs to be more firmly anchored in risks that prevail in a given context. The second and third iterations of IOE’s manual emphasize more strongly the kind of environment and climactic risks and the analysis undertaken to recognize those risks in IFAD operations.

Future methodological directions could build on the existing normative framework. Two potential lines of enquiry that could be better explored in future evaluations are appropriateness and adequacy of interventions addressing climate risks and accompanying climate adaptation needs. Appropriateness links interventions more explicitly to prevailing climate change-induced risks in a given area and seeks to identify whether an intervention is appropriate for a given risk. Although the evaluation questions in IOE’s methodological iterations do implicitly recognize the risks and responding interventions in a given context, they do not do so explicitly. Adequacy of interventions pertains to whether the intensity of an intervention, the matrix of interventions, and interventions’ coherence is sufficient.

However, such evolution would require a newer paradigm of evaluation methodology, methods, and specialized skill sets to understand the climactic risks in a given context. This would require evaluations to assess two aspects of a program/project. First, project design would need to include a climatic risk assessment for the program area. Second, evaluations would have to validate the robustness of the climactic risk assessment process, which would require the capacity of evaluation teams to undertake climate risk assessment through qualitative and quantitative methods.

IOE has come a long way in terms of conducting evaluations, constantly learning as an evaluation unit, and upgrading its methodology periodically. As the operations of IFAD have become more focused on the issues around natural resource management and climate change adaptation, IOE has had more occasions to build its knowledge base. In the next years, IOE will have an opportunity to undertake another revision of its methodology when it revises its evaluation manual and incorporate the new methods and experiences it has piloted.