A way to address the challenge of assessing complex environmental and development interventions that require comparing and cross-checking evidence from diverse disciplines is to apply a sequenced, purposive approach in the conduct of an evaluation. That is what the IEO has done with strategic country cluster evaluations (SCCEs). SCCEs focus on a limited set of common themes across clusters of countries and/or portfolios that involve a critical mass of GEF investments toward comparable or shared environmental challenges and that have gained substantial experience with GEF programming over the years. Starting from aggregate portfolio analysis to identify trends and cases of positive and absent or negative change, SCCEs are designed to dive deeply into those themes and unpack them through purposive evaluative inquiry. SCCE design is based on a conceptual analysis framework, an approach the GEF IEO developed earlier at the country level,Footnote 1 to enable comparison of findings across geographic regions and/or portfolios. In addition to the aggregate portfolio analysis, SCCEs use geospatial analysis to identify change on key environmental outcome indicators over time. Targeted field verifications follow in specific hot spots selected based on the findings of the geospatial and portfolio analyses. The purpose of field verifications is to identify and understand the determinants of the observed change or lack thereof.
The identification of factors hindering and/or enabling the sustainability of GEF outcomes was one of the main themes selected by the GEF IEO for deep-dive investigation in SCCEs. In 2017, the IEO completed a desk study on the sustainability of GEF project outcomes (GEF IEO, 2019b).Footnote 2 The study analyzed the IEO datasets of terminal evaluation ratings to assess correlations among sustainability, outcomes, implementation, broader adoption, project design features, country characteristics, and other variables. The analysis took stock of projects for which field verifications were conducted by the IEO at least 2 years after project completion. According to the study, the following contributing factors were at play in those cases where past outcomes were not sustained: (a) lack of financial support for the maintenance of infrastructure or follow-up, (b) lack of sustained efforts from the national executing agency, (c) inadequate political support including limited progress on the adoption of legal and regulatory measures, (d) low institutional capacities of key agencies, (e) low levels of stakeholder buy-in, and (f) inadequate project design characterized by flaws in the theory of change of projects.
The IEO further explored these issues by applying the new SCCE purposive evaluative enquiry approach to three different clusters of country portfolios. The SCCEs’ main objectives were: (a) to provide a deeper understanding of the determinants of the sustainability of the outcomes of GEF support and (b) to assess the relevance and performance/impact of the GEF toward the main environmental challenges from the countries’ perspective. Gender, climate resilience, private sector, and GEF operations in fragile situations were also assessed as cross-cutting issues.
A unique area of SCCE research was the environment vs. socioeconomic development nexus, a concept that is central to sustainable development. This nexus is too often neglected in development interventions, both by donors and developing countries alike (GEF IEO, 2020). Efforts to integrate socioeconomic development with environment conservation/sustainable use both at national and local levels depend on the interest of country governments. Many governments in the least developed countries (LDCs) believe that achieving both at the same time is difficult, and perceive, rather than a nexus, that major trade-offs exist between environment and socioeconomic/livelihoods objectives. Countries differ on: (a) reliance on natural resources, (b) susceptibility to natural disasters, (c) the poor’s dependence on the environment, and (d) the government’s economic development and other priorities. SCCEs investigated if and how the existence of a nexus between socioeconomic development needs and environmental conservation priorities (or lack thereof) contributed to or hindered the observed sustainability of project outcomes.
Applications of the SCCE Approach
The approach discussed in the previous sections has been applied to three clusters of countries, one covering the GEF portfolio of projects and programs in two biomes,Footnote 3 one covering LDCs, and one covering the small island developing states (SIDS) portfolios.Footnote 4 The African biomes covered by the first SCCE were the Sahel and the Sudan-Guinea Savanna. Selection of these two biomes was based on the countries’ comparable land-based environmental challenges. These countries also face challenges related to governance, demographics, migration, conflict, and fragility, which work as drivers for the environmental issues at hand. Most countries in the two selected biomes are LDCs, and half are fragile (World Bank, 2020).
The LDCs SCCE covered 47 countries that are currently designated by the United Nations as LDCs.Footnote 5 Focus on LDCs was based on these countries’ greater challenges related to sustainability of outcomes over several GEF periods (GEF IEO, 2019b) and related economic, social, and environmental challenges. Most LDCs are characterized by a low level of socioeconomic development. They have weak human and institutional capacities, low and unequally distributed income, gender inequality, and scarce domestic financial resources. LDCs often suffer from governance crisis, political instability, and, in some cases, internal and external conflicts. Twenty-eight of the 47 LDCs are fragile (World Bank, 2018). The SIDS SCCE covered 39 small island developing states in the AIMS (Atlantic, Indian Ocean, Mediterranean, and South China Sea), Caribbean, and Pacific regions. The choice to evaluate the SIDS as a strategic country cluster was based on their shared geophysical constraints that result in disproportionately large economic, social, and environmental challenges.
Selection of case study countries in the three SCCEs drew upon sustainability cohorts composed of national and regional projects completed between 2007 and 2014 and having Annual Performance Report (APR) ratings (GEF IEO, 2018b, 2019b, c) to allow for observation of the actual sustainability of outcomes 4–5 years after project completion. Projects in the African biomes and LDC cohorts were classified as: (a) having both outcome and sustainability ratings in the positive range (i.e., highly satisfactory, satisfactory, or moderately satisfactory); (b) having both outcomes and likely sustainability ratings in the negative range (i.e., highly unsatisfactory, unsatisfactory, or moderately unsatisfactory); (c) having either positive outcome and negative likely sustainability ratings, or the inverse; and (d) not having either outcome or sustainability ratings, or both (see Tables 1, 2, and 3).
Also informing the selection of country case studies were trends over time of key environmental outcome indicators at geolocated project sites, with the aim of identifying cases of positive and absent or negative change. Country case study selection started with the identification of the main environmental challenges faced by the countries covered by the respective SCCE. These challenges were classified by biome in the case of the African biomes SCCE and by geographic country category in the case of the LDCs and SIDS SCCEs. Projects with both positive and negative outcome and sustainability ratings in each portfolio were tagged to each environmental challenge.
Guided by the mapping of countries and projects to environmental challenges, the IEO selected countries with the largest number of national and regional projects with positive and negative outcome and sustainability ratings. This method ensured the largest number of observable data points and coverage of possible factors affecting sustainability. The countries selected also included those in which projects addressed the most commonly shared environmental challenges. In the African biomes SCCE, these were deforestation and land degradation, threats to biodiversity, and desertification. In the LDCs SCCE, these were deforestation and land degradation, and biodiversity loss. Water-related challenges were also important and included water quality and quantity, threats to marine resources, and coastal and coral reef degradation.
The application of the pre-mission selection process based on outcomes and sustainability ratings was accompanied by typical logistics and organizational considerations such as site accessibility and seasonality. In Bhutan, evaluators did not make the final selection of project sites to visit until after discussion upon arrival in the country with stakeholders in the Gross National Happiness Commission, relevant line ministries, and technical agencies such as the National Soil Services Center. For example, in the case of sustainable land management (SLM), these discussions resulted in the LDC SCCEs evaluation team visiting a site in Zhemgang District, selected out of three possible sites to logistically coordinate with site visits to the other projects in the sustainability cohort and in consideration of road conditions in the mountainous country. This choice was made because the SLM project sites are located in areas of high incidence of land degradation that are inhabited by most of the country’s poorest and most vulnerable communities. Although the terminal evaluation had rated the project’s outcomes in the positive and sustainability in the negative range, the evaluation team could verify that the SLM measures introduced by the project were still in operation 5 years after the project was completed. Selecting Zhemgang District for a site visit allowed the evaluation team to observe, 5 years postcompletion, the main sustainability factors fostering positive SLM results in mountainous ecosystems alongside unforeseen hindering factors. The team could verify the status of SLM measures introduced by the project and directly collect information on their continued use and maintenance from the remote rural communities living in those highly degraded lands. Photos 1 and 2 show meetings to finalize selection of sites and interview rural communities in Bhutan.
For the African biomes SCCE, once the evaluation team had selected the countries and projects based on the pre-mission selection process described above, they prepared geospatial maps for each project site prior to the missions to the country. Once in the country, evaluators used these maps to select the sites to visit in the field verification mission (see Fig. 1). This ensured the conduct of field observations in specific project locations selected both in highly degraded areas and in areas where vegetation had actually increased.
The evaluation team shared these maps with stakeholders (on a laptop/smartphone in the field, or on paper in local offices) to stimulate discussions and identification of the key factors at play driving the change observed in the map—see Photos 3 and 4. Local technicians, locally elected representatives, and community members all confirmed the environmental changes in the areas indicated in the maps and provided additional insights on when, how, and why those changes occurred. For example, in Tolo (Guinea), areas of increasing vegetation were subject to intense afforestation efforts accompanied by strict enforcement measures by local government forestry technicians. In Kaback, the anti-salt dikes built with GEF support in highly degraded coastal areas were insufficient in both height and width to withstand water intrusion. Attempts were made in Konimodouya and Katonko to change the approach by building a more robust dike, but these too could not withstand the rising sea-level pressure.
The purposive selection processes described above allowed for an in-depth, more granular, and comprehensive understanding of which specific factors have influenced the observed sustainability following project completion. In Tolo, the watershed identified for relocating farmers from the Bafing Lake had insufficient water for irrigation, a case of poor project design (see Box 1). In Zhemgang, the quality of project design led to the highly positive observed sustainability postcompletion (see Box 2).
Box 1: Field Visit in a Site Selected Based on Pre-mission Analysis: Tolo, Bafing Lake (Guinea)
The GEF project applied a coherent ecosystem approach to the whole watershed, working with all the stakeholders involved. Evaluators selected two sites to visit in Tolo: The first was on protection measure to rehabilitate the Bafing Lake banks, and the second involved community-based farming in the adjacent watershed. The lake is a source for 50% of the water going to the Senegal river. Around the lake is a community village. One of the project objectives was to reduce deforestation around the lake that leads to erosion and water loss from the lake basin. Deforestation is due to land clearing for slash-and-burn, itinerant agriculture. The local forest department enforces a forest-cutting ban around the lake. The project relocated the farmer community around the lake to a watershed 2 km from the village, where communities could practice horticulture. This delocalization measure was informed by a socioeconomic study followed by intensive participatory activities and negotiations, which provided a management arrangement for the distribution of land in the watershed and included granting some compensation measures to the farmers.
Years after the delocalization of the activity from the lake, the ecosystem of the lake banks has been slowly rehabilitated through intense reforestation measures (see Photo 5). The area has become green, with no agricultural activities around the lake, favoring the settling in of a small micro-climate that benefits the whole ecosystem. It was reported that years ago, one could cross the lake by foot in April due to damage from deforestation. The banks around the lake, once degraded from unsustainable agriculture activities, are now green.
Access to water remains the key impediment for agriculture in the Mamou region. The two hectares of watershed where the farmers have been delocalized has an irrigation system with canals that allows water to be spread on the field and six groundwater wells, all of which are thanks to the Community Land Management Project investments. The mission found this area underused. Farmers reported that despite the investments made, they can have enough irrigation water only for 6 months in a year (see Photo 6).
Box 2: Field Visit to a Site Selected Based on Pre-mission Analysis: Zhemgang District, (Bhutan)
The project aimed to strengthen institutional and community capacity for anticipating and managing land degradation. SLM practices were piloted in three geogs (groups of villages), where farmers were trained in SLM techniques. The project sites were in areas of high incidence of land degradation that were inhabited by the country’s poorest and most vulnerable communities. The project resulted in an increase in farmers practicing SLM techniques, a reduction in sediment flows in selected watersheds, regeneration of degraded forest land, and improved grazing land in the pilot geogs. The postcompletion site visit to a pilot geog in a remote area in Zhemgang noted continued practice of SLM techniques such as land terracing, hedgerows, fruit orchards, tree plantations, and irrigation systems. Selling produce both in the district and in Gelephu on the border with India has provided increased income for residents. Villagers confirmed in interviews that more land is under cultivation, and 60% of households continue using SLM techniques learned from the project. The remainder of the households discontinued using SLM due to shortages of water and losses caused by wildlife such as bears and wild boars. The government has provided some electric fencing, but it is not sufficient. The continued practice of SLM techniques has also helped improve and retain soil and convert shifting land cultivation to sustainable land cover (see Photo 7).
Among the project outcomes were the preparation and implementation of the 2007 Land Policy Act that incorporated SLM principles in programs and policies including the National Land Policy, the Forestry Policy, the National Adaptation Program of Action, and the National Biodiversity Action Plan. SLM principles have been incorporated in the government’s 12th five-year plan (2018–2023) and in plans on poverty reduction and increased food security.
Key factors driving postcompletion sustainability were good project design and government support, including highly relevant objectives in line with government priorities and relevant activities to achieve the stated objectives. The project design was guided by a bottom-up approach with participatory planning that focused on community priorities, phased implementation allowing for adjustment throughout implementation based on learning from pilots, decentralization to strengthen the role of communities and local authorities, use of knowledge and information on farmer incentives, and an integrated multisectoral approach. Before the completion of the project, institutional, financial, technical, and policy arrangements were made for sustaining its outcomes.
Geospatial Analysis Following Project Field Visits
The IEO conducted targeted geospatial analysis once teams returned from the missions, using the geographic coordinates collected with GPS tracking software apps installed in the team members’ smartphones and the information gathered during field observations. Both in Bhutan and Guinea, this analysis showed increased vegetation despite lower precipitations in the project sites visited, providing complementary data to shed more light on the observed changes (see Figs. 2 and 3). The reforested areas evidenced by the satellite photos taken in 2012 and 2019 on the Bafing lake basin in Guinea (Fig. 4) are the result of GEF-induced farmer relocation and afforestation activities accompanied by strict government enforcement. This temporary project success depends on the farmers continuing to practice horticulture in the watershed where they have been relocated.
In Zhemgang, both forest and vegetation cover in pastures have increased since the onset of the project. In Fig. 5, the 2010 image clearly shows large areas of relatively bare ground, which are subsequently covered by vegetation in 2018. The findings of the post-mission geospatial analysis of the SLM project confirmed the field visit finding of improved sustainability of outcomes years after project completion.