Methodological Approach of the GEF IEO’s Climate Change Mitigation Impact Evaluation: Assessing Progress in Market Change for Reduction of CO₂ Emissions

Abstract

This chapter presents the methodological approach adopted in the evaluation of GEF support to market change for climate change mitigation in four emerging markets: China, India, Mexico and Russia. The evaluation was completed in October 2013. This evaluation included 18 completed and fully evaluated GEF mitigation projects covering various sectors with opportunities for renewable energy, energy efficiency and methane emission reduction. A theory of change approach was used to undertake a comparative analysis across projects aiming to tease out changes across diverse markets or markets segments in different countries as a consequence of GEF support. While attention was given to the extent to which projects resulted in actual greenhouse gas (GHG) emission reductions, more emphasis was placed on understanding the extent and forms by which GEF projects contributed to long term market changes resulting in GHG emission reductions and assessing the added value of GEF support in the context of multiple factors affecting market change.

1 Introduction

The Global Environment Facility (GEF) is a partnership for international cooperation to address global environmental issues related to biodiversity, climate change, international waters, land degradation, and chemicals and waste.¹ Since its inception in 1991 GEF has provided more than US 14.5 billion dollars for addressing these concerns, of which at least $ 4 billion has been provided to support activities that directly address climate change mitigation.² Within the GEF partnership, The GEF Independent Evaluation Office (GEF IEO) has the central role of ensuring the independent evaluation function.³

The OECD DAC ‘Glossary of Key Terms in Evaluation and Results Based Management’ (OECD 2002) defines impact as “Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended.” The OECD DAC’s Principles for Evaluation of Development Assistance’ (OECD 1991) defines evaluation as “an assessment, as systematic and objective as possible, of an on-going or completed project, programme or policy, its design, implementation and results.” Thus, impact evaluations may be understood as systematic and objective assessment of the long-term effects of a development intervention. The impact evaluations undertaken by GEF IEO seek to gauge the long term effects of GEF support, how these were achieved and how GEF’s effectiveness in achieving them may be improved. These evaluations have a strong focus on learning.

The GEF IEO undertook “Climate Change Mitigation Impact Evaluation ⁴ to assess impact and learn lessons from GEF supported climate change mitigation projects. This paper discusses the methodological approach adopted for the evaluation, the challenges faced and choices made in developing and implementing the evaluation, which was carried out by the GEF Independent Evaluation Office in four emerging economies: China India, Mexico and Russia.⁵ The evaluation was implemented from 2012 to 2013.

The purpose of the evaluation was to promote accountability and learning about GEF’s mitigation programme and across GEF overall. It assesses the extent and ways in which GEF support contributes to market change to reduce CO2 emissions and mitigates climate change, and derives lessons to improve the effectiveness of future GEF support.

The evaluation concluded that GEF projects achieved significant direct GEF emission reduction, although indirect emission reduction – which is difficult to measure – may account for much larger reduction. The evaluation found that of the 18 projects covered, in 17 cases there was broader adoption of promoted technologies, approaches and strategies, beyond the direct scope of the project. It found that the projects that demonstrated high progress towards long term impact were those that had adopted comprehensive approaches to address market barriers and specifically targeted supportive policy frameworks. The evaluation found that the methodologies being used by project teams to measure GHG emissions and to calculate ex-post emissions reduction at project completion were inconsistent and contained uncertainties.

The experience gained through conducting the evaluation made methodological challenges in evaluating GEF support salient. It was challenging to draw conclusions and lessons from a large diversity of projects that GEF finances and the wide range of sectors that it covers. Another challenge is the assessment of GEF contributions to change when multiple actors, factors and conditions affect outcomes. Similarly, inconsistency and inaccuracies in measurement pose difficulties. This paper presents methodology adopted to evaluate the contributions of GEF support to initiatives seeking to reduce climate change emissions.

2 Utility as a Guiding Factor to Define What Needs to Be Evaluated

The initial step was to identify the overall topic and the key questions that the evaluation would address. The key criteria were the extent to which the evaluation could provide useful information to inform future GEF support on climate change and the extent to which there were sufficient completed projects to carry out an impact evaluation. The climate change mitigation strategies and programs supported by the GEF were reviewed. Given that most climate change mitigation projects supported by the GEF aim at transforming markets for reducing green-house gas emissions, this emphasis became a starting point for developing the evaluation questions. Evaluation questions were developed in consultation with the GEF Secretariat staff especially those from the climate change mitigation program, and GEF Partner Agencies that were responsible for implementing these projects on the ground. This process led to three overall evaluative questions. These are: (1) What have been the GEF contributions to GHG emission reduction and avoidance? (2) What has been the progress made by GEF supported activities towards transforming markets for climate change mitigation? And (3) What are the impact pathways and factors affecting further progress towards market transformation?

The composition and trends of the GEF mitigation portfolio were analyzed to identify the types of projects that GEF has been supporting and where this support was more concentrated. The next step was to identify a set of projects from which relevant lessons could be derived in addressing climate change mitigation while simultaneously assessing the results of GEF support. Based on consultations with the stakeholders, a decision was made to focus on the major emerging economies based on their respective share within the GEF climate change mitigation portfolio, the potential for climate change mitigation, and their continued importance for future GEF support in this area. Due to budget and time considerations, it was difficult to cover more than four countries. Selection of these countries was based on an iterative process of portfolio analysis and consultations with key stakeholders. Firstly, the GEF climate change mitigation portfolio in all the emerging economies was compared. Based on criteria of overall size of the climate change mitigation portfolio, share in the climate change mitigation portfolio approved before 2002, share in the technology transfer portfolio; and, share in STAR⁶ allocation for climate change mitigation, six countries were identified for further consideration. These were Brazil, China, India, Mexico, Russia and South Africa (Table 9.1). Further analysis showed that GEF climate change mitigation portfolios in China, India and Mexico stand out both in terms of total cumulative GEF funding and total GEF funding in projects that were approved before 2002. Among the remainder, GEF IEO had completed a Country Portfolio Evaluation⁷ in Brazil when the preparation for the Climate Change Mitigation (CCM) impact evaluation started. Therefore, to avoid evaluation fatigue, Brazil was dropped. In South Africa the GEF climate change mitigation portfolio was relatively small compared to other major emerging economies both in terms of completed projects and projects that were under implementation. Therefore, it too was dropped. Russia, where the portfolio of completed projects was also relatively small, was selected because a sizable amount of investment was under implementation and it also had the third largest allocation for climate change mitigation for GEF-5 (2010–2014) period.

Table 9.1 GEF CCM portfolio in the countries considered in this evaluation (in US $ million)

3 Defining the Scope of the Evaluation

Among the four selected countries the extent of coverage was based on substantive and operational considerations. Completed projects that addressed concerns that were still relevant for GEF and likely to receive funding in future were considered (Table 9.2). In India and Mexico all completed full size projects were included. In Russia two of the three completed projects were included.⁸ However, in China where the GEF climate change mitigation portfolio was the largest, only some of the completed full size projects were selected so as to keep the cost of the evaluation manageable. In selecting projects in China, it was ensured that the major targeted markets such as wind energy, solar energy, and transportation, which were also covered in at least one of the other three countries, were represented. While implementation success was a criteria in selection of the projects within China, the eventual outcomes and long term impacts were not considered for selection. Thus, the completed projects covered as part of this evaluation are representative of GEF’s support to climate change mitigation in the countries covered by the evaluation.

Table 9.2 Technologies/Markets addressed by projects covered by the evaluation

In all 18 completed climate change mitigation projects were covered. These projects account for more than US $ 180 million in GEF funding and more than US $ 680 million in total financing. GEF requires that the project proponents also seek co-financing from other sources so that GEF funds only the incremental costs of implementing the projects. Total GEF funding for covered projects ranged from US $ 1.0 million to $ 40 million, whereas total financing ranged from US $ 3.0 million to $ 284 million.

The date for start of implementation of the projects covered through the evaluation ranged from 1992 to 2007 and their completion dates ranged from 1997 to 2012. Although the projects were designed to be implemented for a duration of 3–6 years, during implementation several projects needed extension. As a result, the actual duration of project implementation ranged from about 4 years to 12 years. Inclusion of projects that ended at different points in time meant that at the time the evaluation was conducted, different time duration had elapsed post project completion. That this difference is a factor was established when comparison was made between the observed progress to impact at the point of project completion and at the time evaluation was conducted.

4 Assessing Impacts of GEF Support

An intervention theory of change is meant to explain how inputs and activities will lead to outputs and impacts and to make explicit the key assumptions about how impacts will be achieved. Many publications discuss the use of theories of change in evaluation (Chen 1990; van den Berg and Todd 2011; Weiss 1972). These approaches are particularly well suited to evaluate specific projects or programs. The climate change evaluation included a wide variety of projects covering diverse technologies and markets that are affected by different factors and conditions including policy instruments, institutions, and interactions among producers, suppliers and consumers. Thus, this specific challenge required an overall framework that allowed systematic comparison among such different interventions. Since its inception, GEF has been supporting generation of global environmental benefits in different focal areas. However, for a long period there was no consistent overall conceptual framework that was applicable across its different focal areas to assess how GEF intends to achieve the global environmental benefits. The GEF IEO has, however, found that having such an explicit framework is important for its impact evaluation work. GEF IEO has prepared a generic framework for the development of theories of change (TOC framework) to facilitate these comparative analysis (GEF IEO 2012, 2013, 2014). This general framework was used as a basis to assess impact of GEF climate change mitigation activities (Fig. 9.1).

Fig. 9.1

General framework for GEF theory of change (Source: GEF IEO 2014)

The generic TOC framework shows that the GEF support seeks to change behavior and institutions by focusing on three broad realms of intervention: generation and sharing of knowledge and information; development of institutional capacities; and, testing implementation strategies for behavior change. Through its support to one or more of these realms, GEF support aims to bring about conditions and behavior that if broadly adopted can result in transformations in the long run. The framework identifies the following as pathways for broader adoption: sustainability , mainstreaming, replication, upscaling and market change. Depending on the intervention, one or more of these pathways can be at play. Thus while some carbon emission reduction can take place over the short run, emission reduction at scale is assumed to take place gradually over a longer period of time as behavior changes and systems transform. It is also assumed that the extent and trajectory of change is likely to be affected by multiple factors, some of which may have been addressed by the project, while others may not have been envisioned during project preparation and/or addressed through project design. In time, the spread of tested approaches and behavior that reduce environmental stress (carbon emission reductions) result in changes leading to improved environmental status and human wellbeing. This process is assumed to be unpredictable, non-linearand affected by multiple confounding factors, thus requiring constant attention and adaptation to emerging conditions (Zazueta and Garcia 2014). In case of GEF support for climate change mitigation activities the expected long term impact includes reduction in GHG emissions through the transformation of the structure and function of targeted markets (Fig. 9.1).

The TOC framework was used to develop theory of change for specific project clusters that were covered in the evaluation. The specific theories relevant to the projects were used in this evaluation to develop instruments that would ensure comparability of the information gathered.

Information gathered on the specific contributions of GEF support to conditions (knowledge and information, institutional capacities and effective implementation strategies) to reduce CO2 emissions as well as expected impact pathways , along with information on the rival hypothesis on observed changes, formed a basis to assess GEF contributions to observed changes in the targeted markets.

5 Understanding the System Targeted by the Intervention

The definition of the system which the intervention seeks to change has a strong bearing on the factors that the evaluation will consider in its analysis. While the climate change phenomena take place at different scales including local, national, regional and global, to determine the specific evaluation questions that needed to be asked, in addition to the underlying project’s theory of change for the given intervention, the evaluation also focused on understanding the system that the GEF supported activity was trying to influence, including system boundaries, system components, interactions among components and emergent properties characteristic of each system (Holling 2001; Wörlen and Consult n.d.). While acknowledging that processes affecting climate change take place in various interlinked scales, it is important to identify the relevant system boundaries pertinent to the evaluation. For this evaluation the targeted market/technology was the unit of the analysis and the system boundaries were set at the national scale. Attention was given to identifying various components, and to the interactions among these components or segments of the market targeted by the project. Special attention was given to identifying market barriers to change and how these barriers affect the functioning of the system and the systems likely emergent properties. Wörlen (2014) and Wörlen and Consults (n.d.), have analyzed changes in the status of market barriers addressed by climate change mitigation projects, including those supported by the GEF. The evaluation built on their work to assess changes in targeted market barriers and factors contributing to change.

Subsequent steps focused on assessing the extent and the way in which specific elements of the intervention’s theory of change interacted with elements of the system (Mayne 2008). The focus of enquiry was on how the intervention became part of the system and the changes (intended or unintended) which this brought about (Garcia and Zazueta 2015). This perspective seeks to emphasize the interconnectedness of the intervention and elements of the system unlike other contextual perspectives that emphasize the effects of context on the intervention (Pawson et al. 2004; Blamey and Mackenzie 2007). This approach was woven into the instruments, which were designed to gather information on the GEF activities and on the links of these activities to support provided by other actors that were relevant to the targeted market. The instruments also took into account the extent the activities undertaken by the other actors were influenced by the GEF support and vice-versa.

The decision to restrict the system boundaries at the national scale was influenced by the fact that flow of information and learning is easier, and policy framework more consistent, within the national boundaries. Similarly, barriers related to suppliers, finance and expertise are more consistent within a country than among countries. GEF projects too are generally geared towards influencing the targeted markets at the national scale (Eberhard and Tokle 2004). This, however, does not mean that the systems at the national scale are insular and may not be affected by factors that have origins in other countries. The evaluation itself documented three instances – ILUMEX (GEF 575), BRT (GEF 1155) and Landfill (GEF 784) in Mexico which had been replicated/scaled up in other Latin-American countries.

6 Measurement of Emission Reduction Benefits

The direct and indirect tons of CO2 emission reductions for each project, were small when compared to global emissions needed to have any effect on climate change mitigation. However, this analysis is important to assess the extent to which GEF supported approaches work and to determine if there is a potential for wider application. To determine the level of GHG emissions, assumptions made by the project proponents on the benefit stream of the technologies promoted by a given project including the estimated duration of the benefit stream were recorded; and, the expected GHG emission reduction – including the changes in the measurements of the underlying indicators for calculating emission reduction – expected at the project start, realized during project implementation period, and revised estimates at the point the evaluation was conducted, were noted. This information, along with information provided in the terminal evaluations of the completed projects and that was gathered through interviews and documents accessed during field verification, formed a basis to prepare revised estimates of the GHG emission benefits.

The evaluation found that although most of the GEF projects covered by the evaluation tracked direct and indirect emission reduction and/or avoidance, in most instances regular monitoring of the emissions related benefits stopped at project completion. Moreover, the information on the indicators specified in the project M&E plan was not being gathered and analyzed regularly. Methodological approaches used by different project proponents to track emission reduction and/or avoidance were often inconsistent. Table 9.3 lists the type of errors that were encountered. To address these errors, the evaluation team recalculated the emission reduction benefits using the available information. Although results for individual projects differed from what had been calculated by the project proponents, the overall figure at the portfolio level were similar.

Table 9.3 Types of Errors encountered in GHG Calculations among projects covered by the evaluation

The evaluation found that of 18 projects, 16 resulted in direct GHG emission reduction. Aggregate direct emission reduction is estimated to be about 6 million tons of CO2 equivalent per year. However, of the 16 projects that were assessed to have had direct GHG emission reduction impact, for two projects the extent of GHG emission reduction could not be ascertained. Of the 16, for three projects actual GHG emission reduction exceeded expectations at the start of the project. For the remainder actual achievement was lower than the expectations. Among the projects, the China TVE II (GEF 622) alone contributed a third of the direct emission reductions achieved by the 18 projects covered by the evaluation. It was found that the key determinants of the scale of the direct GHG emission reduction achieved included market size, maturity of the promoted technology, and the emission factor for the country, which were positively correlated to the scale of direct emission reduction achieved. Projects that tend to address the prevalent market barriers more comprehensively tended to achieve emission reduction at a higher scale. Overly optimistic projection of the expected benefits – which probably also makes project more attractive during appraisal – was also a reason why several projects had lower than expected direct emission reduction benefits.

Of the 18 projects, 14 led to indirect GHG emission reduction. Of these, in 11 instances quantitative assessment of the indirect GHG emission reduction was possible – for the other three projects, the information required to carry out this analysis was not available. Overall, the indirect emission reduction was assessed to be ten times more than direct reductions.

Some of the projects provided relevant actors an opportunity to learn about new technologies and approaches, whereas others were geared towards providing support to the locally nurtured initiatives on climate change mitigation. Of the 14 projects where indirect GHG emission reductions were reported, 9 projects were part of the ongoing process within the country for addressing barriers related to the targeted market/technology. In five instances the GEF supported project supported the first application of the promoted technology in the country. Project design and delineation of project boundary were assessed to be a major factor on whether GHG are counted as direct or indirect result of the project.

7 Assessing Market Change

More important than the carbon emission reductions is the extent to which projects contributed to change that in the long term will result in the needed market, technological and behavioral transformations. To bring about these changes, the projects covered through the evaluation addressed barriers related to different sectors and markets. These projects promoted technologies and removal of barriers to markets/technologies on wind energy, biomass energy, methane, hydro power, solar energy, industrial efficiency, efficient lighting, building efficiency, and, transportation. The instruments developed for the evaluation tracked barriers and changes in these markets/technologies in four spheres: Consumers/users; supply chain and infrastructure; financing; and, policy environment . Based on applicability, market barriers considered in the analysis for a sphere included: information gap on promoted technology or approaches, lack of interest or motivation to adopt, lack of relevant expertise, lack of access to relevant mitigation technologies, lack of cost effectiveness, and lack of a viable model. The instrument developed for analysis also captured the intensity with which the given project targeted each of the barriers prevalent in the given market, the specific activities that the GEF implemented to target each barrier, efforts by other actors in addressing the given barrier, and the extent to which the change evident in the status of the barrier could be attributed to the given project. Information on these indicators was gathered through desk review of available information, field verification, interviews and information from independent sources.

Of the 18 projects covered through the evaluation, for 14 projects market change was observed. The observed changes in the targeted markets may be classified into four categories: adoption of higher quality product/technology in the market (8 projects); reduction of production/technology cost (7 projects); availability of more and/or better suppliers (12 projects); and greater demand for promoted product/technology (7 projects). Generally, achievement of improvement in availability and quality of suppliers and improvements in products were linked to each other and often due to a requirement to meet a predetermined quality standard or to achieve a certification.

8 Establishing Causality and Accounting for Alternative Hypothesis

Determining the causal variables was more demanding than determining the observed change. The log frames and other logic models that articulate a project’s theory of change identify its expected effects. The theories also sketch the expected pathways through which the project outputs and outcomes would lead to its expected long term impacts. While a project’s theory of change provides a useful tool to understand its rationale, there are two main limitations in relying on it as a basis for an impact evaluation. Firstly, although the necessary conditions predicted by a project’s theory of change may have been met the observed change may have been due to factors that were independent of the project. Second, there may be some uncertainty involved in how and when the intended impacts manifest, particularly in the cases in which important causal relationships may be non-linear in nature. Focusing entirely on the project’s theory of change has a risk of overstating GEF’s role in effecting the observed change or may lead to neglect of conditions that are impeding future progress. Furthermore, exclusive focus on the causal links proposed by a project’s theory of change can function as blinders that constraints an evaluation in recording and assessing the unintended impacts of the project (Garcia and Zazueta 2015). Therefore, in addition to taking into account the given project’s theory of change, the evaluation also addressed other factors that may have a causal relationship with the observed change but were independent of the GEF project. The evaluation also searched for alternative explanations that could explain the observed change and assessed their merit in contributing to the observed change vis-à-vis a given GEF project. During the visits to the field the evaluation team gathered information on this issue from different stakeholders such as project implementers, beneficiaries, other agencies that were not involved in project implementation but were familiar with the project, and government officials.

Despite limitations of the theory of change approach, for the most part it remains a useful basis for tracking a given project’s provable impact pathways . As the general framework of GEF’s theory of change suggests, for any observed change to be attributable to GEF project, the behaviors promoted by the project should be adopted by the targeted actors within a market. This broader adoption in turn creates a basis to assess the progress towards the expected long term transformations. To assess this progress, the evaluation tracked the intensity, the scale, and the processes through which it was taking place.

Of the 18 projects in 17 instances there was evidence that broader adoption was taking place through one or more of the following processes: sustaining project supported activities; mainstreaming; replication; scaling up; and, market change (Table 9.4). For each of these processes, the manner in which it was happening and the extent to which it was linked with the GEF project was determined. In 14 cases the evaluation was able to establish causal links between the project activities and the progress made. This involved linking specific activities or components of the GEF supported projects with the intended observed outcomes based on the information gathered through terminal evaluations and interviews conducted and documents accessed during field verifications. The next stage was to also assess the effects of the other actors and factors that could account for the observed results. Based on the qualitative assessment of the information gathered, in ten cases the evaluation was able to discard rival theories and establish primacy of the GEF supported project in effecting the observed change. For example, in India the technologies and approaches promoted by projects on Photo Voltaic Systems (PVMTI GEF 112) and Hilly Hydel (GEF 386) were scaled up at the national level with significant link established with the underlying project. For four other projects (India Alternative Energy, PVMTI and Energy Efficiency; Mexico Agriculture) although causal links were established for some of the changes, these could not be established for others and rival theories were also difficult to discard.

Table 9.4 Progress to impact and causal links to GEF projects

In the remaining four cases, for two projects no link or very tenuous link could be established between the GEF project and the changes observed. In Russia Boilers project (GEF ID 292) although there was some evidence of scale-up it was not linked to the activities supported by the project. Similarly in the case of the Wind project in Mexico, at project end important regulatory changes had be undertaken by the Mexican government, while the project design included such reforms as an important intended outcome, the evaluation found that other factors accounted for such reforms, and that the contributions to these changes by GEF supported activities were marginal. In the two remaining cases (china FCB I and II) the evaluation did not have enough evidence to assess the causal links of the project with the changes observed.

9 Assessing What Would Have Happened If GEF Support Had Not Taken Place

The work presented so far in this paper assesses the extent to which market change took place, whether there is a causal link between GEF support and the changes and whether there are alternative explanations for the observed changes. However, projects take place through partnerships which include governments, other donors and civil society organizations. While linked to GEF support, changes can also be a result of other factors and conditions, some which might not be readily apparent. To assess GEF contribution more fully, understanding GEF role within the change process is also important. Thus, the evaluation also needs to assess the extent to which the given project (or a comparable activity) would have taken place without GEF support. For each of the 18 projects the evaluation carried out an inquiry to assess the extent to which other factors (projects, activities, events) could have bring about or contributed to the observed changes. This was done through interviews with key informants, including people whom had been part of the process and other third parties in the countries, as well as through analysis of publications, gray literature and other relevant reports. The findings are summarized in Table 9.5.

Table 9.5 Added value of GEF financing

The analysis shows that of the 18 projects, 8 were assessed to be very unlikely or not likely to have taken place without GEF support and 9 projects were very likely or likely to have taken place without GEF support. However, the likelihood that a project would have taken place without GEF support does not mean that the support did not bring additional value.

Of the nine projects that were very likely or likely to have taken place without GEF support, in seven instances the GEF support was assessed to have accelerated the process of the project (or comparable activity) being implemented. In two instances it was assessed that the GEF support to the project allowed its design and implementation to be of a higher quality than would have otherwise been possible. Overall it was assessed that GEF support did add value in addressing concerns related to eight of these targeted markets. In one of these nine cases the evaluation was not able to determine if GEF support added value. In summary in 16 of the cases GEF support contributed to the desired change.

10 The Critical Role of Indicators in Impact Evaluation

Determining the extent to which expected long term impacts have taken place requires that relevant impact indicators are being tracked. Often there are severe gaps in base line information (Tokle and Uitto 2009), the results of the project may not be monitored consistently – especially after project completion – and/or the methods used to track changes may not be appropriate. The evaluation found that projects were generally tracking too many indicators and often not tracking them well. Even though projections of GHG emission reduction benefits extended over the post project completion period, often projects made no provisions for gathering data after project completion. Furthermore there are inconsistencies in calculating GHG emission reduction benefits and calculations of these varied considerably across projects. A major recommendation of the evaluation was that GEF should improve its methodology of GHG emission reduction calculations, which was accepted by the GEF Council in its November 2013 meeting. As a follow up to the Council decision, GEF Secretariat has developed new (and updated) methodologies for GHG emission reduction calculation.

11 Conclusion

The project’s theory of change are an important resource to track the extent that projects realized their intended results. Nevertheless there is a need to go beyond a project’s theory of change for a fuller understanding of “why” and “how” the observed change took place. While the theory of change is useful, it by itself is not sufficient because other factors may turn out to be more influential in effecting the observed change. Under these conditions dependence on the project’s theory of change runs the risk of functioning as a blinder hindering assessment of unexpected factors and making it difficult to get a handle over the unintended consequence and alternative explanations to the results realized. This risk can be mitigated going beyond a mere assessment the extent to which intended results of projects were achieved and the causal links between project activities and results and carry out an analysis of the forms in which the project intervention interacts with other components of the system. In other words to fully assess the contributions of an intervention vis-à-vis other interventions and other factors, the evaluator must assess project interventions as part of the system that is targeted. This requires the evaluator to develop a good understanding of the system that the project has targeted, including the system boundaries, components, interactions among components and the unexpected changed resulting from these interactions.