The evaluation assessed the Climate Change Sub-programme in three areas of focus, corresponding to three distinct but strongly related clusters of evaluation questions (see Table 6.2). First, the evaluation assessed the strategic relevance and appropriateness of sub-programme objectives and strategy. It analysed the clarity and coherence of the CCSP’s vision, objectives and intervention strategy, within the changing global, regional and national context, and the evolving overall mandate and comparative advantages of UNEP
. Second, the evaluation assessed the overall performance of the CCSP in terms of effectiveness (i.e. achievement of outcomes), sustainability
, up-scaling and catalytic effects. It also reviewed the potential or likelihood that outcomes were leading towards impact
. Which outcomes were assessed, was determined by a reconstruction of the sub-programme’s Theory of Change
(see below). Third, the evaluation examined the factors affecting performance in more detail: intervention design issues, organizational aspects, partnerships etc. that affected the overall performance of the sub-programme.
These areas of focus were not addressed in sequence but simultaneously as they are strongly linked to each other and dynamic as shown in Fig. 6.1. For instance, elements of strategic relevance of UNEP’s involvement in Climate Change determine the scope and scale of the sub-programme and shape the kinds of products, services and delivery mechanisms are used to reach core objectives. Decisions surrounding strategic relevance of the CCSP thereby also influence the administrative, management and implementation structure, and other factors that affect performance. Sub-programme performance, in turn, affects funding availability and programme orientation. Progress made on expected accomplishments and impact
also changes the priority needs of countries and other stakeholders, justifying strategic adjustments to sub-programme objectives and strategies.
As illustrated in Fig. 6.2, the evaluation was conducted at five units of analysis. The two upper units are UNEP
corporate and the Sub-programme itself. Considering the vast number and high variety of interventions, and highly diverse institutional arrangements and other factors affecting performance under the CCSP, neither UNEP or the sub-programme as a whole were the most practical and straightforward level at which to conduct analysis. They were also not the best level at which to attribute performance and uncover lessons learned.
Therefore, three lower units of analysis were used, which, combined, would provide sufficient information and analysis for the assessment of the sub-programme as a whole. The main unit of analysis was the sub-programme component (adaptation
etc.). At that level, performance could be most easily attributed to the line managers and partners delivering against the Expected Accomplishments of each component. The components were also the best units of analysis for learning, as they were usually better defined and delimited, and less complex than the sub-programme as a whole, but still provided the opportunity to see linkages between interventions either within or between main areas of intervention.
Another useful aggregated level of analysis was the country, where it was possible to obtain insights on the linkages (complementarities and synergies) between projects within a component, between the different components of the CCSP, and also between the CCSP and other sub-programmes within one, confined geographical and political space. The evaluation team visited six countries selected on the basis of geographical spread (spanning the regions of Latin America, Africa, Europe, West Asia, and Asia and the Pacific), presence of the sample projects (see next paragraph) and diversity of UNEP support on climate change in the country. A country case study was prepared for each visited country.
The lowest unit of analysis was the individual project. This was the most appropriate level to unveil factors affecting performance, but as the resources for the evaluation were limited only a sample of projects could be looked at in sufficient depth. The evaluation team prepared rapid reviews of 19 projects – about one third of the entire portfolio. Projects were selected on the basis of four criteria: thematic area (adaptation, mitigation
or REDD), project size (based on estimated cost), project scope (global, regional or national) and maturity.
The evaluation made use of a
Theory of Change
(ToC) approach to address several evaluation questions. A ToC depicts the logical sequence of desired changes (also called “causal pathways” or “results chains”) to which an intervention, programme, strategy etc. is expected to contribute. It shows the causal linkages between changes at different results levels (outputs, outcomes, intermediate states and impact
), and the actors and factors influencing those changes. Initially inspired by guidance provided by the
Global Environment FundFootnote 6 the UNEP
Evaluation Office has been systematically using a ToC approach in project and sub-programme evaluations
The ToC for each component of the CCSP, and then for the CCSP as a whole, was reconstructed based on a review of strategic documents and UNEP
staff interviews, and using best practice in determining correct results levels. Figure 6.3 presents the overall reconstructed ToC for the CCSP. The reconstructed ToC helped identify the expected outcomes of UNEP’s work on Climate Change and the intermediary changes between outcomes and desired impact. Thus, it allowed to cluster outputs and define summary direct outcome statements cutting across components, which would prove very useful to frame data collection and synthesize findings on sub-programme effectiveness.
The reconstruction of the ToC also helped to determine the key external factors affecting the achievement of outcomes, intermediary states and impact
, namely the drivers that UNEP
could influence through awareness raising, partnerships etc., and the assumptions that were outside UNEP’s control. As these were key determinants of the likelihood of impact
, upscaling and sustainability
of the sub-programme, it was important to identify them early on so that adequate information on their status could be collected in the course of the data collection phase.
The reconstructed ToC was also used to assess the internal logic and coherence of the formal results framework of the sub-programme. Therefore, the formal results framework comprised of the Sub-programme objective, Expected Accomplishments and Programme of Work Outputs was compared with the reconstructed ToC, and differences between the two were pointed out. For instance, in the formal results framework the results levels at which Expected Accomplishments and Programme of Work Outputs had been set were inconsistent between and within components, some cause-to-effect relationships were either non-existent or had been overlooked, and several key drivers and assumptions had been neglected.
As explained above, attribution of large-scale, global changes to
UNEP’s work was difficult due to the largely normative nature of UNEP’s work. Casual pathways from UNEP outputs to impact
on the environment and human living conditions tended to be very long, with many external factors coming into play all along the causal pathways. The reconstructed ToC was used to assess the likelihood of impact by considering four distinct elements:
UNEP’s effectiveness in achieving its expected direct outcomes. This included verification of progress on output delivery and, most importantly, of the extent to which UNEP outputs led to increased stakeholder capacity, for instance: enhanced access to information and technological know-how, enabling policies and regulatory frameworks, or increased access to climate change finance.
The validity of the ToC. The purpose was to prove the causal connection between UNEP
direct outcomes and results higher-up the causal pathways. This was done by applying logic, through interviews with key stakeholders, and through analysis of evaluative evidence
of progress towards impact at the country or lower geographical levels.
The presence of drivers and validity of assumptions. The evaluation had to collect adequate evidence, mostly through desk review and key informant interviews, to verify the presence of an adequate enabling environment
in supported countries.
Early signs of large-scale progress on medium-term outcomes, intermediate states and impact. In itself this was not evidence of
UNEP’s contribution to higher-level changes, but was still necessary to inform stakeholders about global trends. Also, if UNEP’s contribution to direct outcomes had been established, the ToC was very likely to be valid, and the required drivers were present and assumptions were valid, then the likelihood of UNEP’s contribution to impact was very high even though it remained unquantifiable.