Pathway to Impact: Supporting and Evaluating Enabling Environments for Research for Development
The chapter presents a research for development program’s shift from a Logframe Approach to an outcome and results-based management oriented Monitoring, Evaluation and Learning (MEL) system. The CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS) is designing an impact pathway-based MEL system that combines classic indicators of process in research with innovative indicators of change. We have developed a methodology for evaluating with our stakeholders factors that enable or inhibit progress towards behavioral outcomes in our sites and regions. Our impact pathways represent our best understanding of how engagement can bridge the gap between research outputs and outcomes in development. Our strategies for enabling change include a strong emphasis on partnerships, social learning, gender mainstreaming, capacity building, innovative communication and MEL that focuses on progress towards outcomes.
It presents the approach to theory of change, impact pathways and results-based management monitoring, evaluation and learning system. Our results highlight the importance of engaging users of our research in the development of Impact Pathways and continuously throughout the life of the program. Partnerships with diverse actors such as the private sector and policy makers is key to achieving change, like the attention to factors such as social learning, capacity building, networking and institutional change when generating evidence on climate smart technologies and practices. We conclude with insights on how the theory of change process in CGIAR can be used to achieve impacts that balance the drive to generate new knowledge in agricultural research with the priorities and urgency of the users and beneficiaries of these research results.
Evaluating the contribution of agricultural research to development has always been a challenge. Research alone does not lead to impact, but research does generate knowledge which actors, including development partners, can put into use to generate development outcomes. In CCAFS we are finding that a theory of change approach to research program design, implementation and evaluation is helping us bridge the gap between knowledge generation and development outcomes.
KeywordsResults-based management Impact pathway Monitoring Learning and evaluation Theory of change
Global poverty has been reduced over the past 25 years. The developing regions overall saw a 42 % reduction in the prevalence of undernourished people between 1990–1992 and 2012–2014 (FAO 2015). Despite major investments of the international community in reducing poverty and food insecurity, an estimated 805 million people were chronically undernourished in 2012–2014 (FAO 2015), almost all of whom live in developing countries. There are large regional differences in terms of the progress that has been made against poverty and hunger: in South Asia it has been limited, and in sub-Saharan Africa it has actually gone backwards since 1990–1992 (FAO 2015). There is much to be done to reach the targets for 2030 as articulated in the Sustainable Development Goals (UN 2015a). Research for development (R4D) has played a significant role in reducing food insecurity over the last decades and will continue to play a critical role moving forward.
R4D is a set of applied research approaches that aim to directly contribute towards achieving international development targets through innovation. In this, there is a wide range of understanding of the concept. In this chapter we focus on agricultural research for development as operationalized by CGIAR. The underlying assumption is that research within R4D is done with broader development outcomes in mind, e.g. demand-led prioritization of research, participatory and action research and stakeholder involvement (Harrington and Fisher 2014).
Agricultural R4D has a long history. CGIAR was founded in 1971 as a response to address global hunger in India, Pakistan and other South Asian countries. The adoption of improved agricultural practices and technologies developed by CGIAR, including high-yielding rice and wheat varieties, fertilizers, pesticides and irrigation, has proven to be a powerful instrument of the Green Revolution in fighting hunger in that part of the world. CGIAR currently comprises 15 international agricultural research centers that collectively aim to increase agricultural productivity, reduce poverty and enhance environmental sustainability. Renkow and Byerlee (2010) and Raitzer and Kelly (2008) reviewed evidence of impact across the centers and concluded that there have been strong positive impacts of CGIAR research relative to investment. Another way to describe CGIAR’s success is to show a world without it (Evenson and Gollin 2003): focusing on the impact of crop improvement research from 1965 to 1998 provided counterfactual scenarios of the global food system: developing countries would produce 7–8 % less food; their cultivated area would be 11–13 million hectares greater at the expense of primary forests and other fragile environments; and 13–15 million more children would be malnourished.
However, agricultural R4D has not realized its full potential: the world food system continues to face challenges of persistent food insecurity and rural poverty in many parts of the developing world. The adoption of improved agricultural technologies and practices by farmers has often been less than expected, when considering their demonstrated benefits, primarily due to a supply-led approach to their development and dissemination, with limited attention paid to context specificity, to farmer’s priorities beyond increased agricultural productivity, and to the socio-economic, political and institutional contexts within which smallholder farmers operate. Many studies have shown that ‘scientifically proven’ technologies alone are not the only key to get to impact. If a technology gets adopted or adapted, it is often not so much because of its quality and suitability but because of good social management and implementation processes (Hartmann and Linn 2008; Pachico and Fujisaka 2004). New challenges like population growth and climate change are adding complexity to the mission of CGIAR and other R4D organizations.
Within this context, this chapter aims to describe the journey towards a new R4D approach based on theory of change (TOC) and impact pathway thinking for program implementation, monitoring, learning and evaluation (MEL). It illustrates lessons of broad applicability regarding results-based management (RBM) and adaptive management approach to tackling complex development challenges through R4D. The key messages are summarized in Box 4.1. The chapter starts by describing a case study within CGIAR, where TOC combined with IPs and learning-based approaches were employed to build an outcome-focused RBM approach to R4D. It then analyses the main findings, focusing on program design and systems for planning and reporting, as well as a MEL framework within an impact pathways approach. The chapter concludes with lessons for required institutional change as well as for MEL practitioners, researchers and policy makers.
Box 4.1: Key Messages
Overall, RBM can offer many elements and approaches to help with strategic program design, but it needs to be adapted to the specific context of a program, institution, or organization. It requires some enabling conditions and an environment to support an outcome-focused R4D program.
Buy-in from the top, healthy balance between given structures but allowing for creativity in designing processes.
Investing in facilitation and process – and bringing the three elements of MEL together is key and requires resources (time and money).
Flexible condition, rigid system to allow adaptive management and learning (liberating structures).
The ‘three thirds’ principle: one third partnerships, ownership and buy-in externally from partners, one third capacity enhancement at all levels internally and externally, and one third cutting-edge science.
System support – building an online platform and working towards a one-stop-shop (database).
CGIAR is a global agricultural research partnership for a food secure future. Its science is carried out by 15 research centers with 10,000 scientists working in 96 countries and a host of partners in national and regional research institutes, civil society organizations, academia, development organizations, and the private sector (CGIAR 2015a). Its work contributes to the global effort to tackle poverty, hunger and major nutritional imbalances, and environmental degradation. The 15 CGIAR Centers have different foci and operate semi-autonomously in pursuing their specific research agendas, ranging from promoting the productivity of specific crops, livestock, and fish commodities to production systems in specific agro-ecologies and research on policies natural resource management (Raitzer and Kelly 2008).
The challenges of demonstrating wide-reaching impact through R4D are compounded by a rapidly growing human population, climate change and other complexities of our time. The human population has almost doubled from 3.8 billion in 1971 to 7.3 billion in 2014 (UN 2015b). With an expected extra two to three billion people to feed over the next 40 years, this will require targeted research efforts to achieve not just growing 70 % more food but making 70 % more food available on the plate to keep up with rapidly rising demand (WWAP 2012). Climate change is already affecting agriculture in many developing countries, and the effects will become increasingly challenging in the future. Higher temperatures, shifting disease and pest pressures, and more frequent and severe droughts and flooding will affect agricultural production and place increasing pressure on water and other natural resources (IPCC 2013). Climate change impacts are increasing the vulnerabilities of populations that are already struggling with food insecurity and poverty, even in the relatively conservative scenario of a global 2-degree temperature rise (Thornton et al. 2014a).
The increasing complexity of the challenges, particularly with regard to their impacts on poor and vulnerable populations, requires a rethinking of our approach to R4D. CGIAR has taken on this challenge by broadening its portfolio of major new initiatives for strategic research. A first round of some half-a-dozen ‘Challenge Programs’ were mandated to develop new R4D models over a period of up to 10 years, starting in 2002 (CGIAR 2015b). Box 4.2 describes one example of these programs, focusing on water and food.
Box 4.2: Challenge Program on Water and Food
The CGIAR Challenge Program on Water and Food (CPWF) piloted new ways of increasing the resilience of social and ecological systems through better water management for food production. From 2002 to 2013, the program supported more than 120 research projects in ten of the world’s largest river basins (Hall et al. 2014; Harrington and Fisher 2014). The program early on developed IPs and theories of change for its R4D river basin programs. From a Monitoring and Evaluation (M&E) perspective this included results-based and adaptive management as well as learning-oriented approaches. The insights and knowledge gained from CPWF’s 12 years of work are being integrated into another CGIAR Research Program on Water, Land and Ecosystems.
In a second round, from 2010 onwards, 16 CGIAR Research Programs (CRPs) were set up in a 5-year first phase (CGIAR 2015c). The major reorientation of the R4D portfolio was in the move from an output focus to an outcome focus. Success was now to be measured in terms of the CRP’s contribution to behavioral changes, manifested in changes in knowledge, attitudes and skills and practices of a wide set of non-research next users, including development practitioners, farmers and policy makers.
Through approaches such as results-based management, theories of change and impact pathways, the term outcome came into focus. Organizations such as the International Development Research Center (IDRC) were early developers of M&E tools to capture and measure outcomes through their ‘Outcome Mapping’ methodology (Earl et al. 2001). Within CGIAR, ‘Participatory Impact Pathways Analysis’ (PIPA) (Douthwaite et al. 2003, 2007) was developed under the CPWF to unpack processes and mechanisms in the realm of outcomes.
Towards the end of the first phase, 4 of the 16 CGIAR research programs were tasked to develop a comprehensive, suitable and lean results-based management approach for research for development, initially for a period of 12 months. The following section describes how the CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS) developed and implemented its RBM trial and highlights the main lessons learnt.
Box 4.3: About the CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS)1
The CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS) is a strategic partnership of CGIAR and Future Earth led by the International Center for Tropical Agriculture (CIAT). CCAFS brings together the world’s best researchers in agricultural science, development research, climate science and Earth System science to identify and address the most important interactions, synergies and trade-offs between climate change, agriculture and food security. For more information see ccafs.cgiar.org
As an R4D program working on addressing the complexities of climate change, agriculture and food security, the main goal of CCAFS is to improve the livelihoods of the most vulnerable and poor people in target countries in Asia, Africa and Latin America. While CCAFS is at the cutting edge of generating demand-driven science products, it also plays a bridging role: to transform credible scientific evidence and results into development outcomes. A key underlying principle the CCAFS management team subscribes to is the “Three-Thirds Principle”: one third of effort engaging with partners to decide what needs to be done and how; one third on doing the actual research, often in partnership; and one third on sharing results in appropriate formats and strengthening capacity of next users to utilize the research to achieve outcomes and impact. Deep engagement with stakeholders with the support from a wide network of partners to get science-based solutions to practical problems is fundamental to the CCAFS approach (CCAFS 2014).
CCAFS has been one of the programs at the forefront of testing and paving the way for moving a multi-million dollar R4D program from a logframe approach to an outcome-focused approach. Additionally, it has put in place a MEL mechanism for programmatic RBM, including elements of adaptive management.
4.4 Getting to the Right Mix
CCAFS was clearly committed to an outcome-focused R4D program from its inception. It became increasingly clear that a logframe approach (LFA) was not the most ideal way of doing R4D differently. In particular, when moving from a Challenge Program to a CGIAR Research Program with increasing complexities of partnerships, engagement and CGIAR integration, the limits of a logframe became apparent (as described in section 4.5.1). The program’s vision of contributing towards development outcomes increasingly required a different approach: one that acknowledged the importance of stakeholder engagement and capacity development. As a result, monitoring the annual contribution of CCAFS and its partners towards development outcomes becomes increasingly complex.
While a wide range of MEL approaches and methodologies with an outcome focus exist (e.g. PIPA, Outcome Mapping, Outcome Harvesting), none provides a blue-print solution that can just be rolled out. The approaches were designed to address the particular needs of a specific program or organization. Thus, to adapt these approaches to a new program, it is key to select the right mix of elements creating a conceptual framework in support of the program’s specific TOC and MEL requirements. Springer-Heinze et al. (2003) advocate a holistic approach to impact evaluation and program monitoring with quantitative and qualitative elements, based on an impact pathway that can accommodate different stakeholder views, allows for reflection, and emphasizes capacity of research organizations. Cummings (1997) compares RBM, LFA and Project Lifecycle Management and would welcome more discussions and learning among the different approaches. According to Bazeley (2004), ‘The “mixing” may be nothing more than a side-by-side or sequential use of different methods, or it may be that different methods are being fully integrated in a single analysis’. Applying a mix of methodologies in a programmatic MEL framework raises certain terminological, definitional, paradigmatic and methodological issues, including over-interpretation of numbers, single dimensionality, and disregarding ‘outliers’ from the analysis (Bazeley 2004). However, mixed methods also provide opportunities to address the respective shortcomings of any single method as applied in practice.
CCAFS in its early years worked with various logframe elements in planning and reporting. Limitations of the more traditional LFA resulted in experimentation with elements of TOC that were integrated within the logframe, in order to more effectively capture the complexity of activities, partners and anticipated outcomes of the program. The limitations of this single method approach resulted in CCAFS deciding to operationalize a modular MEL approach, described in the next section. The findings and analysis section explains CCAFS’s approach over time. With the limitations in mind CCAFS is aiming for a more holistic approach in line with Springer-Heinze et al. (2003).
4.5 Findings and Analysis
4.5.1 Moving Away from a Logframe
Milestones output targets
Means of verification
Objective 4.1 Explore and jointly apply approaches and methods that enhance knowledge to action linkages with a wide range of partners at local, regional and global levels
Outcome 4.1: Appropriate adaptation and mitigation strategies mainstreamed into national policies in at least 20 countries, in the development plans of at least five economic areas (e.g. ECOWAS, EAC, South Asia) covering each of the target regions, and in the key global processes related to food security and climate change
Output 4.1.1 For each region, coherent and plausible futures scenarios to 2030 and looking out to 2050 that examine potential development outcomes under a changing climate and assumptions of differing pathways of economic development; developed for the first time in a participative manner with a diverse team of regional stakeholders
Milestone 184.108.40.206 Capacity built among three regional teams of diverse stakeholders trained in scenarios approaches and engaging with policymakers in their countries/regions and in global climate change processes and with the Earth System Science Partnership community; Methodological briefs, papers
Regional scenarios partners actively participating in regional food security debates and global climate change processes. Number of partners using/citing scenarios; number of regional partners trained in scenarios participating in regional food security debates and global climate change processes
CCAFS and partner websites and reports; newspaper and other media reports
Partners remain engaged and help communicate scenario research results widely and to inform key decision makers
Regional Agricultural Research Organizations; Regional policy organizations; International NGOs; Regional NGOs; Private Sector; Farmers Organizations; Regional Meteorological Organizations;
R4D programming over the last few decades has commonly been based on a logframe approach (LFA). The LFA was initially developed for United States Agency for International Development (USAID) in 1970 and adopted by a range of international organizations, including agricultural R4D (Schubert et al. 1991). The approach has been widely required by funding agencies and has thus been used for project planning, management and evaluation and adheres to a relatively rigid framework. It tends to prescribe a hierarchy of objectives converging on a single goal, a set of measurable and time-bound indicators of achievement, checkable sources of information, and assumptions of other impinging factors (Gasper 2000). In the R4D context, the underlying assumption is that development agencies, communication units, ministry staff and other people who could use the findings are able to source the scientific evidence, understand it, know how to implement and apply it, and convey this to people who they think need them. In this case both research and development have their mandates, responsibilities and clearly defined boundaries.
While this has been a useful approach for several decades, it is debatable whether it is entirely suitable for ensuring the use of research results and their translation into outcomes. Crawford and Bryce (2003) note that although much of the literature promotes the use of the LFA for the purposes of M&E, it has proven inadequate and evidence for its usefulness is lacking. The LFA does not pay enough attention to involving key stakeholders in a joint process, emphasizing the stakeholder networks needed to achieve impact, providing managers with the information needed both to learn and to report to their funding agencies, and establishing a research framework to examine the critical processes of change that projects seek to initiate and sustain (Douthwaite et al. 2008).
CCAFS has gone through several iterations of the logframe that was employed for planning and reporting (CCAFS 2015c). In 2010, a limited version was used (CCAFS 2010) while more elements were added in the following years. Planning and reporting elements were pre-determined to some extent by requirements from CGIAR, though for internal purposes additional elements were added in response to the limitations that were identified from year to year.
4.6 Testing the Waters with Theory of Change and Results-Based Management in CCAFS
In addition to the use of logframe elements within the CCAFS planning and reporting system, at program design stage CCAFS also explicitly included a research theme entitled ‘Knowledge to Action’ in its portfolio (Jost et al. 2014a). The team was experimenting with strategies of getting from research outputs to development outcomes. This theme was tasked with research, not with creating an operational mechanism for CCAFS per se. It was only in year 3, when CCAFS started working in two new target regions, that opportunities presented themselves to trial a TOC approach within this new component of the R4D portfolio (Jost and Sebastian 2014; Jost et al. 2014b). Very early on it became clear that a new way of thinking needs to take effect for the whole program in order to plan for and capture outcomes more effectively and include engagement and capacity enhancement as key strategic elements (Thornton et al. 2014b). As a consequence, the ‘Knowledge into Action’ theme was then mainstreamed into the whole CCAFS program with its four research and five regional programs.
The opportunity to trial an alternative approach of RBM was taken up enthusiastically (Thornton et al. 2014b; Jost et al. 2014c). Theory of change, impact pathways and results-based management offer practical mechanisms to potentially enhance program design and its monitoring, learning and evaluation and help CCAFS to create an operational program management framework that is better suited to deal with the complexities at hand.
Closely linked to this, CCAFS and partners also started experimenting with learning-based approaches within R4D recognizing the need to include mechanisms that challenge business as usual and support institutional learning and innovation to ensure that research contributes to development outcomes, see Box 4.4.
Box 4.4: Why Learning
Learning-based approaches are useful in supporting transformational change across institutions and stakeholders. One such approach is social learning. We understand social learning to be a facilitated process of planning, implementing, reflecting, and adapting. It can effectively foster an institutional learning culture and pave the way for climate resilient food systems and sustainable development outcomes. For more information see Kristjanson et al. (2014), Gonsalves (2013), and Harvey et al. (2013) and ccafs.cgiar.org/social-learning-and-climate-change.
4.7 Trialing Results-Based Management in CCAFS
CCAFS decided to trial a RBM approach for one of its research themes, Policies and institutions for climate-resilient food systems, fast-tracking the extension phase for this particular theme. A new portfolio of six multiannual regional projects was set up and these were each tasked from the beginning with designing their project using a TOC approach (Schuetz et al. 2014a). TOC are key elements of CCAFS’ approach to RBM.
RBM builds on the same logical causal chain and is more explicit about output-use. Within R4D output-use refers to strategies that directly engage the next-users in the research process, e.g. through stakeholder platforms and user-oriented communication products. At the turn of the century, many development and funding agencies, including USAID, Department for International Development, IDRC, UNDP and the World Bank, reformed their performance management systems and M&E approaches towards a RBM approach (Binnendijk 2000; Bester 2012; Mayne 2007a, b). At the time, these organizations faced a number of common challenges: how to establish an effective performance measurement system, how to deal with analytical issues of attributing impacts and aggregating results, how to ensure a distinct yet complementary role for evaluation, and how to establish organizational incentives and processes that will stimulate the use of performance information in management decision-making (Binnendijk 2000). These early experiences with RBM have informed further development of the approach.
Early on, IDRC has attempted to unpack the in-between area of outcomes and were at the forefront of developing means to measure outcomes through the Outcome Mapping methodology (Earl et al. 2001). To show that R4D contributes to the desired behavioral changes, i.e. outcomes, that enable long-term positive impacts is a particular challenge, as it requires more qualitative monitoring than dealing with quantitative means of measuring alone (Young and Mendizabal 2009; Springer-Heinze et al. 2003). Evaluators generally agree that it is good practice to first formalize a project’s TOC, and then monitor and evaluate the project against this ‘logic model’ (e.g. Chen 2005). The TOC is a mental model made explicit by involving as many people as possible in its design. Key principles of the Participatory Impact Pathways Analysis also include reflecting on these models, regularly validating the assumptions that were made, and adjusting program management accordingly (Douthwaite et al. 2013).
Comparison between research, research for development and international development
Criteria/elements for RBM, TOC, IPs
Research centers with a key scientific focus
Interdisciplinary research programs around a development challenge and partnership approach
NGOs, development aid agencies, UN agencies
Mandate and performance focus
Responsibility for achieving impact
Provision of solid science and technologies
Type of communication, knowledge management
More traditional/corporate communications
Communications for development, engagement
Type of partners
International, regional, national research partners
International and national research partners, and development agencies
Local/ district implementing agencies, central/national governments
Focused on quantitative measuring of publications, quality of journals, citations,
Forward looking external evaluation, learning-based approach, contribution (not attribution), balanced quantitative with qualitative measures
Focused on traditional impact assessment, quantitative measuring including baselines
Timeframe for achieving outcomes/impact
Often not considered
Achieving outcomes at scale within 5 years and impact within 15 years
Long term impacts 10–20 years at large scale
Languages of products
Both international and locally appropriate languages
Both international and locally appropriate languages
4.8 Building Capacity and Learning Within the Program for Theory of Change Approach
The RBM trial project teams were thrown in at the deep end. Used to a more traditional LFA, they were tasked with shifting to a TOC and learning-based approach for planning their projects within the trial. It was quickly apparent that capacity to plan projects using this new approach had to be built within CCAFS (and wider CGIAR).
Using TOC approaches within R4D requires the strengthening of capacities of scientists to do research differently and work with non-research partners for impact, but also of institutions to facilitate such a shift. Several authors highlight the importance of building capacity for institutional learning (Hall et al. 2003; Horton and Mackay 2003; Eade 1997; Springer-Heinze et al. 2003). Eade (1997) emphasizes a capacity-building approach, training of staff in a variety of relevant skills, and the dynamic and long-term nature of the process when looking at types of social organization of NGOs engaged in development theory and practice. Johnson et al. (2003) show that participation of non-research stakeholders early on in the research process is important, as it can inform institutional learning in research organizations to change priorities and practices. It can also enhance the relevance of agricultural technologies and the capacity of these stakeholders to design their own action research processes (Johnson et al. 2003). Horton and Mackay (2003) outline the links between M&E, learning and institutional change and highlight the importance of institutional learning as a means to develop the capacities of the organization and of individual researchers, as well as empowering non-research partners as key stakeholders in the process.
CCAFS worked with expert facilitators and trainers from PIPA to implement a 1-week training course on using TOC for project and program planning (Alvarez et al. 2014). Participants were chosen strategically so that capacity would be available in the CGIAR Centers at the point in time when CGIAR proposals would need to be developed following the TOC principles. In addition to project representatives, CCAFS science officers representing all themes and regions participated, in order to build in-house capacity of TOC champions. The training, in combination with TOC facilitation guides (version 1: Jost et al. 2014d; version 2: Schuetz et al. 2014d) and learning notes (CCAFS 2015a), helped highlight the opportunities (and constraints) of rolling out RBM to a whole R4D program. An online community of practice (and wikispace) was established and allowed for continued documentation and exchange of experiences.
4.9 CCAFS’ Results-Based Management Trial: Insights from Researchers and Partners
CCAFS’ approach to RBM is centered on adaptive management, regular communications between program and projects, and facilitated learning within projects. Besides periodic virtual meetings, trial participants were surveyed for a more in-depth and standardized reflection, and for capturing lessons and achievements from their experience (Schuetz et al. 2014b, c). These lessons also formed the basis for the progress report to CO (Thornton et al. 2014c). Ten months into the RBM trial, the progress report summarized project participant experiences, as well as the programmatic perspective.
From the programmatic level, reflections and lessons by the CCAFS Program Management Committee have been documented in the CO progress report, as well as in the series of learning notes (CCAFS 2015a). It was a great learning experience to have an RBM trial with the six projects and to be allowed to test and tryout what is required to make the shift from a LFA to an approach that is much more people-focused, learning-focused and outcome-focused. The approach to developing the IPs was simplified over time, mostly in relation to a reduction in the type and number of indicators and level of complexity so that the wider group of people who were expected to work with them would continue to buy in to the approach (Schuetz et al. 2014d).
The survey results show that there are many people within CGIAR Centers and CCAFS partners who are willing to take on the challenge to develop new ways of collaborating and working beyond delivering outputs towards outcomes (Schuetz et al. 2014b). From the survey, the RBM trial team found that the projects had made considerable progress, but also that making fundamental shifts in the way of working take time and (initially at least) additional resources. It requires iterative and continuous processes. Staffing, or the profile of project team members, and project team composition are emerging as key factors for success. Project staff has acknowledged that they may require additional skills beyond disciplinary expertise, such as skills in coordination, facilitation, engagement, communications, and participatory and learning-oriented M&E. The RBM trial team is using the findings from the survey to explore how additional support can be provided in such areas as engaging with stakeholders and using RBM.
4.10 Rolling Out Results-Based Management for CCAFS as a Whole
Experience in CPWF also shows that an intense process is required to finalize the program portfolio and allow for the appropriate triangulation and harmonization between thematic perspective, regional context and individual project proposals to ensure programmatic coherence, cohesion and its relevance and potential for impact (Hall et al. 2014; Biswas et al. 2008). This requires intense bilateral virtual preparation between research and regional teams, facilitated face-to-face time (e.g. in the form of workshops or writeshops), and follow-up work. Intensive workshops bring together project leaders, key national and regional partners and core program staff within a respective regional or thematic focus. The workshops/writeshops can bring together selected projects in a region as a team that will continue to work together over a period of time. It is key that the agenda is designed to hone the individual project IPs towards a coherent and cohesive regional and global R4D program that complements other ongoing initiatives and contributes to the given development goals.
While it took a considerable amount of effort, the iterative development of the CCAFS TOCs and impact pathways was done in a resource-efficient way. It started off mostly virtually and intensely facilitated, building on CCAFS Phase 1 engagement and regional priorities, and was completed in five regional face-to face meetings with key next-users and stakeholders within the CGIAR research community (Schuetz et al. 2014e, f). Building on the learning-based approach to developing a suitable TOC approach for CCAFS, a series of learning notes was written to document the RBM trial experiences and the rolling out of the approach to the whole program (CCAFS 2015a). The TOC development and facilitation process, and guidance documentation were revised to make them leaner, more contextualized and easier to implement (Schuetz et al. 2014d). The TOC building process is one key component in the CCAFS MEL system that was developed to support the new approach in a comprehensive manner (CCAFS 2015b).
4.11 Implementing a Modular MEL System for CCAFS
Understanding of why the program and projects are believed to contribute to the outcomes sought – the TOC.
Setting meaningful performance expectations/targets for key results (outputs and outcomes).
Measuring and analyzing results and assessing the contribution being made by the program to the observed outcomes/impact.
Deliberately learning from this evidence and analysis to adjust delivery and, periodically, modify or confirm program design, i.e. have an adaptive management in place.
Reporting on the performance achieved against expectations – outcomes accomplished and the contribution being made by the program.
A modular system can best meet the demands of the program as a whole and its projects, as well as the wider CGIAR system (see below; Thornton et al. 2014d). Some elements are prescribed by CGIAR governance bodies, including the carrying out of baselines, independent impact assessments, and periodic external evaluations. Programmatic flexibility exists within the day-to-day operational MEL, as a system is required that allows enough flexibility and adaptability to be applied to the different types of projects and programs.
Harmonization of TOCs: the framework for this modular approach is set through the TOC development across CCAFS thematic and regional operations, describing how CCAFS flagships, regions and projects anticipate changes in next-user behavior and practices, and their role in it. Investment in the development, harmonization and use of IPs and more elaborated TOC: (1) ensures that CCAFS plan of work is targeted at achieving outcomes and requires that tasks addressing the ‘use of outputs’ are built into each activity plan; (2) strategically encourages communication and collaboration among colleagues within research, regions and projects and guides exchanges across disciplines and regions; and (3) revisits the trajectory of CCAFS contributions to change and uses them as an ex-ante impact assessment.
Indicators & Baselines: In preparation for a harmonization process, as described above, indicators and outcome target numbers to which the program and projects will be held accountable were defined by the regional and research program leaders. The regionally and thematically aggregated targets were then checked against what individual projects proposed to contribute towards an agreed set of target values. Additionally, a programmatic baseline at site level was conducted at the beginning of the program to be able to compare achievements against these later on, with respect to behavior and practice change of farmers. Furthermore, projects are responsible for conducting specific baselines to monitor progress over time within their respective thematic and regional foci.
Reflexive spaces & activities: These need to be built in systematically to ensure that the key elements of adaptive management are operationalized. Adaptive management provides for flexibility and corrective actions to strengthen predictive capacity, which is essential when working in a constantly changing, complex environment. In working with TOCs, we make assumptions as to how we anticipate change will happen, but we know that change does not always happen as predicted, and so reflexive spaces are critical for allowing us to make well-documented and well-justified adjustments in response to the insights gained through our work.
Planning and reporting support: First, an online planning and reporting platform (P&R) collects project information at project inception, so that projects population the system once, and build on this for follow-up planning, reporting and learning. Project teams are guided in their TOCs/IPs-building from the beginning and use this as basis for monitoring annual progress. Thematic and regional programmatic goals/frameworks are prefilled by the program team, while projects map their individual contributions into these. Second, an MEL support pack provides practical mechanisms and tools to ensure a balanced quantitative and qualitative monitoring.
Assessment and bonus: Feedback loops, spaces for justification of changes and learning are weaved into the P&R to allow for systematic and strategic adaptive management throughout. Evaluation and synthesis are done from the regional and thematic perspectives after project reporting, to facilitate reporting to funding agencies, but also to minimize double counting of outcome target numbers and facilitate learning and knowledge brokerage across the program portfolio and beyond. Evaluation criteria include traditional output focused criteria, as well as progress towards outcomes, partnership and learning. Incentive mechanisms are being introduced, recognizing that these do not always have to relate to budgetary bonuses.
Institutional transformation and learning: Through feedback loops and reflexive spaces the program’s evaluative learning-oriented culture is also built into the system to ensure that the program is not only capturing ‘are we doing the right thing?’, ‘are we doing it right?’, but also ‘how do we know we are getting it right?’ (Kristjanson et al. 2014; van Epp and Garside 2014).
Chapter 14 ( Adaptation Processes in Agriculture and Food Security: Insights from Evaluating Behavioral Changes in West Africa) of this book describes an example of how this has been operationalized in a regional program of CCAFS.
4.12 Implications for Policy, Practice and Research
In this section we list some practical implications for a research-for-development organization that is considering moving to an approach based on RBM and TOC (Schuetz et al. 2015).
Working along TOCs and impact pathways has major implications for M&E. It implies a move to contribution rather than attribution, to acknowledge the role and inputs of partners and other actors both in achieving outcomes and in providing evidence for those outcomes. Building in triple-loop learning can make a major contribution to reflection and to supporting adaptive management, so that project teams can better deal with uncertainty. At the same time, not everything can be measured; this highlights the need for narratives that can complement and support more quantitative information.
The three thirds principle implies different budgeting and funding structures, so that appropriate levels of resources are allocated to capacity building, communications and engagement with the wide range of different partners likely to be needed. These elements need to be budgeted for explicitly within a project life-cycle, rather than as an after-thought. At the same time, there is still much work to be done on how to monitor outcomes effectively, evaluate the real share of contribution towards the observed change, and assess value for money. Similarly, delivery of outcomes, especially at scale, may take time for research-for-development programs. Longer funding cycles could be expected to facilitate this considerably.
The CCAFS experience has highlighted several operational principles for programmatic RBM. First, there is a need to focus on people and users, on utilizing M&E as a tool to help achieve outcomes, and on accountability – it is the people within organizations that make behavioral and practice changes happen. Second, there should be an emphasis on learning through M&E activities. Robust knowledge needs to be generated that can feed into developmental policy and investment decision making, and this in turn requires a cumulative and catholic approach to choice of impact assessment methods at different levels (Maredia 2009). Third, adaptive management needs to be encouraged, as a key element of RBM. As a tool that is based on learning processes, it can improve long-run management outcomes. The challenge in using it is to find the balance between gaining knowledge to improve management in the future and achieving the best short-term outcome based on current knowledge. Fourth, the development and implementation of an online platform is a great investment for capacity development. Planning, reporting and evaluation procedures need to be as simple as possible while still providing (most of) the information needed for effective and timely management.
Sharing findings along the way is a good way to foster the inclusive involvement of as wide a range of stakeholders as possible in project planning and implementation. Encouraging researchers to get early drafts of findings out to potential users for feedback from early on is one way to build a learning culture and to encourage open-mindedness.
Rigid application of just one specific approach most likely will not work. Whether it is the adoption of a technology, an M&E methodology, a learning approach or a scientific result, it is often not the whole package that is attractive to users but specific pieces. We need to allow users to cherry pick while ensuring that the relevant linkages remain intact so that the context is not lost for others who may want other cherries.
Solutions that are good enough rather than optimal. In many domains of knowledge and practice there is no best practice or option, particularly when the problem is complex and resources are constrained. CCAFS made considerable changes once it had started to implement an approach based on TOC and impact pathways, and in time moved towards a leaner and simpler model. Time will tell if some of the details inevitably lost in this process will need to be added back in, but the notion of “good enough” systems needs to be a key guiding principle.
Addressing tensions across scale. CCAFS is still in the process of embedding TOCs for the different organizational units of the program, in order to provide a flexible framework that allows for aggregation of output, outcomes and targets across the different units. For example, targets need to be framed locally with users and beneficiaries, and voiced in such a way as to allow the flexibility to deal with uncertainty and emerging priorities and opportunities. New investments of time and effort may be needed to identify and work with non-traditional partners to promote behavioral change in shared IPs.
Providing value for money. Many funding agencies now require that grantees demonstrate value for money. The Deutsche Gesellschaft für Zusammenarbeit states that its ‘work is systematically geared towards results, the yardstick by which we measure the success of our work. We want to help achieve tangible positive changes on the ground’ (GIZ 2015). Some have critiqued the whole notion of payment by results as applied to development and research-for-development on the basis that it provides perverse incentives that actually diminishes cost-effectiveness (see Chambers 2014). As noted above, there is much work still to do on appropriate measurement mechanisms, but this does not diminish the need to demonstrate accountability.
Balancing science and outcomes. Research is often curiosity-driven, and traditional indicators of success center on peer-reviewed publications in high-profile academic journals. In today’s highly competitive research environment another crucial success factor relates to fundraising: the ability to write and win competitive research proposals. Neither of these motivations for research is guaranteed to deliver development outcomes. For CGIAR and its research programs, it is still early days, but preliminary results suggest that “successful RBM” relates to effective and efficient research leading to outcomes, with a minimum of perverse incentives. The building of an IP with a narrative TOC forces researchers to give some thought to what lies between solid science, great technologies, and their positive developmental impact. A mix of an outcome-focused TOC with people and partners at the core, and a RBM approach that allows us to monitor, reflect, evaluate, and learn, are key elements for a programmatic MEL strategy – coupled with great science.
Requests by funding agencies for a move towards outcome-oriented research programs are having considerable impact on the way in which research is conceived, planned, implemented and evaluated. A key requirement for such work is flexibility – the flexibility to adjust so that the outcome orientation works as a support mechanism and enabler rather than a one-size-fits-all straitjacket without any space for innovation, serendipity and creativity. The shift to a R4D approach based on TOC is fostering massive change, much of it for the better, in our view. However, it also comes with considerable challenges. Defining the necessary changes, and developing new processes and mechanisms, need time and resources, which are often grossly underestimated and inadequately planned for. Some of these challenges arise because of the nature of research: the results are not known from the start, unlike in engineering where the outcomes are generally much less uncertain. Another challenge is that CGIAR is a R4D organization, not a development organization, and it is still in the process of sorting out how to balance the need to do great science with the need for impact. We need to avoid the results-based focus being to the disadvantage of the science, and development being seen to be in competition with the science. Rather, they need to be seen as complementary, enabling, and liberating.
CCAFS started as a Challenge Program (2009–2011), and then became a CRP with Phase 1 (2012–2014) and an Extension Phase (2015–2016). The proposal for Phase 2 (2017–2022) is currently under development. We acknowledge support to CCAFS from the CGIAR Fund Council, European Union, and International Fund for Agricultural Development. We also acknowledge the inputs of many people in the work and activities described here.
- Alvarez, S., Jost, C., Schuetz, T., Förch, W., Schubert, C., & Kristjanson, P. (2014). Lessons in theory of change from the introductory training on theories of change, impact pathways and monitoring & evaluation (CCSL Learning Brief No. 10). Copenhagen: CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS). http://hdl.handle.net/10568/52992.Google Scholar
- Bazeley, P. (2004). Issues in mixing qualitative and quantitative approaches to research. In R. Buber, J. Gadner, & L. Richards (Eds.), Applying qualitative methods to marketing management research (pp. 141–156). Basingstoke: Palgrave Macmillan.Google Scholar
- Bester, A. (2012). Results-based management in the United Nations Development System: Progress and challenges. A report prepared for the United Nations Department of Economic and Social Affairs, for the Quadrennial Comprehensive Policy Review Final Report.Google Scholar
- Binnendijk, A. (2000). Results-based management in the development co-operation agencies: A review of experience. DAC Working Party on Aid Evaluation Report. OECD. www.oecd.org/development/evaluation/dcdndep/31950852.pdf.
- Biswas, A., Palenberg, M., & Bennet, J. (2008). Report of the first external review of the water and food challenge program review. CGIAR Science Council. www.sciencecouncil.cgiar.org/system/files_force/ISPC_CPER_WaterFood.pdf?download=1
- CCAFS. (2010). Activity workplans and budgets for 2010. www.ccafs.cgiar.org/publications/ccafs-activity-workplans-and-budgets-2010
- CCAFS. (2011). Annual workplan 2011 for theme leaders and regional program leaders. cgspace.cgiar.org/bitstream/handle/10568/10214/workplan2011themeleaders1.pdf
- CCAFS. (2014). Annual report 2014. ccafs.cgiar.org/research/annual-report/2014
- CCAFS. (2015a). How we work: Impact pathways. ccafs.cgiar.org/impact-pathways
- CCAFS. (2015b). Planning, monitoring, learning and evaluation. ccafs.cgiar.org/planning-monitoring-learning-and-evaluation
- CCAFS. (2015c). Key strategic documents. ccafs.cgiar.org/key-strategic-documentsGoogle Scholar
- CGIAR. (2015a). www.cgiar.org
- CGIAR. (2015b). CGIAR challenge programs. www.cgiar.org/our-strategy/challenge-programs/
- CGIAR. (2015c). CGIAR research programs. www.cgiar.org/our-strategy/cgiar-research-programs/
- Chambers, R. (2014). Perverse payment by results: Frogs in a pot and straitjackets for obstacle courses. Blog. participationpower.wordpress.com/2014/09/03/perverse-payment-by-results-frogs-in-a-pot-and-straitjackets-for-obstacle-courses/
- Evenson, R. E., & Gollin, D. (Eds.). (2003). Crop variety improvement and its effects on productivity. Wallingford: CABI Publishing.Google Scholar
- Douthwaite, B., Alvarez, S., Cook, S., Davies, R., George, P., Mackay, R., & Rubiano, J. (2007). Participatory impact pathways analysis: A practical application of program theory in R4D. The Canadian Journal of Program Evaluation, 22(2), 127–159.Google Scholar
- Douthwaite, B., Kamp, K., Longley, C., Kruijssen, F., Puskur, R., Chiuta, T., Apgar, M., & Dugan, P. (2013). Using theory of change to achieve impact in AAS (AAS working paper). aas.cgiar.org/sites/default/files/publications/files/AAS-theory-of-change-impact.pdf
- Eade, D. (1997). Capacity-building: An approach to people-centred development. Oxford: Oxfam. policy-practice.oxfam.org.uk/publications/capacity-building-an-approach-to-people-centred-development-122906Google Scholar
- Earl, S., Carden, F., & Smutylo, T. (2001). Outcome mapping, building learning and reflection into development programs. Ottawa: IDRC Publications.Google Scholar
- FAO. (2015). Climate change and food systems: Global assessments and implications for food security and trade. Rome: FAO. www.fao.org/documents/card/en/c/2d309fca-89be-481f-859e-72b27a3ea5dc/.Google Scholar
- GIZ. (2015). Results-based monitoring. https://www.giz.de/rd-en/aboutgiz/518.html
- Gonsalves, J. (2013). A new relevance and better prospects for wider uptake of social learning within CGIAR (CCAFS Working Paper No. 37). Copenhagen: CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS).Google Scholar
- Harrington, L. W., & Fisher, M. J. (Eds.). (2014). Water scarcity, livelihoods and food security, research and innovations for development. New York: Routledge.Google Scholar
- Hartmann, A., & Linn, J. F. (2008). Scaling up: A framework and lessons for development effectiveness from literature and practice (Wolfensohn Center for Development Working Paper 5). Brookings Institute.Google Scholar
- Harvey, B., Ensor, J., Garside, B., Woodend, J., Naess, L. O., & Carlile, L. (2013). Social learning in practice: A review of lessons, impacts and tools for climate change (CCAFS Working Paper No. 38). Copenhagen: CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS).Google Scholar
- IPCC. (2013). In T. F. Stocker, D. Qin, G.-K. Plattner, M. Tignor, S. K. Allen, J. Boschung, A. Nauels, Y. Xia, V. Bex, & P. M. Midgley (Eds.), Climate change 2013: The physical science basis. Contribution of working group I to the fifth assessment report of the intergovernmental panel on climate change. Cambridge, UK: Cambridge University Press.Google Scholar
- Jost, C., Kristjanson, P., Alvarez, S., Schuetz, T., Foerch, W., Cramer, L., & Thornton, P. (2014c). Lessons in theory of change: Experiences from CCAFS (CCSL Learning Brief No 6). Copenhagen: CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS). http://hdl.handle.net/10568/35184.Google Scholar
- Maredia, M. K. (2009). Improving the proof – evolution of and emerging trends in impact assessment methods and approaches in agricultural development (IFPRI discussion paper). Washington, DC: IFPRI.Google Scholar
- Mayne, J. (2007a). Best practices in results-based management: A review of experience a report for the United Nations Secretariat volume 1: Main report. wpqr4.adb.org/lotusquickr/cop-mfdr/pagelibrary482571ae00516aa5.nsf/0/92A8DA95E6556A1A482576AF0029BB29/$file/john_mayne_un_1.pdf.
- Mayne, J. (2007b). Challenges and lessons in implementing results-based management. Evaluation, 13(1), 187–109.Google Scholar
- Pachico, D., & Fujisaka, S. (Eds.). (2004). Scaling up and out: Achieving widespread impact through agricultural research (Vol. 3). Cali: CIAT.Google Scholar
- Schubert, B., Nagel, U. J., Denning, G. L., & Pingali, P. L. (1991). A logical framework for planning agricultural research programs. Manila: IRRI.Google Scholar
- Schuetz, T., Crame, L., Foerch, W., Jost, C., Alvarez, S., Thornton, P., & Kristjanson, P. (2014a). Summary for the CCAFS flagship 4 projects kick-off meeting 28–29 January 2014: Result-based management trial. http://hdl.handle.net/10568/35407
- Schuetz, T., Schubert, C., Thornton, P., Förch, W., & Cramer, L. (2014b). Lessons and insights from the CCAFS results-based management trial. Summary from CCAFS Flagship 4 RBM trial projects survey 2014. CGIAR Research Program on Climate Change, Agriculture and Food Security. Report cgspace.cgiar.org/bitstream/handle/10568/53103/FP4_RBM%20Survey%20results_Final.pdf
- Schuetz, T., Förch, W., Schubert, C., Thornton, P., & Cramer, L. (2014c). Lessons and insights from CCAFS results-based management trial (CCSL Learning Brief No 12). Copenhagen: CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS). https://cgspace.cgiar.org/handle/10568/56629
- Schuetz, T., Förch, W., & Thornton, P. (2014d). Revised CCAFS theory of change facilitation guide. Copenhagen: CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS). http://hdl.handle.net/10568/56873
- Schuetz, T., Förch, W., Thornton, P., Wollenberg, L., Hansen, J., Jarvis, A., Coffey, K., Bonila-Findji, O., Loboguerrero Rodriguez, A. M., Martínez Barón, D., Aggarwal, P., Sebastian, L., Zougmoré, R., Kinyangi, J., Vermeulen, S., Radeny, M., Moussa, A., Sajise, A., Khatri-Chhetri, A., Richards, M., Jost, C., & Jay, A. (2014e). Lessons in theory of change from a series of regional planning workshops (CCSL Learning Brief No 11). Copenhagen: CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS). Link: http://hdl.handle.net/10568/52990
- Schuetz, T., Förch, W., Thornton, P., Wollenberg, L., Hansen, J., Jarvis, A., Coffey, K., Bonilla-Findji, O., Loboguerrero Rodriguez, A.-M., Aggarwal, P., Sebastian, L., Zougmoré, R., Kinyangi, J., Jost, C., & Jay, A. (2014f). Lessons in theory of change from a series of regional planning workshops. Copenhagen: CCAFS. Full report: http://hdl.handle.net/10568/52315
- Schuetz, T., Förch, W., & Thornton, P. (2014g). CCAFS monitoring and evaluation strategy. Copenhagen: CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS). http://hdl.handle.net/10568/41913
- Thornton, P. K., Ericksen, P. J., Herrero, M., & Challinor, A. J. (2014a). Climate variability and vulnerability to climate change: a review. Global Change Biology, 20(11), 3313–3328.Google Scholar
- Thornton, P., Förch, W., Cramer, L., Vasileiou, J. C., & Kristjanson, P. (2014b). Lessons learned from the Flagship 4 results-based management trial. Copenhagen: CCAFS. http://hdl.handle.net/10568/35188
- Thornton, P., Schuetz, T., Förch, W., & Campbell, B. (2014c). The CCAFS flagship 4 trial on results-based management: Progress report. cgspace.cgiar.org/bitstream/handle/10568/52261/CCAFS%20RBM%20trial%20report%202014_FINAL.pdf
- Thornton, P., Förch, W., Schuetz, T., Barahona, C., Abreu, D., & Tobón, H. (2014d). Guiding principles or “propositions” on monitoring and evaluation in CCAFS. Copenhagen: CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS). http://hdl.handle.net/10568/51830
- UN. (2015a). Transforming our world: The 2030 agenda for sustainable development.Google Scholar
- UN. (2015b). World population prospects. The 2015 revision. esa.un.org/unpd/wpp
- UNDP. (2007). Evaluation of results-based management at UNDP, achieving results. Evaluation Office. http://web.undp.org/evaluation/documents/thematic/rbm/rbm_evaluation.pdf
- Van Epp, M., & Garside, B. (2014). Monitoring and evaluating social learning: A framework for cross-initiative application (CCAFS Working Paper no. 98). Copenhagen: CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS).Google Scholar
- Vogel, I. (2012). ‘Theory of Change’ in international development. Review Report for the UK Department of International Development.Google Scholar
- WWAP. (2012). The United Nations world water development report 4: Managing water under uncertainty and risk. Paris: UNESCO.Google Scholar
- Young, J., & Mendizabal, E. (2009). Helping researchers become policy entrepreneurs (ODI briefing paper No. 53). www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/1730.pdf.
Open Access This chapter is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, duplication, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the work’s Creative Commons license, unless indicated otherwise in the credit line; if such material is not included in the work’s Creative Commons license and the respective action is not permitted by statutory regulation, users will need to obtain permission from the license holder to duplicate, adapt or reproduce the material.