Advertisement

The European Journal of Development Research

, Volume 30, Issue 5, pp 892–913 | Cite as

Factors Affecting the Relative Success of Collaborative Forestry Research Projects in Indonesia

  • Anthony G. BartlettEmail author
Open Access
Original Article

Abstract

The success of research for development projects is of keen interest to project funders and participating researchers, and underpins project impact. This paper reports a qualitative investigation of factors identified by project researchers as affecting relative success in ten collaborative forestry research projects in Indonesia. Interviews with 33 project participants identified 30 factors that influence project success. The most frequently identified factors were scientists’ commitment and collaboration; collaborative scoping and design; funding and equipment; effective communications and networks; implementation flexibility, monitoring and review; and skills mix and time allocations. The relative success of projects was evaluated through an analysis of project records, and examination of three projects of different relative success provided evidence of relationships between relative success and the identified success factors. As most of the success factors relate to project design or implementation, this knowledge can assist funders, research managers and project staff to improve project success.

Keywords

project success factors evaluation international development projects forestry research Indonesia 

Le succès de la recherche pour les projets de développement est d’un vif intérêt pour les bailleurs de fonds et les chercheurs qui y participent, et sous-tend l’impact des projets. Cet article rend compte d’une enquête qualitative sur les facteurs identifiés par des chercheurs comme les clés d’un succès relatif dans dix projets de recherche forestière en Indonésie. Des entretiens avec 33 participants du projet ont permis d’identifier les 30 facteurs qui influent sur le succès du projet. Les facteurs les plus fréquemment identifiés étaient l’engagement et la collaboration des scientifiques; la démarche collaborative dans l’identification et la conception du projet; le financement et l’équipement; une communication et des réseaux efficaces; la souplesse de mise en œuvre, le suivi et la révision du projet; et la diversité des compétences et la répartition du temps. La réussite relative des projets a été évaluée au moyen d’une analyse des documents du projet et l’examen de trois projets différents ayant eu un succès relatif a permis de fournir les preuves d’une corrélation entre le succès relatif du projet et les facteurs de succès identifiés. Comme la plupart des facteurs de réussite se rapportent à la conception ou à la mise en œuvre du projet, cette connaissance peut aider les bailleurs de fonds, les directeurs de recherche et le personnel de projet à améliorer le succès des projets.

Introduction

Many developed countries fund agricultural and natural resource management programs and projects in developing countries through their official development assistance (ODA) programs. While such projects can generate significant benefits to farmers and rural communities (Raitzer, 2003; Lindner et al, 2013), the poor performance and mixed success of many ODA projects have long been a concern (Yalegama et al, 2016; Ika et al, 2012). This challenge can be exacerbated in research-for-development projects, as the relationships between research-based knowledge and action are complex and often poorly understood (van Kerkhoff and Lebel, 2006). Understanding the factors that influence project success, referred to as success factors or critical success factors (Ika et al, 2012), enhances the ability of donors and implementing agencies to realise desired outcomes (Khang and Moe, 2008). However, surprisingly little has been documented on ODA project success factors (Diallo and Thuillier, 2004; Yalegama et al, 2016), which can vary according to the type of project and stage of the project life cycle (Pinto and Mantel, 1990) and the context in which the project is conducted (Ika and Donnelly, 2017).

Research evaluation is challenging because, even in the most efficient system, there is typically a lag of many years for the full impact of the research to emerge (Buxton, 2011); hence, impact assessments undertaken soon after a project concludes tend to under-estimate research impacts (Arnold, 2012). Not all impacts are easy to measure, and therefore impact assessments mostly focus on measurable economic and social impacts, with very few addressing environmental impacts (Weißhuhn et al, 2017). It is challenging to identify factors that contribute to project success in a consistent and meaningful way in the forestry sector in general, and for forestry research projects in particular. As Henderson (2000) observes, because of the complex nature and long production cycles of forestry systems, forestry research generally requires long-term commitments and multi-faceted programs to generate substantial impacts.

Research funders may also want to compare the relative success of projects addressing different topics or conducted in different contexts, and of successive projects addressing the same topic. In this general context, Bartlett (2016a) proposed a methodology for evaluating the relative success of collaborative ODA research projects, based on scoring against eight evaluation criteria. Bartlett et al (2017) applied this methodology to a sample of Australian Centre for International Agricultural Research (ACIAR) forestry projects in Vietnam, and complemented it with interviews of project leaders and researchers, to investigate the factors perceived to enhance or diminish success of these projects. Results demonstrated how such an analysis could assist implementing organisations to improve the likelihood of project success.

The definition of ‘success’ itself can be contested and controversial (McLeod et al, 2012). In this study, success is defined in terms of two primary dimensions, specifically related to the purpose of research for development projects, drawing on the approaches used by Pearce (2010) and Bartlett et al (2017). The first dimension, termed ‘achievements’, is the extent to which planned research outputs are achieved and adopted by ‘next users’, such as the participating scientists; the second dimension, termed ‘impacts’, is the extent of the impacts resulting from wider adoption of the research outputs by ‘end users’, such as forest growers.

This paper continues this series of investigations, and reports a qualitative study involving ten collaborative forestry research projects between Australia and Indonesia supported by ACIAR. It addresses three questions: What differences exist in the level of success achieved by these projects? What are the factors that are considered by project leaders and researchers to affect the relative success of these projects? and Is there evidence that the way these factors have been managed in individual projects has affected their relative success? The results are relevant to both researchers and international development practitioners, because greater knowledge about research for development (R4D) (sensu lato Høgh-Jensen et al, 2010) project success factors can assist those responsible for project design and implementation to improve project effectiveness.

The Australian Centre for International Agricultural Research

The Australian Centre for International Agricultural Research (ACIAR) is a federally funded agency that commissions collaborative agriculture, fisheries and forestry research projects in developing countries. ACIAR funds R4D projects conducted by Australian or Consultative Group on International Agricultural Research (CGIAR) scientists working collaboratively with scientists from the partner countries to address a research priority identified by the partner country. ACIAR projects seek to generate knowledge, technologies and capacity to achieve better decision making, changed agricultural practices and policies that, in turn, generate positive scientific, economic, social or environmental impacts (ACIAR, 2014). These projects involve capacity building and research activities and, where relevant, develop an understanding of the farming and forestry systems as well as policy settings to enhance the prospects that the knowledge and technologies developed will be adopted. Over a 30-year period, ACIAR has invested over AUD 100 million to fund 150 forestry research projects, with the greatest number of projects implemented in Indonesia, Vietnam and Papua New Guinea (Bartlett, 2016b).

ACIAR regularly evaluates the impacts and effectiveness of a sample of its projects, including quantifying their economic returns (ACIAR, 2014). An ACIAR study by Pearce (2010) identified 14 factors that were considered important to achieving successful project outcomes, but it did not examine how these factors may have contributed to different levels of success in different projects.

ACIAR’s Support for Forestry Research in Indonesia

In 2011, Indonesia’s forestry sector, based on each of natural and planted forests, contributed USD 14.57 billion to the national economy (FAO, 2014). Indonesia’s diverse natural forests have been heavily exploited for timber production over the past 50 years, and rates of conversion to agriculture have been high (Tsujino et al, 2016). Nevertheless, Indonesia retains the eighth largest area of forest in the world, with about 91 million hectares (53 per cent of its land area) classified as forest (FAO, 2015).

Indonesian farmers have a long history of planting trees and allowing natural regeneration of trees on private land. Smallholders grow trees as a ‘living savings account’, though their returns are constrained by poor knowledge of silviculture, timber standards and markets, and complicated regulations governing timber trading (Roshetko et al, 2013). These smallholders supply timber to thousands of wood manufacturing industries (Perdana and Roshetko, 2015), but many of these suffer from inefficient value chains and inappropriate processing and manufacturing techniques for small-diameter logs (Wibowo et al, 2013).

Indonesia has encouraged the development of large-scale timber plantations. In 2014, the area of fast-growing acacia and eucalypt plantations was 1.5 million hectares, with 800,000 hectares located in large estates managed by plantation companies on Sumatra (Harwood and Nambiar, 2014). However, the viability of fast-growing plantations based on these exotic species is threatened, due to the increasing impacts of damaging diseases such as Ganoderma (Francis et al, 2014) and Ceratocystis (Tarigan et al, 2011), as well as restrictions on the use of peatlands (Jauhiainen et al, 2012).

ACIAR’s forestry projects in Indonesia have covered a broad range of themes in the context of forest-based development described above; they have included technical, social and policy aspects of plantation and smallholder forestry systems (Mendham and Hardiyanto, 2011; Rohadi et al, 2012), climate change (Irawan and Tacconi, 2009) and value adding of timber and non-timber forest products (Cunningham et al, 2011; Purnomo et al, 2014). From 1987 to December 2015, ACIAR completed 21 forestry research projects in Indonesia (Table 1), representing about one-fifth of all forestry projects commissioned by ACIAR over three decades (Bartlett, 2016b). An ACIAR impact study of 12 completed ACIAR forestry projects in Indonesia (Lindner, 2011) reported high returns on investment overall, but evidence of impact from only some of the projects. These results highlight the need for improved understanding of the factors that affect project outcomes and impacts.
Table 1

Summary information for ACIAR’s completed Indonesian forestry projects, with those selected for study highlighted, and those for phase 3 evaluation identified

ACIAR project code

Duration

Value AUD m

Research theme 1

Title of project

FST/2009/051

2011–2015

1.873

T2

T4

Increasing productivity and profitability of Indonesian smallholder plantations

FST/2008/030

2011–2015

0.898

T6

Overcoming constraints to community-based commercial forestry in Indonesia

FST/2007/119

2008–2013

1.012

T5

Mahogany and teak furniture: improving value chain efficiency and enhancing livelihoods

FST/2007/052

2008–2014

1.450

T10

Improving governance, policy and institutional arrangements for REDD in Indonesia

FST/2006/117

2009–2014

1.001

T5

Improving added-valued furniture production from plantation timber in the Jepara region

FST/2005/1772

2007–2011

0.810

T6

Improving profitability from smallholder teak agroforestry

SMAR/2006/011

2006–2009

0.273

T7

Enterprise development, value chains and evaluation of non-timber forest products

FST/2004/058

2006–2010

0.703

T2

Improving water and nutrient management in Indonesian and Australian plantations

FST/2003/0482

2006–2010

0.710

T4

Management of fungal root rot in plantation acacias in Indonesia

FST/2003/025

2005–2007

0.400

T6

Community partnerships for plantation forestry in eastern Indonesia and Australia

FST/2001/105

2003–2007

0.641

T10

Impacts of decentralisation on sustainable forest management, development and livelihoods

FST/2001/020

2001–2004

0.302

T6

Facilitating development of agroforestry systems as alternatives to slash-and-burn agriculture

FST/2000/123

2001–2006

0.679

T4

Heart rots in plantation hardwoods in Indonesia and southeast Australia

FST/2000/1222

2001–2003

0.394

T1

Application of molecular marker technologies for genetic improvement of forest plantation species

FST/2000/001

2002–2005

0.795

T9

Impacts of fire and its use for sustainable land and forest management

FST/1999/035

2002–2007

1.143

T6

The impact of changing agroforestry mosaics on catchment water yield and quality in SE Asia

FST/1998/096

2000–2004

2.209

T1

Domestication of Australian trees for reforestation and agroforestry

FST/1998/085

1999–2001

0.153

T4

The taxonomy of Hypsipyla robusta and allied species

FST/1993/709

1993–1996

0.135

T6

Agroforestry solutions to rehabilitate Imperata grasslands

FST/1990/043

1991–1995

0.437

T3

Multi-purpose tree and sandalwood silviculture in Indonesia

FST/1986/013

1987–1991

0.451

T3

Fuelwood and sandalwood silviculture in eastern Indonesia

1ACIAR forestry program research themes as described in Bartlett (2016b)

Theme 1: Domestication and improvement of Australian trees

Theme 2: Silviculture for Australian trees

Theme 3: Domestication and silviculture of non-Australian trees

Theme 4: Forest health and biosecurity

Theme 5: Value-added processing and treatment of wood

Theme 6: Agroforestry and community forestry

Theme 7: Non-timber forest products

Theme 9: Fire management

Theme 10: Forestry and environment policies

2Phase 3 evaluation projects

FST/2005/177 – high achievements/high impacts

FST/2003/048 – high achievements/low impacts

FST/2000/122 – low achievements/low impacts

Methods

The methods for this study follow those developed by Bartlett (2016a) and refined in a companion study by Bartlett et al (2017), involving three phases as outlined below. Here, success factors, which were identified from information provided by project researchers, are considered to be factors that can enhance or diminish project success, but they are not in themselves indicators of project success. The evaluation of relative success of the case study projects was undertaken by the author prior to identification of the success factors, using information from a variety of sources in ACIAR project records, as described below. The research protocol was approved by the Australian National University Human Ethics Committee (protocol no. 2014/051).

Selection of Projects for the Case Study

Ten of the 21 ACIAR forestry projects completed in Indonesia between 1987 and 2015 (Table 1) were selected for the study, taking into account the following factors:
  • Focussing on medium to large research projects conducted entirely in Indonesia; these included some projects that were part of a longer-term program;

  • Ensuring representation of projects from across the ten research themes, five of which were represented;

  • Including some projects commissioned through the CGIAR international agricultural research centres;

  • Having adequate project records available for analysis and being able to locate researchers involved in a project for interview.

In this sample, eight projects were led by Australian research agencies and two by CGIAR centres. Each project involved collaboration with scientists from various Indonesian partner organisations, including the national Forestry Research and Development Agency (FORDA), universities, non-governmental organisations and private-sector companies. The selected projects included two that continued long-term research commenced in three earlier projects and included many of the same project team members. One of these successor projects combined research on tree diseases and plantation productivity previously undertaken in two separate projects.

Phase 1: Identification of Project Success Factors

Thirty-three scientists from a range of partner organisations were identified for interview from records of the ten projects. They were selected using a purposive strategy because they had worked as project leaders, Indonesian project coordinators or collaborating researchers on one or more of the selected projects, and were still able to be contacted. The interviewees comprised 7 scientists from Australian agencies, 9 scientists from the CGIAR centres and 17 scientists from Indonesian partner agencies. They were interviewed individually by the author using a standard set of questions (see Bartlett et al, 2017), which asked them to describe what they thought constituted success for an ACIAR project, and to nominate five factors that can enhance, and five factors that can diminish, project success. Their views on aspects of the design and implementation of each project, and other contextual factors, were also sought.

HyperRESEARCH1 qualitative data analysis software was used to assist analysis of interview data by aggregating responses to specific questions into single reports and searching the data for commonly used phrases and similar concepts. This enabled the author to establish participants’ perspectives on the definition of project success, and facilitated aggregation of thematic aspects of the responses into two lists, of factors that either enhance or diminish project success. Participants’ responses about factors affecting project success and about each project’s design and implementation were analysed and results were aggregated into two groups: those from the Indonesian participants, and those from the Australian and CGIAR participants. The frequency with which each success factor was identified by each group was recorded, and complementary expressions of the same factor from the two lists identified, as the basis for preparing concisely worded statements of the factors identified as enhancing or diminishing project success.

Phase 2: Evaluation of Relative Success of the Case Study Projects

In this study, the relative success of each of the ten projects was evaluated using qualitative data, drawn from internal ACIAR project records, and the score-card matrix methodology described by Bartlett (2016a). The records included: project documents; annual reports; annual assessments and mid-term reviews conducted by the program manager; final reports; external end-of-project reviews; adoption studies and external impact assessments; project-related publications; and written correspondence between ACIAR and project staff. These data provided a degree of triangulation by presenting the perspectives of research program managers and external reviewers of projects, as well as those of project participants.

As explained by Bartlett (2016a), scores were assigned for four criteria related to research achievements: project design, results achieved, collaboration and publications, and for four criteria related to research impacts: capacity building outcomes, scientific outcomes, economic outcomes and social and policy outcomes. For each criterion, the available evidence was considered and a score assigned by the author, to the nearest 0.5, up to the maximum score. The types of evaluation questions, maximum scores and nature of the evidence sought are presented in Table 2. Scores totalling ten were assigned for each of research achievements and research impacts. Scores of 0.0–5.0 were categorised as low achievements or low impacts; scores of 5.1–10.0 were categorised as high achievements or high impacts. This classification generates four categories of project success: high achievements/high impacts, high achievements/low impacts, low achievements/low impacts and low achievements/high impacts. A companion study (Bartlett et al, 2017) demonstrated this categorisation to be helpful in relating success factors to levels of relative success.
Table 2

Evaluation questions, maximum scores and evidence guidance for the eight project evaluation criteria

Criterion

Score

Evaluation questions

Evidence sought

Project design

2

How well was the project designed in terms of specific activities to address objectives and to facilitate adoption?

Consideration of research strategy and nature of research and dissemination activities planned; Composition of project team; Level of funding provided and co-contributions from partners; Findings from any mid-term review

Results achieved

4

What has been achieved in terms of completed activities and specified outputs?

Identification of the quality of actual achievements compared with planned outputs; Adaptation of methods and activities to enhance outcomes; Methods and level of dissemination of results; Findings from any end-of-project review

Collaboration

2

How well did the project team collaborate in conducting the research, and what new skills did the scientists gain?

Information about collaboration in correspondence and reports; Effectiveness of in-country coordination; Joint authorship of reports; Level of networking developed and extent of within-project capacity building activities

Publications

2

What is the relative magnitude and quality of publications produced?

Quality of information in final report; Amount and quality of project reports, including consideration of local language publications; Number of published journal articles; Quality of website information

Capacity building

2

What is occurring as a result of the enhanced capacity?

Evidence of enhanced capacity of project scientists; Appraisal of how well these skills are being utilised; Local scientists’ contributions to scientific publications

Scientific outcomes

4

How has the body of scientific knowledge been enhanced, and how is this knowledge being used?

Number of international journal publications and citations; Continuation of related research; Evidence of networking between scientists; Appraisal of scientific contributions to international development

Economic outcomes

2

Has the research led to improved livelihoods or facilitated economic development?

Indications of improved productivity, greater access to markets and higher prices for products; Indications of costs or losses avoided; Indications of greater employment levels or wages; Indications of new enterprises established

Social and/or policy outcomes

2

What changes to the social circumstances of project beneficiaries or the enabling policy environment have occurred that the project has contributed towards?

Indications of enhanced social capital including strengthening of community institutions; Evidence of empowerment of women and disadvantaged groups; More equitable benefit sharing from common property resources; Evidence of new or changed policies or effective input to policy processes

Phase 3: Identification of Relationships Between Success Factors and the Level of Relative Success Achieved by Different Projects

Three projects representing different success categories were selected (Table 1) for a more detailed analysis, with supporting information presented in Appendix 1.

For each selected project, interview responses from the project leader and two Indonesian participants were further analysed to identify any references to the way the success factors identified in the phase 1 analysis had enhanced or diminished success. The ACIAR project records were reviewed to identify evidence about the way these success factors may have influenced the project’s success. Using these two sources of information, subjective ratings were assigned for the apparent influence of each of these success factors on the project’s success. The following five-category rating system was used:

Strongly enhances—presence of factor appears to have strongly enhanced success

Enhances—presence of factor appears to have enhanced success

Neutral—no evidence that the factor enhanced or diminished success

Diminishes—absence of factor appears to have diminished success

Strongly diminishes—absence of factor appears to have strongly diminished success.

Results

Interpreting Success in a Collaborative Research Project

The views expressed by participants on what constitutes project success varied considerably, with some articulating factors that influence success rather than what success meant to them. Several participants noted that an individual project in a long-term program of research could be considered successful even if the project outputs could not be widely adopted at the end of the project. The thematic analysis enabled a common definition of success to be developed from participants’ responses: a successful ACIAR forestry research project in Indonesia was one which uses good but flexible scientific methods to achieve the planned outputs, enhances the capacity of partners, facilitates ongoing scientific networks, and disseminates the results to achieve impacts for the intended beneficiaries.

Identification of Success Factors

The thematic analysis of participants’ responses on the factors that can enhance or diminish project success identified 26 factors that were considered to enhance, and 29 factors considered to diminish, project success; when taken as a whole, there were 30 different factors identified that influence project success (Table 3). While most factors which diminish success were the converse of those that enhance success, there were three factors identified that diminish success (continuity of partner institutions and team; experience of project leader in country; external factors: policies, markets, environmental, security) and one factor that enhances success (collaboration with international scientists), for which there was no converse factor identified by participants.
Table 3

Success factors, showing participants’ views on aspects that enhance or diminish project success

Factor no.

Success factor

Participants’ views on factors that can enhance success (ES) or diminish success (DS)

1

Collaborative scoping and design

ES: Shared research agenda and good collaboration on scoping and design

DS: Inadequate consultation with partners and too ambitious or poorly focussed design

2

Skills mix and time allocations

ES: Having diversity of skilled and experienced scientists with sufficient time allocations

DS: Team with narrow skills mix, inexperienced or overcommitted scientists

3

Funding, facilities and equipment

ES: Adequate funding and other resources, including donor and partner contributions

DS: Inadequate funding or facilities to undertake planned activities

4

Scientists’ commitment, collaboration and focus

ES: Dedicated and focussed scientists and collaborative team work

DS: Scientists lacking interest, commitment or focus and poor collaboration within team

5

Team and technical capacity building

ES: Supporting capacity building, informal and formal study

DS: Poor focus on capacity building of project partners

6

Mutual benefit of research topic

ES: Selection of research issue with mutual benefits

DS: Research does not provide mutual benefits or linkages between activities in each country

7

Selection and commitment of partner institutions

ES: Effective selection and ongoing commitment of project partners

DS: Poor support or conflict with partners or too many partners

8

Site selection and scientific rigour of trials

ES: Appropriate sites for research trials with good scientific design and stakeholder support

DS: Inappropriate trial location or poor scientific discipline in trial establishment

9

Leadership and management

ES: Good leadership and effective project planning and oversight

DS: Poor leadership and inefficient project management

10

Strong, culturally appropriate team relationships

ES: Respect of culture, patience and developing friendships

DS: Poor relationships or misunderstandings within team

11

Time spent on in-country collaboration

ES: Sufficient resourcing to enable adequate time of external researchers in country

DS: Inadequate travel funds or other restrictions limit in-country collaboration

12

Effective communications and research networks

ES: Good communications within project and effective dissemination of knowledge

DS: Poor communications between team members and failure to disseminate results to stakeholders

13

Links to impact pathway and user benefits

ES: Results linked to stakeholder benefits

DS: Lack of benefits for stakeholders from research

14

Implementation flexibility, monitoring and review

ES: Flexibility to adapt activities and appropriate monitoring and review of progress

DS: No flexibility to adapt, poor monitoring or no review

15

Continuity of partner institutions and team

ES: Not identified

DS: Changes in project staff or structures of partner institutions

16

Duration of project

ES: Duration long enough to implement activities and build partnerships

DS: Duration too short to implement activities or to obtain and publish results

17

Donor influence on design

ES: Effective support from donor to enable collaborative design

DS: Donor insisting on design elements not supported by project researchers

18

Long-term research collaborations

ES: Long-term relationships supported via follow-on projects

DS: Lack of follow-on research projects

19

Continuation of research post project

ES: Agencies continue research after project or clear exit strategy

DS: No funding available after project or no exit strategy

20

Alignment with national development objectives

ES: Research relevant to national policies and programs

DS: Project not relevant to national policies and programs

21

Experience of project leader in country

ES: Not identified

DS: Naivety of project leader about local context

22

Trust within team

ES: Trust between project participants

DS: Lack of trust within team or of confidence with stakeholders

23

Local government and community support

ES: Good support from local government and communities

DS: Poor collaboration or conflicts with local government or communities

24

Engagement with private sector

ES: Effective engagement of private-sector partners in conduct and adoption of research

DS: Lack of engagement or support from private-sector partners

25

Publication and dissemination of results

ES: Effective dissemination of scientific and extension information

DS: Ineffective dissemination of scientific or extension information

26

External factors: policies, markets, environmental, security

ES: Not identified

DS: External factors influencing research facilities, trials or markets and lack of appropriate supporting policies

27

Engagement of policy actors

ES: Effective engagement of policy actors

DS: Inability to engage policy makers

28

Willingness to adopt innovation

ES: Not identified

DS: Culture, finance or risk limit adoption of technologies

29

User champions

ES: Engagement of farmer or industry champions

DS: Poor selection of or lack of commitment of champions

30

Collaboration with international scientists

ES: Benefits from collaboration with international scientists

DS: Not identified

The 17 Indonesian participants and the group of 16 Australian and CGIAR participants generated a total of 424 responses related to individual success factors. The frequency of identification of each of the 30 factors considered to enhance or diminish project success is shown in Figure 1. The two most frequently identified factors, which together represented 18 per cent of the responses, were: scientists’ commitment, collaboration and focus (40 responses); and collaborative scoping and design (35 responses). Thirteen of the success factors (nos. 1–7, 9, 12, 14, and 23–25) together represented 74 per cent of the responses, and so can be considered as the most important factors affecting project success identified in this study.
Figure 1

Frequency of identification of the 30 factors considered by the Indonesian and the Australian/CGIAR groups of respondents to enhance or diminish project success.

Most of the success factors were consistent across the two country groups (Indonesian and Australian/CGIAR), but some differences were apparent. Indonesian participants more frequently identified success factors such as: scientists’ commitment, collaboration and focus; funding, facilities and equipment; effective communications and research networks; and engagement with the private sector. Australian/CGIAR participants more frequently identified key success factors such as: selection and commitment of partner institutions; publication and dissemination of results; and engagement of policy actors. These differences probably reflect a combination of cultural, experiential and institutional differences between the two groups of researchers, as well as the different challenges each experienced in conducting research projects in the Indonesian context. The Indonesian scientists placed a stronger emphasis on having staff that were committed, adequate funding and good communications within the team, while the Australian and CGIAR scientists were more concerned about the importance of institutional commitment and effective dissemination of results, including into the policy arena.

Evaluation of the Relative Success of Projects

The results of the evaluation of project achievements and project impacts for each of the 10 case study projects are shown in Figure 2. The 10 projects had different levels of apparent success, with scores ranging from 3 to 9 for research achievements and 2 to 7 for research impacts. In the evaluation of research achievements, nine projects (90 per cent) received scores of more than five, whereas in the evaluation of research impacts only four projects (40 per cent) received scores of more than five. Only four projects (40 per cent) achieved scores of more than five for both achievements and impacts. The evaluation methodology proved informative: even when projects received the same overall evaluation scores for research achievements and research impacts (as was the case for FST/2006/117 and FST/2007/119), they received different scores for the constituent criteria.
Figure 2

Overall and constituent project achievement and impact scores for the 10 case study Indonesian forestry projects.

The case study projects represent three categories of project success (Figure 3): one project with low achievements and low impacts, five projects with high achievements but low impacts, and four projects with high achievements and high impacts. No projects were categorised with the unlikely combination of low achievements yet high impacts.
Figure 3

Case study project impact and evaluation scores and assignment to success categories.

This study showed that subsequent projects on the same research topic may not always result in improved achievements and impacts compared with those from a precursor project. There were two projects that directly followed on from other projects: Project FST/2008/030 continued research on community forestry commenced in FST/2003/025. FST/2009/051 was a multidisciplinary project that continued research on plantation productivity and tree diseases commenced under two separate projects (FST/2004/058 and FST/2003/048). The results of the relative success evaluations for these related projects are shown in Figure 4.
Figure 4

Changes in relative success of related projects.

A project which commenced long-term research on root rot disease (FST/2003/048) received a high score for research achievements but a low score for research impact. The research was continued in a successor project (FST/2009/051) which received a similar evaluation score for achievements but a higher score for impacts, driven by increased scientific impacts from the ongoing research. Conversely, this same project (FST/2009/051), which also continued research on productivity of short-rotation plantations commenced under another project (FST/2004/058), achieved lower scores for both achievements and impacts than were achieved in that precursor project. The reason for this ‘unexpected’ result was that ACIAR combined the two different research themes into one project but did not provide sufficient financial resources to support all the required research activities. A third project (FST/2008/030) continued research on community forestry commenced in another project (FST/2003/025). Both projects received similar scores for research impacts, but the successor project had a higher score for achievements, as improved collaboration within the team led to completion of a higher proportion of planned activities and more publications.

A project which researched the application of molecular markers in tree breeding (FST/2000/122) received low scores for both research achievements and research impacts, reflecting an inadequate project duration of only 2 years with no follow-on phase of research. However, the Indonesian partner was still using the scientific capacity some 12 years after the project concluded, demonstrating that a relatively unsuccessful project may result in some enduring impacts. The finding on the importance of having long-term funding commitments for research programs to achieve substantial impacts is consistent with the findings of other studies of collaborative research endeavours, including an evaluation of Australia’s Cooperative Research Centre program (Allen Consulting Group, 2012).

Evidence of Success Factors in Selected Projects

The author assessed the apparent influence of each of the 30 success factors identified by project participants (Table 3) on the success of the three projects chosen to represent different evaluated levels of relative success (Table 1), using both interview responses and evidence from project records. This assessment is presented in Table 4.
Table 4

Expression of success factors within three projects with different evaluated levels of success, with the 13 most frequently identified factors shown in bold italics

This analysis showed that, for the project evaluated as having high achievements and high impacts, there was good evidence that about two-thirds of the success factors had strongly enhanced the project’s success. Conversely, for the project evaluated as having low achievements and low impacts, it was apparent that about half of these factors had not been appropriately addressed and thereby had contributed to the diminished success of the project. The project with high achievements but low impacts had a lesser number of the factors that appeared to strongly enhance project success than did the project with high achievements and high impacts, and some factors, such as project duration, effective communications and monitoring and review, had contributed to diminished success. These relationships were more evident in information from the project records than from the interview responses, perhaps because the project-related interview questions did not directly address how the particular success factors may have influenced the project. These results demonstrate that project records, including external review reports, can provide evaluators with both positive and negative project performance-related information.

The analysis also showed that there is a reasonably clear relationship between the presence of those success factors which can be influenced during project design (nos. 1–3, 6, 7, 16, 17, 20 and 21) and evaluated levels of project research achievement and impact. The high achievements/high impacts project showed evidence of almost all of these factors either strongly enhancing or enhancing success, while in the low achievements/low impacts project, the evidence suggested that inadequate attention to over half of these factors had either strongly diminished or diminished success. This demonstrates the importance of careful attention to these factors in the design of research projects.

Discussion

Various authors (Bartlett et al, 2017; Baynes et al, 2015; Byron, 2001; Pearce, 2010) have examined the factors that influence the success of forestry development initiatives, and Pearce (2010) examined project-level factors that affect the success of ACIAR projects. The main purpose of studies such as these is to improve understanding of the factors that enhance or diminish success of ODA-funded projects, so that those responsible for project design and implementation can take them into account to improve project effectiveness. The findings of this study both confirm and supplement those from these previous studies.

The 14 success factors identified by Pearce (2010) as relevant to ACIAR research projects were all identified in this study, as were the 22 success factors identified in a companion study of ACIAR forestry research projects in Vietnam (Bartlett et al, 2017). The relative frequency of the factors differed between Vietnam and Indonesia, and a further eight success factors were identified by the Indonesian study participants. The apparent relationship between the presence of these success factors and the evaluated level of relative project success found by Bartlett et al (2017) for the Vietnam projects was also evident in this study.

The most notable differences in the factors identified in this study, when compared with the Vietnam study, were in the substantially increased frequency of two factors: effective communications and research networks (no. 12) and implementation flexibility, monitoring and review (no. 14), and the inclusion of three new factors in the 13 most frequently identified factors, viz. local government and community support (no. 23), engagement with the private sector (no. 24) and publication and dissemination of results (no. 25).

The eight success factors identified for the first time in this study were:

Local government and community support (no. 23)—this reflects the decentralised responsibility for forestry in Indonesia, and the need to have active participation of communities and smallholders to enhance the prospects of adoption of the forestry innovations from many projects.

Engagement with the private sector (no. 24)—this recognises the importance of the private sector in both smallholder and industrial forestry systems in Indonesia, and reflects a research focus on topics relevant to these systems: plantation productivity, disease management, timber and non-timber value chains and wood processing.

Publication and dissemination of results (no. 25)—this reflects the desirability and challenges of preparing and disseminating scientific articles and appropriate extension materials within the timeframe of a research project, in a research system that did not historically have a strong emphasis on academic writing, particularly in English.

External factors: policies, markets, environmental, security (no. 26)—this reflects a range of factors that are outside the control of projects but can affect project achievements, including unsupportive policies, access to markets, unforeseen diseases, natural disasters and political or security issues that limit travel to research sites.

Engagement of policy actors (no. 27)—this recognises that, in Indonesia’s dynamic and decentralised political system, it can be difficult for researchers to achieve effective engagement with relevant policy actors.

Willingness to adopt innovation (no. 28)—this reflects the constraints on the capacity of some end users, including smallholders and small enterprises, to adopt innovations, for example because of risk aversion or lack of access to the finance needed to utilise a technology.

User champions (no. 29)—this reflects the benefits that can arise from having effective user champions actively engaged in a research project and, conversely, the challenges that exist when such champions are not present or are unable to lead adoption.

Collaboration with international scientists (no. 30)—this reflects the benefits that come from networking and collaboration with skilled international scientists and the challenges that many developing-country scientists have in accessing or capitalising on such collaborations.

These results illustrate how the factors that influence project success may be both common and different between projects; For example, the factor ‘mutual benefit of research topic’ was not considered to have influenced the success of a teak agroforestry project, whereas its absence was considered to have diminished success in a molecular marker project. Differences are likely to be attributable to both differences in the nature of the research itself, and in the local contexts within which the research and adoption occur. This shows the importance of having a flexible, content-driven approach to considering the relevance of and managing the individual success factors during project design and implementation, rather than a pre-determined list that is presumed to apply universally. While some of the identified success factors are closely related, for example ‘collaborative scoping and design’ and ‘mutual benefit of research topic’, they have been listed separately so that the subtle differences can be considered, as appropriate.

The identification in this study of the three new frequently identified success factors (nos. 23–25), which relate to engagement of relevant stakeholders beyond the project team and publication of project results, is also important. The identification of the factor expressed as publication and dissemination of results refers to preparation of a range of communications materials, such as journal articles, technical reports, information and policy briefs, training manuals, field guides, websites and blogs. It also relates to ensuring that the information is effectively disseminated to the stakeholders, who either will benefit directly from the research findings or have responsibilities for policies or programs that affect adoption of research findings. This finding is likely to reflect both the strong pressures on Australian, international and Indonesian scientists to publish research results, as well as the recognition that the results have to be appropriately communicated to end users to facilitate adoption. The identification of factors related to engagement with key external stakeholders – the private sector, policy actors, local communities and user champions – emphasises the importance of factors that facilitate the relevance of research to, and knowledge of research results by, their ultimate users. This in turn is likely to affect the prospects for adoption and thereby the magnitude of the impacts from the research investment.

In this study, over 80 per cent of the factors identified as affecting project success, including all of the 13 most frequently identified factors, relate to either project design or project implementation. Therefore, paying close attention to success factors related to project design, particularly the degree of collaboration with partners on project design, the quality of the research design, the selection and commitment of partner organisations and the time allocations for the collaborating scientists, is likely to enhance prospects of the project’s success. Likewise, project success will also be influenced by how well project teams pay attention to those success factors that can be influenced during project implementation. The most important of these factors are the commitment, focus and collaboration of the partner scientists, the effectiveness of leadership and communication processes, the degree of capacity building undertaken, and the flexibility the project has to modify its activities and approaches in response to feedback from monitoring and review.

Conclusions

Since the agreement of the Paris Declaration on Aid Effectiveness (OECD, 2005), and in the context of significant global negative externalities such as climate change and the global financial crisis (Haddad, 2012), there has been an increased interest in understanding both how aid effectiveness is evaluated and which factors contribute to the success of aid programs and projects. As Ofir (2010) notes, there is a need for deeper understanding of the essential and sufficient conditions for success, and also of the context necessary to achieve successful implementation and sustained impacts from agricultural research. This study has contributed to this learning, both by reinforcing the conclusions of an earlier companion study in Vietnam (Bartlett et al, 2017) and by broadening the understanding of which factors enhance or diminish the success of international collaborative forestry research projects. This study also demonstrated the utility of conducting evaluations of the relative success of related projects, through the finding that subsequent projects on the same research topic do not necessarily result in improved achievements and impacts relative to a precursor project.

As in the companion study, the results from this study suggest that there was a good convergence of assessment amongst project participants about the most important factors influencing project success, with about three-quarters of the responses relating to 13 of the identified success factors. This suggests that the majority of research project participants have a good understanding of the factors that influence the success of collaborative forestry research projects, which is consistent with the view of Haddad (2012) that the agricultural development evidence base needs to be broadened beyond the views of evaluation experts. It is encouraging that all of these ‘most important’ factors can be influenced by research program managers, or project leaders and researchers, during project design and implementation.

This study also provides further evidence of the linkages between the identified success factors and the success of research projects. It is likely that the effectiveness of international collaborative research projects in forestry and similar sectors could be improved if research program managers and project leaders considered which of these factors might be most relevant to a particular project, and then took appropriate action to address the relevant factors during project design and implementation. Collaborative research projects, in either the forestry or other sectors, are not limited to the international level; for example, both Australia and Germany have Cooperative Research Centre programs (Turpin et al, 2011; Schröder et al, 2014). It would be informative for further research to explore the application of relative success evaluations, the generality of the factors identified here, and our understanding of how identified success factors relate to the success of projects, in different national and international contexts.

Note

  1. 1.

    Researchware, Inc. (http://www.researchware.com/, accessed 13/06/2014).

     

Notes

Acknowledgements

The support of ACIAR’s previous and current Chief Executive Officers, Dr. Nick Austin and Prof. Andrew Campbell, to undertake postgraduate study and to allow access to all the research project records is gratefully acknowledged. A/Prof. Lorrae van Kerkhoff and Prof. Peter Kanowski (from Australian National University) and Prof. Neil Byron (from University of Canberra) provided academic guidance; Prof Kanowski provided extensive constructive input to various drafts. The 33 scientists interviewed provided insightful comments on the research projects used in the study. We thank the two external reviewers for their constructive comments, and Clive Hilliker for improving some of the figures and tables.

References

  1. ACIAR. (2014). ACIAR Strategic Plan 2014–18. Canberra: Australian Centre for International Agricultural Research.Google Scholar
  2. Allen Consulting Group. (2012). The Economic, Social and Environmental Impacts of the Coopertative Research Centres Program: A Report to the Department of Industry, Innovation, Science, Research and Tertiary Education. Canberra: Allen Consulting Group.Google Scholar
  3. Arnold, E. (2012). Understanding long-term impacts of R&D funding: The EU framework programme. Research Evaluation 21: 332–343.CrossRefGoogle Scholar
  4. Bartlett, A.G. (2016a). Evaluating relative success of donor-funded collaborative research projects. Research Evaluation 25: 405–415.Google Scholar
  5. Bartlett, A.G. (2016b). The evolution and impacts of ACIAR’s forestry research program over three decades. Australian Forestry 79: 171–188.CrossRefGoogle Scholar
  6. Bartlett, A.G., Kanowski, P.J., Van Kerkhoff, L. and Byron, R.N. (2017). Identifying factors that influence the success of forestry research projects implemented in developing countries: case study results from Vietnam. Forestry 90: 413–425.CrossRefGoogle Scholar
  7. Baynes, J., Herbohn, J., Smith, C., Fisher, R. and Bray, D. (2015). Key factors which influence the success of community forestry in developing countries. Global Environmental Change 35: 226–238.CrossRefGoogle Scholar
  8. Buxton, M. (2011). The payback of ‘Payback’: Challenges in assessing research impact. Research Evaluation 20: 259–260.CrossRefGoogle Scholar
  9. Byron, N. (2001). Keys to smallholder forestry. Forests, Trees and Livelihoods 11: 279–294.CrossRefGoogle Scholar
  10. Cunningham, A.B., Ingram, W., Daos Kadati, W., Howe, J., Sujatmoko, S., Refli, R., Liem, J.V., Tari, A., Maruk, T., Robianto, N., Sinlae, A., Ndun, Y., Made Maduarta, I., Sulistyohardi, D. and Koeslutat, E. (2011). Hidden Economies, Future Options: Trade in Non-timber Forest Products in Eastern Indonesia. ACIAR Technical Reports No. 77. Canberra: Australian Centre for International Agricultural Research.Google Scholar
  11. Diallo, A. and Thuillier, D. (2004). The success dimensions of international development projects: The perceptions of African project coordinators. International Journal of Project Management 22: 19–31.CrossRefGoogle Scholar
  12. Eyles, A., Beadle, C., Barry, K., Francis, A., Glen, M. and Mohammed, C. 2008. Management of fungal root-rot pathogens in tropical Acacia mangium plantations. Forest Pathology 38: 332–355.CrossRefGoogle Scholar
  13. FAO. (2014). Contribution of the forestry sector to national economies, 1990–2011. In: A. Lebedys, A. and Y. Li (eds.) Forest Finance Working Paper FSFM/ACC/09. Rome: Food and Agriculture Organization of the United Nations.Google Scholar
  14. FAO. (2015). Global Forest Resources Assessment 2015: Desk Reference. Rome, Italy: Food and Agriculture Organization of the United Nations.Google Scholar
  15. Francis, A., Beadle, C., Puspitasari, D., Irianto, R., Agustini, L., Rimbawanto, A., Gafur, A., Hardiyanto, E., Junarto, Hidyati, N., Tjahjono, B., Mardai, U., Glen, M. and Mohammed, C. (2014). Disease progression in plantations of Acacia mangium affected by red root rot (Ganoderma philippii). Forest Pathology 44: 447–459.Google Scholar
  16. Haddad, L. (2012). Five assumptions of dominant thinking in international development. Development 55: 34–44.CrossRefGoogle Scholar
  17. Harwood, C.E. and Nambiar, E.K.S. (2014). Sustainable plantation forestry in South-East Asia. ACIAR Technical Reports No. 84. Canberra: Australian Centre for International Agricultural Research.Google Scholar
  18. Henderson, J.S. (2000). Evaluating the impact of forestry research for development. International Forestry Review 2: 191–199.Google Scholar
  19. Høgh-Jensen, H., Oelofse, M. and Egelyng, H. (2010). New challenges in underprivileged regions call for people-centered research for development. Society & Natural Resources 23: 908–915.CrossRefGoogle Scholar
  20. Ika, L.A., Diallo, A. and Thuillier, D. (2012). Critical success factors for World Bank projects: An empirical investigation. International Journal of Project Management 30: 105–116.CrossRefGoogle Scholar
  21. Ika, L.A. and Donnelly, J. (2017). Success conditions for international development capacity building projects. International Journal of Project Management 35: 44–63.CrossRefGoogle Scholar
  22. Irawan, S. and Tacconi, L. 2009. Reducing emissions from deforestation and forest degradation (REDD) and decentralized forest management. International Forestry Review 11: 427–438.CrossRefGoogle Scholar
  23. Jauhiainen, J., Hooijer, A. and Page, S.E. (2012). Carbon dioxide emissions from an Acacia plantation on peatland in Sumatra, Indonesia. Biogeosciences 9: 617–630.CrossRefGoogle Scholar
  24. Khang, D. B. and Moe, T.L. (2008). Success criteria and factors for international development projects: A life-cycle-based framework. Project Management Journal, 39: 72–84.CrossRefGoogle Scholar
  25. Lindner, R. (2011). The economic impact in Indonesia and Australia of investment in plantation forestry research, 1987–2009. ACIAR Impact Assessment Series Report No. 71. Canberra: Australian Centre for International Agricultural Research.Google Scholar
  26. Lindner, R., Mcleod, P. and Mullen, J. (2013). Returns to ACIAR’s investment in bilateral agricultural research. ACIAR Impact Assessment Series No 86. Canberra: Australian Centre for International Agricultural Research.Google Scholar
  27. Mcleod, L., Doolin, B. and Macdonell, S.G. (2012). A perspective-based understanding of project success. Project Management Journal 43: 68–86.CrossRefGoogle Scholar
  28. Mendham, D. and Hardiyanto, E. (2011). Realising genetic gains in Indonesia and Australian plantations through water and nutrient management. ACIAR Final Reports No. FR2011-07. Canberra: Australian Centre for International Agricultural Research.Google Scholar
  29. OECD (2005). Paris declaration on aid effectiveness. Paris: Organisation for Economic Cooperation and Development.Google Scholar
  30. Ofir, Z. (2010). Commentary on ‘The sorry state of M&E in agriculture: can people-centred approaches help?’. IDS Bulletin 41: 26–27.CrossRefGoogle Scholar
  31. Pearce, D. (2010). Lessons learned from past ACIAR impact assessments, adoption studies and experience. ACIAR Impact Assessment Series Report No 69. Canberra: Australian Centre for International Agricultural Research.Google Scholar
  32. Pearce, D. and Alford, A. (2015). Adoption of ACIAR project outputs 2015. ACIAR Adoption Studies Report No. 12. Canberra: Australian Centre for International Agricultural Research.Google Scholar
  33. Perdana, A. and Roshetko, J.M. (2015). Survival strategy: Traders of smallholder teak in Indonesia. International Forestry Review 17: 461–468.CrossRefGoogle Scholar
  34. Pinto, J.K. and Mantel, S.J. (1990). The causes of project failure. IEEE Transactions on Engineering Management 37: 269–276.CrossRefGoogle Scholar
  35. Pramono, A.A., Fauzi, M.A., Widyani, N., Heriansyah, I. and Roshetko, J.M. (2011). Managing smallholder teak plantations: Field guide for farmers. Bogor: Center for International Forestry Research (CIFOR).Google Scholar
  36. Purnomo, H., Achdiawan, R., Melati, Irawati, R. H., Sulthon, Shantiko, B. and Wardell, A. (2014). Value-chain dynamics: strengthening the institution of small-scale furniture producers to improve their value addition. Forests, Trees and Livelihoods 23: 87–101.Google Scholar
  37. Raitzer, D.A. (2003). Benefit-cost meta-analysis of investment in International Agricultural Research Centres of the CGIAR. Rome: Food and Agricultural Organisation of the United Nations.Google Scholar
  38. Rohadi, D., Roshetko, J., Perdana, A., Blyth, M., Nuryartono, N., Kusumowardani, N., Pramono, A., Widyani, N. and Fauzan, A. (2012). Improving economic outcomes for smallholders growing teak in agroforestry systems in Indonesia. ACIAR Final Reports No. FR2012-11. Canberra: Australian Centre for International Agricultural Research.Google Scholar
  39. Roshetko, J.M., Rohadi, D., Perdana, A., Sabastian, G., Nuryartono, N., Pramono, A.A., Widyani, N., Manalu, P., Fauzi, M.A., Sumardamto, P. and Kusumowardhani, N. (2013). Teak agroforestry systems for livelihood enhancement, industrial timber production, and environmental rehabilitation. Forests, Trees and Livelihoods 22: 241–256.CrossRefGoogle Scholar
  40. Schröder, S., Welter, F., Leisten, I., Richert, A. and Jeschke, S. (2014). Research performance and evaluation—empirical results from collaborative research centers and clusters of excellence in Germany. Research Evaluation 23: 221–232.CrossRefGoogle Scholar
  41. Tarigan, M., Roux, J., Van Wyk, M., Tjahjono, B. and Wingfield, M.J. 2011. A new wilt and die-back disease of Acacia mangium associated with Ceratocystis manginecans and C. acaciivora sp. nov. in Indonesia. South African Journal of Botany 77: 292–304.CrossRefGoogle Scholar
  42. Tsujino, R., Yumoto, T., Kitamura, S., Djamaluddin, I. and Darnaedi, D. 2016. History of forest loss and degradation in Indonesia. Land Use Policy 57: 335–347.CrossRefGoogle Scholar
  43. Turpin, T., Woolley, R. and Garrett-Jones, S. (2011). Cross-sector research collaboration in Australia: the Cooperative Research Centres Program at the crossroads. Science and Public Policy 38: 87–97.CrossRefGoogle Scholar
  44. Van Kerkhoff, L. and Lebel, L. (2006). Linking knowledge and action for sustainable development. Annual Review of Environment and Resources. Palo Alto: Annual Reviews.Google Scholar
  45. Weißhuhn, P., Helming, K. and Ferretti, J. (2017). Research impact assessment in agriculture—a review of approaches and impact areas. Research Evaluation rvx034–rvx034.Google Scholar
  46. Wibowo, L.R., Race, D.H. and Curtis, A.L. (2013). Policy under pressure: Policy analysis of community-based forest management in Indonesia. International Forestry Review 15: 398–405.CrossRefGoogle Scholar
  47. Yalegama, S., Chileshe, N. and Ma, T. (2016). Critical success factors for community-driven development projects: A Sri Lankan community perspective. International Journal of Project Management 34: 643–659.CrossRefGoogle Scholar

Copyright information

© The Author(s) 2018

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  1. 1.Fenner School of Environment and SocietyThe Australian National University, ACIARCanberraAustralia

Personalised recommendations