Introduction

Many developed countries fund agricultural and natural resource management programs and projects in developing countries through their official development assistance (ODA) programs. While such projects can generate significant benefits to farmers and rural communities (Raitzer, 2003; Lindner et al, 2013), the poor performance and mixed success of many ODA projects have long been a concern (Yalegama et al, 2016; Ika et al, 2012). This challenge can be exacerbated in research-for-development projects, as the relationships between research-based knowledge and action are complex and often poorly understood (van Kerkhoff and Lebel, 2006). Understanding the factors that influence project success, referred to as success factors or critical success factors (Ika et al, 2012), enhances the ability of donors and implementing agencies to realise desired outcomes (Khang and Moe, 2008). However, surprisingly little has been documented on ODA project success factors (Diallo and Thuillier, 2004; Yalegama et al, 2016), which can vary according to the type of project and stage of the project life cycle (Pinto and Mantel, 1990) and the context in which the project is conducted (Ika and Donnelly, 2017).

Research evaluation is challenging because, even in the most efficient system, there is typically a lag of many years for the full impact of the research to emerge (Buxton, 2011); hence, impact assessments undertaken soon after a project concludes tend to under-estimate research impacts (Arnold, 2012). Not all impacts are easy to measure, and therefore impact assessments mostly focus on measurable economic and social impacts, with very few addressing environmental impacts (Weißhuhn et al, 2017). It is challenging to identify factors that contribute to project success in a consistent and meaningful way in the forestry sector in general, and for forestry research projects in particular. As Henderson (2000) observes, because of the complex nature and long production cycles of forestry systems, forestry research generally requires long-term commitments and multi-faceted programs to generate substantial impacts.

Research funders may also want to compare the relative success of projects addressing different topics or conducted in different contexts, and of successive projects addressing the same topic. In this general context, Bartlett (2016a) proposed a methodology for evaluating the relative success of collaborative ODA research projects, based on scoring against eight evaluation criteria. Bartlett et al (2017) applied this methodology to a sample of Australian Centre for International Agricultural Research (ACIAR) forestry projects in Vietnam, and complemented it with interviews of project leaders and researchers, to investigate the factors perceived to enhance or diminish success of these projects. Results demonstrated how such an analysis could assist implementing organisations to improve the likelihood of project success.

The definition of ‘success’ itself can be contested and controversial (McLeod et al, 2012). In this study, success is defined in terms of two primary dimensions, specifically related to the purpose of research for development projects, drawing on the approaches used by Pearce (2010) and Bartlett et al (2017). The first dimension, termed ‘achievements’, is the extent to which planned research outputs are achieved and adopted by ‘next users’, such as the participating scientists; the second dimension, termed ‘impacts’, is the extent of the impacts resulting from wider adoption of the research outputs by ‘end users’, such as forest growers.

This paper continues this series of investigations, and reports a qualitative study involving ten collaborative forestry research projects between Australia and Indonesia supported by ACIAR. It addresses three questions: What differences exist in the level of success achieved by these projects? What are the factors that are considered by project leaders and researchers to affect the relative success of these projects? and Is there evidence that the way these factors have been managed in individual projects has affected their relative success? The results are relevant to both researchers and international development practitioners, because greater knowledge about research for development (R4D) (sensu lato Høgh-Jensen et al, 2010) project success factors can assist those responsible for project design and implementation to improve project effectiveness.

The Australian Centre for International Agricultural Research

The Australian Centre for International Agricultural Research (ACIAR) is a federally funded agency that commissions collaborative agriculture, fisheries and forestry research projects in developing countries. ACIAR funds R4D projects conducted by Australian or Consultative Group on International Agricultural Research (CGIAR) scientists working collaboratively with scientists from the partner countries to address a research priority identified by the partner country. ACIAR projects seek to generate knowledge, technologies and capacity to achieve better decision making, changed agricultural practices and policies that, in turn, generate positive scientific, economic, social or environmental impacts (ACIAR, 2014). These projects involve capacity building and research activities and, where relevant, develop an understanding of the farming and forestry systems as well as policy settings to enhance the prospects that the knowledge and technologies developed will be adopted. Over a 30-year period, ACIAR has invested over AUD 100 million to fund 150 forestry research projects, with the greatest number of projects implemented in Indonesia, Vietnam and Papua New Guinea (Bartlett, 2016b).

ACIAR regularly evaluates the impacts and effectiveness of a sample of its projects, including quantifying their economic returns (ACIAR, 2014). An ACIAR study by Pearce (2010) identified 14 factors that were considered important to achieving successful project outcomes, but it did not examine how these factors may have contributed to different levels of success in different projects.

ACIAR’s Support for Forestry Research in Indonesia

In 2011, Indonesia’s forestry sector, based on each of natural and planted forests, contributed USD 14.57 billion to the national economy (FAO, 2014). Indonesia’s diverse natural forests have been heavily exploited for timber production over the past 50 years, and rates of conversion to agriculture have been high (Tsujino et al, 2016). Nevertheless, Indonesia retains the eighth largest area of forest in the world, with about 91 million hectares (53 per cent of its land area) classified as forest (FAO, 2015).

Indonesian farmers have a long history of planting trees and allowing natural regeneration of trees on private land. Smallholders grow trees as a ‘living savings account’, though their returns are constrained by poor knowledge of silviculture, timber standards and markets, and complicated regulations governing timber trading (Roshetko et al, 2013). These smallholders supply timber to thousands of wood manufacturing industries (Perdana and Roshetko, 2015), but many of these suffer from inefficient value chains and inappropriate processing and manufacturing techniques for small-diameter logs (Wibowo et al, 2013).

Indonesia has encouraged the development of large-scale timber plantations. In 2014, the area of fast-growing acacia and eucalypt plantations was 1.5 million hectares, with 800,000 hectares located in large estates managed by plantation companies on Sumatra (Harwood and Nambiar, 2014). However, the viability of fast-growing plantations based on these exotic species is threatened, due to the increasing impacts of damaging diseases such as Ganoderma (Francis et al, 2014) and Ceratocystis (Tarigan et al, 2011), as well as restrictions on the use of peatlands (Jauhiainen et al, 2012).

ACIAR’s forestry projects in Indonesia have covered a broad range of themes in the context of forest-based development described above; they have included technical, social and policy aspects of plantation and smallholder forestry systems (Mendham and Hardiyanto, 2011; Rohadi et al, 2012), climate change (Irawan and Tacconi, 2009) and value adding of timber and non-timber forest products (Cunningham et al, 2011; Purnomo et al, 2014). From 1987 to December 2015, ACIAR completed 21 forestry research projects in Indonesia (Table 1), representing about one-fifth of all forestry projects commissioned by ACIAR over three decades (Bartlett, 2016b). An ACIAR impact study of 12 completed ACIAR forestry projects in Indonesia (Lindner, 2011) reported high returns on investment overall, but evidence of impact from only some of the projects. These results highlight the need for improved understanding of the factors that affect project outcomes and impacts.

Table 1 Summary information for ACIAR’s completed Indonesian forestry projects, with those selected for study highlighted, and those for phase 3 evaluation identified
Full size table

Methods

The methods for this study follow those developed by Bartlett (2016a) and refined in a companion study by Bartlett et al (2017), involving three phases as outlined below. Here, success factors, which were identified from information provided by project researchers, are considered to be factors that can enhance or diminish project success, but they are not in themselves indicators of project success. The evaluation of relative success of the case study projects was undertaken by the author prior to identification of the success factors, using information from a variety of sources in ACIAR project records, as described below. The research protocol was approved by the Australian National University Human Ethics Committee (protocol no. 2014/051).

Selection of Projects for the Case Study

Ten of the 21 ACIAR forestry projects completed in Indonesia between 1987 and 2015 (Table 1) were selected for the study, taking into account the following factors:

  • Focussing on medium to large research projects conducted entirely in Indonesia; these included some projects that were part of a longer-term program;

  • Ensuring representation of projects from across the ten research themes, five of which were represented;

  • Including some projects commissioned through the CGIAR international agricultural research centres;

  • Having adequate project records available for analysis and being able to locate researchers involved in a project for interview.

In this sample, eight projects were led by Australian research agencies and two by CGIAR centres. Each project involved collaboration with scientists from various Indonesian partner organisations, including the national Forestry Research and Development Agency (FORDA), universities, non-governmental organisations and private-sector companies. The selected projects included two that continued long-term research commenced in three earlier projects and included many of the same project team members. One of these successor projects combined research on tree diseases and plantation productivity previously undertaken in two separate projects.

Phase 1: Identification of Project Success Factors

Thirty-three scientists from a range of partner organisations were identified for interview from records of the ten projects. They were selected using a purposive strategy because they had worked as project leaders, Indonesian project coordinators or collaborating researchers on one or more of the selected projects, and were still able to be contacted. The interviewees comprised 7 scientists from Australian agencies, 9 scientists from the CGIAR centres and 17 scientists from Indonesian partner agencies. They were interviewed individually by the author using a standard set of questions (see Bartlett et al, 2017), which asked them to describe what they thought constituted success for an ACIAR project, and to nominate five factors that can enhance, and five factors that can diminish, project success. Their views on aspects of the design and implementation of each project, and other contextual factors, were also sought.

HyperRESEARCH1 qualitative data analysis software was used to assist analysis of interview data by aggregating responses to specific questions into single reports and searching the data for commonly used phrases and similar concepts. This enabled the author to establish participants’ perspectives on the definition of project success, and facilitated aggregation of thematic aspects of the responses into two lists, of factors that either enhance or diminish project success. Participants’ responses about factors affecting project success and about each project’s design and implementation were analysed and results were aggregated into two groups: those from the Indonesian participants, and those from the Australian and CGIAR participants. The frequency with which each success factor was identified by each group was recorded, and complementary expressions of the same factor from the two lists identified, as the basis for preparing concisely worded statements of the factors identified as enhancing or diminishing project success.

Phase 2: Evaluation of Relative Success of the Case Study Projects

In this study, the relative success of each of the ten projects was evaluated using qualitative data, drawn from internal ACIAR project records, and the score-card matrix methodology described by Bartlett (2016a). The records included: project documents; annual reports; annual assessments and mid-term reviews conducted by the program manager; final reports; external end-of-project reviews; adoption studies and external impact assessments; project-related publications; and written correspondence between ACIAR and project staff. These data provided a degree of triangulation by presenting the perspectives of research program managers and external reviewers of projects, as well as those of project participants.

As explained by Bartlett (2016a), scores were assigned for four criteria related to research achievements: project design, results achieved, collaboration and publications, and for four criteria related to research impacts: capacity building outcomes, scientific outcomes, economic outcomes and social and policy outcomes. For each criterion, the available evidence was considered and a score assigned by the author, to the nearest 0.5, up to the maximum score. The types of evaluation questions, maximum scores and nature of the evidence sought are presented in Table 2. Scores totalling ten were assigned for each of research achievements and research impacts. Scores of 0.0–5.0 were categorised as low achievements or low impacts; scores of 5.1–10.0 were categorised as high achievements or high impacts. This classification generates four categories of project success: high achievements/high impacts, high achievements/low impacts, low achievements/low impacts and low achievements/high impacts. A companion study (Bartlett et al, 2017) demonstrated this categorisation to be helpful in relating success factors to levels of relative success.

Table 2 Evaluation questions, maximum scores and evidence guidance for the eight project evaluation criteria
Full size table

Phase 3: Identification of Relationships Between Success Factors and the Level of Relative Success Achieved by Different Projects

Three projects representing different success categories were selected (Table 1) for a more detailed analysis, with supporting information presented in Appendix 1.

For each selected project, interview responses from the project leader and two Indonesian participants were further analysed to identify any references to the way the success factors identified in the phase 1 analysis had enhanced or diminished success. The ACIAR project records were reviewed to identify evidence about the way these success factors may have influenced the project’s success. Using these two sources of information, subjective ratings were assigned for the apparent influence of each of these success factors on the project’s success. The following five-category rating system was used:

Strongly enhances—presence of factor appears to have strongly enhanced success

Enhances—presence of factor appears to have enhanced success

Neutral—no evidence that the factor enhanced or diminished success

Diminishes—absence of factor appears to have diminished success

Strongly diminishes—absence of factor appears to have strongly diminished success.

Results

Interpreting Success in a Collaborative Research Project

The views expressed by participants on what constitutes project success varied considerably, with some articulating factors that influence success rather than what success meant to them. Several participants noted that an individual project in a long-term program of research could be considered successful even if the project outputs could not be widely adopted at the end of the project. The thematic analysis enabled a common definition of success to be developed from participants’ responses: a successful ACIAR forestry research project in Indonesia was one which uses good but flexible scientific methods to achieve the planned outputs, enhances the capacity of partners, facilitates ongoing scientific networks, and disseminates the results to achieve impacts for the intended beneficiaries.

Identification of Success Factors

The thematic analysis of participants’ responses on the factors that can enhance or diminish project success identified 26 factors that were considered to enhance, and 29 factors considered to diminish, project success; when taken as a whole, there were 30 different factors identified that influence project success (Table 3). While most factors which diminish success were the converse of those that enhance success, there were three factors identified that diminish success (continuity of partner institutions and team; experience of project leader in country; external factors: policies, markets, environmental, security) and one factor that enhances success (collaboration with international scientists), for which there was no converse factor identified by participants.

Table 3 Success factors, showing participants’ views on aspects that enhance or diminish project success
Full size table

The 17 Indonesian participants and the group of 16 Australian and CGIAR participants generated a total of 424 responses related to individual success factors. The frequency of identification of each of the 30 factors considered to enhance or diminish project success is shown in Figure 1. The two most frequently identified factors, which together represented 18 per cent of the responses, were: scientists’ commitment, collaboration and focus (40 responses); and collaborative scoping and design (35 responses). Thirteen of the success factors (nos. 1–7, 9, 12, 14, and 23–25) together represented 74 per cent of the responses, and so can be considered as the most important factors affecting project success identified in this study.

Figure 1
figure 1

Frequency of identification of the 30 factors considered by the Indonesian and the Australian/CGIAR groups of respondents to enhance or diminish project success.

Full size image

Most of the success factors were consistent across the two country groups (Indonesian and Australian/CGIAR), but some differences were apparent. Indonesian participants more frequently identified success factors such as: scientists’ commitment, collaboration and focus; funding, facilities and equipment; effective communications and research networks; and engagement with the private sector. Australian/CGIAR participants more frequently identified key success factors such as: selection and commitment of partner institutions; publication and dissemination of results; and engagement of policy actors. These differences probably reflect a combination of cultural, experiential and institutional differences between the two groups of researchers, as well as the different challenges each experienced in conducting research projects in the Indonesian context. The Indonesian scientists placed a stronger emphasis on having staff that were committed, adequate funding and good communications within the team, while the Australian and CGIAR scientists were more concerned about the importance of institutional commitment and effective dissemination of results, including into the policy arena.

Evaluation of the Relative Success of Projects

The results of the evaluation of project achievements and project impacts for each of the 10 case study projects are shown in Figure 2. The 10 projects had different levels of apparent success, with scores ranging from 3 to 9 for research achievements and 2 to 7 for research impacts. In the evaluation of research achievements, nine projects (90 per cent) received scores of more than five, whereas in the evaluation of research impacts only four projects (40 per cent) received scores of more than five. Only four projects (40 per cent) achieved scores of more than five for both achievements and impacts. The evaluation methodology proved informative: even when projects received the same overall evaluation scores for research achievements and research impacts (as was the case for FST/2006/117 and FST/2007/119), they received different scores for the constituent criteria.

Figure 2
figure 2

Overall and constituent project achievement and impact scores for the 10 case study Indonesian forestry projects.

Full size image

The case study projects represent three categories of project success (Figure 3): one project with low achievements and low impacts, five projects with high achievements but low impacts, and four projects with high achievements and high impacts. No projects were categorised with the unlikely combination of low achievements yet high impacts.

Figure 3
figure 3

Case study project impact and evaluation scores and assignment to success categories.

Full size image

This study showed that subsequent projects on the same research topic may not always result in improved achievements and impacts compared with those from a precursor project. There were two projects that directly followed on from other projects: Project FST/2008/030 continued research on community forestry commenced in FST/2003/025. FST/2009/051 was a multidisciplinary project that continued research on plantation productivity and tree diseases commenced under two separate projects (FST/2004/058 and FST/2003/048). The results of the relative success evaluations for these related projects are shown in Figure 4.

Figure 4
figure 4

Changes in relative success of related projects.

Full size image

A project which commenced long-term research on root rot disease (FST/2003/048) received a high score for research achievements but a low score for research impact. The research was continued in a successor project (FST/2009/051) which received a similar evaluation score for achievements but a higher score for impacts, driven by increased scientific impacts from the ongoing research. Conversely, this same project (FST/2009/051), which also continued research on productivity of short-rotation plantations commenced under another project (FST/2004/058), achieved lower scores for both achievements and impacts than were achieved in that precursor project. The reason for this ‘unexpected’ result was that ACIAR combined the two different research themes into one project but did not provide sufficient financial resources to support all the required research activities. A third project (FST/2008/030) continued research on community forestry commenced in another project (FST/2003/025). Both projects received similar scores for research impacts, but the successor project had a higher score for achievements, as improved collaboration within the team led to completion of a higher proportion of planned activities and more publications.

A project which researched the application of molecular markers in tree breeding (FST/2000/122) received low scores for both research achievements and research impacts, reflecting an inadequate project duration of only 2 years with no follow-on phase of research. However, the Indonesian partner was still using the scientific capacity some 12 years after the project concluded, demonstrating that a relatively unsuccessful project may result in some enduring impacts. The finding on the importance of having long-term funding commitments for research programs to achieve substantial impacts is consistent with the findings of other studies of collaborative research endeavours, including an evaluation of Australia’s Cooperative Research Centre program (Allen Consulting Group, 2012).

Evidence of Success Factors in Selected Projects

The author assessed the apparent influence of each of the 30 success factors identified by project participants (Table 3) on the success of the three projects chosen to represent different evaluated levels of relative success (Table 1), using both interview responses and evidence from project records. This assessment is presented in Table 4.

Table 4 Expression of success factors within three projects with different evaluated levels of success, with the 13 most frequently identified factors shown in bold italics
Full size table

This analysis showed that, for the project evaluated as having high achievements and high impacts, there was good evidence that about two-thirds of the success factors had strongly enhanced the project’s success. Conversely, for the project evaluated as having low achievements and low impacts, it was apparent that about half of these factors had not been appropriately addressed and thereby had contributed to the diminished success of the project. The project with high achievements but low impacts had a lesser number of the factors that appeared to strongly enhance project success than did the project with high achievements and high impacts, and some factors, such as project duration, effective communications and monitoring and review, had contributed to diminished success. These relationships were more evident in information from the project records than from the interview responses, perhaps because the project-related interview questions did not directly address how the particular success factors may have influenced the project. These results demonstrate that project records, including external review reports, can provide evaluators with both positive and negative project performance-related information.

The analysis also showed that there is a reasonably clear relationship between the presence of those success factors which can be influenced during project design (nos. 1–3, 6, 7, 16, 17, 20 and 21) and evaluated levels of project research achievement and impact. The high achievements/high impacts project showed evidence of almost all of these factors either strongly enhancing or enhancing success, while in the low achievements/low impacts project, the evidence suggested that inadequate attention to over half of these factors had either strongly diminished or diminished success. This demonstrates the importance of careful attention to these factors in the design of research projects.

Discussion

Various authors (Bartlett et al, 2017; Baynes et al, 2015; Byron, 2001; Pearce, 2010) have examined the factors that influence the success of forestry development initiatives, and Pearce (2010) examined project-level factors that affect the success of ACIAR projects. The main purpose of studies such as these is to improve understanding of the factors that enhance or diminish success of ODA-funded projects, so that those responsible for project design and implementation can take them into account to improve project effectiveness. The findings of this study both confirm and supplement those from these previous studies.

The 14 success factors identified by Pearce (2010) as relevant to ACIAR research projects were all identified in this study, as were the 22 success factors identified in a companion study of ACIAR forestry research projects in Vietnam (Bartlett et al, 2017). The relative frequency of the factors differed between Vietnam and Indonesia, and a further eight success factors were identified by the Indonesian study participants. The apparent relationship between the presence of these success factors and the evaluated level of relative project success found by Bartlett et al (2017) for the Vietnam projects was also evident in this study.

The most notable differences in the factors identified in this study, when compared with the Vietnam study, were in the substantially increased frequency of two factors: effective communications and research networks (no. 12) and implementation flexibility, monitoring and review (no. 14), and the inclusion of three new factors in the 13 most frequently identified factors, viz. local government and community support (no. 23), engagement with the private sector (no. 24) and publication and dissemination of results (no. 25).

The eight success factors identified for the first time in this study were:

Local government and community support (no. 23)—this reflects the decentralised responsibility for forestry in Indonesia, and the need to have active participation of communities and smallholders to enhance the prospects of adoption of the forestry innovations from many projects.

Engagement with the private sector (no. 24)—this recognises the importance of the private sector in both smallholder and industrial forestry systems in Indonesia, and reflects a research focus on topics relevant to these systems: plantation productivity, disease management, timber and non-timber value chains and wood processing.

Publication and dissemination of results (no. 25)—this reflects the desirability and challenges of preparing and disseminating scientific articles and appropriate extension materials within the timeframe of a research project, in a research system that did not historically have a strong emphasis on academic writing, particularly in English.

External factors: policies, markets, environmental, security (no. 26)—this reflects a range of factors that are outside the control of projects but can affect project achievements, including unsupportive policies, access to markets, unforeseen diseases, natural disasters and political or security issues that limit travel to research sites.

Engagement of policy actors (no. 27)—this recognises that, in Indonesia’s dynamic and decentralised political system, it can be difficult for researchers to achieve effective engagement with relevant policy actors.

Willingness to adopt innovation (no. 28)—this reflects the constraints on the capacity of some end users, including smallholders and small enterprises, to adopt innovations, for example because of risk aversion or lack of access to the finance needed to utilise a technology.

User champions (no. 29)—this reflects the benefits that can arise from having effective user champions actively engaged in a research project and, conversely, the challenges that exist when such champions are not present or are unable to lead adoption.

Collaboration with international scientists (no. 30)—this reflects the benefits that come from networking and collaboration with skilled international scientists and the challenges that many developing-country scientists have in accessing or capitalising on such collaborations.

These results illustrate how the factors that influence project success may be both common and different between projects; For example, the factor ‘mutual benefit of research topic’ was not considered to have influenced the success of a teak agroforestry project, whereas its absence was considered to have diminished success in a molecular marker project. Differences are likely to be attributable to both differences in the nature of the research itself, and in the local contexts within which the research and adoption occur. This shows the importance of having a flexible, content-driven approach to considering the relevance of and managing the individual success factors during project design and implementation, rather than a pre-determined list that is presumed to apply universally. While some of the identified success factors are closely related, for example ‘collaborative scoping and design’ and ‘mutual benefit of research topic’, they have been listed separately so that the subtle differences can be considered, as appropriate.

The identification in this study of the three new frequently identified success factors (nos. 23–25), which relate to engagement of relevant stakeholders beyond the project team and publication of project results, is also important. The identification of the factor expressed as publication and dissemination of results refers to preparation of a range of communications materials, such as journal articles, technical reports, information and policy briefs, training manuals, field guides, websites and blogs. It also relates to ensuring that the information is effectively disseminated to the stakeholders, who either will benefit directly from the research findings or have responsibilities for policies or programs that affect adoption of research findings. This finding is likely to reflect both the strong pressures on Australian, international and Indonesian scientists to publish research results, as well as the recognition that the results have to be appropriately communicated to end users to facilitate adoption. The identification of factors related to engagement with key external stakeholders – the private sector, policy actors, local communities and user champions – emphasises the importance of factors that facilitate the relevance of research to, and knowledge of research results by, their ultimate users. This in turn is likely to affect the prospects for adoption and thereby the magnitude of the impacts from the research investment.

In this study, over 80 per cent of the factors identified as affecting project success, including all of the 13 most frequently identified factors, relate to either project design or project implementation. Therefore, paying close attention to success factors related to project design, particularly the degree of collaboration with partners on project design, the quality of the research design, the selection and commitment of partner organisations and the time allocations for the collaborating scientists, is likely to enhance prospects of the project’s success. Likewise, project success will also be influenced by how well project teams pay attention to those success factors that can be influenced during project implementation. The most important of these factors are the commitment, focus and collaboration of the partner scientists, the effectiveness of leadership and communication processes, the degree of capacity building undertaken, and the flexibility the project has to modify its activities and approaches in response to feedback from monitoring and review.

Conclusions

Since the agreement of the Paris Declaration on Aid Effectiveness (OECD, 2005), and in the context of significant global negative externalities such as climate change and the global financial crisis (Haddad, 2012), there has been an increased interest in understanding both how aid effectiveness is evaluated and which factors contribute to the success of aid programs and projects. As Ofir (2010) notes, there is a need for deeper understanding of the essential and sufficient conditions for success, and also of the context necessary to achieve successful implementation and sustained impacts from agricultural research. This study has contributed to this learning, both by reinforcing the conclusions of an earlier companion study in Vietnam (Bartlett et al, 2017) and by broadening the understanding of which factors enhance or diminish the success of international collaborative forestry research projects. This study also demonstrated the utility of conducting evaluations of the relative success of related projects, through the finding that subsequent projects on the same research topic do not necessarily result in improved achievements and impacts relative to a precursor project.

As in the companion study, the results from this study suggest that there was a good convergence of assessment amongst project participants about the most important factors influencing project success, with about three-quarters of the responses relating to 13 of the identified success factors. This suggests that the majority of research project participants have a good understanding of the factors that influence the success of collaborative forestry research projects, which is consistent with the view of Haddad (2012) that the agricultural development evidence base needs to be broadened beyond the views of evaluation experts. It is encouraging that all of these ‘most important’ factors can be influenced by research program managers, or project leaders and researchers, during project design and implementation.

This study also provides further evidence of the linkages between the identified success factors and the success of research projects. It is likely that the effectiveness of international collaborative research projects in forestry and similar sectors could be improved if research program managers and project leaders considered which of these factors might be most relevant to a particular project, and then took appropriate action to address the relevant factors during project design and implementation. Collaborative research projects, in either the forestry or other sectors, are not limited to the international level; for example, both Australia and Germany have Cooperative Research Centre programs (Turpin et al, 2011; Schröder et al, 2014). It would be informative for further research to explore the application of relative success evaluations, the generality of the factors identified here, and our understanding of how identified success factors relate to the success of projects, in different national and international contexts.

Note

  1. 1.

    Researchware, Inc. (http://www.researchware.com/, accessed 13/06/2014).