Background

Research capacity building (RCB) develops individuals and institutions to higher levels of skill and ability to conduct quality research. It aims to improve the ability to conduct, use and promote research through providing training, funding, infrastructure, linkages and career pathways [1–4]. There is an absence of a critical mass of primary health care researchers, research activity and the ability to carry out quality research [5–7]. This is largely due to lack of research skill, time, funding and linkages to universities [8], and a continued gap between research and practice [9]. A strategic approach to RCB is needed to accommodate the complex and multi-disciplinary context of primary health care (PHC) [10].

Historically RCB interventions have focused on developing the research skills of individuals using strategies such as research skills training, quarantined time for research through Fellowships, writing scholarships, mentoring, journal clubs, and small research seeding grants [1, 8, 11–15]. Research networks have been another method used to foster research but have historically focused on assisting individual GPs to cultivate an interest in research by providing research skills training and assistance to individuals for their small research project [7, 16, 17].

More recently a 'whole of system' approach to research capacity building has been proposed which influences different levels of the organisation and allows practitioners to participate at a level that enables them to build on their existing capacity [18]. Cooke provides a framework for conducting and evaluating RCB initiatives with a set of guiding principles to be applied at individual, team, organisation and supra-organisation levels [6]. These principles are to: build skills and confidence, develop linkages and partnerships, ensure research is close to practice, develop appropriate dissemination, invest in infrastructure, and build elements of sustainability/continuity. Similarly, practice-based research networks have been recommended which provide support to research-focused practices rather than individuals [16], and existing networks which target individuals are being encouraged to support a range of practice staff and PHC professionals not just GPs [7]. In order to bring about the culture shift necessary to cultivate quality research in PHC more innovative approaches to RCB are needed [5, 10, 19, 20]; for example, strategies that support PHC teams to conduct research and the 'whole of system' approach described above [5, 6, 16, 18, 21].

Although considerable funds have been invested in RCB activities both in Australia and overseas, few studies have rigorously evaluated specific RCB interventions in primary health care [2, 8, 14, 22–24]. These evaluation studies used either the Research Spider [25] to identify improvements in individuals' research skills [14, 22], and/or traditional process and output measures such as number of conference presentations and research proposals [8, 14]. However, some also developed conceptual frameworks to qualitatively measure the impacts of their intervention [2, 23, 24].

Only two publications were found that described or evaluated the process of implementing a team or settings based approach to RCB [2, 24]. One of these papers provided a qualitative evaluation of the 'Designated Research Team' (DRT) approach, using the six guiding principles for RCB developed by Cooke to evaluate the model [2]. The other paper described the DRT approach using one team as a case study [24]. While tools measuring individual research skill exist, few tools quantitatively identify and measure changes in the organisation's capacity to support research to enable the adequate evaluation of RCB interventions adopting the whole of system approach [26].

This study addresses these gaps in knowledge by providing a team-based approach to RCB for primary health care teams using evidence-based RCB strategies and an evaluation framework using the validated Research Capacity and Culture (RCC) Tool which measures RCB at individual, team and organisation domains [26]. A more detailed description of RCC tool, including its psychometric properties, is provided in the 'Data Collection' section of the Methods. The aim of the study is to evaluate the impacts of an array of RCB initiatives, using a team-based approach to RCB, according to three domains: individual, team and organisation.

Methods

Study design

A non-randomised matched-pair trial design was used to compare the impact of a multi-strategy research capacity building (RCB) intervention. Four intervention teams were matched with four control teams on service role and size.

Study participants and recruitment

Multi-disciplinary PHC teams were recruited from within one geographical district of Queensland Health. Team leaders from multi-disciplinary allied health teams with a primary health care role were approached initially, before a meeting with the team to provide more information. Interested teams applied to participate in the intervention, providing relevant eligibility details. Eligible teams were required to have a PHC focus, have between 5-50 staff, have approval from their manager to participate in the RCB intervention, have an idea for a research project they could develop, and have at least one person in the team with some research experience. As insufficient eligible teams applied to satisfy sample size requirements it was not possible to randomly allocate to intervention or control groups. Therefore suitable control teams were recruited from other districts after matching on size and service role. Control teams completed the same self-report written survey as intervention teams prior to and after the intervention period. No other requirements were placed on controls. Written consent was obtained from individuals in both the intervention and control teams using an informed consent information package. Ethical clearance was obtained from Griffith University and each of the relevant local Queensland Health Ethics Committees.

Variations in team characteristics in relation to the eligibility criteria became evident after baseline data collection. Firstly, preliminary mentoring meetings with teams revealed that teams did not have a research project that was robust and feasible within the intervention time frame. Considerable changes were required to all research project ideas to become viable. Secondly, analysis of the baseline data revealed that most teams did not have a team member with research experience at a level enabling them to mentor others. More often they had experience collecting data for another researcher, not conducting their own ethics-approved research project.

Intervention program

The 'Designated Research Team' (DRT) approach to RCB described by Cooke provides a documented method similar to that taken in this study. The DRT approach worked with selected teams to support them to conduct a small research project, by providing protected time, skills training and mentoring [24]. Similarly, our approach was to work with selected teams to conduct research relevant to their clinical practice. A comprehensive multi-strategic package of RCB strategies previously documented in the literature [1, 8, 11–15] was used to develop research capacity and culture within the participating teams. Training and progress review activities were provided to all four teams as one larger group, enabling collaboration, networking and mutual support of projects. Our strategies included: i) specially tailored research skills training programs, timed to coincide with the appropriate phases of their projects and recorded on DVD for those unable to attend in person; ii) ongoing mentoring for each intervention team to support the research project being undertaken as part of the intervention; iii) writing bursaries to support research funding grant applications; iv) some financial support with direct research costs where grant applications were unsuccessful; v) research Fellowships (quarantined time) for one day per week for one person from each team; vi) infrastructure support such as research software, desk and computer use. All teams had good representation at training regarding developing a research idea and project proposal and all had some representation (although the proportion of staff attending varied) at other training events such as writing an ethics application, writing a funding grant application and modules in either qualitative or quantitative methods. Only one team had an experienced internal mentor in their team. This was the only team successful in obtaining external research grant funds for their project, a pilot for a randomized control trial). One other team chose to not apply for a research fellowship or writing bursary as they felt these resources were not required by their team. Financial support provided by the intervention strategy to support the team-based research projects was allocated based on required research costs for each project (ranging from approximately AU$1,000-AU$21,000).

Data Collection

Data collection for intervention teams occurred in December 2008 (baseline) and March 2010 (follow-up). Baseline and follow-up data collection occurred later for controls (March - June 2009 and May to December 2010); however, the same time period (15 months) between pre- and post-data collection phases occurred for both groups. This delay was due to the matched controls study design and the need to obtain ethical clearance at each district level, once suitable controls had been found.

Tools were specifically designed to measure the effectiveness of this RCB strategy as follows:

i.) A self-report written survey was developed and validated to measure research capacity and culture (RCC Tool) for Organisation, Team and Individual Domains (Cronbach's α = 0.95, 0.96, 0.96 respectively and test-retest reliability of intra-class correlations = 0.77, 0.83, and 0.82 respectively) [26]. For the Organisation and Team Domains the respondent ranks their organisation's or team's success/skill level for all 20 items in each domain. Each item has a response scale of 1-10 with 1 being least success/skill and 10 the highest possible success/skill level. The same response scale system applies to the individual domain where the respondent ranks his or her own success/skill level on 15 items. Data were also collected on research outputs but were not included in the analysis due to poor test-retest reliability of these indicators.

The following qualitative data (items ii) and iii) below) were collected for the purpose of providing contextual information to support the interpretation of the research findings in the discussion section of this paper. A separate paper (unpublished) reports findings from Senior Managers.

ii.) Team leaders from participating teams completed a written survey both pre- and post-intervention to identify changes in team structure, role and management that may impact on the study.

iii.) Senior managers of both intervention and control teams were interviewed both pre- and post-intervention to provide contextual information to identify organisational factors that could impact on the evaluation of the intervention.

Statistical analysis

A minimum number of four people in each of the intervention teams and four in each of the control teams was required and an overall sample of 16 in the intervention group and 16 in the control group was required to demonstrate a change of 15% with a power of 80%, a significance level of 0.05 and a standard deviation of 40. We used linear regression analysis to determine if there were significant differences in baseline domain total scores between intervention and control groups. We adjusted for potential within-team clustering by including variables identifying teams and individuals within teams in our analytic models. For each of the Individual, Team, and Organisation Domains we report the pre- and post-intervention mean for each item. To determine if the mean scores changed significantly over time when comparing control group and intervention group we used a random-effects mixed model. This type of model adjusts for the potential clustering effect of individuals within teams and enables missing data to be intrinsically imputed for those with a baseline score but no follow-up score. Models included time and treatment group as the main effects and time by treatment as the interaction term, and models were performed with and without adjusting for highest education level. Models were analysed with and without adjusting for highest level of qualification as it was thought that formal training may be a potential confounder influencing outcomes. Statistical analysis was performed in Stata v11.2 (Statacorp, College Station, TX, USA).

Results

A total of 69 primary health care professionals from multi-disciplinary teams (four intervention and four control teams) were recruited to the study. Table 1 reports the profession and highest qualification at baseline for each group. Only one intervention team had medical staff (2 Paediatricians) and the matched control team had one Paediatrician. No medical students were part of teams. Few differences were detected between groups for these characteristics. There were no statistically significant differences at baseline between control and intervention groups in total scores for each of the three domains (organisation, team and individual) (Table 2).

Table 1 Baseline Profile of Sample
Full size table
Table 2 Comparison of baseline domain scores# for intervention group compared to control group to determine if any significant differences between groups prior to intervention (using linear regression adjusting for teams and individuals)
Full size table

Table 3 displays the domain scores in the Individual Domain. There was a significantly greater improvement in the total domain scores in the Individual Domain and in six of the 15 items in that domain for the intervention group compared to the control group. These items related to writing a research protocol, securing research funding, submitting an ethics application, analysing qualitative data, writing a research report, providing advice to less experienced researchers and a trend only for writing for publication.

Table 3 INDIVIDUAL DOMAIN item and total scores for Control group compared with Intervention group (using hierarchical modelling to adjust for the potential clustering effect of individuals within teams)
Full size table

Table 4 displays the domain scores in the Team Domain. There was no significant difference in total domain score change between the intervention and control groups but there were for the following Team Domain items: "Has consumer involvement in research activities/planning"; and "Has external partners (e.g. universities) engaged in research"; and a trend of p < 0.1 for "Provides resources to support staff research training".

Table 4 TEAM DOMAIN item and total scores for Control group compared with Intervention group (using hierarchical modelling to adjust for the potential clustering effect of individuals within teams)
Full size table

Table 5 displays the domain scores in the Organisation Domain. There was no significant difference in total domain score but there were for the Organisation Domain item: "Supports a multi-disciplinary approach to research"; and trends of p < 0.1 for Organisation Domain items: "Provides adequate resources to support staff research training" and "Has funds, equipment or admin to support research".

Table 5 ORGANISATION DOMAIN item and total scores for Control group compared with Intervention group (using hierarchical modelling to adjust for the potential clustering effect of individuals within teams)
Full size table

Discussion

This study provides empirical evidence of the effectiveness of a team-based approach to RCB in primary health care. Statistically significant improvements were observed for the intervention group compared to the control group within the Individual Domain (which measures research skills and competences held by the individual) for both the total domain score and for six of the fifteen Individual Domain items. It should be noted, however, that even where there was a significant improvement in scores the actual scores generally remained low for both groups; indicating an ongoing need for RCB support for allied health professionals. Similarly, for the Team and Organisation Domains (which measure the team/organisations capacity to support research and to provide a culture that promotes research) there was no significant improvement in total domain scores but there were for two of the Team Domain items and one of the Organisation Domain items; with additional items demonstrating trends of p < 0.1 (one in the Organisation Domain and two in the Team Domain). In all three domains there were no occasions where improvements were significantly greater for the control group compared to the intervention group.

These findings demonstrate the RCB intervention targeted at teams results in considerable improvements in the research skills held by individuals within those teams and, to a lesser extent, improvements in the teams' and the organisation's capacity to support research and to provide a culture that fosters research. We believe this reduced impact on team and organisation domains was largely due to the type of interventions conducted within this RCB strategy. That is, strategies were targeted at the development of research capacity through skill development and research activity but not directly targeted at changing policy and practice of teams or the organisation. However, a lack of research champions/mentors within teams could also have hindered the development of team-level indicators of research culture. This lack of capacity is reflected across allied health and primary health care [5–7]. Strategies to improve capacity have been identified recently in a study by Pager et. al. Identified strategies include working with staff already motivated to do research and providing them with time and funds to conduct research, opportunities for career advancement linked to research, access to mentors and links to universities [27].

Strengths of the study include the use of a control group, which provides a reference for change that may have occurred regardless of the intervention. The use of a validated tool to quantitatively measure research capacity in all three domains adds to the rigor of the evaluation. The fact the intervention occurred within one large organisation only, not several organisations, limits the generalizability of our findings. However, it also provides the same broad contextual environment for both intervention and control groups which could be beneficial. Because insufficient teams applied to participate in the study we were not able to randomise participation to intervention or control groups. This resulted in a matched-controls design with a time lag between groups, which could dilute the effect of the intervention as the organisation may be further progressed in implementing its research development strategy than when data collection occurred for intervention teams. A larger sample size is likely to have resulted in more items demonstrating statistical significance as many items showed a greater increase for interventions than controls but did not reach significance. This could indicate that our study may be underpowered to detect modest, but important, differences. Due to funding limitations the duration of the intervention was less than desirable to bring about change [28]. A two year intervention is likely more realistic to support a team of novice researchers from inception to manuscript submission stage in the research process.

This RCB initiative occurred within an environment of considerable change. Interviews with senior managers of both intervention and control groups at pre- and post-intervention identified that during the intervention period of 15 months there was an organisation restructure that involved realignment of allied health divisions within the organisation to match new district geographical boundaries. Other changes during that period identified by senior managers were the on-going negotiation of a new pay award, an increased demand on services, and a resource-intensive RCB initiative undertaken by the organisation. Senior managers expressed hope that the award restructure and organisational RCB strategy would provide additional opportunity to support and recognise research; and that the realignment of divisions, along with the establishment of joint research appointments and research centres would result in greater critical mass to conduct research. They reported that unfortunately some of these strategies, in the short term at least, have resulted in unstable ground and increased work load. For example, the slowing down of systems and processes, such as getting ethical clearance; and funding allocation that does not benefit all causing dissention among staff.

At a team or service level, written surveys completed by team leaders at pre- and post-intervention for both intervention and control groups identified high levels of staff turn-over, including management staff, during the intervention period. Team leaders also reported increased workload demands without increased staffing levels, and significant changes impacting on services and staff resulting from the organisation restructure. There were no differences in circumstances for intervention teams compared with controls with the exception that two intervention teams also experienced a new model of care which resulted in a considerably increased workload. This added workload is likely to have negatively impacted on the potential for these teams to participate in the RCB activities and thus diminished potential improvements to RCB available through the initiatives. Similarly, this changing environment brings challenges to the process of supporting teams of novice researchers to conduct research; therefore, consideration to the amount and complexity of change the teams and individuals can cope with needs to be given. This context is not uncommon according to Rosenbeck, 2001 who sees organisational processes as a missing link between research and practice, stating that "complex organisations are characterised by multiple competing goals and fluid involvement of key participants". He emphasises the importance of developing self-sustaining communities of practice within these organisations [21].

The areas in which our study found significant improvements for intervention teams in the Organisation and Team Domains are consistent with those reported in the evaluation of the DRT approach [2]. Cooke et.al, 2008, found a team-based approach effective in developing linkages, collaborations and skills [2]. Similarly, our study found significantly improved resources for skills development and research, improved support for multi-disciplinary research, improved involvement of consumers in research and improved links to external partners such as universities. External linkages have also been reported as important for supporting practice-based research [29]. At an individual level our study found significant improvements in a range of indicators of research skill or capacity, particularly related to research activity skills rather than evidence based practice related skills.

Factors contributing to the success of the research and RCB outcomes for our study are consistent with those described for the DRT approach. Protected time, managerial support, applied timely research skills training, mentorship, critical mass of research experience in teams and an encouraging work environment were all identified as important factors of success [24]. Our intervention included all of these characteristics although, unfortunately there were not experienced researchers in our teams; therefore, no critical mass at baseline, as was the case for the DRT teams.

Some teams had not completed their projects by the time post-intervention data collection occurred. These teams continued on with their projects to the write up stage with minimal support, demonstrating a potential for sustainability beyond the scope of the program. This was consistent with findings from the DRT evaluation which reported that research activity was sustained beyond the intervention period and that a research question derived from clinical practice was an important factor [24].

The greatest impact of our intervention was seen in the Individual Domain. This is largely because the intervention focused more on skill development and applied research activity rather than changes to policy and practice. The feedback meetings conducted for formative evaluation during the intervention period clearly indicated teams were at their maximum capacity due to heavy workloads, so their ability to take on additional tasks such as skill development, research activity or changes to policies and infrastructure to support research was limited. This was due in part to the additional workload generated from organisational change factors. Therefore we focussed on interventions that supported teams to conduct the research project they had committed to. Feedback also highlighted the importance of having sufficient lead time working with teams to develop a viable research project with agreed tasks and time lines, and sufficient supported time to conduct the research activities with RCB support, such as research skills training at appropriate times. The complexity of conducting research in teams revealed additional strengths and challenges to the research process. For example, shared task management and research vision were strengths whereas decision-making and making time where clinicians were available for collaboration was difficult amid heavy clinical loads. Future directions in research to evaluate team-based interventions for RCB would benefit from designing a study with the following characteristics: a randomised intervention and control design using separate organisations for each team and targeting interventions to include refining research related policies and practices; a larger sample; longer intervention period and longer-term follow-up evaluation. The lack of strategies targeted at team and organisation levels are a limitation of this intervention design. However, a longer intervention period would be required to support this as teams indicated they did not have the capacity to take on additional activities beyond conducting their research project and the skills development required to support that.

We found that a team-based approach to RCB was effective for novice researchers with demanding clinical loads. The approach enabled clinicians to conduct a research project close to clinical practice and to share the responsibilities of the research among the group. The approach also enabled RCB implementers to support considerably more people to get experience in research activity than would be possible if supporting individuals each working on a separate project. With a continued need for developing research culture and capacity in PHC this study provides valuable evidence in support of an innovative and efficient approach to RCB.

Conclusions

A team-based approach to RCB is an effective and innovative method of developing research culture and capacity in PHC.