Background

Primary care research aims to improve the quality, effectiveness and cost effectiveness of primary care, in all its manifestations. An essential outcome of this research is to provide the evidence base that primary care services previously lacked [1]. While greater emphasis is given to translational research, and numerous organizations provide support to build and support research capacity in the field of health research [2], factors that influence a positive research culture in health professionals are not fully understood [3]. Different instruments have emerged to assess the topics where professionals’ skills are most lacking and for which priority must be given in the design of appropriate training courses and other capacity building initiatives [2, 4].

The “Wessex Research Network Spider (WReN) Spider” was validated in 1997 as a simple and efficient way of assessing research experience across a large multiprofessional health care group [5]. The WReN Spider, shown in Fig. 1, was designed to assess research experience in order to inform the planning of healthcare professional’s educational programs. Users rate their research experience from 1 = ‘no experience’ to 5 = ‘very experienced’ on 10 limbs (items) relating to discrete components of the research process, including writing a research protocol, analysing and interpreting research results and publishing research. It is unclear to what extent other researchers have adopted and/or possibly adapted the original WReN Spider.

Fig. 1
figure 1

The original WReN Spider as published in 2002 by Smith [5]

Full size image

Therefore, the aim of this study was to systematically review the papers which cited the original WReN Spider publication in order to investigate the uses and developments of the WReN Spider by the research community over a period of 15 years (2002 to 2017) since its development.

Methods

A literature search was conducted in February 2018 on Google Scholar, Scopus and Web of Science to identify all the citations of the original article [5] since its publication in the year 2002. These science citation indexing services were chosen as they provide comprehensive citation searches in different academic and scientific disciplines coupled with ease of accessibility and wide use [6,7,8]. In Web Of Science, the “Cited Reference Search” function was used, using the title of the original publication to search. In Scopus, the “Search Article Title” option was used, and then the function “View cited by”. The same protocol was used in Google Scholar. The search results from all three searches were collated in EndNote. To ensure we had captured all relevant articles we expanded our search with citation chaining searching backwards and forwards in time for materials that are cited by and also cited the articles we had already identified.

We searched for papers in English and those languages where we had the skills to translate within the research team (German, Spanish, French, Italian, Mandarin, Swedish, Italian, Croatian, Burmese). Papers were included if the original WReN Spider publication [5] appeared in the list of references.

We plotted the publications by year and by place before classifying them according to whether they were examples of (i) utilisation of the WReN Spider in its original or an adapted form, or (ii) merely a citation of the original WReN Spider article.

For papers reporting on (i) use of the WReN Spider we extracted the following:

  • Study setting (country & institution e.g. primary care research network)

  • Target audience (profession, number)

  • Study design (e.g. before / after evaluation, survey)

  • Type of use (e.g alone or in conjunction with other instruments, original WReN Spider or modified WReN)

    1. 1)

      original Spider alone original

    2. 2)

      original Spider in conjunction with another assessment

    3. 3)

      modified Spider alone

    4. 4)

      modified Spider in conjunction with another assessment

  • Study findings with respect to Spider measured research experience.

For papers which also adapted the WReN Spider, in addition to the above information, we also extracted the following:

  • Characteristics of modified instrument (conceptual model (if any), number of items, number of domains)

  • Validity of modified instrument.

For the papers that (ii) cited rather than used the WReN Spider in its original or an adapted form we provide a narrative summary of these.

Results

Of the 71 publications retrieved, 32 were found on Google Scholar, whilst the search in Scopus yielded 20 and the Web Of Science yielded 19 publications. In total 31 distinct papers were identified after duplicates were removed (Fig. 2). One paper was indexed in both French and English, so the French duplicate was removed. Two papers had no mention of the WReN Spider in the full-text and therefore were excluded. A further citation was excluded because it was an abstract rather than a full-text publication. Twenty eight papers were included in the final analysis (Fig. 2).

Fig. 2
figure 2

Study flow diagram indicating number of publications screened and finally included

Full size image

Most publications were in English (N = 25, 89%), with others in Chinese (N = 3, 11%). The first citation of the WReN Spider was in 2006, four years after publication of the original article. Citations peaked in 2015 with 8 (26%) (Fig. 3). Papers came from seven different countries (Fig. 4), Australia generating the most (64%, N = 18).

Fig. 3
figure 3

Number of research publications using or citing the WReN Spider per year between 2002 and 2017 (at January 18th, 2018)

Full size image
Fig. 4
figure 4

Number of research publications using or citing the WReN Spider by country between 2002 and 2017

Full size image

With respect to focus of the relevant articles, 15 (54%) papers used the WReN Spider and 13 (46%) cited it.

Papers using the WReN spider

The extracted characteristics of the 15 papers using the WReN Spider are shown in Table 1. In 8 cases the instrument was used once to assess the gaps in professionals’ research experience / skills. In 7 cases it was used pre- and post-training to evaluate the impact of the intervention; these interventions were targeted to Allied Health Professionals (e.g, nurses or occupational therapists), General Practitioners and medical graduate students.

Table 1 Summary of the 15 papers that used the WReN Spider (in chronological order)
Full size table

Only one paper was classified as category 1, using the original WReN Spider alone [9]. Three cohorts of graduating medical student used the WReN Spider for self-assessment of research experiences in the ten areas of research activity included in the original instrument. The WReN Spider was administered to each student prior to, and on completion of a community based research project.

Five studies were classifed as category 2, using the original WReN Spider in conjunction with other assessments [10,11,12,13]. Ried [10] focused on primary care professionals with research experience to evaluate the impact of a grant funding scheme on skills development using the WReN Spider, during phone interviews. As in Ried [14], professional’s level of participation in research was also assessed. Further, researchers were asked to rate the support intervention and to describe the dissemination of their research findings. Ried [11] conducted a similar study to that of 2007, using the WReN on Australian General Practitioners (GP) registrars having attended the training programme ‘Registrar Research Workshop’ to measure their research experience in the 10 WReN Spider limbs. Other measurements included free text responses to rate the workshop as well as confidence and interest in research with a five-point Likert scale. Webster [15] reported using the original WReN Spider to assess changes in the research experience of rural health workers before and after their candidature in the Rural Research Capacity Building Program. A qualitative evaluation was also conducted, aimed at gaining a better understanding of the impacts of the research program from the perspective of all involved in it. Leung [12] used the WReN Spider as a self-assessment instrument for nurses to enable the different components of a research training program to be tailored to the individual needs. The program’s impact was assessed through online questionnaires after each of the 11 sessions and at the end for the overall program. Each questionnaire encompassed 7 topics (session objectives met, session interesting and relevant, utility of knowledge gained, sharing of information planned, readings helpful, instructor evaluation, satisfaction with session) to be rated on a 5-point Likert scale. Schmidt [13] conducted a survey on trainees from the Research Capacity Building Program (RCBP) using a combination of emailed paper questionnaires and phone surveys. The trainee’s workplace managers and staff of the Centre for Research Excellence in Rural and Remote Primary Health Care Research (CRE) were also surveyed. Trainees self-assessed research experience using the WReN Spider instrument. Baseline assessment was completed after an introductory research methods workshop and was reassessedon program completion. Additional questions on support and supervision aspects of the RCBP were rated on a 5-point Likert scale, such as the manager support, supervision from the program organizer, quality of short course in research methods, peer support and teleconference support.

No study was classified as category 3, all remaining studies (N = 9) fell into category 4

Stephens [16] used the WReN Spider to assess both research experience and interest of allied health professionals (N = 132) in Australia. This quantitative data collection was followed by 17 focus groups and individual interviews with participants with the highest level of interest in developing research skills. Data from focus groups and interviews yielded themes within the Dimensional Enhancing Research Capacity (DERC) model.

Ried [14] reported the first use of the WReN Spider amongst primary care professionals since its original publication. In this study, the Spider was used to measure experience but also interest in up-skilling in the ten Spider areas. In addition participants responded to questions about their i) personal and professional background, ii) current level of participation in research and iii) publication and funding achievemnents. Short [17] conducted a mixed-methods study on clinical staff in an emergency department who were asked. All the department members (N = 67) completed a 38-item self-evaluation questionnaire utilizing the WReN Spider to evaluate experience in the ten research activities. In addtion, participants were asked about perceived benefits of and barriers to conducting research projects as well as suggestions for future research projects. A repeated WReN Spider was completed to indicate participant’s level of interest in developing each of the ten skills.

Similarily, Nonoyama [18] used the WReN Spider to measure experience plus interest in up-skilling in each of the Spider limbs in respiratory Health Professionals. The questionnaire consisted of closed ended questions, but open-ended questions were used to collect additional comments. Other information covered in the questionnaire regarded demographic information; barriers and facilitators to conducting research; future directions in respiratory research; and research funding and mentorship.

Harding [19] evaluated a 12-weeks research training scheme for allied health professionals. The WReN Spider was used to analyze changes not only in research experience in each of the 10 original limbs, but also in research interest and confidence of those enrolled in the training. In addition, qualitative methods (in-depth semi-structured interviews) were used to explore the experiences of the professionals and their mentors.

Finch [20] used the WReN Spider using a customized web-based survey to study the current levels of interest, confidence and experience performing specific research tasks of speech language pathologists. The survey included three other sections: consent, demographic information and additional research participation questions. In the latter, respondents were asked how frequently they has completed each of the 10 WReN research activities in the last 5 years.

The WReN spider was extended in three studies [21,22,23]

Harvey [21] identified the capacity to conduct research in terms of interest, experience, confidence and support needs of social work employees as part of a larger study of health practitioners. Although the survey instrument was designed by the authors, questions on research experience were based on the WReN Spider. The scale was modified for each item from rating each level of experience on a level from 1 to 5 to three-point ordinal scales referring to ‘little/ no’, ‘some’ or ‘moderate/very’ experienced. Moreover, the questionnaire consisted of 14 items. The item ‘Writing & Presenting a research report’ from the original WReN Spider was divided into two items: ‘Writing a research report’ and ‘Presenting research’ and three items were added: ‘Writing a literature review’, ‘Developing a research question’ and ‘Writing an ethics application’.

Pighills [22] surveyed occupational therapists with a questionnaire developed by the authors, including questions on research experience based on the categories in the WReN Spider. Authors added questions on experience of writing a literature review, developing research questions and writing an ethics application. The original item ‘Writing and presenting a research report’ was splitted into two components ‘Writing a research report’ and ‘Presenting research’. The expanded WReN Spider was then used to measure level of research experience and need for support in each of the 14 items. Pighills [22] justified their modifications noting they aimed to make their survery more “specific” and “include other components of the research continuum identified in the literature as areas of limited experience”. The survey developed by the authors also included demographic information, staff self appraisal (practice profile, level of research experience and need for support, perceived barriers and enablers for conducting research, anxiety about conducting research), and open ended questions.

Similarly, Pain [23] developed a survey instrument that included questions on research experience of Health Practitioners based on the WReN Spider. Some modifications to the name of the items were introduced. The authors changed the item ‘Writing a protocol’ to the more generic item ‘Writing proposals’ and the item ‘Analysing and interpreting results’ to simply ‘Analysis’, ‘Finding relevant literature’ was changed to ‘Finding literature’ and ‘Critically reviewing literature’ to ‘Reviewing literature’. Like Harvey [21] and [22], the authors split ‘Writing and presenting a research report’ in two items, ‘Report writing’ and ‘Presenting research’. Finally, three items were added: ‘Writing a literature review’, ‘Developing questions’ and ‘Writing ethics’. This group did not provide a rationale for the amendments to the original WReN Spider. This modified 14-items Spider was used for self-rating of experience and support needs. Additional questions in the online survey assessed factors influencing research engagement and participation and barriers and enablers of it.

Papers citing the WReN spider

Thirteen papers (46%) cited the WReN spider

Of the papers just citing the Spider five were from Australia, 3 from China and 2 from Germany. Holden [24], Lazzarini [25], Williams [26], Pan [27], Gong [28] and Wu [29] cited the original paper when citing contemporary instruments measuring participants’ self-rated individual research experience. These cite the fact that the WReN only includes the individual domain as a shortcoming. Janssen [30], Huber [31] and Williams [26] cited the original Spider publication in a list of published evaluation studies of research training programmes. Borkowski, [4] cites the WReN as a tool to measure research experience used in some of the studies included in their review. Holden [32] consulted the items in it when developing the individual level domain of the Research Capacity and Culture (RCC) tool. Eam [33] mentioned the WReN Spider as one of the tools consulted when developing their own instrument to measure research involvement among faculty members. Huber [2] cited the items in the WReN Spider as a way to operationalize their review of instruments to measure individual’s research experience.

These papers related to the evaluation of interventions targeted to Allied Health Professionals (N = 5, 18% of the total number of studies) and nurses (N = 3, 11% of the total number of studies) followed by podiatrists (N = 2, 6% of the total number of studies). The rest of the studies aimed to assess training of specific health professions of faculty members and medical graduate students.

Five papers [24,25,26, 32, 34] employed the validated RCC instrument, validated in Holden [32], that allows measurement of research capacity and culture at organization, team and individual levels with robust scale items. Participants rate each of the 50 items on a 10-point scale (with one considered as the lowest skills or success level and ten the highest possible skill or success level), and items are scored separately for each domain with robust scale items examining three domains. The individual domain was partly based on the ‘WReN Spider’, and items for the other domains were based on previous literature. Out of the 5 studies using the RCC, 4 were published in Australia and 1 in the UK.

Two studies used other validated instruments, the Edmonton Research Orientation Survey (EROS) and the Estabrook’s Research Utilisation Questionnaire and the Research Self-Efficacy Scales (RSES) and Scholarly Activity Scale (SAS). Janssen [30] combined qualitative interviews with different measurement instruments (the EROS and the Estabrook’s Research Utilisation Questionnaire) to measure change in attitude to and capacity of research in allied health professionals working in the acute care setting. Gong [28] used the RSES and SAS to rate the efficacy of research training for Master and PhD students.

Four papers used self-reported survey questionnaires with about 50 items, always coupled with open questions capturing participants’ comments and recommendations. Huber [31] aimed to validate the Local Health Research Capacity Strengthening (LHRCS) in four different training settings in Tanzania on either clinical research skills or on clinical skills necessary for research projects. The questionnaire, that was based on a theoretical model, was modified after the intervention to include four subscales with 19 items, three global impression items and open questions for participants’ comments and recommendations. Eam [33] used self-reported survey questionnaires in 10 Cambodian universities, containing 47 key items organized into three sections, exploring how individual factors and institutional factors affect involvement in research activities. The items in this last section were partly based on the activities included in the WReN Spider, with some modifications done on the language to ease understanding in the Cambodian academic context. Pan [27] used a survey to revise scientific research ability of nurses with self-evaluating rating scales with 37 items in 6 categories: academic writing capability, document analysis capability, capability to conduct research, capability to design research, capability to do literature review, capability to identify research problems. The survey was designed by doctors and matrons and initially included 50 items, that were reduced to 37 after a pilot with nurses. Wu [29] followed a similar approach and consulted with experts and matrons to formulate 36 items for survey, based on their experience and on a literature review. The 35 items finally included were divided into 5 categories: papers and research projects, research awards and results, capability to conduct research, capability to identify problems and professional training, communication capability.

Huber [2] conducted a systematic review to optimize and systemize future efforts in the Health Research Capacity Development (HRCD) field by providing overview of the Needs assessment, Monitoring and Evaluation (NaME) activities at the individual and organizational level with a focus on methods, instruments and instruments used. This review used the WReN Spider to operationalize the “Research” category of NaME framework used to assess each instrument. The review concluded there was a need for a coherent and transparent taxonomy of Health Research Capacity Development (HRCD) to maximize the benefits of future studies in the field.

Borkowski [4] conducted a systematic review on the research culture of allied health professionals. The review found that different instruments to measure research capacity and culture were used in the original studies reviewed, including the RCC instrument and the WReN Spider, and suggested that collaborative efforts with external partners and research leadership were needed to intensify allied health professionals research culture and capacity.

Discussion

The ‘Wessex Research Network (WReN) Spider’ is an instrument developed and validated 20 years ago to assess research experience of members of a UK Primary Care Research Network. The network had been established to stimulate research awareness and involvemement amonst health care workers in General Practices. This bibliometric study, tracing the use and development of the instrument, found 31 citations of the WReN Spider in the published literature. Of these 18 were studies which have used the Spider to assess the training needs of health professionals or were before and after studies to evaluate the impact of training initiatives. The WReN Spider has been most used in the areas where activites to grow research capacity are in the early stages of development, such as nursing and allied health professions [35,36,37,38]. Its use in Australia may be because its developer promoted its use at a national conference in 2001, when Primary Care research initiatives were in their infancy [39,40,41]. We contacted the authors of these studies who described the characteristics of the WReN Spider most valued; these characteristics were its relevance and being a “straight-forward and brief tool to use”. Whilst the WReN Spider continues to be used, it is frequently enhanced with the use of additional questions that explore the wider issues of research success, including collaborators, resource and environment and also with a qualitative component to the evaluation. Some adjustments have been made to the research activities represented in the legs of the Spider.

The WReN Spider inaugural paper presented data supporting the validity of the instrument in the setting in which it was used. Interestingly none of the subsequent studies have repeated a validation process in their local context, with perhaps cultural and linguistic differences. This may have been for purely pragmatic reasons, as validation studies are often unfunded and there may not have been the resources available. Another factor might be the misunderstanding generated by the shorthand we frequently use when describing measurement instruments. The tendency is to talk about instruments being ‘validated’ rather than the measurements generated by the instruments which are considered valid. If it is really the measurements to which claims of validity pertain then it becomes apparent that validation is a continuous process rather than a one-off activity. From a pure psychometric point of view, validation needs to be repeated in each new setting or population. Interestingly, the WReN Spider has only been used in anglophone countries. Often the need to linguistically validate a measurement instrument (i.e. to translate it rigorously in order to ensure linguistic equivalence) triggers awareness regarding psychometric validation also. This lack of awareness of the necessity for ongoing validation and a paucity of resources (time, staff, money, skills) means it will remain an often neglected area.

There is very little direct critique of the WReN Spider structure with the five point scale on ten legs. Short [17] highlights the potential risk of reports with a high rate of missing data, with as many as 12% of the participants not completing the 10 items.

Missing data may reflect that there is no option for the respondent to indicate that experience in the research area being assessed is not relevant to them. For example, the WReN Spider measures experience both in qualitative and in quantitative research, while many researchers may focus their activities on one or the other methodology. Similarly the assessment of two activities on a single scale (for example ‘writing and presenting a research report’, and ‘analysing and interpreting results’) may discourage a respondent completing the scale if they have divergent experience of these two discrete skills. Further, the role of social desirability bias arising from the desire of the respondent not to appear inexperienced is unclear. It may result in missing data or data skewed to the positive end of the response spectrum.

The WReN Spider structure has been maintained in the 10 studies that measure constructs beyond experience, including confidence or interest in up-skilling in each of the 10 items. Many of the shortcomings of the WReN Spider are expressed implicitly in the manuscripts, through the changes, additions and modifications made by other researchers to the instrument. Qualitative components added to the questionnaire have enabled a better understanding of the barriers and motivators for individuals undertaking research and the specific impact of interventions (research training and funding) on research activity.

Already we have a WReN Spider with ten appendages which challenges the physical characteristics of an arachnid with four pairs of legs, but as the research culture becomes more sophisticated we may need more legs to represent activites such as good clinical practice (GCP), good research practice and patient and public involvement (PPI) in research. As Pighills [22] reminds us, the WReN Spider is an interim or intermediate measure of success because “traditionally impact is measured by grants acquired and peer-reviewed articles published, rather than experience or skills”. The WReN Spider will continue to be a process measure, but the move away from grant income and bibliometric prowess as measures of success to valuing the health and health sector benefits of research is to be welcomed. However, it calls for the addition of yet another Spider leg, one that addresses the activities linked to the preparation of case studies summarising impact.

The original WReN Spider focused on the individual, but research is becoming much more of a team activity and a multidisciplinary activity. The need for a more holistic approach to improving the research culture of health professionals is now recognised and individuals need to be considered as part of a team and an organization. Cheetham [42] defined research culture as the “structure that gives [research behaviour] significance and that allows us to understand and evaluate the research activity.” Research culture cannot be developd merely by addressing the skills and experiences of the individual, but needs to focus on the individual and the organisation synergistically [43]. This inclusive vision of research activities has been addressed in the Australian Research Capacity and Culture (RCC) instrument which measures the research skills of individuals together with the research team’s and the organization’s capacity to support research. The RCC instrument, with its multidimensional approach to improving research culture, also provides an opportunity for participants to determine their perceptions of their team’s and organisation’s research skills. As health research becomes more integrated into industry and health care there may be a need for a supra-organisational domain to be developd to capture these activities.