Background

Writing a thesis is a fundamental step for post-graduate studies globally [1]. This process inculcates knowledge and skills for scientific enquiry, critical thinking, systematic problem-solving, and appraisal of scientific and lay claims. This if often followed by dissemination of the thesis results to the scientific community, of which publication in a peer-reviewed journal is the highest and most respected form [2]. However, how much of this thesis work appears only in theses, and its citation particularly in public policy-related work, in low- and middle-income countries is the subject of our systematic review.

Our first aim was to conduct a systematic review of literature that identifies and assesses the effectiveness of approaches that increase productivity (proportion published) or the application (proportion cited) of post-graduate students’ research. Our second aim was to assess the determinants of post-graduate students’ research productivity and use.

Methods

We registered our protocol a priori in PROSPERO (CRD42016042819) and thereafter published it in a peer-reviewed journal [3]. We thus present an overview of the methodological approach and highlight differences between the protocol and the actual conduct of this systematic review.

Search strategy

Electronic search

We report the electronic search for the PubMed/Medline database only (https://www.ncbi.nlm.nih.gov/pubmed/). We combined terms using Boolean logic ‘OR’ for synonyms and ‘AND’ across elements of PICOS (Population Intervention Comparison Outcome and Study design), as follows: ‘medicine’, ‘nursing’, ‘dentistry’, ‘pharmacy’ and ‘public health’ described the professional fields of interest; while ‘degree’, ‘doctor’, ‘post-doctor’, ‘master’, ‘fellow’, ‘resident’, ‘student’, ‘trainee’, ‘graduate’ and ‘post-graduate’ were intermediate transitional terms for the population of interest, setting or interventions. The terms describing the interventions of interest included were ‘mentor’, ‘grant’, ‘fund’, ‘supervise’, ‘workshop’, ‘seminar’, ‘conference’, ‘manuscript writing’, ‘scientific writing’, ‘academic writing’, ‘scholarly writing’, ‘grants writing’, ‘capacity-building’ and ‘research’.

We defined ‘productivity’ as the proportion of dissertations from which at least one manuscript was published and ‘use’ as the proportion of dissertations cited in peer-reviewed articles, technical reports or policy-related documents. Thus, the search terms for the outcome of interest were ‘abstract’, ‘thesis’, ‘dissertation’, ‘publication’, ‘poster session’, ‘poster presentation’, ‘book chapter’, ‘technical report’, ‘policy brief’, ‘policy dialog’, ‘evidence-informed policy’, ‘evidence-based policy’, ‘evidence-informed health policy’, ‘evidence-based health policy’, ‘decision-making’, ‘policy-making’ and ‘dissemination’.

We restricted these search terms to the title or abstract, and included terms for the outcomes to maximise relevance and efficiency. Further, in order to minimise the risk of an empty review, we did not enter specific terms for the study design as we intended to use all evidence types to describe the available range of interventions. See the full search string in Additional file 1. We found articles in French, Persian and Spanish, and used Google translator (https://translate.google.com/) for English translations during screening and full-text review.

Additional searches

In our targeted search, we screened the reference lists of included publications and retrieved full texts of articles likely to be eligible for inclusion. We contacted authors of included articles for any literature that they may be aware of.

Selection of studies

Data management

Using EndNote software version X7 (Thomson Reuters, 2015) we imported all identified titles, excluded duplicates, and screened and grouped these into relevant eligibility categories as described in our Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) flow chart [4].

Minimising bias in study identification and selection

A second reviewer (RS), an Information Scientist, validated the electronic search in PubMed by performing an independent and duplicate search. Similarly, a third reviewer (MO), screened all full texts excluded by the first reviewer (EAO). We resolved differences by discussion and consensus.

Criteria for considering inclusion of studies

We included published studies reporting at least one outcome of interest, and reporting on post-graduate research conducted in a low- and middle-income country.

Exclusion criteria for ineligible studies

Our exclusion criteria were studies about research conducted by Bachelor’s degree or undergraduate students’ or established university faculty not identified as post-graduate students, studies conducted in high-income settings or in low-income settings but by students from high-income settings, qualitative designs, non-empirical studies, syntheses, editorials or perspectives, studies published earlier than 1990, or those that were totally irrelevant.

Data abstraction

We adapted a data extraction tool for observational studies we had developed for a previous systematic review [5]. We then abstracted administrative, study design and primary outcome data on productivity measured as proportion of publications. We further abstracted our secondary outcomes of use of the research as measured by citations, time to publication, conference abstracts and additional outcomes describing the nature of the post-graduate students’ research (predominant types of research, first authorship status of student).

Handling of missing data

We denoted variables that were missing as not reported. We did not employ any statistical methods for handling missing data, neither did we contact authors for additional information.

Risk of bias of assessment of included studies

We adapted a tool we used in a published systematic review to assess for the risk of bias in the included studies [5]. We considered the following seven aspects of bias [6, 7]: selection bias due to sampling or proportion of responders or baseline characteristics (and confounding), detection bias due to reliability of measurements used, and bias due to method of data analysis used for overall outcome and reporting biases. We categorised risk of bias as high, moderate or low, guided by the descriptive assessment questions in our tool (Additional file 2).

Synthesis of included studies

We employed a structured synthesis in which the units of analysis were findings from a single primary study. First, we described the characteristics of the included primary studies. Using the command ‘metaprop’ in Stata version 14.1 (Stata, College Station, Texas, USA), we constructed forest plots for proportions of published post-graduate work for both fixed and random effects models. The kind of data we abstracted would not permit assessment of the measures of effect, comparing two groups. We thus evaluated single group prevalence of publications, conference abstracts and predominant types of studies of the post-graduate work. We visually explored heterogeneity by inspecting the forest plots and statistically quantifying this using the I-squared statistic and tested for significance using Cochran’s Q. As we found a high level of heterogeneity we conducted a meta-regression testing the variables of duration of study, period of study and geographical region, before conducting a sensitivity analysis by excluding the very large study contributing 93% of the overall combined information. Finally, we wrote a narrative synthesis for the results for which we were unable to perform quantitative meta-analysis.

Results

Differences between the published protocol and the actual study

In conducting this review, we employed practical approaches to circumvent unanticipated methodological challenges. We did not contact heads of academic or research departments in target universities as key informants, nor did we search additional grey literature-specific databases due to resource and time constraints of completing this doctoral project. Secondly, we did not find a single article reporting on the effects of strategies to increase productivity or increase the use of post-graduate students research and therefore synthesis was based on other relevant outcomes to map the field for future studies. Third, we did not assess the overall quality of evidence, as we did not find effectiveness studies or studies with comparison groups testing interventions for increasing productivity or use of post-graduate students research [8].

Systematic review flow, screening and inclusion

Our results are illustrated in the PRISMA flow chart (Fig. 1) and in Table 1, while Table 2 shows a summary of the risk of bias assessments. We retrieved 5080 titles and abstracts from two databases, PubMed (n = 3848) and ERIC (n = 1232). We updated the search in PubMed only and findings are as recent as July 17, 2017, as it provides the bulk of health-related literature. After excluding 33 duplicates, we screened all 5047 titles or abstracts, excluding 5012 mainly due to irrelevance (n = 4659) or various reasons not meeting eligibility criteria (n = 353). We retrieved a total of 44 full text articles, of which 9 were from additional targeted searching; we finally reviewed 14 of them, with only 12 in the statistical meta-analysis. The main reason for excluding the full texts was not containing at least one outcome of interest and the two studies were dropped in the meta-analysis either because they lacked the primary outcome [9] or the primary outcome was not reported in a way to permit synthesis [10].

Fig. 1
figure 1

Flow diagram for systematic review on productivity of post-graduate students’ research

Full size image
Fig. 2
figure 2

Meta-analysis of productivity of post-graduate students’ research in low- and middle-income countries

Full size image
Table 1 Productivity of post-graduate students’ research in low- and middle-income countries
Full size table
Table 2 Risk of bias assessments for studies on productivity of post-graduate students’ research in low- and middle-income countries
Full size table

Description of included studies

The studies we included were from Asia (n = 7), Africa (n = 5) and South America (n = 2), published between 2007 and 2014, including post-graduate students over a 30-year span (1974 to 2014). The smallest study had a sample size of 90, while the largest had 22,625. All studies were about post-graduate students pursuing Masters’ degrees, of which 4 included Doctoral students as well. Importantly, although many studies included cohorts of post-graduate students, all used a cross-sectional analysis. We included 1 study that did not report the primary outcome but described other secondary outcomes, and 7 studies that reported at least an additional outcome of interest.

Findings on the outcomes of interest

Publication proportion of post-graduate students’ research

We included 12 of the 14 review studies in the meta-analysis for the primary outcome, all together contributing work of 27,477 post-graduate students for the meta-analysis (Fig. 2). The proportion of post-graduate students research published ranged from 6% in Turkey [11] to 41% in Togo [12]. We found an average publication proportion of 7% (95% CI 7–8%; Higgins I-squared 0.0% and Cochran’s Q, p < 0.01) and 23% (95% CI 17–29%; Higgins I-squared of 98% and Cochran’s Q, p < 0.01) using fixed and random effects models, respectively.

Meta-regression and sensitivity analyses

After excluding the study by Orzgen et al. [11] as it accounted for 92% of the review sample size, the results were 20% (95% CI 19–21%; Higgins I-squared and Cochran’s Q, p < 0.01) by fixed effects and 24% (95% CI 20–29%; Higgins I-squared of 92% and Cochran’s Q, p < 0.01) by random effects.

Our meta-regression results were not significant for duration of study span, period of the research or geographical region (data not shown).

Citation of post-graduate students’ research

Two studies reported on the citation of studies, which was our surrogate for use of post-graduate students’ research. In Uganda [13], this was 17% overall (95% CI 15–19%) and 4% specific for policy-related documents, while in Turkey [11], the median citation was 1 study (IQR 0.6–2.3).

Determinants of post-graduate students’ research

Only one included study reported on the determinants of productivity or use of post-graduate students’ research, and it suggested that younger students were more likely to publish and that cohort studies were more likely to be published [13].

Additional outcomes

Six studies reported the time to publication from completion of theses by post-graduate students. The earliest average time to publication was 2 years (Cameroon, Egypt) [14, 15], while the rest were 2.3, 2.8 and 3 years in Uganda [13], India [2] and Iran [16], respectively. Only three studies reported the proportion of abstracts presented in conferences as 54% in Togo, 11% in Cameroon and 2% in Uganda [12,13,14]. Post-graduate students were first authors of their work in 23%, 54%, 62% and 80% in Cameroon, India, Egypt and Turkey, respectively, and in nearly all their papers in Uganda [2, 12,13,14, 17]. Cross-sectional studies were the predominant designs of post-graduate students’ research projects (44–80%), while randomised trials were few (3–23%) [2, 9, 13].

Discussion

Principal findings

We report that the majority of post-graduate students in low-income countries infrequently publish their research theses. Secondly, the most published studies are cross-sectional in design with hardly any clinical trials, likely because of feasibility considerations with higher logistic demands, particularly for students. Third, it is apparent that post-graduate students are not the first authors in a significant proportion of their published work. Nevertheless, the evidence of citations suggests that post-graduate students’ research work is used in some form.

Findings in relation to other systematic reviews

Although we did not identify data to support our primary objective, we found a systematic review about interventions for increasing scholarly productivity among residents in the United States and Canada, which are high-income settings [18]. This review mapped the following approaches that were associated with increased scholarly productivity: protected research time, research curricula, research directors, dedicated research days, and research tracks, but with mixed effects on resident presentations or publications. It would be important to extend these single studies by explicitly testing the approaches found to be associated with productivity in low-income countries such as Uganda.

The proportion of publications in our review was relatively low. In a seminal paper by Dickersin et al. [19], students in John Hopkins University in the 1980s were found to publish less than the faculty. Although it is not clear why students published less, we found that students were not necessarily first authors of their work even when they published. Indeed, authorship would be an incentive for career advancement for which students and faculty would benefit. It is possible that power imbalances may explain this finding and, in some instances, discourage would be student authors from publishing.

With only a single study appropriately documenting the determinants of productivity and citation, we could hardly draw firm conclusions. The results herein suggested that younger students were more likely to publish and that cohort studies were more likely to be published. Bullen and Reeve [20] documented that Master of Public Health students in New Zealand mentioned that barriers to publication included a protracted publication process and a negative perception of the importance of the results. Future qualitative investigations in this setting would be informative.

Systematic review strengths and limitations

This is the first systematic review documenting the academic research productivity of postgraduate students in low- and middle-income countries. We have employed a robust and internationally agreed methodology [21] to conduct this systematic review with a sizeable sample of included studies and students.

Nonetheless, we report two main methodological limitations in our review. First, we did not find a single study assessing the effects of interventions to increase academic research productivity of post-graduate students. This could be explained by the identification of all the relevant studies or university reports, limited by grey literature beyond the reach of our review team. Nonetheless, we employed a robust and comprehensive search strategy beyond the electronic search, which enabled us access to articles for over a 40-year span and in languages other than English. Secondly, identifying the outcome of ‘use’ of students’ research in the policy process or decision-making in health remains a challenge, with citation being an imperfect proxy. An additional limitation was the fact that we did not explore grey literature more effectively, largely due to resources constraints.

Implications for future research and policy

It is clear that there is a sheer lack of evidence to effectively assess interventions to increase productivity and use of research conducted by post-graduate students in low- and middle-income countries. The cohort studies that exist did not analyse the data in a manner that permits comparisons of groups exposed to specific approaches, an area that can be strengthened. Future studies should be prospective, employing mixed methods to investigate interventions to increase productivity and citation of post-graduate students’ research as well as to identify unique aspects such as publishing in predatory journals. Additionally, it is becoming more common for early career researchers globally to publish during their candidature (PhD by publication), an area that can be examined further. Policy-makers and actors should invest in supporting research in this area.

Conclusions

We report a low productivity of post-graduate students’ research in low- and middle-income countries, including the use of post-graduate students’ research in evidence-informed health policy. Secondly, we did not find a single study that assessed strategies to increase productivity and use of post-graduate students’ research in evidence-informed health policy, a subject for future research.