Skip to main content
SpringerLink
Log in

Scientometric impact assessment of a research policy instrument: the case of rating researchers on scientific outputs in South Africa

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

The influence of the National Research Foundation’s (NRF) rating system on the productivity of the South African social science researchers is investigated scientometrically for the period from 1981 to 2006. Their output performance is mainly indicated by their research publications. Following international best practice in scientometrics as well as the behavioural reinforcement theory, we employed the “before/after control impact (BACI) method”, as well as the well known econometric breakpoint test as proposed by Chow. We use as control group the publications in the field of clinical medicine. The field is not supported by NRF and hence clinical medicine researchers are not affected by the evaluation and rating system. The findings show a positive impact of the NRF programme on the research outputs of social sciences researchers and the implementation of the programme has increased the relevant population of research articles by an average of 24.5% (during the first 5 years) over the expected number of publication without the programme. The results confirm the scientometric findings of other studies (e.g. that of Nederhof) that ratings promulgate research productivity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. Approximately US $14,000.

References

  • Arenas, J. L. D., Valles, J., & Arenas, M. (2000). Educational research in Mexico: Socio-demographic and visibility issues. Educational Research, 42, 85–90.

    Article  Google Scholar 

  • Berkhout, F., Hertin, J., & Jordan, A. (2002). Socio-economic futures in climate change impact assessment: Using scenarios as “learning machines”. Global Environmental Change, 12, 83–95.

    Article  Google Scholar 

  • Bertrand, M., Djankov, S., Hanna, R., & Mullainathan, S. (2006). Does corruption produce unsafe drivers? National Bureau of Economic Research NBER Working Paper #12274, Cambridge, MA.

  • Bouabid, H., Dalimi, M., & Elmajid, Z. (2010). Impact evaluation of the voluntary early retirement policy on research and technology outputs of the faculties of science in Morocco. Scientometrics, 85, 125–132.

    Google Scholar 

  • Caspary, G. (2009). Assessing, mitigating and monitoring environmental risks of large infrastructure projects in foreign financing decisions: The case of OECD country public financing for large dams in developing countries. Impact Assessment and Project Appraisal, 27, 19–32.

    Article  Google Scholar 

  • Chow, C. G. (1960). Tests of equality between sets of coefficients in two linear regressions. Econometrica, 28, 591–605.

    Article  MathSciNet  MATH  Google Scholar 

  • Duflo, E., Glennerster, R., & Kremer, M. (2006). Using randomization in development economics research: A toolkit. National Bureau of Economic Research (NBER), Technical Working Paper 333, Cambridge, MA.

  • Groot, T., & Garcia-Valderrama, T. (2006). Research quality and efficiency—An analysis of assessments and management issues in Dutch economics and business research programmes. Research Policy, 35, 1362–1376.

    Article  Google Scholar 

  • Gujarati, D. N. (2003). Basic econometrics. New York: McGraw-Hill.

    Google Scholar 

  • Habib, A., & Morrow, S. (2007). Research, research productivity and the state in South Africa. Journal of Higher Education in Africa, 5, 113–130.

    Google Scholar 

  • Jain, A., Garg, Kc., Sharma, P., & Kumar, S. (1998). Impact of Serc’s funding on research in chemical sciences. Scientometrics, 41, 357–370.

    Article  Google Scholar 

  • Johnston, R. (1995). Research impact quantification. Scientometrics, 34(3), 415–426.

    Article  Google Scholar 

  • Kim, M. J. (2007). A bibliometric analysis of the effectiveness of Korea’s biotechnology stimulation plans with a comparison with four other Asian nations. Scientometrics, 72, 371–388.

    Article  Google Scholar 

  • Kremer, M. (2003). Randomized evaluations of educational programmes in developing countries: Some lessons. American Economic Review, 93, 102–106.

    Article  Google Scholar 

  • Leigh, A. (2003). Employment effects of minimum wages: Evidence from a quasi-experiment. Australian Economic Review, 36, 361–373.

    Article  Google Scholar 

  • Miguel, S., Moya-Anegon, F., & Herrero-Solana, V. (2010). The impact of the socioeconomic crisis of 2001 on the scientific system of Argentina from the scientometric perspective. Scientometrics, 85, 495–507.

    Article  Google Scholar 

  • Momtaz, S. (2002). Environmental impact assessment in Bangladesh: A critical review. Environmental Impact Assessment Review, 22, 163–179.

    Article  Google Scholar 

  • Morgan, R. K. (2008). Institutionalising Health Impact assessment: The New Zealand experience. Impact Assessment and Project Appraisal, 26, 2–6.

    Article  Google Scholar 

  • National Research Foundation. (n.d.). NRF profile, [online]. Available from http://www.nrf.ac.za/about_overview.php. Last accessed 23 June 2011.

  • Nederhof, A. J. (1988). Changes in publication patterns of biotechnologists: An evaluation of the impact of government stimulation programmes in six industrial nations. Scientometrics, 15, 475–484.

    Article  Google Scholar 

  • Nederhof, A. J. (2008). Policy impact of bibliometric rankings of research performance of departments and individuals in economics. Scientometrics, 74, 163–174.

    Article  Google Scholar 

  • Pouris, A. (1991). Effects of funding policies on research publications in South Africa. South African Journal of Science, 87, 78–81.

    Google Scholar 

  • Pouris, A. (2007). The national research foundation rating system: Why scientists let their ratings lapse. South African Journal of Science, 103, 439–442.

    Google Scholar 

  • Raitzer, D. A., & Kelley, T. G. (2008). Assessing the contribution of impact assessment to donor decisions for international agricultural research. Research Evaluation, 17, 187–199.

    Article  Google Scholar 

  • Shadish, R. W., Cook, D. T., & Campbell, T. D. (2003). Experimental and quasi experimental designs for generalised causal inference. Boston: Houghton Mifflin Company.

    Google Scholar 

  • Skinner, B. F. (1969). Contingencies of reinforcement: A theoretical analysis. Englewood Cliffs, NJ: Prentice Hall.

    Google Scholar 

  • Smith, P. E., Orvos, R. D., & Cairns, J., Jr. (1993). Impact assessment using the before-after-control-impact (BACI) model: Concerns and comments. Canadian Journal of Fisheries and Aquatic Sciences, 50, 627–637.

    Article  Google Scholar 

  • Tassey, G. (2003). Methods for assessing the economic impacts of government R&D. National Institute of Standards and Technology, [Online]. Available at http://www.nist.gov/director/planning/upload/report03-1.pdf.

  • Tertiary Education Commission (TEC) (2007). PBRF evaluating research excellence. The 2006 assessment. [Online]. Available at http://www.tec.govt.nz/Documents/Publications/pbrf-full-report-2006.pdf. Last accessed 23 June 2011.

  • Tien, F. F. (2007). Faculty research behaviour and career incentives: the case of Taiwan. International Journal of Educational Development, 27, 1–7.

    Article  Google Scholar 

Download references

Acknowledgments

The authors wish to thank Mrs A Pouris, Grants Director, NRF and two anonymous referees for their constructive comments of an earlier version of the article. The normal caveat applies.

Author information

Authors and Affiliations

  1. Department of Economics, University of Pretoria, Pretoria, 0002, South Africa

    Roula Inglesi-Lotz

  2. Institute for Technological Innovation, University of Pretoria, Pretoria, South Africa

    Anastassios Pouris

Authors
  1. Roula Inglesi-Lotz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anastassios Pouris
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Roula Inglesi-Lotz.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Inglesi-Lotz, R., Pouris, A. Scientometric impact assessment of a research policy instrument: the case of rating researchers on scientific outputs in South Africa. Scientometrics 88, 747–760 (2011). https://doi.org/10.1007/s11192-011-0440-8

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-011-0440-8

Keywords

Search

Navigation