, 88:747

Scientometric impact assessment of a research policy instrument: the case of rating researchers on scientific outputs in South Africa



The influence of the National Research Foundation’s (NRF) rating system on the productivity of the South African social science researchers is investigated scientometrically for the period from 1981 to 2006. Their output performance is mainly indicated by their research publications. Following international best practice in scientometrics as well as the behavioural reinforcement theory, we employed the “before/after control impact (BACI) method”, as well as the well known econometric breakpoint test as proposed by Chow. We use as control group the publications in the field of clinical medicine. The field is not supported by NRF and hence clinical medicine researchers are not affected by the evaluation and rating system. The findings show a positive impact of the NRF programme on the research outputs of social sciences researchers and the implementation of the programme has increased the relevant population of research articles by an average of 24.5% (during the first 5 years) over the expected number of publication without the programme. The results confirm the scientometric findings of other studies (e.g. that of Nederhof) that ratings promulgate research productivity.


Scientometrics Incentives Research policy Quasi experimental design South Africa 


  1. Arenas, J. L. D., Valles, J., & Arenas, M. (2000). Educational research in Mexico: Socio-demographic and visibility issues. Educational Research, 42, 85–90.CrossRefGoogle Scholar
  2. Berkhout, F., Hertin, J., & Jordan, A. (2002). Socio-economic futures in climate change impact assessment: Using scenarios as “learning machines”. Global Environmental Change, 12, 83–95.CrossRefGoogle Scholar
  3. Bertrand, M., Djankov, S., Hanna, R., & Mullainathan, S. (2006). Does corruption produce unsafe drivers? National Bureau of Economic Research NBER Working Paper #12274, Cambridge, MA.Google Scholar
  4. Bouabid, H., Dalimi, M., & Elmajid, Z. (2010). Impact evaluation of the voluntary early retirement policy on research and technology outputs of the faculties of science in Morocco. Scientometrics, 85, 125–132.Google Scholar
  5. Caspary, G. (2009). Assessing, mitigating and monitoring environmental risks of large infrastructure projects in foreign financing decisions: The case of OECD country public financing for large dams in developing countries. Impact Assessment and Project Appraisal, 27, 19–32.CrossRefGoogle Scholar
  6. Chow, C. G. (1960). Tests of equality between sets of coefficients in two linear regressions. Econometrica, 28, 591–605.MathSciNetMATHCrossRefGoogle Scholar
  7. Duflo, E., Glennerster, R., & Kremer, M. (2006). Using randomization in development economics research: A toolkit. National Bureau of Economic Research (NBER), Technical Working Paper 333, Cambridge, MA.Google Scholar
  8. Groot, T., & Garcia-Valderrama, T. (2006). Research quality and efficiency—An analysis of assessments and management issues in Dutch economics and business research programmes. Research Policy, 35, 1362–1376.CrossRefGoogle Scholar
  9. Gujarati, D. N. (2003). Basic econometrics. New York: McGraw-Hill.Google Scholar
  10. Habib, A., & Morrow, S. (2007). Research, research productivity and the state in South Africa. Journal of Higher Education in Africa, 5, 113–130.Google Scholar
  11. Jain, A., Garg, Kc., Sharma, P., & Kumar, S. (1998). Impact of Serc’s funding on research in chemical sciences. Scientometrics, 41, 357–370.CrossRefGoogle Scholar
  12. Johnston, R. (1995). Research impact quantification. Scientometrics, 34(3), 415–426.CrossRefGoogle Scholar
  13. Kim, M. J. (2007). A bibliometric analysis of the effectiveness of Korea’s biotechnology stimulation plans with a comparison with four other Asian nations. Scientometrics, 72, 371–388.CrossRefGoogle Scholar
  14. Kremer, M. (2003). Randomized evaluations of educational programmes in developing countries: Some lessons. American Economic Review, 93, 102–106.CrossRefGoogle Scholar
  15. Leigh, A. (2003). Employment effects of minimum wages: Evidence from a quasi-experiment. Australian Economic Review, 36, 361–373.CrossRefGoogle Scholar
  16. Miguel, S., Moya-Anegon, F., & Herrero-Solana, V. (2010). The impact of the socioeconomic crisis of 2001 on the scientific system of Argentina from the scientometric perspective. Scientometrics, 85, 495–507.CrossRefGoogle Scholar
  17. Momtaz, S. (2002). Environmental impact assessment in Bangladesh: A critical review. Environmental Impact Assessment Review, 22, 163–179.CrossRefGoogle Scholar
  18. Morgan, R. K. (2008). Institutionalising Health Impact assessment: The New Zealand experience. Impact Assessment and Project Appraisal, 26, 2–6.CrossRefGoogle Scholar
  19. National Research Foundation. (n.d.). NRF profile, [online]. Available from Last accessed 23 June 2011.
  20. Nederhof, A. J. (1988). Changes in publication patterns of biotechnologists: An evaluation of the impact of government stimulation programmes in six industrial nations. Scientometrics, 15, 475–484.CrossRefGoogle Scholar
  21. Nederhof, A. J. (2008). Policy impact of bibliometric rankings of research performance of departments and individuals in economics. Scientometrics, 74, 163–174.CrossRefGoogle Scholar
  22. Pouris, A. (1991). Effects of funding policies on research publications in South Africa. South African Journal of Science, 87, 78–81.Google Scholar
  23. Pouris, A. (2007). The national research foundation rating system: Why scientists let their ratings lapse. South African Journal of Science, 103, 439–442.Google Scholar
  24. Raitzer, D. A., & Kelley, T. G. (2008). Assessing the contribution of impact assessment to donor decisions for international agricultural research. Research Evaluation, 17, 187–199.CrossRefGoogle Scholar
  25. Shadish, R. W., Cook, D. T., & Campbell, T. D. (2003). Experimental and quasi experimental designs for generalised causal inference. Boston: Houghton Mifflin Company.Google Scholar
  26. Skinner, B. F. (1969). Contingencies of reinforcement: A theoretical analysis. Englewood Cliffs, NJ: Prentice Hall.Google Scholar
  27. Smith, P. E., Orvos, R. D., & Cairns, J., Jr. (1993). Impact assessment using the before-after-control-impact (BACI) model: Concerns and comments. Canadian Journal of Fisheries and Aquatic Sciences, 50, 627–637.CrossRefGoogle Scholar
  28. Tassey, G. (2003). Methods for assessing the economic impacts of government R&D. National Institute of Standards and Technology, [Online]. Available at
  29. Tertiary Education Commission (TEC) (2007). PBRF evaluating research excellence. The 2006 assessment. [Online]. Available at Last accessed 23 June 2011.
  30. Tien, F. F. (2007). Faculty research behaviour and career incentives: the case of Taiwan. International Journal of Educational Development, 27, 1–7.CrossRefGoogle Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2011

Authors and Affiliations

  1. 1.Department of EconomicsUniversity of PretoriaPretoriaSouth Africa
  2. 2.Institute for Technological InnovationUniversity of PretoriaPretoriaSouth Africa

Personalised recommendations