Advertisement

Scientometrics

, Volume 33, Issue 2, pp 169–185 | Cite as

Quasi-quantitative measures of research performance: An assessment of construct validity and reliability

  • P. S. Nagpaul
Article

Abstract

This paper argues that research performance is essentially a multidimensional concept which cannot be encapsulated into a single universal criterion. Various indicators used in quantitative studies on research performance at micro or meso-levels can be classified into two broad categories: (i) objective or quantitative indicators (e.g. counts of publications, patents, algorithms or other artifacts of research output) and (ii) subjective or qualitative indicators which represent evaluative judgement of peers, usually measured on Likert or semantic differential scales. Because of their weak measurement properties, subjective indicators can also be designated as quasi-quantitative measures. This paper is concerned with the factorial structure and construct validity of quasi-quantitative measures of research performance used in a large-scale empirical study carried out in India. In this study, a reflective measurement model incorporating four latent variables (R & D effectiveness, Recognition, User-oriented effectiveness and Administrative effectiveness) is assumed. The latent variables are operationalized through thirteen indicators measured on 5-point semantic differential scales. Convergent validity, discriminant validity and reliability of the measurement model are tested through LISREL procedure.

Keywords

Latent Variable Construct Validity Research Performance Measurement Model Research Output 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Notes and references

  1. 1.
    See for example,P. S. Nagpaul, Conceptual and methodological problems in the measurement of research performance, In:P. S. Nagpaul (Ed.),Organization and Efficiency of Research Groups, National Institute of Science, Technology and Development Studies, New Delhi, 1987.Google Scholar
  2. 2.
    M. R. Moser, Measuring R & D performance,Research Management, 5 (1985) 31–33.Google Scholar
  3. 3.
    K. S., Cameron, Measuring organizational effectiveness in institutions of higher education,Administrative Science Quarterly, 23 (1978) 604–632.Google Scholar
  4. 4.
    F. M. Andrews (Ed.),Scientific Productivity, Cambridge University Press, 1979.Google Scholar
  5. 5.
    P. H. Birnbaum, Predictors of long term research performance, In:S. R. Epton, et al (Eds)Managing Interdisciplinary Research, John Wiley & Sons, New York, 1974;P. S. Nagpaul,op. cit. Ref. 1. Conceptual and methodological problems in the measurement of research performance, In:P. S. Nagpaul (Ed.),Organization and Efficiency of Research Groups, National Institute of Science, Technology and Development Studies, New Delhi, 1987.Google Scholar
  6. 6.
    G. Cole, Classifying research units by performance and influence: A typology of Round I data, In:F. M. Andrews, (Ed.),op. cit. Scientific Productivity, Cambridge University Press, 1979. Ref. 4, 353–404; M. F. Fox, Publication productivity among scientists: A critical review,Social Studies of Science, 13 (1983) 285–305.Google Scholar
  7. 7.
    G. Cole,op. cit. Ref. 6 Classifying research units by performance and influence: A typology of Round I data, In:F. M. Andrews, (Ed.),op. cit. Scientific Productivity, Cambridge University Press, 1979. Ref. 4, 353–404; M. F. Fox, Publication productivity among scientists: A critical review,Social Studies of Science, 13 (1983) 285–305.Google Scholar
  8. 8.
    V. Stolte-Heiskanen, Trends and problems in the evaluation of research, In:E. Kaukonen, V. Stolte-Heiskanen (Eds),Science Studies and Science Policy: Proceedings of a Finnish Bulgarian Symposium, Academy of Finland (Coordination Research Group for Science of Science Studies) and University of Tampere, Finland, 1984.Google Scholar
  9. 9.
    P. S. Napaul,op. cit. Ref. 1..Google Scholar
  10. 10.
    L. Carter, A new look at NSF,Science, 204 (1979) 1064–1065.Google Scholar
  11. 11.
    V. Stolte-Heiskanen,op. cit. Ref. 8..Google Scholar
  12. 12.
    In this study, a research unit is operationally defined as a unit that has at least one project, has a total expected life span of at least one year and comprises at least four core members among whom there is one scientist who is the head of the unit.Google Scholar
  13. 13.
    A. H. Vande Ven, A. L. Delbeg, K. Koenig, Jr, Determinants of coordination modes within organizations,American Sociological Review, 41 (1976) 322–338.Google Scholar
  14. 14.
    S. Katz, N. L. Johnson (chief Eds)Encyclopedia of Statistical Sciences, Vol. 1 (1982) 189–191.Google Scholar
  15. 15.
    D. T. Campbell, D. W. Fiske, Convergent and discriminant validation by the multitrait-multimethod matrix,Psychological Bulletin, 56 (1959) 81–105.PubMedGoogle Scholar
  16. 16.
    D. T. Campbell, D. W. Fiske,op. cit. Ref. 15..PubMedGoogle Scholar
  17. 17.
    R. P. Bagozzi, L. W. Phillips, Representing and testing organizational theories: A holistic construct,Administrative Science Quarterly, 27 (1980) 459–489.Google Scholar
  18. 18.
    L. J. Cronbach, Test validation, In:R. L. Thorndike (Ed.)Educational Measurement, 2nd ed., American Council on Education, Washington, DC.Google Scholar
  19. 19.
    F. M. Lord, M. R. Novick,Statistical Theories of Mental Test Scores, Addison Wesley, Reading, MA, 1968.Google Scholar
  20. 20.
    N. Venkatraman, Strategic orientation of business enterprises: the constructs, dimensionality and measurement,Management Science, 11 (1989) 942–962.Google Scholar
  21. 21.
    C. E. Werts, R. L. Linn, K. G. Joreskog, Interclass reliability estimates: testing structural assumptions,Educational and Psychological Measurement, 34 (1974) 25–33.Google Scholar
  22. 22.
    D. D. Duncan,Introduction to Structural Equation Models, Academic Press, New York, 1975, p. 113.Google Scholar
  23. 23.
    S. K. Reddy, Effects of ignoring correlated measurement errors in structural equation models,Educational and Psychological Measurement, 52 (1992) 549–570.Google Scholar
  24. 24.
    J. C. Anderson, D. W. Gerbing, The effect of sampling error on convergence, improper solutions and goodness of fit indices for maximum likelihood confirmatory factor analysis,Psychometrika, 49 (1984) 155–173;R. P. Bagozzi, Issues in the application of covariance structure analysis: a further comment,Journal of Consumer Research, 9 (1983) 449–450;C. Fornell, Issues in the application of covariance structure analysis; a comment,Journal of Consumer Research, 9 (1983) 443—447.Google Scholar
  25. 25.
    S. K. Reddy,op. cit. Ref. 23..Google Scholar
  26. 26.
    R. P. Bagozzi, A holistic methodology for modeling consumer response to innovation,Operations Research, 31 (1983) 128–176.Google Scholar
  27. 27.
    K. Joreskog, D. Sorbom,LISREL 7.16: Analysis of Linear Structured Relationship by Maximum Likelihood and Least Squares Method, International Educational Services, Chicago, 1989.Google Scholar
  28. 28.
    D. N. Lawley, A. E. Maxwell,Factor Analysis as a Statistical Method, Butterworth, London, 1971.Google Scholar
  29. 29.
    M. W. Browne, R. Cuddeck, Alternative ways of assessing model fit,Sociological Methods and Research, 21 (1992) 230–258.Google Scholar
  30. 30.
    K. A. Bollen, A new incremental fit index for general structural equation models,Psychometrika, 51 (1992) 335–373.Google Scholar
  31. 31.
    S. Mulaik, L. James, J. Van Alstine, N. Bennett, S. Lino, C. Stilwell, Evaluation of goodness-of-fit indices for structural equation models,Psychological Bulletin, 105 (1987) 430–445.Google Scholar
  32. 32.
    P. M. Bentler, D. G. Bonnet, Significance tests and goodness of fit in the analysis of covariance structures,Psychological Bulletin, 88 (1980) 588–606.Google Scholar
  33. 33.
    P. M. Bentler, D. G. Bonnet,op. cit. Ref. 32..Google Scholar
  34. 34.
    K. A. Bollen,op. cit. Ref. 30..Google Scholar
  35. 35.
    R. K. Parkes, Field dependence and the differentiation of neurotic syndromes, In:P. Cuttanece, R. Ecob (Eds),Structural Modeling by Example, Cambridge University Press, Cambridge, 1987.Google Scholar
  36. 36.
    See for example.J. F. A. Spangenberg, W. Buijink, W. Alfennar, Some incentives and constraints of scientific performance in departments of economics: Part I. Predictor criterion relations,Scientometrics, 18 (1990) 241–268;F. M. Andrews,op. cit. Ref. 4.Scientific Productivity, Cambridge University Press, 1979.Google Scholar
  37. 37.
    H. Morita-Lou,Science and Technology Indicators for Development, United Nations Science and Technology for Development Series, Westview Press, London, 1985.Google Scholar

Copyright information

© Akadémiai Kiadó 1995

Authors and Affiliations

  • P. S. Nagpaul
    • 1
  1. 1.National Institute of Science, Technology and Development StudiesNew Delhi(India)

Personalised recommendations