Advertisement

Scientometrics

, Volume 32, Issue 2, pp 195–218 | Cite as

Criteria for the analysis of scientific quality

  • K. Buchholz
Article

Abstract

One of the major questions in science research is addressed in detail, that is the problem of evaluation of research work both by objective characterization, accessible to proof, and by adequate characterization, referring to the content and cognitive level of the work under investigation. A short discussion of established methods by science indicators as well as by peer review compiles merits and shortcomings of these methods. A short review refers to a few approaches towards the development of criteria for an improved assessment and characterization of research work and their shortcomings are discussed. Notably for the evaluation of medium or low range quality no reliable method is available. Therefore a systematic compilation of criteria which covers the full range of excellence to failure with respect to scientific quality is developed and a comprehensive list of criteria is presented which should provide a basis both for objective and adequate characterization of publications.

Keywords

Research Work Science Research Full Range Reliable Method Systematic Compilation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References and notes

  1. 1.
    P. Weingart, M. Winterhager,Die Vermessung der Forschung, Campus Verlag, Frankfurt, 1984.Google Scholar
  2. 2.
    H. D. Daniel, R. Fisch, Research performance evaluation in the German university sector,Scientometrics, 19 (1990) 349–361.Google Scholar
  3. 3.
    W. van den Daele, W. Krohn, P. Weingart, Die politische Steuerung der wissenschaftlichen Entwicklung, in the same eds.,, Geplante Forschung, p. 11 Suhrkamp, Frankfurt, 1979.Google Scholar
  4. 4.
    P. Weingart, M. Winterhager,op. cit. note 1,, p. 78, 87.Google Scholar
  5. 5.
    H. J. Block, W. Krull, What are the consequences? Reflections on the impact of evaluations conducted by a science policy advisory body,Scientometrics, 19 (1990) 427–437.Google Scholar
  6. 6.
    D. Lindsey, The relationship between performance indicators for academic research and funding: Developing a measure of return on investment in science,Scientometrics, 19 (1991) 221–234.Google Scholar
  7. 7.
    H. R. Spiegel, Initiatives for the promotion of science of science — The Stifterverband für die Deutsche Wissenschaft,Scientometrics, 19 (1990) 505–512.Google Scholar
  8. 8.
    H. D. Daniel, R. Fisch, Research performance evaluation in the German university sector,Scientometrics, 19 (1990) 349–361. With respect to political decisions intending to save funds the so-called law of Price seems remarkable, which states that half of the scientific work is produced by a population of the square root (\(\sqrt N \)) of all N scientists in a special field (which would correspond to only 1 to 3% of scientific staff in specialities in universities in Germany where 1.000 to 10.000 are active (loc. cit. 8). Government interests obviously would aim at identifying these.Google Scholar
  9. 9.
    H. D. Daniel, The development and use of research performance indicators for UK universities—A survey, In:Beiträge zur Hochschulforschung, 4 (1990) 269–288, Bayerisches Staatsinstitut für Hochschulforschung und Hochschulplanung (D).Google Scholar
  10. 10.
    B. R. Martin, Bibliometric indicators and the evaluation of British university research, inop. cit. note 9, In:Beiträge zur Hochschulforschung, 4 (1990) 269–288, Bayerisches Staatsinstitut für Hochschulforschung und Hochschulplanung (D). 349–384.Google Scholar
  11. 11.
    I. Hartmann, F. Neidhardt, Peer review at the Deutsche Forschungsgemeinschaft,Scientometrics, 19 (1990) 419–425.Google Scholar
  12. 12.
    H. D. Daniel, R. Fisch, Introduction: Quantitative science and technology indicators studies in the Federal Republic of Germany,Scientometrics, 19 (1990) 327–329.Google Scholar
  13. 13.
    T. Finkenstaedt, Points in an ongoing debate, inop. cit. note 9, In:Beiträge zur Hochschulforschung, 4 (1990) 269–288, Bayerisches Staatsinstitut für Hochschulforschung und Hochschulplanung (D). 415–428.Google Scholar
  14. 14.
    L. E. Miner, S. McDonald, Reliability of peer reviews,J. Soc. Res. Administrators, 13 (1981) 21–25.Google Scholar
  15. 15.
    I. J. Mitroff, D. E. Chubin, Peer review at the NSF, A dialectal policy analysis,Social Studies of Science, 9 (1979) 199–232.Google Scholar
  16. 16.
    D. Edge, Quantitative measures of communication in science: a critical review, in:History of Science, 17 (1979) 102–134.Google Scholar
  17. 17.
    P. Weingart, M. Winterhager,op. cit. note 1,, 98–105.Google Scholar
  18. 18.
    D. P. Hamilton, Publishing by—and for?—the numbers,Science, 250 (1990) 1331–1332.Google Scholar
  19. 19.
    D. H. Lowry, N. J. Rosebrough, A. L. Farr, R. J. Randal, Method for quantitative protein analysis;J. Biol. Chem., 193 (1951) 265; this paper has not been a major contribution to understanding of protein chemistry, but is one of the papers most frequently cited in biochemistry; it has not been considered in a “retrospect on proteins”, a historical review.20 PubMedGoogle Scholar
  20. 20.
    P. R. Srinivason, J. S. Fruton, J. T. Edsall (Eds), The origins of modern biochemistry,Ann. New York Acad. Sci., Vol. 325, New York, 1979.Google Scholar
  21. 21.
    P. Weingart, M. Winterhager,op. cit. note 1,. 132.Google Scholar
  22. 22.
    H. Grupp, On the supplementary functions of science and technology indicators. The case of West German telecommunications R&D,Scientometrics, 19 (1990) 447–472.Google Scholar
  23. 23.
    A. Maelicker, Wer zitiert wen?Nachr. Chem. Techn. Lab., 37 (1989) 266–268.Google Scholar
  24. 24.
    P. Weingart, R. Sehringer, M. Winterhager, Bibliometric indicators for assessing strength and weakness of West German Science, inHandbook of Quantitative Studies of Science and Technology,A. F. J. Van Raan (Ed.), North Holland Elsevier Science Publishers, 1988, p. 391–430.Google Scholar
  25. 25.
    P. Weingart, M. Winterhager,op. cit. note 1.,. 208, 212.Google Scholar
  26. 26.
    K. R. Popper, Logik der Forschung (8th edition), J. C. B. Mohr, Tübingen, 1984, 7, 8 (see also K. R. Popper,The Logic of Science Discovery, London 1980, Hutchinson, 10th edition).Google Scholar
  27. 27.
    J. Lakatos, Falsification and the methodology of scientific research programmes, in:J. L. Musgrave (Ed.),Criticism and the Growth of Knowledge, Cambridge University Press, London, 1970, p. 118.Google Scholar
  28. 28.
    P. Weingart, M. Winterhager,op. cit. note 1,, 157.Google Scholar
  29. 29.
    Ibid.,, 162, 168.Google Scholar
  30. 30.
    Ibid.,. 249.Google Scholar
  31. 31.
    Ibid.,, 201, 202.Google Scholar
  32. 32.
    Ibid.,, 228, 250.Google Scholar
  33. 33.
    Physics Survey Committee, National Research Council:Physics in Perspective, Vol. 1, National Academy of Sciences, Washington, 1972, p. 393.Google Scholar
  34. 35.
    Wissenschaftsrat, Empfehlungen zu Organisation, Planung und Förderung der Forschung; Bewertungskriterien, Bonn, 1975, p. 168.Google Scholar
  35. 36.
    J. M. Chase, Normative criteria for scientific publication,The American Sociologist, 5 (1970) 262–265.Google Scholar
  36. 37.
    H. Hertz,Die Prinzipien der Mechanik, Einleitung (Introduction), 2nd. ed., J. A. Barth, Leipzig, 1910.Google Scholar
  37. 38.
    E. Mach,Die Mechanik in ihrer Entwicklung, 7th ed., F. A. Brockhaus, Leipzig, 1912.Google Scholar
  38. 39.
    L. Boltzmann,Vorlesungen über Maxwells Theorie der Elektrizität und des Lichtes, J. A. Barth, Leipzig, 1891.Google Scholar
  39. 40.
    “Wissenschaftliche Theorien sind allgemeine Sätze” mit dem Ziel, die Welt “zu rationalisieren, zu erklären und zu beherrschen”.—Scientific theories are laws, aiming at rationalizing, explaining and governing the world (Popper,op. cit., note 26, p. 31).K. R. Popper, Logik der Forschung (8th edition), J. C. B. Mohr, Tübingen, 1984, 7,8 (see also K.R. Popper,The Logic of Science Discovery, London 1980, Hutchinson, 10th edition).Google Scholar
  40. 42.
    E. Garfield,Current Contents, 18 (1986) (No. 1) 3–10.Google Scholar
  41. 43.
    H. Hertz,op. cit. note 37,. p. 4, “...Bild (der) Physik von den Dingen der sinnlichen Welt...” Hertz donates the image of physics representing the things of visible and sensuous world. He however points out the high level of theory which must be reflected also in “images”. In his introduction he develops a rather logic concept of mechanics which is strongly oriented towards logics in mathematics (p. 4, 8). “...logische Zulässigkeit des entworfenen Bildes...” (p. 39). Hertz claims that “images” must correspond to logic proof.Google Scholar
  42. 44.
    E. Mach,op. cit. note 38,, p. 73, 125; “...skizzenhafte Regel,...” “...gedankliche Nachbildung der Tatsachen...”, “...Abstraktion, Idealisierung...”; Mach speaks of sketch rules, mental reproduction of facts, abstraction, idealization.Google Scholar
  43. 45.
    L. Boltzmann,op. cit. note 39,, Vol. I, p. 13; “Die Theorien sind blosse Bilder der Naturprozesse”. Theories are considered to be images of processes in nature.Google Scholar
  44. 46.
    E. Mach,op. cit. note 38,. p. 125: “Das Experiment entscheidet...”; Mach points out that experiments decide on theories.Google Scholar
  45. 47.
    Fortschritt in der Theorie “prognostiziert neue Konsequenzen, neue Effekte und eröffnet damit neue Überprüfungs — bzw. Falsifikationsmöglichkeiten”. — Progress in theory means prognosis of new consequences, new affects and this opens new possibilities of experimental proof or falsification, respectively;Popper,op. cit. note 26,K. R. Popper, Logik der Forschung (8th edition), J. C. B. Mohr, Tübingen, 1984, 7, 8 (see also K. R. Popper,The Logic of Science Discovery, London 1980, Hutchinson, 10th edition), p. 51, see also p. 72.Google Scholar
  46. 49.
    A. Lavoisier,Traité élémentaire de chimie, 2nd ed., Vol. 2, p. 3, Cuchet, Paris 1793; “je suis loin de prétendre que ceux qui veulent prendre des connaissances exactes en Chimie, puissent se dispenser... de se familiariser avec les instruments...”; Lavoisier points out the high relevance of experimental methods for exact knowledge in chemistry.Google Scholar
  47. 50.
    J. F. Robyt, F. Walseth, The mechanism of acceptor reaction of leuconostoc mesenteroides dextransucrase,Carbohydrate Res., 61 (1978) 433–445.Google Scholar
  48. 51.
    K. Buchholz, Innovationen und Entwicklungen im Bereich der Biotechnologie, in:Technik und Gesellschaft,M. E. A. Schmutzler,H. Winter (eds), Springer, Wien, 1981, p. 71 (further examples on theoretical background, experimental and technical progress in Biotechnology are described).Google Scholar
  49. 52.
    H. D. Daniel, Forschungsleistungen wissenschaftlicher Hochschulen im Vergleich, in:H.D. Daniel, R. Fisch (Eds),Evaluation von Forschung, Universitätsverlag Konstanz, 1988, p. 93–104.Google Scholar
  50. 53.
    T. S. Kuhn,Die Structur wissenschaftlicher Revolutionen (Suhrkamp Verlag, Frankfurt, 1967) (see alsoThe Structure of Scientific Revolutions, Chicago 1962).Google Scholar
  51. 54.
    Such a school was that ofPatat, a chemical engineer, Institut für Technische Chemie, Technical University of Munich, in the sixties,F. Patat, D. Behrens, Forschung auf dem Gebiet der Technischen Chemie, in:Angewandte Forschung in der BRD, DFG, Wiesbaden, 1957, p. 71.Google Scholar
  52. 55.
    For chemical engineering seeK. Buchholz, Verfahrenstechnik (Chemical Engineering) — its development, present state and structure,Soc. Stud. Sci., 9 (1979) 33–62.Google Scholar
  53. 57.
    K. Buchholz, Die gezielte Förderung und Entwicklung der Biotechnologie, In:W. Van den Daele, W. Krohn, P. Weingart,Geplante Forschung, Suhrkamp, Frankfurt, 1979, p. 64 (see specifically p. 86–92).Google Scholar

Copyright information

© Akadémiai Kiadó 1995

Authors and Affiliations

  • K. Buchholz
    • 1
  1. 1.Institute for Carbohydrate Technology at the Technical University BraunschweigBraunschweig(Germany)

Personalised recommendations