Qualitative Judgement of Research Impact: Domain Taxonomy as a Fundamental Framework for Judgement of the Quality of Research
The appeal of metric evaluation of research impact has attracted considerable interest in recent times. Although the public at large and administrative bodies are much interested in the idea, scientists and other researchers are much more cautious, insisting that metrics are but an auxiliary instrument to the qualitative peer-based judgement. The goal of this article is to propose availing of such a well positioned construct as domain taxonomy as a tool for directly assessing the scope and quality of research. We first show how taxonomies can be used to analyze the scope and perspectives of a set of research projects or papers. Then we proceed to define a research team or researcher’s rank by those nodes in the hierarchy that have been created or significantly transformed by the results of the researcher. An experimental test of the approach in the data analysis domain is described. Although the concept of taxonomy seems rather simplistic to describe all the richness of a research domain, its changes and use can be made transparent and subject to open discussions.
KeywordsResearch impact Scientometrics Stratification Rank aggregation Multicriteria decision making Semantic analysis Taxonomy
Unable to display preview. Download preview PDF.
- ACM (2012), The 2012 ACM Computing Classification System, https://www.acm.org/publications/class-2012.
- ARAGNÓN, A.M. (2013), “A Measure for the Impact of Research”, Scientific Reports, 3, Article number: 1649.Google Scholar
- DORA (2013). San Francisco Declaration on Research Assessment (DORA), http://www.ascb.org/files/SFDeclarationFINAL.pdf.
- HALLANTIE, T. (2016), ”What It Takes to Succeed in FET-Open”, https://ec.europa.eu/digital-single-market/en/blog/what-it-takes-succeed-fet-open.
- METRIC TIDE (2016), “The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management”, http://www.hefce.ac.uk/pubs/rereports/Year/2015/metrictide/Title,104463,en.html.
- MIRKIN, B. (2013), “On the Notion of Research Impact and Its Measurement”, Institute of Control Problems, Moscow (in Russian), Control in Large Systems, Special Issue: Scientometry and Experts in Managing Science, 44, 292–307.Google Scholar
- MIRKIN, B., and ORLOV, M. (2013), “Methods for Multicriteria Stratification and Experimental Comparison of Them”, Preprint (in Russian) WP7/2013/06, Higher School of Economics, Moscow, 31 pp.Google Scholar
- MIRKIN, B., and ORLOV, M. (2015). “Three Aspects of the Research Impact by a Scientist: Measurement Methods and an Empirical Evaluation”, in Optimization, Control, and Applications in the Information Age, eds. A. Migdalas, and A. Karakitsiou, Springer Proceedings in Mathematics and Statistics, 130, pp. 233–260.Google Scholar
- OSTERLOH, M., and FREY, B.S. (2014), “Ranking Games”, Evaluation Review, Sage, pp. 1–28.Google Scholar
- SCHAPIRE, R.E. (1990), “The Strength of Weak Learnability”, Machine Learning, 5(2), 197–227.Google Scholar
- SIDIROPOULOS, A., KATSAROS, D., and MANOLOPOULOS, Y. (2014), “Identification of Influential Scientists vs. Mass Producers by the Perfectionism Index”, Preprint, ArXiv:1409.6099v1, 27 pp.Google Scholar
- SNOMED CT (2016), IHTSDO, International Health Terminology Standards Development Organization, SNOMEDCT, Systematized Nomenclature of Medicine, Clinical Terms, http://www.ihtsdo.org/snomed-ct.
- SUN, Y., HAN, J., ZHAO, P., YIN, Z., CHENG, H., and WU, T. (2009), “RankClus: Integrating Clustering with Ranking for Heterogeneous Information Network Analysis”, EDBT ’09 Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology, New York: ACM, pp. 565–576.Google Scholar
- THOMSON REUTERS (2016), “Thomson Reuters Intellectual Property and Science”, (Acquisition of the Thomson Reuters Intellectual Property and Science Business by Onex and Baring Asia Completed, Independent business becomes Clarivate Analytics), http://ip.thomsonreuters.com.
- UNIVERSITY GUIDE (2016), “The Complete University League Guide”, http://www.thecompleteuniversityguide.co.uk/league-tables/methodology.