Skip to main content

Advertisement

SpringerLink
Log in

Research on developmental evaluation based on the "four abilities" model: evidence from early career researchers in China

  • Published:
Quality & Quantity Aims and scope Submit manuscript

Abstract

Based on the theory of developmental evaluation, this paper develops a new development-oriented research evaluation model called "four abilities", with "basic development ability", "process development ability", "achievement ability" and "influence ability" as key dimensions, which can be applied in different situations such as recruitment, performance appraisal, promotion evaluation, tenure review, and selection of honorary academic awards. Based on the data from the Youth Fund of the Ministry of Management Science of National Natural Science Foundation of China (NSFC) from 2014 to 2018, this paper takes the developmental evaluation of early career researchers as an example, treating the existing research output as the development goal and the "development ability" of early career researchers as the explanatory factor. Then the propensity score matching method (PSM) method is adopted to control the sample self-selection bias in the way of reverse order evaluation, so as to explain whether the guiding indicator "development ability" is conducive to the development of researchers. The results indicate that strong “process development ability” of newly recruited researchers in the first 3 years of employment period can significantly promote their future research performance, which has important implications for the current evaluative culture overemphasizing short-term output. Through the application of the “four abilities” model integrating various dimensions composed of potential, capacity, output and impact, researchers at different career stages are guided to concentrate more on long-term academic mission and to achieve better career development based on their differentiated development needs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Data availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

References

  • Acker, S., Webber, M.: Made to measure: early career academics in the Canadian university workplace. High. Educ. Res. Dev. 36(3), 541–554 (2017)

    Article  Google Scholar 

  • Aksnes, D.W., Langfeldt, L., Wouters, P.: Citations, citation indicators, and research quality: an overview of basic concepts and theories. SAGE Open 9(1), 2158244019829575 (2019)

    Article  Google Scholar 

  • Berning, N., Nunning, A., Schwanecke, C.: (Trans-)national criteria norms and standards in literary studies: a comparative analysis of criteria-based ex ante evaluation forms of funding proposals in the humanities. Germanisch-Romanische Monatsschrift 65(1), 115–135 (2015)

    Google Scholar 

  • Brink, A.G., Lowe, D.J., Victoravich, L.M.: The Effect of evidence strength and internal rewards on intentions to report fraud in the Dodd-Frank regulatory environment. Audit. J. Pract. Theory 32(3), 87–104 (2013)

    Article  Google Scholar 

  • Coryn, C.L.S., Hattie, J.A., Scriven, M., Hartmann, D.J.: Models and mechanisms for evaluating government-funded research: an international comparison. Am. J. Eval. 28(4), 437–457 (2007)

    Article  Google Scholar 

  • Dahler-Larsen, P.: The evaluation society. In: The Evaluation Society. Stanford University Press (2011)

  • de Rijcke, S., Wouters, P.F., Rushforth, A.D., Franssen, T.P., Hammarfelt, B.: Evaluation practices and effects of indicator use—a literature review. Res. Eval. 25(2), 161–169 (2015)

    Article  Google Scholar 

  • Dickson, R., Saunders, M.: Developmental evaluation: lessons for evaluative practice from the SEARCH Program. Evaluation 20(2), 176–194 (2014)

    Article  Google Scholar 

  • Enders, J., de Boer, H., Weyer, E.: Regulatory autonomy and performance: the reform of higher education re-visited. High. Educ. 65, 5–23 (2013)

    Article  Google Scholar 

  • Falk-Krzesinski, H.J., Tobin, S.C.: How do I review thee? Let me count the ways: a comparison of research grant proposal review criteria across US federal funding agencies. J. Res. Adm. 46(2), 79–94 (2015)

    Google Scholar 

  • Gamble, J.: A developmental evaluation primer. The J.W. McConnell Family Foundation, Ottawa (2008)

    Google Scholar 

  • Geuna, A., Martin, B.R.: University research evaluation and funding: an international comparison. Minerva 41(4), 277–304 (2003)

    Article  Google Scholar 

  • Guba, E.G., Lincoln, Y.S.: Fourth Generation Evaluation. Sage (1989)

    Google Scholar 

  • Guillory, J.: Valuing the humanities, evaluating scholarship. Profession 28–38 (2005)

  • Hammarfelt, B.: Assessing academic careers: The peer review of professorial candidates. Peer review in an Era of Evaluation: Understanding the Practice of Gatekeeping in Academia, 347–370 (2022)

  • Hammarfelt, B., Haddow, G.: Conflicting measures and values: how humanities scholars in Australia and Sweden use and react to bibliometric indicators. J. Am. Soc. Inf. Sci. 69(7), 924–935 (2018)

    Google Scholar 

  • Han, S., Xu, X.: How far has the state ‘stepped back’: an exploratory study of the changing governance of higher education in China (1978–2018). High. Educ. 78(5), 931–946 (2019)

    Article  Google Scholar 

  • Hu, Z., Wu, Y.: Research on analysis of influence factors and prediction for scientific and technological outputs—an approach based on multiple linear regression and BP neural network. Stud. Sci. Sci. 07, 992–1004 (2012). https://doi.org/10.16192/j.cnki.1003-2053.2012.07.004

    Article  Google Scholar 

  • Huang, Y.T., Xu, J.: Surviving the performance management of academic work: evidence from young Chinese academics. High. Educ. Res. Dev. 39(4), 704–718 (2020)

    Article  Google Scholar 

  • Huang, Y.T., Pang, S.K., Yu, S.L.: Academic identities and university faculty responses to new managerialist reforms: experiences from China. Stud. High. Educ. 43(1), 154–172 (2018)

    Article  Google Scholar 

  • Hug, S.E., Aeschbach, M.: Criteria for assessing grant applications: a systematic review. Palgrave Commun. 6(1), 1–15 (2020)

    Article  Google Scholar 

  • Hug, S.E., Ochsner, M., Daniel, H.-D.: Criteria for assessing research quality in the humanities: a Delphi study among scholars of English literature, German literature and art history. Res. Eval. 22(5), 369–383 (2013). https://doi.org/10.1093/reseval/rvt008

    Article  Google Scholar 

  • Hume, K.M., Giladi, A.M., Chung, K.C.: Factors impacting successfully competing for research funding: an analysis of applications submitted to the Plastic Surgery Foundation. Plast. Reconstr. Surg. 135(2), 429–435 (2015)

    Article  Google Scholar 

  • Hu-Ping, S., Pan-Pan, Z.: What factors of the applicants are influencing the output performance of research projects? A back-chaining evaluation based on the data of National Science Foundation. Stud. Sci. Sci. 32(9), 1378 (2014)

    Google Scholar 

  • Kaplan, R.S., Norton, D.P.: The balanced scorecard: measures that drive performance. Harv. Bus. Rev. 83(7), 172 (2005)

    Google Scholar 

  • Lamont, M.: How Professors Think: Inside the Curious World of Academic Judgment. Harvard University Press (2009)

    Book  Google Scholar 

  • Leviton, L.C., Hughes, E.F.X.: Research on the utilization of evaluations. Eval. Rev. 5(4), 525–548 (1981a)

    Article  Google Scholar 

  • Leviton, L.C., Hughes, E.F.: Research on the utilization of evaluations: a review and synthesis. Eval. Rev. 5(4), 525–548 (1981b)

    Article  Google Scholar 

  • Lincoln, Y.S., Guba, E.G.: The roots of fourth generation evaluation. Eval. Roots Tracing Theor. Views Influ. 1(1), 225–241 (2004)

    Google Scholar 

  • Lo, W.Y.W.: Soft power, university rankings and knowledge production: distinctions between hegemony and self-determination in higher education. Comp. Educ. 47(2), 209–222 (2011)

    Article  Google Scholar 

  • Manathunga, C., Selkrig, M., Sadler, K., Keamy, K.: Rendering the paradoxes and pleasures of academic life: using images, poetry and drama to speak back to the measured university. High. Educ. Res. Dev. 36(3), 526–540 (2017)

    Article  Google Scholar 

  • Mok, K.H., Wang, L.: The impacts of neo-liberalism on higher education in China. J. Crit. Educ. Policy Stud. 5(1), 316–348 (2007)

    Google Scholar 

  • Nederhof, A.J.: Bibliometric monitoring of research performance in the social sciences and the humanities: a review. Scientometrics 66(1), 81–100 (2006)

    Article  Google Scholar 

  • O’Meara, K., Terosky, A., Neumann, A.: Faculty Careers and Work Lives: A Professional Growth Perspective. ASHE Higher Education Report. Jossey-Bass, Hoboken (2008)

    Google Scholar 

  • Ochsner, M., Peruginelli, G.: National research evaluation systems and the social sciences. In: Handbook on Research Assessment in the Social Sciences, pp. 416–433. Edward Elgar Publishing (2022)

  • Ochsner, M., Kulczycki, E., Gedutis, A.: The diversity of European research evaluation systems. In: STI 2018 Conference Proceedings, pp. 1235–1241. Centre for Science and Technology Studies (CWTS) (2018)

  • Ochsner, M., Kancewicz-Hoffman, N., Hołowiecki, M., Holm, J.: Overview of peer review practices in the SSH (2020)

  • Patton, M.Q.: A world larger than formative and summative. Eval. Pract. 17(2), 131–144 (1996)

    Article  Google Scholar 

  • Patton, M.Q.: Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use. Guilford Press (2010)

    Google Scholar 

  • Reymert, I., Jungblut, J., Borlaug, S.B.: Are evaluative cultures national or global? A cross-national study on evaluative cultures in academic recruitment processes in Europe. High. Educ. 82(5), 823–843 (2020)

    Article  Google Scholar 

  • Rolfe, G.: The University in Dissent: Scholarship in the Corporate University. Routledge (2013)

    Google Scholar 

  • Rosenbaum, P.R., Rubin, D.B.: Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. Am. Stat. 39(1), 33–38 (1985)

    Google Scholar 

  • Scriven, M.: The methodology of evaluation. In: Tyler, R.W., Gagné, R.M., Scriven, M. (eds.) Perspectives of Curriculum Evaluation, vol. 1, pp. 39–83. Rand McNally, Chicago, IL (1967)

    Google Scholar 

  • Scriven, M.: Evaluation Thesaurus. Sage (1991)

    Google Scholar 

  • Sheldon, K.M., Krieger, L.S.: Understanding the negative effects of legal education on law students: a longitudinal test of self-determination theory. Pers. Soc. Psychol. Bull. 33(6), 883–897 (2007)

    Article  Google Scholar 

  • Spence, C.: ‘Judgement’ versus ‘metrics’ in higher education management. High. Educ. 77(5), 761–775 (2019)

    Article  Google Scholar 

  • Stake, R.E.: The countenance of educational evaluation. Teach. Coll. Rec. 68(7), 1–15 (1967)

    Article  Google Scholar 

  • Strathern, M. (ed.): Routledge, London (2000)

    Google Scholar 

  • Sutherland, K.A.: Constructions of success in academia: an early career perspective. Stud. High. Educ. 42(4), 743–759 (2015)

    Google Scholar 

  • Theoharakis, V., Hirst, A.: Perceptual differences of marketing journals: a worldwide perspective. Mark. Lett. 13(4), 389–402 (2002)

    Article  Google Scholar 

  • Thorndike, E.L.: Measurement in education. Teach. Coll. Rec. 22(5), 1–5 (1921)

    Article  Google Scholar 

  • Tian, M., Lu, G.: What price the building of world-class universities? Academic pressure faced by young lecturers at a research-centered University in China. Teach. High. Educ. 22(8), 957–974 (2017)

    Article  Google Scholar 

  • Torrance, H.: The research excellence framework in the United Kingdom: Processes, consequences, and incentives to engage. Qual. Inq. 26(7), 771–779 (2020)

    Article  Google Scholar 

  • Tyler, R.W.: Trends in teaching: how research is affecting our understanding of the learning process. Sch. Rev. 59(5), 263–272 (1951)

    Article  Google Scholar 

  • Van Thiel, S., Leeuw, F.L.: The performance paradox in the public sector. Public Perform. Manag. Rev. 25(3), 267–281 (2002)

    Article  Google Scholar 

  • Vidovich, L., Yang, R., Currie, J.: Changing accountabilities in higher education as China ‘Opens Up’ to globalisation. Glob. Soc. Educ. 5(1), 89–107 (2007)

    Google Scholar 

  • Wang, H., Zhi, Q., Fei, J.: The Influence of Youth Science Foundation on Research Performance for Youth University Teachers in China—Evidence from 1995 to 2013 NSFC Program Data for Life Sciences. Educational Research, Beijing (2016)

    Google Scholar 

  • Yuan, R.: Understanding higher education-based teacher educators’ identities in Hong Kong: a sociocultural linguistic perspective. Asia-Pac. J. Teach. Educ. 44(4), 379–400 (2016)

    Article  Google Scholar 

  • Zacharewicz, T., Lepori, B., Reale, E., Jonkers, K.: Performance-based research funding in EU member states—a comparative assessment. Sci. Public Policy 46(1), 105–115 (2019)

    Article  Google Scholar 

Download references

Funding

This work is supported by Humanity and Social Science Youth foundation of Ministry of Education of China (22YJC880074).

Author information

Authors and Affiliations

  1. School of Business Administration, Southwestern University of Finance and Economics, Chengdu, China

    Song Jing & Qingzhao Ma

  2. School of Public Administration, Southwestern University of Finance and Economics, Chengdu, China

    Siyi Wang, Hanliang Xu & Zhuolin Wu

  3. Graduate School of Arts and Sciences, Columbia University, New York, USA

    Tian Xu

  4. Sichuan Vocational and Technical College of Communications, Chengdu, China

    Xia Guo

Authors
  1. Song Jing
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qingzhao Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Siyi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hanliang Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tian Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xia Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Zhuolin Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Siyi Wang.

Ethics declarations

Conflict of interest

The above authors declare that they have no conflict of interest or personal relationships that could have appeared to influence the work reported in this paper. All authors have participated in (a) conception and design, or analysis and interpretation of the data; (b) drafting the article or revising it critically for important intellectual content; and (c) approval of the final version.

Human and animal rights

This article does not contain any studies with human participants performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jing, S., Ma, Q., Wang, S. et al. Research on developmental evaluation based on the "four abilities" model: evidence from early career researchers in China. Qual Quant 58, 681–704 (2024). https://doi.org/10.1007/s11135-023-01665-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11135-023-01665-0

Keywords

Search

Navigation