, Volume 109, Issue 2, pp 927–951 | Cite as

Assessing the maturity of a research area: bibliometric review and proposed framework

  • Heather Keathley-HerringEmail author
  • Eileen Van Aken
  • Fernando Gonzalez-Aleu
  • Fernando Deschamps
  • Geert Letens
  • Pablo Cardenas Orlandini


In recent years, many disciplines have begun to adopt more systematic and standardized approaches to evaluate the impact and development of a research area with a stronger emphasis on quantitative techniques. In particular, identifying and analyzing the published literature have become important exercises for many disciplines and methods such as systematic literature review and bibliometric analysis have become more regularly used to obtain a deeper understanding of a research area. One concept that is of particular interest is the maturity, or level of development, of a research area. While this concept has been mentioned in many works, it has not yet been formalized, resulting in a lack of consensus concerning the definition of research area maturity and analysis techniques to assess maturity. Therefore, most assessments of research area maturity consider only a subset of the possible criteria with significant differences in the metrics and analyses used among different disciplines. Due to the inconsistencies in the definition and assessment of this concept, a comprehensive synthesis of this literature area is needed. This paper presents the results of a study to identify and analyze the literature, define the maturity of a research area, and synthesize the criteria for assessing maturity. The results are used to develop a generalized maturity assessment framework that establishes a comprehensive set of criteria, which can be adapted for use across a variety of research areas.


Research area Maturity Systematic literature review Conceptual framework Bibliometric analysis 


  1. Archambault, É., & Larivière, V. (2010). The limits of bibliometrics for the analysis of the social sciences and humanities literature. World Social Science Report, 251–254.Google Scholar
  2. Becheikh, N. (2010). How to improve knowledge transfer strategies and practices in education? Answers from a systematic literature review. Research in Higher Education Journal, 7, 1–21.Google Scholar
  3. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. doi: 10.1002/9780470743386.
  4. Bornmann, L. (2015). Alternative metrics in scientometrics: A meta-analysis of research into three altmetrics. Scientometrics, 103(3), 1123–1144. doi: 10.1007/s11192-015-1565-y.CrossRefGoogle Scholar
  5. Borrego, M., & Bernhard, J. (2011). The emergence of engineering education research as an internationally connected field of inquiry. Journal of Engineering Education, 100(1), 14–47. doi: 10.1002/j.2168-9830.2011.tb00003.x.CrossRefGoogle Scholar
  6. Brookes, B. C. (1990). Biblio-, sciento-, infor-metrics? What are we talking about? In L. Egghe, & R. Rousseau (Eds.), Informetrics 89/90 (pp. 31–43). Diepenbeek.Google Scholar
  7. Budi, I., Aji, R. F., & Widodo, A. (2013). Prediction of research topics on science & technology (S&T) using ensemble forecasting. International Journal of Software Engineering and Its Applications, 7(5), 253–268. doi: 10.14257/ijseia.2013.7.5.23.CrossRefGoogle Scholar
  8. Campbell Resource Center. (n.d.). Retrieved January 23, 2015, from
  9. Carlile, P. R., & Christensen, C. M. (2004). The cycles of theory building in management research, version 5.0, Harvard Business School Working Knowledge. Accessed 21 March 2011.
  10. Cheon, M. J., Groven, V., & Sabherwal, R. (1993). The evolution of empirical research in IS. Information & Management, 24(3), 107–119. doi: 10.1016/0378-7206(93)90060-7.CrossRefGoogle Scholar
  11. Cobo, M. J., López-Herrera, A. G., Herrera-Viedma, E., & Herrera, F. (2011). An approach for detecting, quantifying, and visualizing the evolution of a research field: A practical application to the fuzzy sets theory field. Journal of Informetrics, 5(1), 146–166. doi: 10.1016/j.joi.2010.10.002.CrossRefzbMATHGoogle Scholar
  12. Cooper, H., & Hedges, L. V. (2009). Research synthesis as a scientific process. In The handbook of research synthesis and meta-analysis (pp. 3–17). Retrieved from
  13. Cooper, H., Hedges, L., & Valentine, J. (2009). The handbook of research synthesis and meta-analysis. Psychological Review April 2006 (Vol. 113).Google Scholar
  14. Cronin, P., Ryan, F., Coughlan, M., Reid, E. R., Harris, B., Vessali, K. V., et al. (2007). Undertaking a literature review: A step-by-step approach. Computers, Environment and Urban Systems, 60(1), 1–15. doi: 10.1177/107808747000500401.Google Scholar
  15. Diallo, S. Y., Lynch, C. J., Gore, R., & Padilla, J. J. (2016). Identifying key papers within a journal via network centrality measures. Scientometrics, 107(3), 1–16. doi: 10.1007/s11192-016-1891-8.CrossRefGoogle Scholar
  16. Drew, C. H., Pettibone, K. G., Finch, F. O., Giles, D., & Jordan, P. (2016). Automated research impact assessment: A new bibliometrics approach. Scientometrics, 106(3), 987–1005. doi: 10.1007/s11192-015-1828-7.CrossRefGoogle Scholar
  17. Gagnon, R. J., & Ghosh, S. (1991). Assembly line research: Historical roots, research life cycles and future directions. Omega, 19(5), 381–399. doi: 10.1016/0305-0483(91)90056-Y.CrossRefGoogle Scholar
  18. Garfield, E. (2006). Citation indexes for science. A new dimension in documentation through association of ideas. International Journal of Epidemiology, 35(5), 1123–1127. doi: 10.1093/ije/dyl189.CrossRefGoogle Scholar
  19. Garfield, E. (2009). From the science of science to scientometrics visualizing the history of science with HistCite software. Journal of Informetrics, 3(3), 173–179.CrossRefGoogle Scholar
  20. Grover, V. (2012). The information systems field: Making a case for maturity and contribution. Journal of the Association for Information Systems, 13(Special Issue), 254–272.MathSciNetGoogle Scholar
  21. Harvey, L. J., & Myers, M. D. (1995). Scholarship and practice: The contribution of ethnographic research methods to bridging the gap. Information Technology & People, 8(3), 13–27. doi: 10.1108/09593849510098244.CrossRefGoogle Scholar
  22. Hicks, D. (1999). The difficulty of achieving full coverage of international social science literature and the bibliometric consequences. Scientometrics, 44(2), 193–215.CrossRefGoogle Scholar
  23. Higgins, J., & Green, S. (Eds.). (2011). Chapter 4: Guide to the contents of a Cochrane protocol and review. Cochrane handbook for systematic reviews of interventions. The Cochrane Collaboration. doi: 10.1002/9780470712184.
  24. Hood, W. W., & Wilson, C. S. (2001). The literature of bibliometrics and informetrics scientometrics. Scientometrics, 52(2), 291–314.CrossRefGoogle Scholar
  25. Houy, C., Fettke, P., & Loos, P. (2010). Empirical research in business process management—Analysis of an emerging field of research. Business Process Management Journal, 16(4), 619–661. doi: 10.1108/14637151011065946.CrossRefGoogle Scholar
  26. Karuga, G., Lowry, P., & Richardson, V. (2007). Assessing the impact of premier information systems research over time. Communications of the Association for Information Systems, 19, 115–131. Retrieved from
  27. Keathley, H., Bean, A., Chen, T., Vila, K., Ye, K., & Gonzalez-Aleu, F. (2015). Bibliometric analysis of author collaboration in engineering management research. In Proceedings of the 2015 international annual conference, American Society for Engineering Management, October 7-10, 2015.Google Scholar
  28. Keathley, H., Bean, A., Chen, T., Vila, K., Ye, K., & Gonzalez-Aleu, F. (2016). Bibliometric analysis of research design characteristics in engineering management research. In Proceedings of the 2016 industrial and systems engineering research conference, Institute of Industrial Engineers, May 21-24, 2016.Google Scholar
  29. Li, X., Hu, D., Dang, Y., Chen, H., Roco, M. C., Larson, C. A., et al. (2009). Nano Mapper: An Internet knowledge mapping system for nanotechnology development. Journal of Nanoparticle Research, 11(3), 529–552. doi: 10.1007/s11051-008-9491-z.CrossRefGoogle Scholar
  30. Lim, J., Rong, G., & Grover, V. (2007). An inductive approach to documenting the “core” and evolution of the IS field. Communications of the Association for Information Systems, 19(32), 665–691. Retrieved from
  31. Maloni, M. J., Carter, C. R., & Carr, A. S. (2009). Assessing logistics maturation through author concentration. International Journal of Physical Distribution & Logistics Management, 39(3), 250–268. doi: 10.1108/09600030910951728.CrossRefGoogle Scholar
  32. Maloni, M., Carter, C. R., & Kaufmann, L. (2012). Author affiliation in supply chain management and logistics journals: 2008–2010. International Journal of Physical Distribution & Logistics Management,. doi: 10.1108/09600031211202481.Google Scholar
  33. Mongeon, P., & Paul-Hus, A. (2016). The journal coverage of Web of Science and Scopus: A comparative analysis. Scientometrics, 106(1), 213–228. doi: 10.1007/s11192-015-1765-5.CrossRefGoogle Scholar
  34. Moody, D. L. (2000). Building links between IS research and professional practice: Improving the relevance and impact of IS research. In Proceedings of the twenty first international conference on information systems (Issue 1, pp. 351–360). Retrieved from
  35. Neely, A. (2005). The evolution of performance measurement research: Developments in the last decade and a research agenda for the next. International Journal of Operations & Production Management, 25(12), 1264–1277. doi: 10.1108/01443570510633648.CrossRefGoogle Scholar
  36. Nie, K., Ma, T., & Nakamori, Y. (2009). An approach to aid understanding emerging research fields—The case of knowledge management. Systems Research and Behavioral Science, 26(6), 629–643. doi: 10.1002/sres.926.CrossRefGoogle Scholar
  37. Nissen, M. E. (1995). A focused review or the reengineering literature: Expert frequently asked questions. Quality Management Journal, 3(3), 52–66.Google Scholar
  38. Pasqualine, A., Plytiuk, C. F., da Costa, S. E. G., & de Lima, E. P. (2012). Performance management in healthcare: A bibliometric review. In IIE Annual Conference. Proceedings (p. 1). Institute of Industrial Engineers-Publisher.Google Scholar
  39. Patra, S. K., Bhattacharya, P., & Verma, N. (2006). Bibliometric study of literature on bibliometrics. DESIDOC Bulletin of Information Technology, 26(1), 27–32. Retrieved from
  40. Paul-Hus, A., Desrochers, N., & Costas, R. (2016). Characterization, description, and considerations for the use of funding acknowledgement data in Web of Science. Scientometrics,. doi: 10.1007/s11192-016-1953-y.Google Scholar
  41. Perianes-Rodriguez, A., & Ruiz-Castillo, J. (2016). A comparison of two ways of evaluating research units working in different scientific fields. Scientometrics, 106(2), 539–561. doi: 10.1007/s11192-015-1801-5.CrossRefGoogle Scholar
  42. Piro, F. N., Rørstad, K., & Aksnes, D. W. (2016). How does prolific professors influence on the citation impact of their university departments? Scientometrics, 107(3), 1–21. doi: 10.1007/s11192-016-1900-y.CrossRefGoogle Scholar
  43. Porter, A. L., & Detampel, M. J. (1995). Technology opportunities analysis. Technological Forecasting and Social Change, 49(3), 237–255. doi: 10.1016/0040-1625(95)00022-3.CrossRefGoogle Scholar
  44. Pritchard, A. (1969). Statistical bibliography or bibliometrics. Journal of Documentation, 25, 348.Google Scholar
  45. Schmenner, R. W., Wassenhove, L. Van, Ketokivi, M., Heyl, J., & Lusch, R. F. (2009). Too much theory, not enough understanding. Journal of Operations Management, 27(5), 339–343. doi: 10.1016/j.jom.2009.07.004.CrossRefGoogle Scholar
  46. Schoepflin, U., & Glänzel, W. (2001). Two decades of “Scientometrics” An interdisciplinary field represented by its leading journal. Scientometrics, 50(2), 301–312. doi: 10.1023/A:1010577824449.CrossRefGoogle Scholar
  47. Small, H. (1973). Co-citation in the scientific literature: A new measure of the relationship between two documents. Journal of the American Society for Information Science, 24(4), 265–269.CrossRefGoogle Scholar
  48. Small, H. (1999). Visualizing science by citation mapping. Journal of the American Society for Information Science, 50(9), 799–813. doi: 10.1002/(SICI)1097-4571(1999)50:9<799:AID-ASI9>3.0.CO;2-G.CrossRefGoogle Scholar
  49. Smith, D. R. (2012). Impact factors, scientometrics and the history of citation-based research. Scientometrics, 92(2), 419–427.CrossRefGoogle Scholar
  50. Stone, K. B. (2012). Four decades of lean: A systematic literature review. International Journal of Lean Six Sigma, 3(2), 112–132. doi: 10.1108/20401461211243702.CrossRefGoogle Scholar
  51. Sud, P., & Thelwall, M. (2014). Evaluating altmetrics. Scientometrics, 98(2), 1131–1143. doi: 10.1007/s11192-013-1117-2.CrossRefGoogle Scholar
  52. Sun, X., Lin, H., Xu, K., & Ding, K. (2015). How we collaborate: Characterizing, modeling and predicting scientific collaborations. Scientometrics,. doi: 10.1007/s11192-015-1597-3.Google Scholar
  53. Taylor, A., & Taylor, M. (2009). Operations management research: Contemporary themes, trends and potential future directions. International Journal of Operations & Production Management, 29(12), 1316–1340. doi: 10.1108/01443570911006018.CrossRefGoogle Scholar
  54. Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14, 207–222. doi: 10.1111/1467-8551.00375.CrossRefGoogle Scholar
  55. Walsh, D., & Downe, S. (2005). Meta-synthesis method for qualitative research: A literature review. Journal of Advanced Nursing,. doi: 10.1111/j.1365-2648.2005.03380.x.Google Scholar
  56. Wendler, R. (2012). The maturity of maturity model research: A systematic mapping study. Information and Software Technology, 54(12), 1317–1339. doi: 10.1016/j.infsof.2012.07.007.CrossRefGoogle Scholar
  57. Zhao, Y., & Zhao, R. (2016). An evolutionary analysis of collaboration networks in scientometrics. Scientometrics, 107(2), 759–772. doi: 10.1007/s11192-016-1857-x.CrossRefGoogle Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2016

Authors and Affiliations

  • Heather Keathley-Herring
    • 1
    Email author
  • Eileen Van Aken
    • 2
  • Fernando Gonzalez-Aleu
    • 3
  • Fernando Deschamps
    • 4
    • 5
  • Geert Letens
    • 6
  • Pablo Cardenas Orlandini
    • 2
  1. 1.Department of Industrial Engineering and Management SystemsUniversity of Central FloridaOrlandoUSA
  2. 2.Grado Department of Industrial and Systems EngineeringVirginia TechBlacksburgUSA
  3. 3.Department of EngineeringUniversidad de MonterreySan Pedro Garza GarciaMexico
  4. 4.Polytechnic SchoolPontifical Catholic University of ParanaCuritibaBrazil
  5. 5.Department of Mechanical EngineeringFederal University of ParanaCuritibaBrazil
  6. 6.Department of Economics, Management and LeadershipRoyal Military AcademyBrusselsBelgium

Personalised recommendations