Skip to main content
SpringerLink
Log in

Exploring a prototype framework of web-based and peer-reviewed “European Educational Research Quality Indicators” (EERQI)

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Digitization, the Internet, and information or webometric interdisciplinary approaches are affecting the fields of Scientometrics and Library and Information Science. These new approaches can be used to improve citation-only procedures to estimate the quality and impact of research. A European pilot to explore this potential was called “European Educational Research Quality Indicators” (EERQI, FP7 # 217549). An interdisciplinary consortium was involved from 2008 to 2011. Different types of indicators were developed to score 171 educational research documents. Extrinsic bibliometric and citation indicators were collected from the Internet for each document; intrinsic indicators reflecting content-based quality were developed and relevant data gathered by peer review. Exploratory and confirmatory factor analysis and structural modeling were used to explore statistical relationships among latent factors or concepts and their indicators. Three intrinsic and two extrinsic latent factors were found to be relevant. Moreover, the more a document was related to a reviewer’s own area of research, the higher the score the reviewer gave concerning (1) significance, originality, and consistency, and (2) methodological adequacy. The conclusions are that a prototype EERQI framework has been constructed: intrinsic quality indicators add specific information to extrinsic quality or impact indicators, and vice versa. Also, a problem of “objective” impact scores is that they are based on “subjective” or biased peer-review scores. Peer-review, which is foundational to having a work cited, seems biased and this bias should be controlled or improved by more refined estimates of quality and impact of research. Some suggestions are given and limitations of the pilot are discussed. As the EERQI development approach, instruments, and tools are new, they should be developed further.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Alexander, F. (2012). Assessing information taxonomies using epistemology and the sociology of science. Journal of Documentation, 68(5), 725–743.

    Article  Google Scholar 

  • Alexander, F. (2014). Devising a framework for assessing the subjectivity and objectivity of information taxonomy projects. Journal of Documentation, 70(1), 4–24.

    Article  Google Scholar 

  • Barnett, G. A., Park, H. W., Jiang, K., Tang, C., & Aguillo, I. F. (2014). A multi-level network analysis of web-citations among the world’s universities. Scientometrics, 99, 5–26.

    Article  Google Scholar 

  • Bridges, D. (2009). Research quality assessment: Impossible science, possible art? British Educational Research Journal, 35(4), 497–517.

    Article  Google Scholar 

  • Campanario, J. M. (2014). The effect of citations on the significance of decimal places in the computation of journal impact factors. Scientometrics, 99, 289–298.

    Article  Google Scholar 

  • Centre for Higher Education Research and Information (CHERI), & Hobsons Research. (2008). Counting What is Measured or Measuring What Counts? League tables and their impact on higher education institutions in England. Report to HEFCE. http://www.hefce.ac.uk/data/year/2008/countingwhatismeasuredormeasuringwhatcounts/. Accessed March 25, 2014.

  • Cerchiello, P., & Giudici, P. (2014). On a statistical h index. Scientometrics, 99, 299–312.

    Article  Google Scholar 

  • Dam Christensen, H. (2014). The framing of scientific domains: About UNISIST, domain analysis and art history. Journal of Documentation, 70(2), 261–281.

    Article  Google Scholar 

  • Devena, S. E., Gay, C. E., & Watkins, M. W. (2013). Confirmatory factor analysis of the WISC-IV in a hospital referral sample. Journal of Psychoeducational Assessment, 31(6), 591–599.

    Article  Google Scholar 

  • Eccles, K. E., Thelwall, M., & Meyer, E. T. (2012). Measuring the web impact of digitised scholarly resources. Journal of Documentation, 68(4), 512–526.

    Article  Google Scholar 

  • EERQI project. (2010). State of the Art Reports on EERQI Project Parts. Preparatory meeting for the second EERQI workshop, 18–19 March. http://eerqi.eu/de/page/final-report. Accessed January 11, 2011.

  • EERQI project. (2011). European Educational Research Quality Indicators. Project final report. http://eerqi.eu/de/page/final-report. Accessed June 23, 2012.

  • Finnemann, N. O. (2014). Research libraries and the internet: On the transformative dynamic between institutions and digital media. Journal of Documentation, 70(2), 202–220.

    Article  Google Scholar 

  • Fragkiadaki, E., & Evangelidis, G. (2014). Review of the indirect citations paradigm: Theory and practice of the assessment of papers, authors and journals. Scientometrics, 99, 261–288.

    Article  Google Scholar 

  • Fry, T. D., & Donohue, J. M. (2014). Exploring the author affiliation index. Scientometrics, 98, 1647–1667.

    Article  Google Scholar 

  • Gogolin, I. (2008). European Educational Research Quality Indicators (EERQI). (Project 217549). FP7 Collaborative project. Hamburg: Universität Hamburg.

  • Gogolin, I., Åström, F., & Hansen, A. (2014) (Eds.). Assessing quality in European Educational Research. Indicators and approaches. Wiesbaden: Springer.

  • Gradmann, S. (2014). From containers to content to context: The changing role of libraries in eScience and eScholarship. Journal of Documentation, 70(2), 241–260.

    Article  Google Scholar 

  • Gradmann, S., Haveman, F., & Oltersdorf, J. (2014). Studies in correlative assessing of intrinsic and extrinsic indicators of quality. In I. Gogolin, F. Åström, & A. Hansen (Eds.), Assessing quality in European Educational Research. Indicators and approaches (pp. 60–84). Wiesbaden: Springer.

  • Gradmann, S., Sieber, J., & Stoye, D. (2011). Extrinsic indicators used in EERQI. Berlin: Humboldt-Universität, Institut für Bibliotheks- und Informationswissenschaft.

    Google Scholar 

  • Hjørland, B. (2012). Is classification necessary after Google? Journal of Documentation, 68(3), 299–317.

    Article  Google Scholar 

  • Hjørland, B., Nielsen, H. J., & Høyrup, H. (2014). Introduction to the special issue: Perspectives on research libraries. Journal of Documentation, 70(2), 198–201.

    Article  Google Scholar 

  • Lee, C. A. (2011). A framework for contextual information in digital collections. Journal of Documentation, 67(1), 95–143.

    Article  Google Scholar 

  • Locoro, A., David, J., & Euzenat, J. (2014). Context-based matching: Design of a flexible framework and experiment. Journal on Data Semantics, 3(1), 25–46.

    Article  Google Scholar 

  • Lorentzen, D. G. (2014). Webometrics benefitting from web mining? An investigation of methods and applications of two research fields. Scientometrics, 99, 409–445.

    Article  Google Scholar 

  • Lu, K., Ajiferuke, I., & Wolfram, D. (in press). Extending citer analysis to journal impact evaluation. Scientometrics.

  • Martínez, M. A., Herrera, M., López-Gijón, J., & Herrera-Viedma, E. (2014). H-Classics: Characterizing the concept of citation classics through H-index. Scientometrics, 98, 1971–1983.

    Article  Google Scholar 

  • Mehta, V., & Hull, D. M. (2013). Structural validity of the professional development profile of the LoTi digital-age survey. Journal of Psychoeducational Assessment, 31(1), 61–71.

    Article  Google Scholar 

  • Mooij, T. (2008). Intermediate conceptual framework and procedures to construct EERQI. Contribution to the international project ‘European Educational Research Quality Indicators’ (Project 217549; FP7). Nijmegen, The Netherlands: Radboud University, ITS.

  • Mooij, T. (2011). European Educational Research Quality Indicators (EERQI): A first prototype framework of intrinsic and extrinsic indicators. Paper presented at the final conference of the international collaborative project ‘European Educational Research Quality Indicators’ (EU Project 217549; FP7) in Brussels, University Foundation, March 15–16, 2011. Nijmegen, The Netherlands: Radboud University, ITS.

  • Mooij, T. (2014). A prototype empirical framework of intrinsic and extrinsic EERQI indicators. In I. Gogolin, F. Åström, & A. Hansen (Eds.), Assessing quality in European Educational Research. Indicators and approaches (pp. 121–138). Wiesbaden: Springer.

  • Mooij, T., & Fettelaar, D. (2013). School and pupil effects on secondary pupils’ feelings of safety in school, around school, and at home. Journal of Interpersonal Violence, 28(6), 1240–1265.

    Article  Google Scholar 

  • Muthén, L. K., & Muthén, B. O. (1998–2010). Mplus user’s guide (6th edn). Los Angeles, CA: Muthén & Muthén.

  • Nielsen, H. J., & Hjørland, B. (2014). Curating research data: The potential roles of libraries and information professionals. Journal of Documentation, 70(2), 221–240.

    Article  Google Scholar 

  • Nolin, J., & Åström, F. (2010). Turning weakness into strength: Strategies for future LIS. Journal of Documentation, 66(1), 7–27.

    Article  Google Scholar 

  • Olsen, B. I., Lund, N. W., Ellingsen, G., & Hartvigsen, G. (2012). Document theory for the design of socio-technical systems: A document model as ontology of human expression. Journal of Documentation, 68(1), 100–126.

    Article  Google Scholar 

  • Rostami, F., Mohammadpoorasl, S., & Hajizadeh, M. (2014). The effect of characteristics of title on citation rates of articles. Scientometrics, 98, 2007–2010.

    Article  Google Scholar 

  • Sándor, Á., & Vorndran, A. (2009). Detecting key sentences for automatic assistance in peer reviewing research articles in educational sciences. http://dl.acm.org/citation.cfm?id=1699757. Accessed March 10, 2011.

  • Sándor, Á., & Vorndran, A. (2014a). Highlighting salient sentences for reading assistance. In I. Gogolin, F. Åström, & A. Hansen (Eds.), Assessing quality in European Educational Research. Indicators and approaches (pp. 43–55). Wiesbaden: Springer.

  • Sándor, Á., & Vorndran, A. (2014b). Enhancing relevance ranking of the EERQI search engine. In I. Gogolin, F. Åström, & A. Hansen (Eds.), Assessing quality in European Educational Research. Indicators and approaches (pp. 56–59). Wiesbaden: Springer.

  • Selek, S., & Saleh, A. (2014). Use of h index and g index for American academic psychiatry. Scientometrics, 99, 541–548.

    Article  Google Scholar 

  • Sieber, J., & Stoye, D. (2011). Description of a measure. Berlin: Humboldt-Universität, Institut für Bibliotheks- und Informationswissenschaft. http://edoc.hu-berlin.de/oa/reports/reJ3Xv4PJ82ZM/PDF/29B6vgnyGba6.pdf Accessed June 27, 2012.

  • Su, X., Deng, S., & Shen, S. (2014). The design and application value of the Chinese Social Science Citation Index. Scientometrics, 98, 1567–1582.

    Article  Google Scholar 

  • Taylor, A. (2013). Examination of work task and criteria choices for the relevance judgment process. Journal of Documentation, 69(4), 523–544.

    Article  Google Scholar 

  • Technopolis. (2009a). Identification and dissemination of lessons learned by institutions participating in the Research Excellence Framework (REF) bibliometrics pilot. Results of the round one consultation—Report to HEFCE. http://www.hefce.ac.uk/data/year/2009/identificationanddisseminationoflessonslearnedbyinstitutionsparticipatingintheresearchexcellenceframeworkrefbibliometricspilotresultsoftheroundoneconsultation/. Accessed Feb 21, 2012.

  • Technopolis. (2009b). Identification and dissemination of lessons learned by institutions participating in the Research Excellence Framework (REF) bibliometrics pilot. Results of the round two consultation—Report to HEFCE. http://www.hefce.ac.uk/data/year/2009/identificationanddisseminationoflessonslearnedbyinstitutionsparticipatingintheresearchexcellenceframeworkrefbibliometricspilotresultsoftheroundtwoconsultation/. Accessed Feb 21, 2012.

  • Teodorescu, D., & Andrei, T. (2014). An examination of “citation circles” for social sciences journals in Eastern European countries. Scientometrics, 99, 209–231.

    Article  Google Scholar 

  • Thomson Reuters. (2013). Journal Citation Reports ®. http://thomsonreuters.com/products_services/science/science_products/a-z/journal_citation_reports/. Accessed June 20, 2013.

  • Williams, R. B. (2011). Citation systems in the biosciences: A history, classification and descriptive terminology. Journal of Documentation, 67(6), 995–1014.

    Article  Google Scholar 

  • Yu, T., Yu, G., Li, P.-Y., & Wang, L. (in press). Citation impact prediction for scientific papers using stepwise regression analysis. Scientometrics.

Download references

Acknowledgments

The pilot ‘European Educational Research Quality Indicators’ (EERQI) was EU funded under the Socio-Economic Sciences and Humanities Theme (FP7 # 217549). The project was developed and conducted by an interdisciplinary European research consortium. I want to thank all partners for their collaboration. Moreover, I am grateful to Daan Fettelaar MSc for his assistance in performing the necessary analyses and to Dr. Maureen Snow Andrade for her check of the appropriate use of English language.

Author information

Authors and Affiliations

  1. Radboud University Nijmegen (ITS), Toernooiveld 212, 6525 EC, Nijmegen, The Netherlands

    Ton Mooij

  2. Open Universiteit in the Netherlands (Welten Institute), Valkenburgerweg 177, 6419 AT, Heerlen, The Netherlands

    Ton Mooij

  3. Radboud University, ITS, PO Box 9048, 6500 KJ, Nijmegen, The Netherlands

    Ton Mooij

Authors
  1. Ton Mooij
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ton Mooij.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mooij, T. Exploring a prototype framework of web-based and peer-reviewed “European Educational Research Quality Indicators” (EERQI). Scientometrics 102, 1037–1055 (2015). https://doi.org/10.1007/s11192-014-1429-x

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-014-1429-x

Keywords

Search

Navigation