Skip to main content
SpringerLink
Log in

Evaluating research-oriented teaching: a new instrument to assess university students’ research competences

  • Published:
Higher Education Aims and scope Submit manuscript

Abstract

Several concepts have been developed to implement research-oriented teaching in higher education in the last 15 years. The definition of research competences, however, has received minor attention so far. Some approaches to modeling research competences describe these competences along the research process but either focus on a specific academic discipline and/or specific facets or offer no empirical verification for cross-disciplinary approaches. We have developed a new approach to model research competences across various academic disciplines in form of a competence model—the RMRC-K-model. This model comprises five dimensions: skills in reviewing the state of research, methodological skills, skills in reflecting on research findings, communication skills, and content knowledge. In this study, we present an instrument (R-Comp) developed on the basis of this cross-disciplinary competence model. The factorial structure of the R-Comp was examined with data from 391 university students in three groups, either enrolled in a Bachelor’s degree, Master’s degree, or a PhD program. The sample represented various academic disciplines. Results from confirmatory factor analyses supported the hypothesized structure of the R-Comp for the five dimensions in accordance with the RMRC-K-model. Additionally, results provided evidence for a more detailed differentiation of all dimensions with sub-dimensions reflecting theorized facets of the model. Both Cronbach’s α and construct reliability H indicated overall good reliability. Despite limitations to this study, there is some first evidence of a valid and reliable assessment of student research competences.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. This sub-sample includes participants from Veterinary Medicine. This study program is a state examination program, licensed and organized by national government agencies. It is separated in two sequences (first and second state examination). Participants in our sub-sample were in the first sequence, which is comparable to the level of a Bachelor’s degree program. Therefore, we classified these participants as Bachelor students to gain a better understanding of our sub-sample in international contexts.

  2. The skill dimensions as well as the knowledge dimension were specified as factors at the same hierarchical level in the empirical model. In pre-studies (Böttcher and Thiel 2016), we examined whether the skills and knowledge dimensions should be modeled separately or in a combined model. Our results indicated that a combined model in line with Simonton’s (2013) conceptualization of competence was the more appropriate solution.

  3. In the empirical model, some theorized facets had to be combined on the basis of theoretical considerations about the research process. Otherwise, some sub-dimensions would have consisted of single indicators.

  4. We thank an anonymous reviewer for his/her suggestion of testing bi-factor models.

  5. Thanks to one of our anonymous reviewers for rising this point.

  6. We are very thankful that one of our anonymous reviewers has drawn attention to this important limitation of our study.

  7. We appreciate our anonymous reviewer for providing this idea.

References

  • Albrecht, L., Arends, H., Burg, A., Jubin, B., Langer, C., Lehmann, J., et al. (2003). FL2 Forschendes Lernen – Lehrende Forschung. https://incom.org/action/open-file/131430. Accessed 6 November 2013.

  • BAK. (2009). Forschendes Lernen - wissenschaftliches Prüfen: Ergebnisse der Arbeit des Ausschusses für Hochschuldidaktik (2nd ed., Schriften der Bundesassistentenkonferenz, Vol. 5). Bielefeld: UVW, Webler.

  • Barnette, J. J. (2000). Effects of stem and likert response option reversals on survey internal consistency: if you feel the need, there is a better alternative to using those negatively worded stems. Educational and Psychological Measurement, 60(3), 361–370. doi:10.1177/00131640021970592.

    Article  Google Scholar 

  • Bauer, W., Bleck-Neuhaus, J., Dombois, R., & Wehrtmann, I. (2013). Forschungsprojekte entwickeln - von der Idee bis zur Publikation (1st ed., UTB, Vol. 4019). Baden-Baden: Nomos.

  • Birkelbach, K. 2005 Über das Messen von Kompetenzen: Einige Überlegungen im Anschluss an ein BMBF-Projekt., Vortrag auf der Herbsttagung der Sektion Berufs- und Wirtschaftspädagogik der DGFE am 20./21. September 2005 in Erfurt.

  • Blömeke, S., & Zlatkin-Troitschanskaia, O. (2013). Kompetenzmodellierung und Kompetenzerfassung im Hochschulsektor: Ziele, theoretischer Rahmen, Design und Herausforderungen des BMBFForschungsprogramms KoKoHs (KoKoHs Working Papers, 1). Berlin & Mainz: Humboldt-Universität & Johannes Gutenberg-Universität.

    Google Scholar 

  • Bollen, K. A., & Davis, W. R. (2009). Two rules of identification for structural equation models. Structural Equation Modeling: A Multidisciplinary Journal, 16(3), 523–536. doi:10.1080/10705510903008261.

    Article  Google Scholar 

  • Böttcher, F., & Thiel, F. (2016). Introduction of a new instrument to assess university students’ research competences: R-Comp. Paper presented at the international Higher Education Conference (HEC), Amsterdam, Netherlands, July, 13.

  • Brew, A. (2006). Research and teaching: Beyond the divide (universities into the twenty-first century). Houndmills, Basingstoke, Hampshire, New York: Palgrave Macmillan.

    Book  Google Scholar 

  • Brown, T. A. (2006). Confirmatory factor analysis for applied research (methodology in the social sciences). New York: Guilford Press.

    Google Scholar 

  • Bühner, M. (2011). Einführung in die Test- und Fragebogenkonstruktion (3rd ed., Psychologie ed.). München, Boston: Pearson Studium.

    Google Scholar 

  • Butts, R. E. (1991). Methodology, metaphysics and the pragmatic unity of science. In Akademie der Wissenschaften zu Berlin (Ed.), Einheit der Wissenschaften: Internationales Kolloquium der Akademie der Wissenschaften zu Berlin (Bonn, 25.-27. Juni 1990) (pp. 23–28, Forschungsbericht, Vol. 4). Berlin, New York: W. de Gruyter.

  • Chang, H.-P., Chen, C.-C., Guo, G.-J., Cheng, Y.-J., Lin, C.-Y., & Jen, T.-H. (2011). The development of a competence scale for learning science: inquiry and communication. International Journal of Science and Mathematics Education, 9(5), 1213–1233. doi:10.1007/s10763-010-9256-x.

    Article  Google Scholar 

  • Dewey, J. (1938). Unity of science as a social problem. Chicago: University of Chicago Press.

    Google Scholar 

  • Diekmann, A. (1995). Empirische Sozialforschung: Grundlagen, Methoden, Anwendungen. Reinbek bei Hamburg: Rowohlt Taschenbuchverl.

    Google Scholar 

  • Geiser, C. (2011). Datenanalyse mit Mplus: Eine anwendungsorientierte Einführung (2nd ed., Lehrbuch). Wiesbaden: VS Verlag für Sozialwissenschaften.

  • Ginns, P., Marsh, H. W., Behnia, M., Cheng, J. H. S., & Scalas, L. F. (2009). Using postgraduate students’ evaluations of research experience to benchmark departments and faculties: issues and challenges. The British Journal of Educational Psychology, 79(Pt 3), 577–598. doi:10.1348/978185408X394347.

    Article  Google Scholar 

  • Griffiths, R. (2004). Knowledge production and the research–teaching nexus: the case of the built environment disciplines. Studies in Higher Education, 29(6), 709–726. doi:10.1080/0307507042000287212.

    Article  Google Scholar 

  • Groß Ophoff, J., Schladitz, S., Lohrmann, K., & Wirtz, M. (2014). Evidenzorientierung in bildungswissenschaftlichen Studiengängen. In K. Drossel, R. Strietholt, & W. Bos (Eds.), Empirische Bildungsforschung und evidenzbasierte Reformen im Bildungswesen (pp. 250–276). Münster, Westf: Waxmann.

  • Hancock, G. R., & Mueller, R. O. (2001). Rethinking construct reliability within latent variable systems. Structural equation modeling: Present and future, 195–216.

  • Healey, M. (2005). Linking research and teaching to benefit student learning. Journal of Geography in Higher Education, 29(2), 183–201. doi:10.1080/03098260500130387.

    Article  Google Scholar 

  • Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. doi:10.1080/10705519909540118.

    Article  Google Scholar 

  • Huber, L. (2009). Warum forschendes Lernen nötig und möglich ist. In L. Huber (Ed.), Forschendes Lernen im Studium: Aktuelle Konzepte und Erfahrungen (pp. 9–35, Motivierendes Lehren und Lernen in Hochschulen, Vol. 10). Bielefeld: UVW, Webler.

  • Klieme, E., Hartig, J., & Rauch, D. (2008). The concept of competence in educational contexts. In E. Klieme, D. Leutner, & J. Hartig (Eds.), Assessment of competencies in educational contexts (pp. 3–22). Toronto: Hogrefe & Huber Publishers.

  • Kline, R. B. (2005). Principles and practice of structural equation modeling (2nd ed., Methodology in the social sciences ed.). New York: Guilford Press.

    Google Scholar 

  • Kossek, B. (2009). Survey: Die forschungsgeleitete Lehre in der internationalen Diskussion. https://ctl.univie.ac.at/fileadmin/user_upload/elearning/Forschungsgeleitete_Lehre_International_090414.pdf. Accessed 17 Dec 2015.

  • Langer, W. (2000). Einführung in sozialwissenschaftliche Skalen-, Index- und Typenkonstruktion, 2000.

  • Marsh, H. W., Rowe, K. J., & Martin, A. (2002). PhD students’ evaluations of research supervision: issues, complexities, and challenges in a nationwide australian experiment in benchmarking universities. The Journal of Higher Education, 73(3), 313–348.

    Google Scholar 

  • Mayer, R. E. (2003). What causes individual differences in cognitive performance? In R. J. Sternberg & E. Grigorenko (Eds.), Perspectives on the psychology of abilities, competencies, and expertise (pp. 263–273). New York: Cambridge University Press.

  • Meijers, A. W. M., van Overveld, K. C. W. A. M., & Perrenet, J. C. (2005). Criteria for academic bachelor’s and master’s curricula. Delft: TU Delft.

  • Mittelstraß, J. (1991). Einheit und Transdisziplinarität. In Akademie der Wissenschaften zu Berlin (Ed.), Einheit der Wissenschaften: Internationales Kolloquium der Akademie der Wissenschaften zu Berlin (Bonn, 25.-27. Juni 1990) (pp. 12–22, Forschungsbericht, Vol. 4). Berlin, New York: W. de Gruyter.

  • Reise, S. P., Moore, T. M., & Haviland, M. G. (2010). Bifactor models and rotations: exploring the extent to which multidimensional data yield univocal scale scores. Journal of Personality Assessment, 92(6), 544–559. doi:10.1080/00223891.2010.496477.

    Article  Google Scholar 

  • Satorra, A., & Bentler, P. M. (2001). A scaled difference chi-square test statistic for moment structure analysis. Psychometrika, 66(4), 507–514.

    Article  Google Scholar 

  • Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the fit of structural equation models: tests of significance and descriptive goodness-of-fit measures. Methods of psychological research online, 8(2), 23–74.

    Google Scholar 

  • Simonton, D. K. (2003). Expertise, competence, and creative ability. In R. J. Sternberg & E. Grigorenko (Eds.), Perspectives on the psychology of abilities, competencies, and expertise (pp. 213–240). New York: Cambridge University Press.

  • Smyth, L., Davila, F., Sloan, T., Rykers, E., Backwell, S., & Jones, S. B. (2015). How science really works: The student experience of research-led education. Higher Education. doi:10.1007/s10734-015-9945-z

  • Stein, J. A., Lee, J. W., & Jones, P. S. (2006). Assessing cross-cultural differences through use of multiple-group invariance analyses. Journal of Personality Assessment, 87(3), 249–258. doi:10.1207/s15327752jpa8703_05.

    Article  Google Scholar 

  • Thiel, F., & Böttcher, F. (2014). Modellierung fächerübergreifender Forschungskompetenzen. Das RMKR-W-Modell als Grundlage der Planung und Evaluation von Formaten forschungsorientierter Lehre. In B. Berendt, H.-P. Voss, & J. Wildt (Eds.), Neues Handbuch Hochschullehre. Lehren und Lernen effizient gestalten. [Teil] I. Evaluation. Fachbereichs- /Studiengangsevaluation (pp. 1–124). Berlin: Raabe.

  • Tremp, P. (2005). Verknüpfung von Lehre und Forschung: Eine universitäre Tradition als didaktische Herausforderung. Beiträge zur Lehrerbildung, 23(3), 339–348.

    Google Scholar 

  • Valter, K., & Akerlind, G. (2010). Introducing students to ways of thinking and acting like a researcher: a case study of research-led education in the sciences. International Journal of Teaching and Learning in Higher Education, 22(1), 89–97.

    Google Scholar 

  • Visser-Wijnveen, G. J., van der Rijst, Roeland M., & van Driel, J. H. (2015). A questionnaire to capture students’ perceptions of research integration in their courses. Higher Education. doi:10.1007/s10734–015–9918-2

  • Weinert, F. E. (2001). Concept of competence: A conceptual clarification. In D. S. Rychen & L. H. Sagalnik (Eds.), Definition and selection of competencies-theoretical and conceptual foundations (pp. 45–65). Kirkland, WA: Hogrefe & Huber.

  • Weinert, F. E. (2002). Schulleistungen - Leistungen der Schule oder der Schüler? In F. E. Weinert (Ed.), Leistungsmessungen in Schulen (2nd ed., pp. 73–86, Beltz Pädagogik). Weinheim [u.a.]: Beltz-Verl.

  • Willison, J., & O’Regan, K. (2007). Commonly known, commonly not known, totally unknown: a framework for students becoming researchers. Higher Education Research & Development, 26(4), 393–409.

    Article  Google Scholar 

  • Yuan, K.-H., & Bentler, P. M. (2000). Three likelihood-based methods for mean and covariance structure analysis with nonnormal missing data. Sociological Methodology, 30, 165–200.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Education and Psychology, School Pedagogy and School Improvement Research, Freie Universität Berlin, Habelschwerdter Allee 45, 14195, Berlin, Germany

    Franziska Böttcher & Felicitas Thiel

Authors
  1. Franziska Böttcher
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Felicitas Thiel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Franziska Böttcher.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Böttcher, F., Thiel, F. Evaluating research-oriented teaching: a new instrument to assess university students’ research competences. High Educ 75, 91–110 (2018). https://doi.org/10.1007/s10734-017-0128-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10734-017-0128-y

Keywords

Search

Navigation