Skip to main content
SpringerLink
Log in

An integrated strategy for the analysis of student evaluation of teaching: from descriptive measures to explanatory models

  • Published:
Quality & Quantity Aims and scope Submit manuscript

Abstract

Over the last decade, the assessment of university teaching quality has assumed a prominent role in the university system with the main purpose of improving the quality of courses offered to students. As a result of this process, a host of studies on the evaluation of university teaching was devoted to the Italian system, covering different topics and considering case studies and methodological issues. Based upon this debate, the contribution aims to present an integrated strategy of analysis which combines both descriptive and model-based methods for the treatment of student evaluation of teaching data. More specifically, the joint use of item response theory and multilevel models allows, on the one hand, to compare courses’ ranking based on different indicators and, on the other hand, to define a model-based approach for building up indicators of overall students’ satisfaction, while adjusting for their characteristics and differences in the compositional variables across courses. The usefulness and the relative merits of the proposed procedure are discussed within a real data set.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. For details visit the website http://www.valmonsrl.it/index.php?p=501.

  2. Among others, see the technical reports of the evaluation committees available on the universities’ websites of Bari, Bologna, Milan, Naples, Padua, Rome, Salerno, Turin and Venice.

  3. A different questionnaire is used for students who declare to not attend the course or to attend less than 50% of the course lectures.

  4. In Italy, the term lyceum refers to a kind of upper secondary schools mostly theoretical and specialized in teaching basic subjects, as preparation for university. On the other side, the upper secondary schools that are no-lyceum, are devoted to teach specific subjects and provide a preparation mainly oriented to a specific professional figure.

  5. Note that the Spearman’s coefficient \(\rho\) takes values between -1 and +1, indicating respectively discordance and rank correlation. In our analysis, for practical reasons and to better highlight the variability of the results, the color scale used in the graphical representation refers only to the positive interval of \(\rho\), from 0 to 1.

  6. The random intercept models are estimated by using the R2MLWIN package in R (Zhang et al. 2016).

  7. The residual variability explained by differences across the 35 degree programs (level-3) was not relevant. Specifically, the amount of variance explained by differences in ratings across degree programs in the null model was about 1.7 % of the total variance and around 0.0 % in the other estimated models including covariates. Thus differences in ratings across degree programs are all explained by differences in the values of the covariates across students. Thus no further levels of clustering of units have been considered in the multilevel analysis presented here.

References

  • Attanasio, M., Capursi, V.: Statistical Methods for the Evaluation of University Systems. Springer, Berlin (2011)

    Book  Google Scholar 

  • Bacci, S., Caviezel, V.: Multilevel IRT models for the university teaching evaluation. J. Appl. Stat. 38(12), 2775–2791 (2011)

    Article  Google Scholar 

  • Baker, F.B.: The Basics of Item Response Theory. ERIC, College Park (2001)

    Google Scholar 

  • Bernardi, L.: Tes—from impressionism to expressionism. In: Capursi, V., Attanasio, M. (eds.) Statistical Methods for the Evaluation of University Systems, pp. 3–14. Springer, Berlin (2011)

    Chapter  Google Scholar 

  • Capursi, V., Librizzi, L.: La qualità della didattica: indicatori semplici o composti. In: Dottor Divago. Discernere valutare e governare la nuova università, FrancoAngeli, Milano (2008)

  • Capursi, V., Porcu, M.: La didattica universitaria valutata dagli studenti: un indicatore basato su misure di distanza fra distribuzioni di giudizi. Atti del Convegno Intermedio della SIS, Processi e Metodi Statistici di Valutazione, Roma 4–6 giugno (2001)

  • Carpita, M., Marasini, D.: Assicurazione e valutazione della qualità nell’università: quale ruolo per gli statistici?. Analysis. Rivista di cultura e politica scientifica 1, 1–11 (2014)

    Google Scholar 

  • Cerchiello, P., Giudici, P.: An integrated statistical model to measure academic teaching quality. Open J. Stat. 2(5), 491–497 (2012)

    Article  Google Scholar 

  • Conte, T., La Rocca, M., Parrella, M., Primerano, I., Vetro, C., Vitale, M.P.: La valutazione della didattica nel sistema universitario. un prototipo software per l’analisi dei questionari degli studenti. Rapporto tecnico, Progetto ORSUC “Osservatorio Regionale Sistema Universitario Campano”, POR Campania FESR 2007-2013 (2015)

  • Crescenzi, F., Mignani, S.: Statistical Methods and Applications from a Historical Perspective: Selected Issues. Springer, Switzerland (2014)

    Book  Google Scholar 

  • De Boeck, P., Wilson, M. (eds.): Explanatory Item Response Models. A Generalized Linear and non Linear Approach. Springer, New York (2004)

  • Fabbris, L.: Effectiveness of University Education in Italy. Physica-Verlag, Heidelberg (2007)

    Book  Google Scholar 

  • Goldstein, H.: Multilevel Statistical Models. Wiley Series in Probability and Statistics, 4th edn. Wiley, Hoboken (2011)

    Google Scholar 

  • Goldstein, H., Spiegelhalter, D.J.: League tables and their limitations: statistical issues in comparisons of institutional performance. J. R. Stat. Soc. Ser. A 159(3), 385–443 (1996)

    Article  Google Scholar 

  • Iannario, M.: Hierarchical CUB models for ordinal variables. Commun. Stat. 41(16–17), 3110–3125 (2012)

    Article  Google Scholar 

  • Kember, D., Leung, D.Y., Kwan, K.: Does the use of student feedback questionnaires improve the overall quality of teaching? Asses. Eval. High. Educ. 27(5), 411–425 (2002)

    Article  Google Scholar 

  • Labovitz, S.: The assignment of numbers to rank order categories. Am. Sociol. Rev. 35(3), 515–524 (1970)

    Article  Google Scholar 

  • Leckie, G., Goldstein, H.: The limitations of using school league tables to inform school choice. J. R. Stat. Soc Ser. A 172(4), 835–851 (2009)

    Article  Google Scholar 

  • Marasini, D., Quatto, P.: Descriptive analysis of student ratings. J. Appl. Quant. Methods 6(4), 125–133 (2011)

    Google Scholar 

  • Marasini, D., Quatto, P.: A family of indices for teaching evaluation: Experiences in italian universities. In: Crescenzi, F., Mignani, S. (eds.) Statistical Methods and Applications from a Historical Perspective, pp. 293–301. Springer, Switzerland (2014)

    Google Scholar 

  • Marsh, H.W.: Students evaluations of university teaching: dimensionality, reliability, validity, potential biases and usefulness. In: Smart, J., Perry, R.P. (eds.) The Scholarship of Teaching and Learning In Higher Education: An Evidence-Based Perspective, pp. 319–383. Springer, Netherlands (2007)

    Chapter  Google Scholar 

  • Monari, P., Bini, M., Piccolo, D., Salmaso, L.: Statistical Methods for the Evaluation of Educational Services and Quality of Products. Springer, Berlin (2009)

    Book  Google Scholar 

  • MURST (2000) Ministero dell’Università e della Ricerca Scientifica e Tecnologica, Osservatorio per la valutazione del sistema universitario, Rapporto finale del gruppo di ricerca. Chiandotto B. Gola M. Questionario di base da utilizzare per l'attuazione di un programma per la valutazione della didattica da parte degli studenti. RdR 1/00 gennaio 2000

  • Rampichini, C., Grilli, L., Petrucci, A.: Analysis of university course evaluations: from descriptive measures to multilevel models. Stat. Methods Appl. 13(3), 357–373 (2004)

    Google Scholar 

  • Ramsden, P.: A performance indicator of teaching quality in higher education: the course experience questionnaire. Stud. High. Educ. 16(2), 129–150 (1991)

    Article  Google Scholar 

  • Samejima, F.: Estimation of ability using a response pattern of graded scores. Psychometrika Monograph Supplement (No. 17) (1969)

  • Sani, C., Grilli, L.: Differential variability of test scores among schools: a multilevel analysis of the fifth-grade INVALSI test using heteroscedastic random effects. J. Appl. Quant. Methods 6(4), 88–99 (2011)

    Google Scholar 

  • Sulis, I., Capursi, V.: Building up adjusted indicators of students evaluation of university courses using generalized item response models. J. Appl. Stat. 40(1), 88–102 (2013)

    Article  Google Scholar 

  • Sulis, I., Porcu, M.: Assessing divergences in mathematics and reading achievement in italian primary schools: a proposal of adjusted indicators of school effectiveness. Soc. Indic. Res. 122(2), 607–634 (2015)

    Article  Google Scholar 

  • Toland, M.D.: Practical guide to conducting an item response theory analysis. J. Early Adolesc. 34(1), 120–151 (2014)

    Article  Google Scholar 

  • Zhang, Z., Parker, R., Charlton, C., Leckie, G., Browne, W.J.: R2MLwiN: A Package to Run MLwiN from within R. J. Stat. Softw. 72(10), 1–43 (2016)

Download references

Acknowledgments

Work supported by POR Campania FESR 2007-2013 project “Osservatorio Regionale Sistema Universitario Campano” - CTS3: Valutazione della didattica e dei servizi per il diritto allo studio. The authors would like to thanks the university administrative staff for the data provided for the analysis and the anonymous reviewers for their helpful comments.

Author information

Authors and Affiliations

  1. Department of Economics and Statistics, University of Salerno, Fisciano, Italy

    Michele La Rocca, Maria Lucia Parrella, Ilaria Primerano & Maria Prosperina Vitale

  2. Department of Social Sciences and Institutions, University of Cagliari, Cagliari, Italy

    Isabella Sulis

Authors
  1. Michele La Rocca
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maria Lucia Parrella
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ilaria Primerano
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Isabella Sulis
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Maria Prosperina Vitale
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maria Prosperina Vitale.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

La Rocca, M., Parrella, M.L., Primerano, I. et al. An integrated strategy for the analysis of student evaluation of teaching: from descriptive measures to explanatory models. Qual Quant 51, 675–691 (2017). https://doi.org/10.1007/s11135-016-0432-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11135-016-0432-0

Keywords

Search

Navigation