Skip to main content
SpringerLink
Log in

Investigating the Judges Performance in a National Competition of Sport Dance

  • Original Research
  • Published:
Social Indicators Research Aims and scope Submit manuscript

Abstract

Many sports, such as gymnastics, diving, figure skating, etc. use judges’ scores to generate a rank for determining the winner of a competition. These judges use some type of rating scale when assessing performances. Human ratings are subject to various forms of error and bias. The overall outcomes may largely depend upon the set of chosen raters. The aim of this paper is to illustrate how results from the Many-Facet Rasch Measurement framework can be used to highlight feedback to judges about their scoring patterns. The purpose is to analytically detect anomalous rater behaviours. We consider the field of Sport Dance, a discipline which enjoys increasing public interest and passion in recent years. We analyze data relating to two national competitions held in Italy in 2018 and 2019.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. Data available at the website https://www.federdanza.it/images/gare/2017_2018/EXPORT/CAMPIONATI_2018/arena_bianca_05/12-grp-d_synchrolat_u11_c/index.htm.

  2. Data available at the website https://www.federdanza.it/images/gare/2018_2019/EXPORT/arena_bianca_05/10-p-grp_synchrolat_u15_c/index.htm.

References

  • Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43(4), 561–573.

    Article  Google Scholar 

  • Bachman, L. F. (2004). Statistical analyses for language assessment. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Carroll, J. D., & Chang, J.-J. (1970). Analysis of individual differences in multidimensional scaling via an \(n\)-way generalization of “eckart-young” decomposition. Psychometrika, 35(3), 283–319.

    Article  Google Scholar 

  • Eckes, T. (2011). Introduction to many-facet Rasch measurement: Analyzing and evaluating rater-mediated assessments. Peter Lang Edition: Language Testing and Evaluation.

  • Engelhard, G. (2002). Large-scale assessment programs for all students: Validity, technical adequacy, and implementation. In Monitoring raters in performance assessments, Chapter 11 (pp. 261–287). Routledge.

  • Farrokhi, F., & Esfandiari, R. (2011). A many-facet rasch model to detect halo effect in three types of raters. Theory and Practice in Language Studies, 1(11), 1531–1540.

    Article  Google Scholar 

  • Harshman, R. (1970). Foundations of the parafac procedure: Models and conditions for an “explanatory” multimodal factor analysis. In: UCLA working papers in phonetics Vol 16 (pp. 1–84).

  • Knoch, U. (2009). Diagnostic assessment of writing: A comparison of two rating scales. Language Testing, 26(2), 275–304.

    Article  Google Scholar 

  • Knoch, U., Read, J., & von Randow, J. (2007). Re-training writing raters online: How does it compare with face-to-face training? Assessing Writing, 12(1), 26–43.

    Article  Google Scholar 

  • Linacre, J. (1989). Many-facet Rasch measurement (I ed.). Chicago: Mesa Press.

    Google Scholar 

  • Linacre, J. (1994). Many-facet Rasch measurement (II ed.). Chicago: Mesa Press.

    Google Scholar 

  • Linacre, J. (2002). Facets, factors, elements and levels. Rasch Measurement Transactions, 16(2), 880.

    Google Scholar 

  • Linacre, J. (2009). Local independence and residual covariance: A study of olympic figure skating ratings. Journal of Applied Measurement, 10(2), 157–169.

    Google Scholar 

  • Linacre, J. (2013). Facets computer program for many-facet Rasch measurement, version 3.71. 4. Beaverton, Oregon: https://www.winsteps.com.

  • Looney, M. (2004). Evaluating judge performance in sport. Journal of Applied Measurement, 5(1), 31–47.

    Google Scholar 

  • Murphy, K., & Balzer, W. (1989). Rater errors and rating accuracy. Journal of Applied Psychology, 74(4), 619.

    Article  Google Scholar 

  • Myford, C., & Wolfe, E. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement, 4(4), 386–422.

    Google Scholar 

  • Myford, C., & Wolfe, E. (2004). Detecting and measuring rater effects using many-facet Rasch measurement: Part II. Journal of Applied Measurement, 5(2), 189–227.

    Google Scholar 

  • Parke, C., Lane, S., & Stone, C. A. (2006). Impact of a state performance assessment program in reading and writing. Educational Research and Evaluation, 12(3), 239–269.

    Article  Google Scholar 

  • Rasch, G. (1960). Studies in mathematical psychology: I. In: Probabilistic models for some intelligence and attainment tests. Nielsen and Lydiche.

  • Roever, C., & McNamara, T. (2006). Language testing: The social dimension. International Journal of Applied Linguistics, 16(2), 242–258.

    Article  Google Scholar 

  • Saal, F., Downey, R., & Lahey, M. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88(2), 413.

    Article  Google Scholar 

  • Tucker, L. R. (1966). Some mathematical notes on three-mode factor analysis. Psychometrika, 31(3), 279–311.

    Article  Google Scholar 

  • Wolfe, E. W., & Dobria, L. (2008). Best practices in quantitative methods, chapter applications of the multifaceted Rasch model (pp. 71–85). Sage Thousand Oaks, CA.

  • Wright, B. D., & Stone, M. H. (1979). Best test design. Chicago: Mesa press.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Bologna, Via Belle Arti 41, 40126, Bologna, Italy

    Laura Anderlucci, Alessandro Lubisco & Stefania Mignani

Authors
  1. Laura Anderlucci
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alessandro Lubisco
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefania Mignani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Laura Anderlucci.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Anderlucci, L., Lubisco, A. & Mignani, S. Investigating the Judges Performance in a National Competition of Sport Dance. Soc Indic Res 156, 783–799 (2021). https://doi.org/10.1007/s11205-019-02256-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11205-019-02256-z

Keywords

Search

Navigation