Abstract
Although teachers are sometimes portrayed as unreliable raters because of their emotional involvement and proximity to students or test-takers, it can be argued that they have more expertise and experience in rating test-takers’ performances than most test developers. Therefore, it seems only logical to include them in the development of rating scales. This applies to both scenarios in which teachers are only responsible for preparing students for High-Stakes exams, and scenarios where teachers are responsible for test preparation as well as the rating of the test performances. Involving teachers in rating scale design can offer test developers access to a wealth of rating experience and thereby increase the validity of the scale. It can also instil an important feeling of ownership in the teachers, which seems indispensable for the promotion of positive attitudes towards High-Stakes exams.
This chapter will outline the potentials and challenges of involving secondary school teachers in the design of rating instruments for a large-scale national High-Stakes exam. Two case studies on teacher involvement in scale development will be presented (writing and speaking). The chapter will compare the two projects, highlighting what was found useful by the involved teachers. It will do so by analyzing teacher statements from retrospective questionnaires (N = 23) about their experience of being involved in one or both of these projects. The chapter will conclude with insights into the importance of teacher involvement in this stage of the test development cycle, and will highlight the usefulness of combining top-down and bottom-up scale development procedures.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Scale development was only one part of the workshops. The teachers also developed test specifications and designed writing tasks during the workshops.
- 2.
As opposed to the writing project, where teachers also performed other tasks during the workshops, the sole purpose of the speaking workshops was scale development.
- 3.
Three of these teachers also took part in the writing project, so their responses were included in both datasets.
References
Alderson, J. C., Nagy, E., & Öveges, E. (Eds.). (2000). English language education in Hungary. Part II: Examining Hungarian learners’ achievements in English. Budapest: The British Council.
Bachman, L. F., & Palmer, A. S. (2010). Language assessment in practice. Oxford: Oxford University Press.
Bundesministerium für Unterricht und Kunst. (1974). Leistungsbeurteilung in Pflichtschulen sowie mittleren und höheren Schulen. Retrieved September 30, 2016, from https://www.ris.bka.gv.at/GeltendeFassung.wxe?Abfrage=Bundesnormen&Gesetzesnummer=10009375
Council of Europe. (2001). Common European framework of reference for languages: Learning, teaching, assessment. Cambridge: Cambridge University Press.
Council of Europe. (2009). Relating language examinations to the Common European framework of reference for languages: Learning, teaching, assessment (CEFR): A manual. Strasbourg: Language Policy Division.
Deygers, B., & Van Gorp, K. (2015). Determining the scoring validity of a co-constructed CEFR-based rating scale. Language Testing, 32, 521–541.
Deygers, B., Van Gorp, K., & Joos, S. (2013). Rating scale design: A comparative study of two analytic rating scales in a task-based test. In E. Galaczi & C. J. Weir (Eds.), Exploring language frameworks: Proceedings from the ALTE Kraków conference, July 2011 (pp. 273–289). Cambridge: Cambridge University Press.
Fulcher, G. (2003). Testing second language speaking. Harlow: Pearson Longman.
Fulcher, G. (2012). Assessment literacy for the language classroom. Language Assessment Quarterly, 9, 113–132.
Fulcher, G., Davidson, F., & Kemp, J. (2011). Effective rating scale development for speaking tests: Performance decision trees. Language Testing, 28, 5–29.
Galaczi, E., French, A., Hubbard, C., & Green, A. (2011). Developing assessment scales for large-scale speaking tests: A multiple-method approach. Assessment in Education: Principles, Policy and Practice, 18, 217–237.
Green, A. (2014). Exploring language assessment and testing. London: Routledge.
Harsch, C., & Martin, G. (2012). Adapting CEF-descriptors for rating purposes: Validation by a combined rater training and scale revision approach. Assessing Writing, 17, 228–250.
Hudson, T. (2005). Trends in assessment scales and criterion-referenced language assessment. Annual Review of Applied Linguistics, 25, 205–227.
Knoch, U. (2011). Rating scales for diagnostic assessment of writing: What should they look like and where should the criteria come from? Assessing Writing, 16, 81–96.
Konzett, C. (2011). Every word counts. Fine-tuning the language of assessment scales: A field report. Paper presented at IATEFL TEASIG 2011: ‘Standards and standardizing in high and low stakes exams: Assessment from classroom to Matura’, Innsbruck.
Spöttl, C., & Green, R. (2009). Going national, standardised and live in Austria: Challenges and tensions. Paper presented at the 6th Annual EALTA Conference, Turku.
Spöttl, C., Kremmel, B., Holzknecht, F., & Alderson, J.C. (2016). Evaluating the achievements and challenges in reforming a national language exam: The reform team’s perspective. Papers in Language Testing and Assessment, 5, 1–22.
Tankó, G. (2005). Into Europe: The writing handbook. Budapest: Teleki Lazlo Foundation and The British Council Hungary.
Turner, C. E. (2000). Listening to the voices of rating scale developers: Identifying salient features for second language performance assessment. The Canadian Modern Language Review, 56, 555–584.
Turner, C. E. (2012). Rating scales for language tests. In C. A. Chapelle (Ed.), The encyclopedia of applied linguistics. Oxford: Wiley-Blackwell.
Upshur, J. A., & Turner, C. E. (1995). Constructing rating scales for second language tests. ELT Journal, 49, 3–12.
Upshur, J. A., & Turner, C. E. (1999). Systematic effects in the rating of second-language speaking ability: Test method and learner discourse. Language Testing, 16, 82–111.
Weigle, S. C. (2002). Assessing writing. Cambridge: Cambridge University Press.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this chapter
Cite this chapter
Holzknecht, F., Kremmel, B., Konzett, C., Eberharter, K., Spöttl, C. (2018). Potentials and Challenges of Teacher Involvement in Rating Scale Design for High-Stakes Exams. In: Xerri, D., Vella Briffa, P. (eds) Teacher Involvement in High-Stakes Language Testing. Springer, Cham. https://doi.org/10.1007/978-3-319-77177-9_4
Download citation
DOI: https://doi.org/10.1007/978-3-319-77177-9_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-77175-5
Online ISBN: 978-3-319-77177-9
eBook Packages: EducationEducation (R0)