Skip to main content
SpringerLink
Log in

Criteria and Bias in Native English Teachers’ Assessment of L2 Pragmatic Appropriacy: Content and FACETS Analyses

  • Regular Article
  • Published:
The Asia-Pacific Education Researcher Aims and scope Submit manuscript

Abstract

Although there are studies on the pragmatic assessment, to date no research has been done on native English raters’ criteria for the assessment of EFL learners’ pragmatic competence. Focusing on this topic, this study pursued two purposes. The first one was to find the criteria for rating the speech act of apology in L2 by native English teachers. The second was to investigate whether there was rater bias in native English teachers’ rating of apology. To this end, 51 native English teachers rated six different pragmatic situations for an apology discourse completion task (DCT) which were accompanied by an L2 learner’s response to each situation. Besides rating, the raters were asked to describe the way they rated the response to each DCT situation. The content analysis of raters’ descriptions revealed five criteria they mostly applied in their rating: expression of apology, situation explanation, repair offer, promise for future, and politeness. FACETS was used to find the rater bias. Results indicated that raters showed different degrees of severity and leniency in their ratings, which contradicts the myth of native speakers being a benchmark in language assessment, including pragmatic assessment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Ang-Aw, H. T., & Goh, C. C. M. (2011). Understanding discrepancies in rater judgement on national-level oral examination tasks. RELC Journal, 42(1), 31–51.

    Article  Google Scholar 

  • Barnwell, D. (1989). ‘Naïve’ native speakers and judgments of oral proficiency in Spanish. Language Testing, 6(2), 152–163.

    Article  Google Scholar 

  • Brown, A. (1995). The effect of rater variables in the development of an occupation-specific language performance test. Language Testing, 12(1), 1–15.

    Article  Google Scholar 

  • Brown, S., & Attardo, S. (2000). Understanding language structure, interaction, and variation. Ann Arbor: The University of Michigan Press.

    Google Scholar 

  • Brown, P., & Levinson, S. (1987). Politeness. Cambridge: Cambridge University Press.

    Google Scholar 

  • Dalmau, S. M., & Gotor, C. H. (2007). From ‘sorry very much’ to ‘I’m ever so sorry’: Acquisitional patterns in L2 apologies by Catalan learners of English. Intercultural Pragmatics, 4(2), 287–315.

    Article  Google Scholar 

  • Du, Y., Wright, B. D., & Brown, W. L. (1996). Differential facet functioning detection in direct writing assessment. Paper presented at the Annual Meeting of the American Educational Research Association, New York.

  • Eckes, T. (2005). Examining rater effects in TestDaF writing and speaking performance assessments: A many-facet Rasch analysis. Language Assessment Quarterly, 2(3), 197–221.

    Article  Google Scholar 

  • Eckes, T. (2008). Rater types in writing performance assessments: A classification approach to rater variability. Language Testing, 25(2), 155–185.

    Article  Google Scholar 

  • Elder, C., Barkhuizen, G., Knock, U., & Randow, J. (2007). Evaluating rater response to an online training program for L2 writing assessment. Language Testing, 24(1), 37–64.

    Article  Google Scholar 

  • Englehard, G. (1994). Examining rater errors in the assessment of written compositions with a many-faceted Rasch model. Journal of Educational Measurement, 31(2), 93–112.

    Article  Google Scholar 

  • Fraser, B. (1981). On apologizing. In F. Coulmas (Ed.), Conversational routine: Explorations in standardized communication situations and prepatterned speech (pp. 259–271). New York: Mouton.

    Google Scholar 

  • Galloway, V. B. (1980). Perceptions of the communicative efforts of American students of Spanish. Modern Language Journal, 64(4), 428–433.

    Article  Google Scholar 

  • Holmes, J. (1990). Apologies in New Zealand English. Language in Society, 19(2), 155–199.

    Article  Google Scholar 

  • Hyland, K., & Anan, E. (2006). Teachers’ perceptions of error: The effects of first language and experience. System, 34(4), 509–519.

    Article  Google Scholar 

  • Johnson, J. S., & Lim, G. S. (2009). The influence of rater language background on writing performance assessment. Language Testing, 26(4), 485–505.

    Article  Google Scholar 

  • Kim, H. (2008). The semantic and pragmatic analysis of South Korean and Australian English apologetic speech acts. Journal of Pragmatics, 40(2), 257–278.

    Article  Google Scholar 

  • Kondo, S. (1997a). Longitudinal study on the development of pragmatic competence in a natural learning context: Perception behind performance. Proceedings of Sophia University Linguistic Society, 12, 35–54.

    Google Scholar 

  • Kondo, S. (1997b). The development of pragmatic competence by Japanese learners of English in a natural learning context: Longitudinal study on interlanguage apologies, MA Thesis, Sophia University, Graduate School of Language and Linguistics.

  • Kondo-Brown, K. (2002). A FACETS analysis of rater bias in measuring Japanese L2 writing performance. Language Testing, 19(1), 3–31.

    Article  Google Scholar 

  • Lee, H. K. (2009). Native and non-native rater behavior in grading Korean students’ English essays. Asia Pacific Education Review, 10(3), 387–397.

    Article  Google Scholar 

  • Lim, G. S. (2011). The development and maintenance of rating quality in performance writing assessment: A longitudinal study of new and experienced raters. Language Testing, 28(4), 543–560.

    Article  Google Scholar 

  • Linacre, J. M., & Wright, B. (1996). FACETS. Chicago: MESA Press.

    Google Scholar 

  • Liu, J. (2006). Assessing EFL learners’ interlanguage pragmatic knowledge: Implications for testers and teachers. Reflections on English Language Teaching, 5(1), 1–22.

    Google Scholar 

  • Lumley, T. (2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19(3), 246–276.

    Article  Google Scholar 

  • Lumley, T. (2005). Assessing second language writing: The rater’s perspective. Frankfurt: Peter Lang.

    Google Scholar 

  • Lumley, T., & McNamara, T. F. (1995). Rater characteristics and rater bias: Implications for training. Language Testing, 12(1), 54–71.

    Article  Google Scholar 

  • Maeshiba, N., Yoshinaga, N., Kasper, G., & Ross, S. (1996). Transfer and proficiency in interlanguage apologizing. In S. Gass & J. Neu (Eds.), Speech acts across cultures (pp. 155–187). Berlin: Mouton de Gruyter.

    Google Scholar 

  • McIntyre, P. N. (1993). The importance and effectiveness of moderation training on the reliability of teachers’ assessment of ESL writing samples. MA Thesis, University of Melbourne.

  • McNamara, T. F. (1996). Measuring second language performance. London: Longman.

    Google Scholar 

  • Nonaka, K. (2000). Apology is not necessary: An in-depth analysis of my own intercultural and intracultural miscommunication. Journal of Hokkaido University of Education at Kushiro, 32, 155–186.

    Google Scholar 

  • Olshtain, E., & Cohen, A. D. (1983). Apology: A speech act set. In N. Wolfson & E. Judd (Eds.), Sociolinguistics and language acquisition (pp. 18–35). Rowley: Newbury House.

    Google Scholar 

  • Orr, M. (2002). The FCE Speaking test: using rater reports to help interpret test scores. System, 30(2), 143–154.

    Article  Google Scholar 

  • Plough, I. C., Briggs, S. L., & Van Bonn, S. (2010). A multi-method analysis of evaluation criteria used to assess the speaking proficiency of graduate student instructors. Language Testing, 27(2), 235–260.

    Article  Google Scholar 

  • Roch, S. G. (2007). Why convene rater teams: an investigation of the benefits of anticipated discussion, consensus, and rater motivation. Organizational Behavior and Human Decision Processes, 104(1), 14–29.

    Article  Google Scholar 

  • Roever, C. (2001). A web-based test of interlanguage pragmalinguistic knowledge: Speech acts, routines, and implicatures. Unpublished doctoral Dissertation, University of Hawai‘i, Honolulu, Hawai‘i.

  • Schaefer, E. (2008). Rater bias patterns in an EFL writing assessment. Language Testing, 25(4), 465–493.

    Article  Google Scholar 

  • Shohamy, E., Gordon, C. M., & Kraemer, R. (1992). The effect of raters’ background and training on the reliability of direct writing tests. Modern Language Journal, 76(1), 27–33.

    Article  Google Scholar 

  • Son, M. (2010). Examining rater bias: An evaluation of possible factors influencing elicited imitation ratings. MA project, Brigham Young University.

  • Song, B., & Caruso, I. (1996). Do English and ESL faculty differ in evaluating the essays of native English-speaking and ESL students? Journal of Second Language Writing, 5(2), 163–182.

    Article  Google Scholar 

  • Sugimoto, N. (1997). A Japan–U.S. comparison of apology styles. Communication Research, 24(4), 349–370.

    Article  Google Scholar 

  • Taguchi, N. (2010). Longitudinal studies in interlanguage pragmatics. In A. Trosborg (Ed.), Handbook of pragmatics (Vol. 7, pp. 333–361)., Pragmatics across languages and cultures Berlin: Mouton de Gruyter.

    Google Scholar 

  • Taguchi, N. (2011). Rater variation in the assessment of speech acts. Pragmatics, 21(3), 453–471.

    Google Scholar 

  • Weigle, S.C. (1994a). Effects of training on raters of English as a second language compositions: Quantitative and qualitative approaches. Ph.D. Dissertation, University of California, Los Angeles.

  • Weigle, S. C. (1994b). Effects of training on raters of ESL compositions. Language Testing, 11(2), 197–223.

    Article  Google Scholar 

  • Weigle, S. C. (1998). Using FACETS to model rater training effects. Language Testing, 15(2), 263–287.

    Article  Google Scholar 

  • Wigglesworth, G. (1993). Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction. Language Testing, 10(3), 305–335.

    Article  Google Scholar 

  • Wigglesworth, G. (1994). Patterns of rater behaviour in the assessment of an oral interaction test. Australian Review of Applied Linguistics, 17(2), 77–103.

    Google Scholar 

  • Yamashita, S. (2008). Investigating interlanguage pragmatic ability. In E. A. Solar & A. Martinez-Flor (Eds.), Investigating pragmatics in foreign language learning, teaching and testing (pp. 201–223). Bristol: Multilingual Matters.

    Google Scholar 

  • Youn, S. J. (2007). Rater bias in assessing the pragmatics of KFL learners using facets analysis. Second Language Studies, 26(1), 85–163.

    Google Scholar 

  • Zhang, Y., & Elder, C. (2011). Judgments of oral proficiency by non-native and native English speaking teacher raters: Competing or complementary constructs? Language Testing, 28(1), 31–50.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Allameh Tabataba’i University, Tehran, Iran

    Zia Tajeddin

  2. Sharif University of Technology, Tehran, Iran

    Minoo Alemi

Authors
  1. Zia Tajeddin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Minoo Alemi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Minoo Alemi.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Tajeddin, Z., Alemi, M. Criteria and Bias in Native English Teachers’ Assessment of L2 Pragmatic Appropriacy: Content and FACETS Analyses. Asia-Pacific Edu Res 23, 425–434 (2014). https://doi.org/10.1007/s40299-013-0118-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40299-013-0118-5

Keywords

Search

Navigation