Abstract
Despite the fact that relatively little attention has been paid to the assessment of second language (L2) pragmatics competence in the fields of L2 pragmatics and language testing (Roever, 2011), there has been a growing body of research on pragmatics assessment since Hudson, Detmer, and Brown (1992, 1995) developed a framework for assessing cross-cultural pragmatics. Hudson et al. developed six prototype pragmatics test instruments: (a) a multiple-choice discourse completion test (DCT), (b) an open-ended written DCT, (c) a language lab DCT, (d) a role play, (e) a self-assessment task, and (f) a role-play self-assessment. Each different test measures written and spoken aspects of pragmatics competence assessed by raters or in a self-assessment format. Hudson et al. also investigated the reliability and validity of their instruments using six analytical rating criteria for raters: (a) ability to use the correct speech act, (b) formulaic expression, (c) amount of speech in a given situation, (d) formality level, (e) directness level, and (f) overall politeness level. These rating criteria reflect diverse factors within pragmatics competence and each rater used the criteria to assess each test item. Since then, researchers following Hudson et al.’s framework in various L2 contexts (e.g., Ahn, 2005; Brown, 2001; Hudson, 2001; Yamashita, 1996; Yoshitake, 1997; Youn, 2008), or developing their own test instruments (e.g., Grabowski, 2009; Liu, 2007; Roever, 2005, 2006; Tada, 2005), have conducted an increasing amount of research on L2 pragmatics assessment. Studies that employed Hudson et al.’s framework in various L2 contexts have consistently reported reasonably high reliability and validity measures for all of the test types except the multiple-choice.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Ahn, R. C. (2005). Five measures of interlanguage pragmatics in KFL (Korean as foreign language) learners. Unpublished doctoral dissertation, University of Hawai’i at Mānoa, Honolulu, HI.
Bond, T. G., & Fox, C. M. (2007). Applying the Rasch Model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum.
Bonk, W. J., & Ockey, G. J. (2003). A many-facet Rasch analysis of the second language group oral discussion task. Language Testing, 20: 89–110.
Brown, J. D. (2001). Pragmatics tests. In K. R. Rose & G. Kasper (eds.), Pragmatics in Language Teaching (pp. 301–325). Cambridge: Cambridge University.
Brown, J. D. (2008). Raters, functions, item types and the dependability of L2 pragmatics tests. In E. A. Soler & A. Martinez-Flor (eds.), InvestigatingPragmatics in Foreign Language Learning, Teaching and Testing (pp. 224–248). Clevedon: Multilingual Matters.
Brown, J. D., & Ahn, R.C. (2011). Variables that affect the dependability of L2 pragmatics tests. Journal of Pragmatics, 43: 198–217.
Eckes, T. (2005). Examining rater effects in TestDaF writing and speaking performance assessments: A many-facet Rasch analysis. Language Assessment Quarterly, 2: 197–221.
Grabowski, K. C. (2009). Investigating the construct validity of a test designed to measure grammatical and pragmatic knowledge in the context of speaking. Unpublished doctoral dissertation. Columbia University, New York.
Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of Item Response Theory. Newbury Park: Sage.
Hudson, T. (2001). Indicators for pragmatic instruction. In K. R. Rose & G. Kasper (eds.), Pragmatics in Language Teaching (pp. 283–300). Cambridge: Cambridge University.
Hudson, T., Detmer, E., & Brown, J. D. (1992). A Framework for Testing Cross-cultural Pragmatics (Technical Report No.2). Honolulu: University of Hawai’i, Second Language Teaching and Curriculum Center.
Hudson, T., Detmer, E., & Brown, J. D. (1995). Developing Prototypic Measures of Cross-Cultural Pragmatics (Technical Report No.7). Honolulu: University of Hawai’i, Second Language Teaching and Curriculum Center.
Kasper, G. (2006). Speech acts in interaction: Towards discursive pragmatics. In K. Bardovi-Harlig, C. Félix-Brasdefer, & A. S. Omar (eds.), Pragmatics and Language Learning, Volume 11 (pp. 281–314). Honolulu, HI: Second Language Teaching and Curriculum Center, University of Hawai’i.
Kasper, G., & Rose, K. R. (2002). Pragmatic Development in a Second Language. Malden, MA: Blackwell.
Knoch, U. (2009). Diagnostic assessment of writing: A comparison of two rating scales. Language Testing, 26: 275–304.
Kondo-Brown, K. (2002). A FACETS analysis of rater bias in measuring Japanese second language writing performance. Language Testing, 19: 3–31.
Kozaki, Y. (2004). Using GENOVA and FACETS to set multiple standards on performance assessment for certification in medical translation from Japanese into English. Language Testing, 21: 1–27.
Kozaki, Y. (2010). An alternative decision-making procedure for performance assessments: Using the multifaceted Rasch model to generate cut estimates. Language Assessment Quarterly, 7: 75–95.
Linacre, J. M. (1989). Many-faceted Rasch Measurement. Chicago: MESA.
Linacre, J. M. (1994). Sample size and item calibration stability. Rasch Measurement Transactions, 7: 328. Retrieved November 1, 2011 from http://www.rasch.org/rmt/rmt74m.htm
Linacre, J. M. (1996). FACETS, version no. 3.0. Chicago: MESA.
Linacre, J. M. (2006). Facets Rasch measurement computer program. Chicago: Winsteps. corn.
Liu, J. (2007). Developing a pragmatic test for Chinese EFL learners. Language Testing, 24: 391–415.
Lynch, B. K., & McNamara, T. F. (1998). Using G-theory and Many-faceted Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants. Language Testing, 15: 158–180.
Matsuno, S. (2009). Self-, peer-, and teacher-assessments in Japanese university EFL writing classrooms. Language Testing, 26: 75–100.
McNamara, T. F. (1996). Measuring Second Language Performance. New York: Addison Wesley Longman.
Roever, C. (2005). Testing ESL pragmatics: Development and validation of a web-based assessment battery. Frankfurt: Peter Lang Publishing.
Roever, C. (2006). Validation of a web-based test of ESL pragmalinguistics. Language Testing, 23: 229–256.
Roever, C. (2011). Testing of second language pragmatics: Past and future. Language Testing, 28: 463–481.
Rose, K. R., & Kasper, G. (eds.) (2001). Pragmatics in Language Teaching. Cambridge: Cambridge University Press.
Sohn, H. M. (1986). Linguistic Expeditions. Seoul, Korea: Hanshin.
Sohn, H. M. (1988). Linguistic devices of Korean politeness. Paper presented at the sixth International conference on Korean linguistics. International Circle of Korean Linguistics & Department of East Asian Studies, University of Toronto. Seoul, Korea: Hanshin.
Sohn, H. M. (1999). Korean: Descriptive grammars. London: Routledge.
Tada, M. (2005). Assessment of ESL pragmatic production and perception using video prompts. Unpublished doctoral dissertation, Temple University, Philadelphia, PA.
Valdés, G. (2001). Heritage language students: Profiles and possibilities. In J. K. Peyton, D. A. Ranard, & S. McGinnis (eds.), Heritage Languages in America: Preserving a national resource (pp. 37–77). McHenry, IL: The Center for Applied Linguistics and Delta Systems.
Yamashita, S. O. (1996). Six Measures of JSL Pragmatics (Technical Report No.14). Honolulu: University of Hawai’i, Second Language Teaching and Curriculum Center.
Yoshitake, S. (1997). Interlanguage competence of Japanese students of English: A multitest framework evaluation. Unpublished doctoral dissertation, Columbia Pacific University, San Rafael, CA.
Youn, S. J. (2008). Rater variation in paper vs. web-based KFL pragmatic assessment using FACETS analysis. Unpublished manuscript, University of Hawai’i, Honolulu, HI.
Editor information
Editors and Affiliations
Copyright information
© 2013 Soo Jung Youn & James Dean Brown
About this chapter
Cite this chapter
Youn, S.J., Brown, J.D. (2013). Item difficulty and heritage language learner status in pragmatic tests for Korean as a foreign language. In: Ross, S.J., Kasper, G. (eds) Assessing Second Language Pragmatics. Palgrave Advances in Language and Linguistics. Palgrave Macmillan, London. https://doi.org/10.1057/9781137003522_4
Download citation
DOI: https://doi.org/10.1057/9781137003522_4
Publisher Name: Palgrave Macmillan, London
Print ISBN: 978-1-137-35213-2
Online ISBN: 978-1-137-00352-2
eBook Packages: Palgrave Language & Linguistics CollectionEducation (R0)