Abstract
We examined the translation of PISA test items based on the theory of test translation error (TTTE), which has proven to allow detection of translation errors with unprecedented levels of detail. Translation error (TE) is defined as the lack of equivalence between the original and translated versions of items on multiple translation error dimensions (TEDs) that involve design, language, and content. According to the theory, TE results not only from poor translation, but also from factors that are beyond the translators’ skills (e.g., languages encode meaning in different ways). We examined the Mexican, Spanish language translation of science and mathematics PISA 2006 items. A panel comprising teachers, translators, a linguist, a test developer, and a measurement specialist examined the translation of 193 text analytical units (55 pieces of introductory text and 138 items) and identified and coded the TEs identified on ten TEDs. For each item, TE was measured as the number of different TEDs on which the review panel identified TEs. To determine which TEDs are critical to student performance, we examined the correlation between TE and item difficulty (percentage of correct answers and mean proportional score, respectively for dichotomous and non-dichotomous items) considering different sets of TEDs. The highest correlations were observed for the sets that included the dimensions, Grammar, Semantics, Register, Information, Construct, and Culture. We also observed different magnitudes of correlations for science and mathematics items and a stronger, statistically significant correlation for translated items whose translation the review panel identified more objectionable than for the rest of the items. These results confirm that language- and content-related TEs may threat the validity of translated items. They speak to the value using the TTTE as a formative evaluation tool that PISA countries can use to operationalize translation guidelines.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Adams, R., Berezner, A., & Jakubowski, M. (2010). Analysis of PISA 2006 preferred items ranking using the percent-correct method (OECD Education Working Papers, No. 46). OECD Publishing. Retrieved June 7, 2011, http://dx.doi.org/10.1787/5km4psmntkq5-en
Allalouf, A. (2003). Revising translated differential item functioning items as a tool for improving cross-lingual assessment. Applied Measurement in Education, 16(1), 55–73.
Backhoff, E., Contreras-Niño, L. A., & Solano-Flores, G. (2011). The theory of test translation error and the TIMSS and PISA international test comparisons. Mexico: National Institute for Educational Evaluation [Sp.].
Backhoff, E., Solano-Flores, G., & Contreras-Niño, L. A. (2010, February 18–19). Analysis of the Mexican Spanish language translation of PISA-2006. Presentation at the Ibero-American Seminar on the theory of test translation error in international comparisons. National Ministry of Education and National Institute for Educational Evaluation, Mexico City, Mexico.
Bybee, R., McCrae, B., & Laurie, R. (2009). PISA 2006: An assessment of scientific literacy. Journal of Research in Science Teaching, 46(8), 865–883.
Church, B. (1991). An examination of the effect that commitment to a hypothesis has on auditors’ evaluations of confirming and disconfirming evidence. Contemporary Accounting Research, 7(2), 513–534.
Creswell, J. W., & Miller, D. L. (2000). Determining validity in qualitative inquiry. Theory into Practice, 39(3), 124–130.
Ercikan, K. (1998). Translation effects in international assessment. International Journal of Educational Research, 29, 543–553.
Ercikan, K., Gierl, M. J., McCreith, T., Puham, G., & Koh, K. (2004). Comparability of bilingual versions of assessments: Sources of incomparability of English and French versions of Canada’s national achievement tests. Applied Measurement in Education, 17(3), 301–321.
Gierl, M. J., Rogers, W. T., & Klinger, D. (1999, April). Using statistical and judgmental reviews to identify and interpret translation DIF. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, QC.
Greenfield, P. M. (1997). You can’t take it with you: Why ability assessments don’t cross cultures. American Psychologist, 52(10), 1115–1124.
Grisay, A. (2003). Translation procedures in OECD/PISA 2000 international assessment. Language Testing, 20(2), 225–240.
Grisay, A. (2007). The challenge of adapting PISA materials into non Indo-European languages: Some evidence from a brief exploration of language issues in Chinese and Arabic. OECD Core A Consortium.
Grisay, A., de Jong, J. H., Gebhardt, E., Berezner, A., & Halleux-Monseur, B. (2007). Translation equivalence across PISA countries. Journal of Applied Measurement, 8(3), 249–266.
Halleux-Monseur, B. (2008). Translation, adaptation and verification of test material in OECD international surveys. Paris: Directorate for Education, Institutional Management in Higher Education Governing Board, OECD.
Hambleton, R. K. (1994). Guidelines for adapting educational and psychological tests: A progress report. European Journal of Psychological Assessment, 10(3), 229–244.
Hambleton, R. K., Merenda, P. F., & Spielberger, C. D. (Eds.). (2005). Adapting educational and psychological tests for cross-cultural assessment. Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.
Harkness, J., van de Vijver, F. J. R., & Mohler, P. (Eds.). (2003). Cross-cultural survey methods. Hoboken, NJ: Wiley.
Mullis, I. V. S., Kelly, D. L., & Haley, K. (1996). Translation Verification Procedures. In M. O. Martin & I. V. S. Mullis (Eds.), Third international mathematics and science study: Quality assurance in data collection. Chestnut Hill, MA: Boston College.
Nettle, D., & Romaine, S. (2000). Vanishing voice: The extinction of the world’s languages. New York: Oxford University Press.
Solano-Flores, G., Backhoff, E., & Contreras-Niño, L. A. (2006). Methodology for evaluating the quality of test translations in international test comparisons: The case of Mexico, TIMSS-1995. [Sp.]. Mexico: National Institute for Educational Evaluation (INEE).
Solano-Flores, G., Backhoff, E., & Contreras-Niño, L. A. (2009). Theory of test translation error. International Journal of Testing, 9, 78–91.
Solano-Flores, G., Backhoff, E., & Contreras-Niño, L. A. (2010, February 18–19). Test translation review sessions: A demonstration. Presentation at the Ibero-American Seminar on the theory of test translation error in international comparisons. National Ministry of Education and National Institute for Educational Evaluation, Mexico City, Mexico.
Solano-Flores, G., Contreras-Niño, L. A., & Backhoff, E. (2005, April 12–14). The Mexican translation of TIMSS-95: Test translation lessons from a post-mortem study. Paper presented at the 2005 annual meeting of the National Council on Measurement in Education. Montreal, Quebec, Canada.
Solano-Flores, G., Contreras-Niño, L. A., & Backhoff, E. (2006). Test translation and adaptation: Lessons learned and recommendations for countries participating in TIMSS, PISA, and other international comparisons. REDIE: Electronic Journal of Educational Research, 8(2). [Sp.] http://redie.uabc.mx/vol8no2/contents-solano2.html
Solano-Flores, G., & Gustafson, M. (2012). Assessment of English language learners: A critical, probabilistic, systemic view. In M. Simon, K. Ercikan, & M. Rousseau (Eds.), Improving large scale assessment in education: Theory, issues, and practice. New York, NY: Taylor & Francis, Routledge.
Solano-Flores, G., Trumbull, E., & Kwon, M. (2003, April 21–25). The metrics of linguistic complexity and the metrics of student performance in the testing of English language learners. Symposium paper presented at the 2003 Annual Meeting of the American Evaluation Research Association. Chicago.
Solano-Flores, G., Trumbull, E., & Nelson-Barber, S. (2002). Concurrent development of dual language assessments: An alternative to translating tests for linguistic minorities. International Journal of Testing, 2(2), 107–129.
Truss, L. (2004). Eats, shoots & leaves. New York: Gotham Books.
van de Vijver, F. J. R., & Poortinga, Y. H. (2005). Conceptual and methodological issues in adapting tests. In R. K. Hambleton, P. F. Merenda, & C. D. Spielberger (Eds.), Adapting educational and psychological tests for cross-cultural assessment. Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer Science+Business Media Dordrecht
About this chapter
Cite this chapter
Solano-Flores, G., Contreras-Niño, L.Á., Backhoff, E. (2013). The Measurement of Translation Error in PISA-2006 Items: An Application of the Theory of Test Translation Error. In: Prenzel, M., Kobarg, M., Schöps, K., Rönnebeck, S. (eds) Research on PISA. Springer, Dordrecht. https://doi.org/10.1007/978-94-007-4458-5_5
Download citation
DOI: https://doi.org/10.1007/978-94-007-4458-5_5
Published:
Publisher Name: Springer, Dordrecht
Print ISBN: 978-94-007-4457-8
Online ISBN: 978-94-007-4458-5
eBook Packages: Humanities, Social Sciences and LawEducation (R0)