Developing an adaptive tool to select, plan, and scaffold oral assessment tasks for undergraduate courses

  • Carrie Demmans Epp
  • Gina Park
  • Christopher Plumb
Development Article


The increased linguistic and cultural diversity of undergraduate classrooms at English language institutions has imposed additional pedagogical and assessment challenges on instructors, many of whom lack the knowledge necessary to design classroom activities and assessments that are fair to all students regardless of students’ background and language abilities. The development of an adaptive instrument for instructors who do not specialize in English language learning represents an attempt to adjust instructional practices to meet this need. This paper reports on the development of an instrument that undergraduate instructors can use to plan their courses at universities where English is the language of instruction. The instrument’s intended use is illustrated through an example that involves the planning of an interdisciplinary undergraduate course. To build this adaptive tool, a taxonomy that describes the relevant components of assessments that involve oral communication was developed and externally reviewed. The questions used in the instrument were then developed and piloted with a group of university undergraduate instructors; after which, the instrument was further refined. Although piloting revealed an increase in instructor awareness of how language abilities relate to assessment, further research is needed to determine the extent to which this tool affects instructor’s classroom or assessment practices.


Assessment Planning Ethics Undergraduates Teaching practices Adaptive instructional tools 



First language


English language learner


International Language Testing Association


International English Language Test System


Test of English as a Foreign Language Internet Based Test


Educational Testing Service


Test of English for International Communication



We would like to thank the reviewers for their guidance. We would also like to thank our instructor, classmates, and participants for their guidance, contributions, and feedback. The first author held W. Garfield Weston and Walter C. Sumner Memorial Fellowships.


  1. Al-Issa, A. S., & Al-Qubtan, R. (2010). Taking the floor: Oral presentations in EFL classrooms. TESOL Journal, 1, 227–246. doi: 10.5054/tj.2010.220425.CrossRefGoogle Scholar
  2. Anaheim University. (2012). Anaheim University—Entrance requirements for accredited online degree and certificate programs. Resource document. Anaheim University. Retrieved December 11, 2012, from
  3. Ang-Aw, H., & Chuen Meng Goh, C. (2011). Understanding discrepancies in rater judgement on national-level oral examination tasks. RELC Journal, 42(1), 31–51. doi: 10.1177/0033688210390226.CrossRefGoogle Scholar
  4. American Psychological Association (1988). Code of fair testing practices in education. Washington, D.C.: Joint Committee on Testing Practices, American Psychological Association.Google Scholar
  5. American Educational Research Association (1999). Standards for educational and psychological testing. Washington, D.C.: American Educational Research Association.Google Scholar
  6. Association of Language Testers in Europe. (2012). The content analysis checklists project. Resource document. Retrieved December 7, 2012, from
  7. Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford, UK: Oxford University Press.Google Scholar
  8. Bachman, L. F., & Palmer, A. S. (2012). Language assessment in practice: Developing language assessments and justifying their use in the real world. Oxford, UK: Oxford University Press.Google Scholar
  9. Bailey, K. (1999). Washback in language testing. Princeton, NJ: Educational Testing Service.Google Scholar
  10. Borup, J., West, R. E., & Graham, C. R. (2012). Improving online social presence through asynchronous video. Internet and Higher Education, 15, 195–203.CrossRefGoogle Scholar
  11. Brown, J. D. (2008). Testing-context analysis: Assessment is just another part of language curriculum development. Language Assessment Quarterly, 5, 275–312.CrossRefGoogle Scholar
  12. Chapelle, C. A., Enright, M., & Jamieson, J. (2010). Does an argument-based approach to validity make a difference? Educational Measurement: Issues and Practice, 29, 3–13.CrossRefGoogle Scholar
  13. Cheng, L., Klinger, D. A., & Zheng, Y. (2007). The challenges of the Ontario Secondary School Literacy Test for second language students. Language Testing, 24, 185–208. doi: 10.1177/0265532207076363.CrossRefGoogle Scholar
  14. Cizek, G. J., Schmid, L. A., & Germuth, A. A. (2011). A Checklist For Evaluating K-12 Assessment Programs. Kalamazoo, MI: The Evaluation Center, Western Michigan University.Google Scholar
  15. Coley, M. (1999). The english language entry requirements of Australian universities for students of non-english speaking background. Higher Education Research & Development, 18, 7–17. doi: 10.1080/0729436990180102.CrossRefGoogle Scholar
  16. Crystal, D. (1991). A dictionary of linguistics and phonetics. Oxford, UK: Basil Blackwell.Google Scholar
  17. Demmans Epp, C., & McCalla, G. I. (2011). ProTutor: Historic open learner models for pronunciation tutoring. In G. Biswas, S. Bull, J. Kay, & A. Mitrovic (Eds.), Artificial intelligence in education (pp. 441–443). Auckland, New Zealand: Springer.CrossRefGoogle Scholar
  18. Demmans Epp, C., Tsourounis, S., Djordjevic, J., & Baecker, R. M. (2013). Interactive event: Enabling vocabulary acquisition while providing mobile communication support. In H. C. Lane, K. Yacef, J. Mostow, & P. Pavlik (Eds.), Artificial intelligence in education (pp. 932–933). Memphis, TN: Springer.CrossRefGoogle Scholar
  19. Deng, C., & Carless, D. (2010). Examination preparation or effective teaching: Conflicting priorities in the implementation of a pedagogic innovation. Language Assessment Quarterly, 7, 285–302. doi: 10.1080/15434303.2010.510899.CrossRefGoogle Scholar
  20. Dix, A., Finlay, J. E., Abowd, G. D., & Beale, R. (2004). Human–computer interaction (3rd ed.). Harlow, England: Pearson/Prentice-Hall.Google Scholar
  21. Educational Testing Service. (2004). ibt/Next generation TOEFL Test: Independent Speaking Rubrics (Scoring Standards). Princeton, NJ: Educational Testing Service.Google Scholar
  22. Educational Testing Service. (2008). Top universities in UK accept TOEFL ® scores. Princeton, NJ: Educational Testing Service.Google Scholar
  23. Educational Testing Service. (2010). User Guide (Speaking & Writing). Princeton, NJ: Educational Testing Service.Google Scholar
  24. Educational Testing Service. (2012). TOEFL Destinations Directory. Princeton, NJ: Educational Testing Service.Google Scholar
  25. Flint, N., & Johnson, B. (2011). Towards fairer university assessment: Recognizing the concerns of students. New York, NY: Taylor & Francis.Google Scholar
  26. Fowler, F. J. (2009). Survey research methods (4th ed.). Thousand Oaks, CA: Sage Publications.Google Scholar
  27. Frost, K., Elder, C., & Wigglesworth, G. (2011). Investigating the validity of an integrated listening-speaking task: A discourse-based analysis of test takers’ oral performances. Language Testing, 29, 345–369. doi: 10.1177/0265532211424479.CrossRefGoogle Scholar
  28. Fulcher, G., & Reiter, R. (2003). Task difficulty in speaking tests. Language Testing, 20(3), 321–344. doi: 10.1191/0265532203lt259oa.CrossRefGoogle Scholar
  29. Gambino, C. P., Acosta, Y. D., & Grieco, E. M. (2014). English-speaking ability of the foreign-born population in the United States: 2012. Washington, DC: US Census Bureau.Google Scholar
  30. Gan, Z. (2013). Task type and linguistic performance in school-based assessment situation. Linguistics and Education, 24(4), 535–544. doi: 10.1016/j.linged.2013.08.004.CrossRefGoogle Scholar
  31. Gilson, C. (1994). Of dinosaurs and sacred cows: The grading of classroom participation. Journal of Management Education, 18, 227–236. doi: 10.1177/105256299401800207.CrossRefGoogle Scholar
  32. Goubeaud, K. (2009). How is science learning assessed at the postsecondary level? Assessment and grading practices in college biology, chemistry and physics. Journal of Science Education and Technology, 19, 237–245. doi: 10.1007/s10956-009-9196-9.CrossRefGoogle Scholar
  33. Heywood, J. (2000). Assessment in higher education: Student learning, teaching, programmes and institutions. Philadelphia, PA: Jessica Kingsley Pub.Google Scholar
  34. Hu, C., Sharpe, L., Crawford, L., Gopinathan, S., Khine, M. S., Moo, S. N., & Wong, A. (2000). Using lesson video clips via multipoint desktop video conferencing to facilitate reflective practice. Journal of Information Technology for Teacher Education, 9(3), 377–388. doi: 10.1080/14759390000200093.CrossRefGoogle Scholar
  35. IELTS. (2007). Handbook 2007. Cambridge, UK: University of Cambridge.Google Scholar
  36. IELTS. (2009). US Recognition List: Educational Institutions, Professional Organizations and Accrediting Bodies Recognizing IELTS. Cambridge, UK: University of Cambridge.Google Scholar
  37. IELTS. (2012a). Institutions: Who accepts IELTS? Cambridge, UK: University of Cambridge. Retrieved December 7, 2012, from
  38. IELTS. (2012b). Speaking: Band descriptors (public version). Cambridge, UK: University of Cambridge. Retrieved November 14, 2012, from
  39. International Language Testing Association (2001). Code of ethics for ILTA.Google Scholar
  40. Ishii, D., & Baba, K. (2003). Locally developed oral skills evaluation in ESL/EFL classrooms: A checklist for developing meaningful assessment procedures. TESL Canada Journal, 21, 79–95.Google Scholar
  41. Joughin, G. (1998). Dimensions of oral assessment. Assessment and Evaluation in Higher Education, 23(4), 367–378.CrossRefGoogle Scholar
  42. Kieffer, M. J., Lesaux, N. K., Rivera, M., & Francis, D. J. (2009). Accommodations for English language learners taking large-scale assessments: A meta-analysis on effectiveness and validity. Review of Educational Research, 79, 1168–1201. doi: 10.3102/0034654309332490.CrossRefGoogle Scholar
  43. King, J. (2002). Preparing EFL learners for oral presentations. Internet TESL Journal, 8(3). Retrieved from
  44. Kunnan, A. J. (2000). Fairness and justice for all. In A. J. Kunnan (Ed.), Fairness and validation in language assessment (pp. 1–14). Cambridge, UK: Cambridge University Press.Google Scholar
  45. Kunnan, A. J. (2004). Test Fairness. In M. Milanovic & C. Weir (Eds.), European language testing in a global context: Proceedings of the ALTE Barcelona conference (pp. 27–48). Cambridge, UK: Cambridge University Press.Google Scholar
  46. Labov, W. (1972). Sociolinguistic patterns. Philadelphia, PN: University of Pennsylvania Press.Google Scholar
  47. Laing, K., & Todd, L. (2012). Fair or foul? Towards practice and policy in fairness in education. Newcastle, UK: Newcastle University. Retrieved April 28, 2014, from
  48. Leki, I. (2007). Undergraduates in a second language: Challenges and complexities of academic literacy development. New York, NY: Lawrence Erlbaum Associates.Google Scholar
  49. MacDonald, K., Alderson, J., & Lai, L. (2004). Selecting and using computer-based language tests (CLBTs) to assess language proficiency: Guidelines for educators. TESL Canada Journal, 21, 93–104.Google Scholar
  50. McNamara, T. F., & Roever, C. (2006). Language testing: The social dimension. Malden, MA: Blackwell Pub.Google Scholar
  51. Melvin, K. B. (1988). Rating class participation: The prof/peer method. Teaching of Psychology, 15, 137–139. doi: 10.1207/s15328023top1503_7.CrossRefGoogle Scholar
  52. NAEP Data Explorer. (2014). 2011 national vocabulary, reading, and writing scores. USA: Institute of Educational Sciences National Center for Education Statistics.Google Scholar
  53. Nielson, J. (1994). Heuristic evaluation. In J. Nielson & R. L. Mack (Eds.), Usability inspection methods (pp. 25–62). New York, NY: John Wiley & Sons.Google Scholar
  54. Nitko, A. J. (2011). Educational assessment of students (6th ed.). Boston, MA: Pearson/Allyn & Bacon.Google Scholar
  55. North, B., & Schneider, G. (1998). Scaling descriptors for language proficiency scales. Language Testing, 15(2), 217–262. doi: 10.1177/026553229801500204.CrossRefGoogle Scholar
  56. Nwoye, O. G. (1992). Linguistic politeness and socio-cultural variations of the notion of face. Journal of Pragmatics, 18, 309–328.CrossRefGoogle Scholar
  57. O’Neil, T., Buckendahl, C., Plake, B., & Taylor, L. (2007). Recommending a nursing-specific passing standard for the IELTS examination. Language Assessment Quarterly, 4, 295–317.CrossRefGoogle Scholar
  58. Otoshi, J., & Heffernen, N. (2008). Factors predicting effective oral presentations in EFL classrooms. Asian EFL Journal, 10(1), 65–78.Google Scholar
  59. Popham, W. J. (2000). Modern educational measurement: Practical guidelines for educational leaders. Boston, MA: Allyn and Bacon Pub.Google Scholar
  60. Robinson, P., Ting, S., & Urwin, J. J. (1995). Investigating second language task complexity. RELC Journal, 26(2), 62–79. doi: 10.1177/003368829502600204.CrossRefGoogle Scholar
  61. Rogers, S. L. (2013). Calling the question: Do college instructors actually grade participation? College Teaching, 61, 11–22. doi: 10.1080/87567555.2012.703974.CrossRefGoogle Scholar
  62. Saville, N. (2003). The process of test development and revision within UCLES EFL. In C. J. Weir & M. Milanovic (Eds.), Continuity and innovation: Revising the Cambridge Proficiency in English Examination, 1913–2002 (pp. 57–120)). Cambridge, UK: Cambridge University Press.Google Scholar
  63. Saville, N. (2005). Setting and monitoring professional standards: A QMS approach. Cambridge ESOL: Research Notes, 22, 2–5.Google Scholar
  64. Scott, M. (1986). Student affective reactions to oral language tests. Language Testing, 3, 99–118.CrossRefGoogle Scholar
  65. Shakya, A., & Horsfall, J. (2000). ESL undergraduate nursing students in Australia: Some experiences. Nursing and Health Sciences, 2, 163–171.CrossRefGoogle Scholar
  66. Shohamy, E. (1982). Affective considerations in language testing. The Modern Language Journal, 66, 13–17. doi: 10.2307/327810.CrossRefGoogle Scholar
  67. Spector, P. (1992). Summated rating scale construction: Introduction. Newbury Park, CA: Sage Publications.CrossRefGoogle Scholar
  68. Taguchi, N. (2007). Task difficulty in oral speech act production. Applin, 28(1), 113–135. doi: 10.1093/applin/aml051.Google Scholar
  69. Trudgill, P. (1983). Sociolinguistics: An introduction to language and society. New York, N.Y.: Penguin.Google Scholar
  70. University of California, San Diego. (2012). GLI—Admissions information. Resource document. University of California, San Diego. Retrieved December 11, 2012, from
  71. Van Moere, A. (2012). A psycholinguistic approach to oral language assessment. Language Testing, 29(3), 325–344. doi: 10.1177/0265532211424478.CrossRefGoogle Scholar
  72. Wall, D., & Alderson, J. C. (1993). Examining washback: the Sri Lankan impact study. Language Testing, 10, 41–69. doi: 10.1177/026553229301000103.CrossRefGoogle Scholar
  73. Webber, K. (2012). The use of learner-centered assessment in US colleges and universities. Research in Higher Education, 53, 201–228.CrossRefGoogle Scholar
  74. Webster, F. (2002). A genre approach to oral presentations. Internet TESL Journal, 8(7). Retrieved from
  75. Welsh, A. J. (2012). Exploring undergraduates’ perceptions of the use of active learning techniques in science lectures. Journal of College Science Teaching, 42, 80–87.Google Scholar
  76. Wieman, C., Perkins, K., & Gilbert, S. (2010). Transforming science education at large research universities: A case study in progress. Change: The Magazine of Higher Learning, 42, 7–14.CrossRefGoogle Scholar
  77. Young, D. J. (1986). The relationship between anxiety and foreign language oral proficiency ratings. Foreign Language Annals, 19, 439–445. doi: 10.1111/j.1944-9720.1986.tb01032.x.CrossRefGoogle Scholar
  78. Zeidner, M., & Bensoussan, M. (1988). College students’ attitudes towards written versus oral tests of English as a foreign language. Language Testing, 5, 100–114.CrossRefGoogle Scholar

Copyright information

© Association for Educational Communications and Technology 2015

Authors and Affiliations

  • Carrie Demmans Epp
    • 1
  • Gina Park
    • 2
  • Christopher Plumb
    • 3
  1. 1.Department of Computer ScienceUniversity of TorontoTorontoCanada
  2. 2.Ontario Institute for Studies in Education (OISE)University of TorontoTorontoCanada
  3. 3.Abu Dhabi National Oil Company Technical InstituteAbu DhabiUnited Arab Emirates

Personalised recommendations