Language Learning in a Cognitive and Immersive Environment Using Contextualized Panoramic Imagery

  • Samuel ChabotEmail author
  • Jaimie Drozdal
  • Yalun Zhou
  • Hui Su
  • Jonas Braasch
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1034)


Immersive technologies augmented with AI for foreign language learning are currently uncommon. The Rensselaer Mandarin Project is a cognitive and immersive classroom that facilitates cultural and foreign language learning beyond a traditional classroom setting. Students perform tasks through intelligent interactions including authentic dialogues with multiple AI agents and multimodal gestures. Street View technology has the potential to further enhance foreign language learning and cultural knowledge acquisition in the Mandarin Project. Real-world panoramic scenes immerse learners in new locations and can provide richer context for more meaningful language learning. Students embark on “field trips” with cultural, historical, and lingual relevance at human-scale. Multiple learning opportunities and interactions are constructed for the various types of scenes students visit: vocabulary games, relevant AI agent dialogues, and informational lessons on cultural elements. The Mandarin Project will be assessed during a six-week classroom experience in the summer of 2019.


Immersive learning Virtual environment Chinese as a foreign language Multimodal interaction 



This research is supported by the Cognitive and Immersive Systems Laboratory (CISL), a collaboration between Rensselaer Polytechnic Institute and IBM Research through IBM’s AI Horizon Network. The authors would like to express gratitude to all of the researchers contributing to the Mandarin Project.


  1. 1.
    The iceberg model, localization and cultural context, August 2018.
  2. 2.
    Allen, D., Divekar, R.R., Drozdal, J., Balagyozyan, L.: The Rensselaer Mandarin Project - A cognitive and immersive language learning environment (2019)Google Scholar
  3. 3.
    Blyth, C.: Immersive technologies and language learning. Foreign Lang. Ann. 51(1), 225–232 (2018)CrossRefGoogle Scholar
  4. 4.
    Board, N.S.C.: World-readiness standards for learning languages (2015)Google Scholar
  5. 5.
    Canto, S., Ondarra, K.J.: Language learning effects through the integration of synchronous online communication: the case of video communication and second life. Lang. Learn. High. Educ. 7(1), 21–53 (2017)Google Scholar
  6. 6.
    Clark, H., Duckworth, S., Heil, J., Hotler, D., Piercy, D., Thumann, L.: The Google Cardboard Book: Explore, Engage, and Educate with Virtual Reality. EdTech Team, Irvine (2017)Google Scholar
  7. 7.
    Divekar, R.R., et al.: Interaction challenges in AI equipped environments built to teach foreign languages through dialogue and task-completion. In: Proceedings of the 2018 on Designing Interactive Systems Conference 2018, pp. 597–609. ACM (2018)Google Scholar
  8. 8.
    Divekar, R.R., et al.: CIRA: an architecture for building configurable immersive smart-rooms. In: Arai, K., Kapoor, S., Bhatia, R. (eds.) IntelliSys 2018. AISC, vol. 869, pp. 76–95. Springer, Cham (2019). Scholar
  9. 9.
    Divekar, R.R., Zhou, Y., Allen, D., Drozdal, J., Su, H.: Building human-scale intelligent immersive spaces for foreign language learning. iLRN 2018 Montana, p. 94 (2018)Google Scholar
  10. 10.
    Güzel, S., Aydin, S.: The effect of second life on speaking achievement. Online Submission 6(4), 236–245 (2016)Google Scholar
  11. 11.
    Hall, E.T.: Beyond Culture. Dobleday & Company, New York (1976)Google Scholar
  12. 12.
    Lee, W.: Using Auditory Cues to Perceptually Extract Visual Data in Collaborative, Immersive Big-Data Display Systems (2017)Google Scholar
  13. 13.
    Lin, T.J., Lan, Y.J.: Language learning in virtual reality environments: past, present, and future. J. Educ. Tech. Soc. 18(4), 486–497 (2015)Google Scholar
  14. 14.
    Oxford, R.L., Gkonou, C.: Interwoven: culture, language, and learning strategies. Stud. Second Lang. Learn. Teach. 8(2), 403–426 (2018)CrossRefGoogle Scholar
  15. 15.
    Sharma, G., Braasch, J., Radke, R.: Interactions in a human-scale immersive environment: the craive-lab. In: Cross-Surface 2016, in Conjunction with the ACM International Conference on Interactive Surfaces and Spaces (2017)Google Scholar
  16. 16.
    Shih, Y.C.: A virtual walk through london: culture learning through a cultural immersion experience. Comput. Assist. Lang. Learn. 28(5), 407–428 (2015)CrossRefGoogle Scholar
  17. 17.
    Shih, Y.C.: Contextualizing language learning with street view panoramas. In: Integrating Multi-User Virtual Environments in Modern Classrooms, pp. 74–91. IGI Global (2018)Google Scholar
  18. 18.
    Taguchi, N., Li, Q., Tang, X.: Learning chinese formulaic expressions in a scenario-based interactive environment. Foreign Lang. Ann. 50(4), 641–660 (2017)CrossRefGoogle Scholar
  19. 19.
    Willis, J.: A Framework for Task-Based Learning. Pearson PTR, Harlow (1996)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Samuel Chabot
    • 1
    • 2
    Email author
  • Jaimie Drozdal
    • 1
    • 2
  • Yalun Zhou
    • 1
    • 2
  • Hui Su
    • 1
    • 2
  • Jonas Braasch
    • 1
    • 2
  1. 1.Rensselaer Polytechnic InstituteTroyUSA
  2. 2.IBM T.J. Watson Research CenterYorktown HeightsUSA

Personalised recommendations