Skip to main content
SpringerLink
Log in

Evaluation of the effectiveness of preschool English learning applications based on touch and voice multimodal interaction technique

  • Long Paper
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

The rising development in information and communication technology affected the rapid presence of new interface designs to meet the needs of user interaction. This includes the needs of the children dominating the use of technology in obtaining effective interaction with the system despite having limitations in cognitive and motor skills. This study aims to evaluate the effectiveness of the ETSApp learning application developed through the performance of users’ as well as their experience while interacting with the application. ETSApp is an English language application based on multimodal interaction techniques combining touch and voice input integrated into preschool curriculum content for children aged four to six years old. Effectiveness evaluation using observation methods and user surveys was conducted on 41 participants comprising 12 kindergarten teachers and 29 kindergarten children from the West Coast and East Coast zones of Peninsular Malaysia. The results of the observation revealed that the percentage of successful interaction between teachers and children was 83% and above for all interaction items. A survey conducted on teachers and children obtained a positive response where they felt that ETSApp was very fun, easy to interact with and effective in helping the teaching and learning process. The development of ETSApp can meet the learning needs based on the Education 4.0, which adapts technology in a more flexible and interactive teaching and learning process, in addition to being an innovative step in preparing children to face emerging challenges.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Auxier, B.Y.B., Anderson, M., Perrin, A., Turner, E.: Parenting children in the age of screens. Pew Res. Cent. Internet, Sci. Tech. 1–28 (2020)

  2. Commission, M.C.A.M.: Internet Users Survey 2020 (2020)

  3. Nawaila, M.B., Kanbul, S., Kani, U.M., Magaji, M.M.: DLMA_NEU: digital literacy mobile application for children. Int. J. Interact. Mob. Technol. 16, 49–64 (2022). https://doi.org/10.3991/ijim.v16i08.25213

    Article  Google Scholar 

  4. Hussain, N.H.: Children’s Multi-Touch and Voice Interaction Model in Active Learning. PhD Dissertation, Universiti Kebangsaan Malaysia (2020)

  5. Hussain, N.H.N.H., Wook, T.S.M.T., Noor, S.F.M., Mohamed, H., Tengku Wook, T.S.M., Mat Noor, S.F., Mohamed, H.: Children’s interaction ability towards multi-touch gestures. Int. J. Adv. Sci. Eng. Inf. Technol. 6, 875–881 (2016)

    Article  Google Scholar 

  6. Brown, Q., Bonsignore, E., Hatley, L., Druin, A., Walsh, G., Foss, E., Brewer, R., Hammer, J., Golub, E.: Clear Panels: a technique to design mobile application interactivity. In: Proceedings of the Conference on Designing Interactive Systems, Aarhus, Denmark, Aug 16–20, 2010. pp. 360–363. ACM digital Library, Aarhus Denmark (2010)

  7. Laili Farhana, M.I., Nadia Akma, A.Z., Maizatul Hayati, M.Y.: Touch gesture interaction of preschool children towards games application using touch screen gadjet. Asia-Pacific J. Inf. Technol. Multimed. 4, 47–58 (2015)

    Article  Google Scholar 

  8. McKnight, L., Cassidy, B.: Children’s interaction with mobile touch-screen devices. Int. J. Mob. Hum. Comput. Interact. 2, 1–18 (2010). https://doi.org/10.4018/jmhci.2010040101

    Article  Google Scholar 

  9. Nacher, V., Jaen, J., Navarro, E., Catala, A., González, P.: Multi-touch gestures for pre-kindergarten children. Int. J. Hum. Comput. Stud. 73, 37–51 (2015). https://doi.org/10.1016/j.ijhcs.2014.08.004

    Article  Google Scholar 

  10. Nor Azah, A.A., Batmaz, F., Stone, R., Chung, P.W.H.: Selection of touch gestures for children’s applications. In: Science and Information Conference 2013. pp. 721–726. IEEE (2013)

  11. Workshop, S.S.: Best Practices: designing touch tablet experiences for preschoolers (2012)

  12. Yu, X., Xue, Y., Zhang, M., Zhu, Z.: An exploration of developing multi-touch virtual learning tools for young. In: ICETC 2010 - 2010 2nd International Conference on Education Technology and Computer. pp. 0–3 (2010)

  13. Behnamnia, N., Kamsin, A., Ismail, M.A.B., Hayati, S.A.: A review of using digital game-based learning for preschoolers. Springer, Berlin Heidelberg (2022)

    Google Scholar 

  14. Blitz-Raith, A.H., Liu, J.: Interactivity in educational apps for young children: a multimodal analysis. Int. J. Instr. 10, 237–254 (2017). https://doi.org/10.12973/iji.2017.10414a

    Article  Google Scholar 

  15. Oviatt, S.: Theoretical foundations of multimodal interfaces and systems. In: The Handbook of Multimodal-Multisensor Interfaces: Foundations, User Modeling, and Common Modality Combinations - Volume 1. pp. 19–50 (2017)

  16. Almeida, N., Silva, S., Teixeira, A.: Design and development of speech interaction: a methodology. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). pp. 370–381 (2014)

  17. Zhao, J., Parry, C.J., Anjos, R. Dos, Anslow, C., Rhee, T.: Voice interaction for augmented reality navigation interfaces with natural language understanding. Int. Conf. Image Vis. Comput. New Zeal. 2020-Novem, (2020). https://doi.org/10.1109/IVCNZ51579.2020.9290643

  18. Zhao, J., William Soukoreff, R., Balakrishnan, R.: Exploring and modeling unimanual object manipulation on multi-touch displays. Int. J. Hum. Comput. Stud. 78, 68–80 (2015)

    Article  Google Scholar 

  19. Neumann, M.M., Neumann, D.L.: The use of touch screen tablets at home and pre-school to foster emergent literacy. J. Early Child. Lit. 17, 203–220 (2017)

    Article  Google Scholar 

  20. Sadiq, R.B., Cavus, N., Ibrahim, D.: Mobile application based on CCI standards to help children learn English as a foreign language. Interact. Learn. Environ. 29, 442–457 (2021). https://doi.org/10.1080/10494820.2019.1579239

    Article  Google Scholar 

  21. Anthony, L.: Physical dimensions of children’s touchscreen interactions: lessons from five years of study on the MTAGIC Project. Int. J. Hum. Comput. Stud. 128, 1–16 (2019)

    Article  Google Scholar 

  22. Oviatt, S., Coulston, R., Tomko, S., Xiao, B., Lunsford, R., Wesson, M., Carmichael, L.: Toward a theory of organized multimodal integration patterns during human-computer interaction. In: Proceedings of the 5th international conference on Multimodal interfaces - ICMI ’03. p. 44 (2003)

  23. Turk, M.: Multimodal interaction: a review. Pattern Recognit. Lett. 36, 189–195 (2014). https://doi.org/10.1016/j.patrec.2013.07.003

    Article  Google Scholar 

  24. Anthony, L., Brown, Q., Tate, B., Nias, J., Brewer, R., Irwin, G.: Designing smarter touch-based interfaces for educational contexts. Pers. Ubiquitous Comput. 18, 1471–1483 (2014). https://doi.org/10.1007/s00779-013-0749-9

    Article  Google Scholar 

  25. Kucirkova, N., Messer, D., Sheehy, K., Fernández Panadero, C.: Children’s engagement with educational iPad apps: insights from a Spanish classroom. Comput. Educ. 71, 175–184 (2014). https://doi.org/10.1016/j.compedu.2013.10.003

    Article  Google Scholar 

  26. Henderson, S., Yeow, J.: iPad in Education: a case study of iPad adoption and use in a Primary School. In: Sciences, 2012 45th Hawaii International Conference on System (ed.) 2012 45th Hawaii International Conference on System Sciences. pp. 78–87. IEEE (2012)

  27. Anthony, L., Brown, Q., Nias, J., Tate, B.: Examining the need for visual feedback during gesture interaction on mobile touchscreen devices for kids. In: Proceedings of the 12th International Conference on Interaction Design and Children - IDC ’13. pp. 157–164 (2013)

  28. Price, S., Jewitt, C., Crescenzi, L.: The role of iPads in pre-school children’s mark making development. Comput. Educ. 87, 131–141 (2015). https://doi.org/10.1016/j.compedu.2015.04.003

    Article  Google Scholar 

  29. Maria, M., Shahbodin, F., Pee, N.C.: Malaysian higher education system towards industry 4.0 - Current trends overview. AIP Conf. Proc. 2016, (2018). https://doi.org/10.1063/1.5055483

  30. Booton, S.A., Hodgkiss, A., Murphy, V.A.: The impact of mobile application features on children’s language and literacy learning: a systematic review. Comput. Assist. Lang. Learn. 36, 400–429 (2023). https://doi.org/10.1080/09588221.2021.1930057

    Article  Google Scholar 

  31. Zhang, Z., Xu, Y., Wang, Y., Yao, B., Ritchie, D., Wu, T., Yu, M., Wang, D., Li, T.J.J.: StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement. Conf. Hum. Factors Comput. Syst. - Proc. (2022). https://doi.org/10.1145/3491102.3517479

  32. Kaldarova, B., Omarov, B., Zhaidakbayeva, L., Tursynbayev, A., Beissenova, G., Kurmanbayev, B., Anarbayev, A.: Applying game-based learning to a primary school class in computer science terminology learning. Front. Educ. 8, 1–10 (2023). https://doi.org/10.3389/feduc.2023.1100275

    Article  Google Scholar 

  33. Lebeau, K., Huey, L.G., Hart, M.: Assessing the quality of mobile apps used by occupational therapists: evaluation using the user version of the mobile application rating scale. JMIR mHealth uHealth. (2019). https://doi.org/10.2196/13019

    Article  Google Scholar 

  34. Hémar-Nicolas, V., Putri Hapsari, H., Angka, S., Olsen, A.: How cartoon characters and claims influence children’s attitude towards a snack vegetable – an explorative cross-cultural comparison between Indonesia and Denmark. Food Qual. Prefer. 87, 104031 (2021). https://doi.org/10.1016/j.foodqual.2020.104031

    Article  Google Scholar 

  35. Tengku Wook, T.S.M., Mohamed, H., Sahari Ashaari, N., Mat Noor, S.F., Muda, Z., Zairon, I.Y., Khaleel, F.L.: User experience evaluation towards interface design of digital footprint awareness application. Asia-Pacific J. Inf. Technol. Multimed. 9, 17–27 (2020). https://doi.org/10.17576/apjitm-2020-0901-02

    Article  Google Scholar 

  36. Schrepp, M., Hinderks, A., Thomaschewski, J.: Design and evaluation of a short version of the user experience questionnaire (UEQ-S). Int. J. Interact. Multimed. Artif. Intell. 4, 103 (2017). https://doi.org/10.9781/ijimai.2017.09.001

    Article  Google Scholar 

  37. Schrepp, M., Hinderks, A., Thomaschewski, J.: Construction of a benchmark for the user experience questionnaire (UEQ). Int. J. Interact. Multimed. Artif. Intell. 4, 40 (2017). https://doi.org/10.9781/ijimai.2017.445

    Article  Google Scholar 

  38. Hinderks, A., Schrepp, M., Thomaschewski, J.: User Experience Questionnaire, https://www.ueq-online.org/

  39. Khanum, M.A., Trivedi, M.C.: Take care: a study on usability evaluation methods for children. Int. J. Adv. Res. Comput. Sci. 3, 3–39 (2012)

    Google Scholar 

  40. Yu, C., Wen, H., Xiong, W., Bi, X., Shi, Y.: Investigating effects of post-selection feedback for acquiring ultra-small targets on touchscreen. In: Conference on Human Factors in Computing Systems - Proceedings. pp. 4699–4710. ACM digital Library, USA (2016)

  41. Radzikowski, K., Wang, L., Yoshie, O., Nowak, R.: Accent modification for speech recognition of non-native speakers using neural style transfer. Eurasip J. Audio, Speech, Music Process (2021). https://doi.org/10.1186/s13636-021-00199-3

    Article  Google Scholar 

  42. Noor Shahirah, M.R., Sharifah Raihan, S.J.: Kajian Fonologi Kesalahan Bunyi Dalam Bahasa Kanak-Kanak. J. Soc. Sci. Humanit. 13, 178–188 (2018)

    Google Scholar 

  43. Badrulzaman, A.H., Ruby Izyan Atika, A.B.: Analisis Akustik Ruang Vokal Kanak-Kanak Melayu. J. Bhs. 11, 48–62 (2011)

    Google Scholar 

  44. Harley, A.: Touch Targets on Touchscreens, https://www.nngroup.com/articles/touch-target-size/

  45. Abdul Razak, N., Alakrash, H., Sahboun, Y.: English language teachers’ readiness for the application of technology towards fourth industrial revolution demands. Asia-Pacific J. Inf. Technol. Multimed. 7, 89–98 (2018). https://doi.org/10.17576/apjitm-2018-0702(02)-08

    Article  Google Scholar 

Download references

Acknowledgements

We thank all the teachers and children from eight kindergartens from the West Coast (Alif Ceria, As Sakeenah and Intelek Idaman Kindergarten) and the East Coast (Qurratu Ayun, Deiana Manjaku, Ehsan Bestari, Mateen Pintar and Iqram Kindergarten) of Peninsular Malaysia for their participation in this study. This research has been sponsored by the Prototype Research Grant Scheme (PRGS/2/2019/ICT04/UKM/02/1).

Funding

This research was funded by the Ministry of Higher Education and, as such, does not require ethical approval, given that the study participants naturally fall within the domain of education. This research was funded by the Prototype Research Grant Scheme (PRGS/2/2019/ICT04/UKM/02/1).

Author information

Authors and Affiliations

  1. Center for Software Technology and Management, Faculty of Information Science and Technology, Universiti Kebangsaan Malaysia, 43600 UKM, Bangi, Selangor, Malaysia

    Tengku Siti Meriam Tengku Wook, Siti Fadzilah Mat Noor, Noraidah Sahari Ashaari & Hazura Mohamed

  2. Faculty of Creative Multimedia and Computing, Islamic University of Selangor, Bandar Seri Putra, 43000, Kajang, Selangor, Malaysia

    Nor Hidayah Hussain

  3. ICS Learning Resources, Kuala Lumpur, Malaysia

    Rasheedah Zakariya

Authors
  1. Tengku Siti Meriam Tengku Wook
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Siti Fadzilah Mat Noor
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Noraidah Sahari Ashaari
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hazura Mohamed
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Nor Hidayah Hussain
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Rasheedah Zakariya
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

TSMTW is the corresponding author who has made the most significant contribution to the writing, which includes all sections. NS helped write Sect. 4.1, Performance of touch and voice interaction. SFMN prepared Figs. 7, 8, 9, 10, and 11. HM and NHH contributed to the user experience Sect. 4.2 by analyzing the data. Finally, RZ played a role in verifying that the English content used was in line with the preschool curriculum in Malaysia. All authors reviewed the manuscript.

Corresponding author

Correspondence to Tengku Siti Meriam Tengku Wook.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Ethical approval

Written human ethics and consent to participate is not applicable because verbal consent was obtained from kindergarten. All participants voluntarily consented to participate in this study after receiving clear information about the purpose, procedures and possible effects of the study. They had the right to withdraw their participation without consequences. The research conducted conforms to the literature review, research design, peer review, validity and reliability, and the use of user experience (UX) theoretical frameworks as a norm or standard to which researchers often adhere.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tengku Wook, T.S.M., Mat Noor, S.F., Sahari Ashaari, N. et al. Evaluation of the effectiveness of preschool English learning applications based on touch and voice multimodal interaction technique. Univ Access Inf Soc (2023). https://doi.org/10.1007/s10209-023-01075-x

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10209-023-01075-x

Keywords

Search

Navigation