Skip to main content
SpringerLink
Log in

Reality Avatar for Customer Conversation in the Metaverse

  • Conference paper
  • First Online:
Human Interface and the Management of Information: Applications in Complex Technological Environments (HCII 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13306))

Included in the following conference series:

Abstract

Much emphasis has been placed on the expansion of the metaverse space, which enables a large number of people to participate concurrently and conduct business or entertainment in their field of interest. A close connection between the metaverse and the real space is expected to significantly increase the availability of the two spaces; however, an effective method for this connection has not yet been presented. In this paper, we propose the XR (cross reality) Telexperience Portal as one method for connecting both remote real space and past space with the metaverse. Furthermore, when people communicate in the metaverse’s 3D space, they require avatars to represent themselves, for which a variety of avatar designs are used. However, highly realistic avatars that hold the identity of the participant are not yet used. In this paper, we propose a method for generating natural facial expressions of a high-reality avatar to become a reliable conversation partner.

Supported by the MIC/SCOPE #191603003, JSPS KAKENHI Grant Number 18H04118 and 18H03283.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Stephenson, N.: Snow Crash. Bantam Books, New York (1993)

    Google Scholar 

  2. Smart, J., Cascio, J., Paffendorf, J.: Pathways to the 3d web: A cross-industry public foresight project. Metaverse Roadmap (2007)

    Google Scholar 

  3. Bailly, G., Perrier, P., Vatikiotis-Bateson, E. (eds.): Audiovisual Speech Processing. Cambridge University Press, New York (2012). https://doi.org/10.1017/CBO9780511843891

  4. Edwards, P., Landreth, C., Fiume, E., Singh, K.: Jali: An animator-centric viseme model for expressive lip synchronization. ACM Trans. Graph. 35(4) (2016). https://doi.org/10.1145/2897824.2925984

  5. Karras, T., Aila, T., Laine, S., Herva, A., Lehtinen, J.: Audio-driven facial animation by joint end-to-end learning of pose and emotion. ACM Trans. Graph. 36(4) (2017). https://doi.org/10.1145/3072959.3073658

  6. Blair, P.: Advanced Animation: Learn How to Draw Animated Cartoons. Walter T, Foster (1947)

    Google Scholar 

  7. Williams, L.: Performance-driven facial animation. In: Proceedings of the 17th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1990, pp. 235–242. Association for Computing Machinery, New York (1990). https://doi.org/10.1145/97879.97906

  8. Ezzat, T., Poggio, T.: Miketalk: a talking facial display based on morphing visemes. In: Proceedings Computer Animation 1998 (Cat. No.98EX169), pp. 96–102 (1998). https://doi.org/10.1109/CA.1998.681913

  9. Parke, F.I.: Computer generated animation of faces. In: Proceedings of the ACM Annual Conference - Volume 1. ACM 1972, pp. 451–457. Association for Computing Machinery, New York (1972). https://doi.org/10.1145/800193.569955

  10. Zhou, Y., Xu, Z., Landreth, C., Kalogerakis, E., Maji, S., Singh, K.: Visemenet: audio-driven animator-centric speech animation. ACM Trans. Graph. 37(4) (2018). https://doi.org/10.1145/3197517.3201292

  11. Chen, L., Maddox, R.K., Duan, Z., Xu, C.: Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7832–7841 (2019)

    Google Scholar 

  12. Zhou, Y., Han, X., Shechtman, E., Echevarria, J., Kalogerakis, E., Li, D.: Makelttalk: Speaker-aware talking-head animation. ACM Trans. Graph. 39(6) (2020). https://doi.org/10.1145/3414685.3417774

  13. Zhang, C., Zhao, Y., Huang, Y., Zeng, M., Ni, S., Budagavi, M., Guo, X.: Facial: Synthesizing dynamic talking face with implicit attribute learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3867–3876 (2021)

    Google Scholar 

  14. Ji, X., Zhou, H., Wang, K., Wu, W., Loy, C.C., Cao, X., Xu, F.: Audio-driven emotional video portraits. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14080–14089 (2021)

    Google Scholar 

  15. Shibatani, M.: Japanese in Concise Encyclopedia of Languages of the World. Elsevier, Tokyo (2006)

    Google Scholar 

  16. Shibatani, M.: English in the Present Day in Encyclopedia of Language & Linguistics, pp. 149–156. Elsevier, Tokyo (2006)

    Google Scholar 

  17. Otake, T., Hatano, G., Cutler, A., Mehler, J.: Mora or syllable? speech segmentation in Japanese. J. Mem. Lang. 32(2), 258–278 (1993)

    Article  Google Scholar 

  18. Lewis, J.P., Anjyo, K., Rhee, T., Zhang, M., Pighin, F.H., Deng, Z.: Practice and theory of blendshape facial models. Eurographics (State of the Art Reports) 1(8), 2 (2014)

    Google Scholar 

  19. Apple: ARFaceAnchor.BlendShapeLocation - Apple Developer Documentation. https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation. Accessed 13 Feb 2022

  20. Russian3dscanner: Wrapping - R3DS Wrap documentation. https://www.russian3dscanner.com/docs/Wrap3/Nodes/Wrapping/Wrapping.html. Accessed 13 Feb 2022

  21. Oculus: Tech Note: Enhancing Oculus Lipsync with Deep Learning. https://developer.oculus.com/blog/tech-note-enhancing-oculus-lipsync-with-deep-learning/. Accessed 13 Feb 2022

  22. Bailly, G.: Learning to speak. sensori-motor control of speech movements. Speech Commun. 22(2), 251–267 (1997). https://doi.org/10.1016/S0167-6393(97)00025-3

  23. Visage-Technologies: Mpeg-4 face and body animation (mpeg-4 fba). In: An Overview, pp. 37–40 (2012)

    Google Scholar 

Download references

Acknowledgment

This work was partially supported by the MIC/SCOPE #191603003, JSPS KAKENHI Grant Number 18H04118 and 18H03283.

Author information

Authors and Affiliations

  1. Tokyo Metropolitan University, 6-6 Asahigaoka, Hino, Tokyo, 191-0065, Japan

    Ryoto Kato, Yusuke Kikuchi & Vibol Yem

  2. The University of Tokyo, 7-3-1 Hongo, Bunkyo, Tokyo, 113-8656, Japan

    Yasushi Ikei

Authors
  1. Ryoto Kato
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yusuke Kikuchi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vibol Yem
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yasushi Ikei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ryoto Kato .

Editor information

Editors and Affiliations

  1. Tokyo University of Science, Tokyo, Saitama, Japan

    Sakae Yamamoto

  2. Tokyo City University, Tokyo, Japan

    Hirohiko Mori

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kato, R., Kikuchi, Y., Yem, V., Ikei, Y. (2022). Reality Avatar for Customer Conversation in the Metaverse. In: Yamamoto, S., Mori, H. (eds) Human Interface and the Management of Information: Applications in Complex Technological Environments. HCII 2022. Lecture Notes in Computer Science, vol 13306. Springer, Cham. https://doi.org/10.1007/978-3-031-06509-5_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-06509-5_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-06508-8

  • Online ISBN: 978-3-031-06509-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation