MediaSync pp 295-317 | Cite as

Media Synchronization in Networked Multisensory Applications with Haptics



In this chapter, we explain the present status of studies for media synchronization in networked multisensory applications with haptics. We also specify the characteristics of haptic media and different features of haptic media from other media such as olfactory, auditory, and visual media. By using such other media together with haptic media, we can get higher realistic sensation when we use the applications for various purposes such as remote education, entertainment, and networked games. When multisensory media streams are transmitted over a network like the Internet, the temporal relationships among the media streams may be disturbed owing to the network delay, delay jitter, and packet loss. Thus, the quality of experience (QoE) may seriously be degraded. To solve this problem, we need to carry out media synchronization control. To achieve a high quality of media synchronization, a number of media synchronization algorithms have been proposed so far. Especially, in networked multisensory applications, we need to take account of human perception of media synchronization errors in the algorithms. This is because the requirements of media synchronization quality depend on types of media. Some algorithms such as Virtual-Time Rendering (VTR) take human perception of the errors into account. For example, VTR tries to accomplish the synchronization by changing the buffering time of each media stream dynamically according to the network delay jitter with several threshold values about the errors. In the algorithms, instead of synchronizing the output timings of media streams exactly, ranges of human perception such as the allowable range, in which users feel that the synchronization error is allowable, the imperceptible range, in which users cannot perceive the error, and the operation range, which is narrower than the imperceptible range and should be kept usually, are taken into account for the sake of high synchronization quality. In this chapter, we explain the algorithms taking account of human perception of the media synchronization errors and enhance other algorithms such as the group (or inter-destination) synchronization control and the adaptive ∆-causality control algorithms for simultaneous output-timing control among multiple terminals by taking account of the perception. It is indispensable to clarify the ranges by QoE assessment in networked multisensory applications. In this chapter, we further make a survey of studies on the assessment. Finally, we discuss the future directions of media synchronization in networked multisensory applications with haptics.


Media synchronization algorithm Networked multisensory applications Haptics Human perception 



Quality of Service (QoS)

QoS is the quality of service which is provided from a layer to its upper layer in network layer model and it is defined as how much the service is faithful to the ideal situation.

Quality of Experience (QoE)

QoE is also called the user-level QoS. QoE is the quality which is perceived subjectively and/or experienced objectively by end-users.

Human perception

Human perception is the mental process in which human beings perceive or recognize an object or idea. In this chapter we quantitatively express human perception of synchronization error as ranges in which users can feel/perceive/allow the synchronization error.


  1. 1.
    Srinivasan, M.A., Basdogan, C.: Haptics in virtual environments: taxonomy, research status, and challenges. J. Comput. Graph. 21(4), 393–404 (1997)CrossRefGoogle Scholar
  2. 2.
    Kovacs, P.T., Murray, N., Rozinaj, G., Sulema, Y., Rybarova, R.: Application of immersive technologies for education: state of the art. In: Proceedings of 9th International Conference on Interactive Mobile Communication Technologies and Learning (IMCL), pp. 283–288, Nov 2015Google Scholar
  3. 3.
    Huang, P., Ishibashi, Y.: QoS control and QoE assessment in multi-sensory communications with haptics. IEICE Trans. Commun., Special Section on Quality of Communication Services Improving Quality of Life, E96-B(2), 392–403 (2013)Google Scholar
  4. 4.
    Ghinea, G., Ademoye, O.: Olfaction-enhanced multimedia: perspectives and challenges. Multimed. Tools Appl. 55(3), 601–626 (2011)Google Scholar
  5. 5.
    Murray, N., Lee, B., Qiao, Y., Muntean, G.: Olfaction-enhanced multimedia: a survey application domains, displays, and research challenges. ACM Comput. Surv. 48(4), 1–34 (2016)CrossRefGoogle Scholar
  6. 6.
    Murray, N., Qiao, Y., Lee, B., Muntean, G.: Subjective evaluation of olfactory and visual media synchronization. In: Proceedings of ACM Multimedia Systems, pp. 162–171, Mar 2013Google Scholar
  7. 7.
    Sithu M., Arima, R., Ishibashi, Y.: Experiment on object recognition with haptic, olfactory, and audio senses in virtual environment. IEICE Technical Report, CQ2015, Mar 2016Google Scholar
  8. 8.
    Ishibashi, Y., Hoshino, S., Zeng, Q., Fukushima, N., Sugawara, S.: QoE assessment of fairness in networked game with olfaction: Influence of time it takes for smell to reach player. Springer’s Multimed. Syst. J. (MMSJ), Special Issue on Network and Systems Support for Games 20(5), 621–631 (2014)Google Scholar
  9. 9.
    Huang, P., Ishibashi, Y., Fukushima, N., Sugawara, S.: QoE assessment of olfactory media in remote ikebana with haptic media. In: Proceedings of IEEE 2012 International Communications Quality and Reliability (CQR) Workshop, May 2012Google Scholar
  10. 10.
    Sithu, M., Ishibashi, Y., Huang, P., Fukushima, N.: Influences of network delay on quality of experience for soft objects in networked real-time game with haptic sense. Int. J. Commun. Netw. Syst. Sci. (IJCNS) 8(11), 440–455 (2015)Google Scholar
  11. 11.
    Sithu, M., Ishibashi, Y., Fukushima, N.: Effects of dynamic local lag control on sound synchronization and interactivity in joint musical performance. ITE Trans Media Technol. Appl., Special Section on Multimed. Transmission System and Services 2(4), 299–309 (2014)Google Scholar
  12. 12.
    Gunn, C., Hutchins, M., Steveson, D., Adcock, M., Youngblood, P.: Using collaborative haptics in remote surgical training. In: Proceedings of the First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, pp. 1–2, Mar 2005Google Scholar
  13. 13.
    Mizuno, H., Ishibashi, Y., Sugawara, S.: A comparison between transmission methods of sense of force in a remote haptic instruction system. In: Proceedings of IEEE ICME, pp. 1202–1205, June/July 2009Google Scholar
  14. 14.
    Zeng, Q., Iwata, K., Ishibashi, Y., Fukushima, N., Sugawara, S.: Influence of network delay in networked ensemble with haptic media, sound, and video. In: National Convention of IPSJ, 3Y-5, Mar 2012Google Scholar
  15. 15.
    Lun, M., Friedman, D., Sankaranarayanan, G., King, H., Fodero, K., Leuschke, R., Hannaford, B.: The RAVEN: design and validation of a telesurgery system. Int. J. Robot. Res. 1(1), 1–16 (2009)Google Scholar
  16. 16.
    ITU E. 800, Terms and definitions related to quality of service and network performance including dependability, Aug 1994Google Scholar
  17. 17.
    ITU-T Rec. P. 10/G. 100 Amendment 1, New appendix I—definition of quality of experience (QoE). Int. Telecommun. Union, Jan 2007Google Scholar
  18. 18.
    Blakowski, G., Steinmetz, R.: A media synchronization survey: reference model, specification, and case studies. IEEE J. Sel. Areas Commun. 14(1), 5–35 (1996)CrossRefGoogle Scholar
  19. 19.
    Huang, Z., Nahrstedt, K., Steinmetz, R.: Evolution of temporal multimedia synchronization principles: a historical viewpoint. ACM Trans. Multimed. Comput. Commun. Appl. 9(1), 34 (2013)Google Scholar
  20. 20.
    Sithu, M., Ishibashi, Y.: Media synchronization control in multimedia communication. IGI Global Emerg. Res. Netw. Multimed. Commun. Syst. Chapter 2, 25–61 (2015)Google Scholar
  21. 21.
    Ishibashi, Y., Tasaka, S., Hasegawa, T.: The Virtual-Time Rendering algorithm for haptic media synchronization in networked virtual environments. In: Proceedings of the 16th International Workshop on Communications Quality and Reliability (CQR), pp. 213–217, May 2002Google Scholar
  22. 22.
    Ishibashi, Y., Kanbara, T., Tasaka, S.: Inter-stream synchronization between haptic media and voice in collaborative virtual environments. In: Proceedings of ACM Multimedia, pp. 604–611, Oct 2004Google Scholar
  23. 23.
    Sannomiya, H., Osada, J., Ishibashi, Y., Fukushima, N., Sugawara, S.: Inter-stream synchronization control with group synchronization algorithm. In: Proceedings of the 2nd IEEE Global Conference on Consumer Electronics (GCCE), pp. 520–524, Oct 2013Google Scholar
  24. 24.
    Ishibashi, Y., Tasaka, S.: A synchronization mechanism for continuous media in multimedia communications. In: Proceedings of IEEE INFOCOM, pp. 1010–1019, Apr 1995Google Scholar
  25. 25.
    Sithu, M., Ishibashi, Y., Fukushima, N.: Effects of dynamic local lag control on sound synchronization and interactivity in joint musical performance. ITE Trans Media Technol. Appl., Special Section on Multimedia Transmission System and Services 2(4), 299–309 (2014)Google Scholar
  26. 26.
    Ishibashi, Y., Tasaka, S.: A distributed control scheme for group synchronization in multicast communications. In: Proceedings of International Symposium on Communications (ISCOM), pp. 317–323, Nov 1999Google Scholar
  27. 27.
    Ishibashi, Y., Hashimoto, Y., Ikedo, T., Sugawara, S.: Adaptive ∆-causality control with adaptive dead-reckoning in networked games. In: Proceedings of the 13th Annual Workshop on Network and Systems Support for Games (NetGames), pp. 75–80, Sep 2007Google Scholar
  28. 28.
    Marshall, A., Wai, Y.: Haptic virtual environment performance over IP networks: a case study. In: Proceedings of IEEE International Symposium on Distributed Simulation and Real-Time Applications, pp. 181–189, Oct 2003Google Scholar
  29. 29.
    El Saddik, A.: The potential of haptics technologies. IEEE Instrum. Meas. Mag. 10(1), 10–17 (2007)CrossRefGoogle Scholar
  30. 30.
    Lin, M., Salisbury, K.: Haptic rendering: beyond visual computing. IEEE Comput. Graph. Appl. 24(2), 22–23 (2004)CrossRefGoogle Scholar
  31. 31.
    Salisbury, K., Conti, F., Barbgli, F.: Haptic rendering: Introductory concepts. IEEE Comput. Graph. Appl. 24(2), 24–32 (2004)CrossRefGoogle Scholar
  32. 32.
    Weller, R., Zachmann, G.: User performance in complex bimanual haptic manipulation with 3DoFs vs. 6DoFs. In: Proceedings of IEEE Haptics, Mar 2012Google Scholar
  33. 33.
  34. 34.
  35. 35.
  36. 36.
  37. 37.
    Sato, M.: Development of string-based force display: SPIDAR. In: Proceedings of the 8th International Conference on Virtual System and Multi Media (VSMM), pp. 1034–1039, Sept 2002Google Scholar
  38. 38.
    Novint Technologies, Inc., Haptic device abstraction layer programmer’s guide, version 1.1.9 Beta, Sept 2007Google Scholar
  39. 39.
    Gosselin, F., Riwan, A.: Design of virtuose 3D: a new haptic interface for teleoperation and virtual reality. In: Proceedings of ICAR, pp. 205–212, Aug 2001Google Scholar
  40. 40.
    Vanderlinde, R.Q., Lammertse, P., Frederiksen, E., Ruiter, B.: The hapticmaster: a new high-performance haptic interface. In: Proceedings of EuroHaptics, pp. 1–5 (2002)Google Scholar
  41. 41.
    Immersion Corporation: CyberGrasp.
  42. 42.
  43. 43.
    Yoo,Y., Sung, M., Kim, N., Jun, K.: An experimental study on the performance of haptic data transmission in networked haptic collaboration. In: Proceedings of International Conference on Advanced Communication Technology, pp. 657–662, Feb 2007Google Scholar
  44. 44.
    Osada, J., Ishibashi, Y., Fukushima, N., Sugawara, S.: QoE comparison of haptic control schemes in remote instruction system with haptic media, video, and voice. In: Proceedings of International Technical Conference on Circuits/Systems, Computers and Communications (ITC-CSCC), July 2012Google Scholar
  45. 45.
    Khademian, B., Hashtrudi-Zaad, K.: Performance issues in collaborative haptic training. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA), pp. 3275–3263, Apr 2007Google Scholar
  46. 46.
    Willaert, B., Corteville, B., Reynaerts, D., Brussel, H., Poorten, E.: Bounded environment passivity of the classical position-force teleoperation controller. In: Proceedings of IEEE/RSJ IROS, Oct 2009Google Scholar
  47. 47.
    Fujimoto, T., Ishibashi, Y., Sugawara, S.: Influences of inter-stream synchronization error on collaborative work in haptic and visual environments. In: Proceedings of Haptics, pp. 113–119, Mar 2008Google Scholar
  48. 48.
    Tatematsu, A., Ishibashi, Y., Fukushima, N., Sugawara, S.: QoE assessment in tele-operation with 3D video and haptic media. In: Proceedings of IEEE ICME Workshop on Hot Topics in Multimedia, July 2011Google Scholar
  49. 49.
    Zeng, Q., Ishibashi, Y., Fukushima, N., Sugawara, S., Psannis, K.E.: Influences of inter-stream synchronization errors among haptic media, sound, and video on quality of experience in networked ensemble. In: Proceedings of IEEE GCCE, pp. 466–470, Oct 2013Google Scholar
  50. 50.
    Ishibashi, Y., Kaneoka, H.: Faieness among game players in networked haptic environments: influence of network latency. In: Proceedings of IEEE ICME, July 2005Google Scholar
  51. 51.
    Ishibashi, Y., Tasaka, S.: Causality and media synchronization control for networked multimedia games: centralized versus distributed. In: Proceedings of NetGames, pp. 42–51, May 2003Google Scholar
  52. 52.
    Hoshino, S., Ishibashi, Y., Fukushima, N., Sugawara, S.: QoE assessment in olfactory and haptic media transmission: Influence of inter-stream synchronization error. In: Proceedings of IEEE CQR, May 2011Google Scholar
  53. 53.
    Nakano, S., Maeda, Y., Ishibashi, Y., Fukushima, N., Huang, P., Psannis, K.E.: Influence of network delay on fairness between players in networked game with olfactory and haptic senses. IEICE Technical Report, CQ2014-94, Jan 2015Google Scholar
  54. 54.
    Tasaka, S. Ishibashi, Y.: Mutually compensatory property of multimedia QoS. In: Conference Record IEEE ICC, pp. 1105–1111, Apr/May 2002Google Scholar
  55. 55.
    Tasaka, S., Ito, Y.: Psychometric analysis of the mutually compensatory property of multimedia QoS. In: Conference Record IEEE ICC, pp. 1880–1886, May 2003Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Seijoh UniversityAichiJapan
  2. 2.Nagoya Institute of TechnologyNagoyaJapan

Personalised recommendations