Advertisement

MediaSync pp 33-71 | Cite as

Evolution of Temporal Multimedia Synchronization Principles

  • Zixia Huang
  • Klara Nahrstedt
  • Ralf Steinmetz
Chapter

Abstract

Ever since the invention of the world’s first telephone in the nineteenth century, the evolution of multimedia applications has drastically changed human life and behaviors, and has introduced new demands for multimedia synchronization. In this chapter, we present a historical view of temporal synchronization efforts with a focus on continuous multimedia (i.e., sequences of time-correlated multimedia data). We demonstrate how the development of multimedia systems has advanced the research on synchronization, and what additional challenges have been imposed by next-generation multimedia technologies. We conclude with a new application-dependent multilocation multi-demand synchronization framework to address these new challenges.

Keywords

Continuous multimedia Temporal synchronization Evolution 

References

  1. 1.
    BELLLABS: The picture of the future. Bell Labs Rec. 47(5), 134–186 (1969)Google Scholar
  2. 2.
    RFC-958: Network Time Protocol (NTP). http://www.ntp.org/. Accessed 28 Apr 2017
  3. 3.
    Cornell University: The CU-SeeMe Project. http://ftp.icm.edu.pl/packages/cu-seeme/html/Welcome.html. Accessed 28 Apr 2017
  4. 4.
    The Cambridge iTV Trial. http://koo.corpus.cam.ac.uk/projects/itv/. Accessed 28 Apr 2017
  5. 5.
    Caltech/CERN Project. http://pcbunn.cithep.caltech.edu/. Accessed 28 Apr 2017
  6. 6.
    IEEE-1588 standard: Precise time synchronization as the basis for real time applications in automation. https://standards.ieee.org/findstds/standard/1588-2008.html. Accessed 28 Apr 2017
  7. 7.
    RFC-5905: Network Time Protocol version 4: Protocol and algorithms specification. http://www.ntp.org/. Accessed 28 Apr 2017
  8. 8.
    Akyildiz, I.F., Yen, W.: Multimedia group synchronization protocols for integrated services networks. IEEE J. Sel. Areas Commun. 14(1), 162–173 (1996)CrossRefGoogle Scholar
  9. 9.
    Anderson, D.P., Homsy, G.: A continuous media I/O server and its synchronization mechanism. IEEE Comput. 24(10), 51–57 (1991)CrossRefGoogle Scholar
  10. 10.
    Arefin, A., Huang, Z., Nahrstedt, K., Agarwal, P.: 4D Telecast: Towards large scale multi-site and multi-view dissemination of 3DTI contents. In: Proceedings of IEEE 32nd International Conference on Distributed Computer Systems (ICDCS), Macau, China, pp. 82–91 (2012)Google Scholar
  11. 11.
    Basilio, C.: Antonio meucci inventore del telefono. Notiziario Tec. Telecommun. Ital. 12(1), 114 (2003)Google Scholar
  12. 12.
    Baxter, B., Scheib, V., Lin, M.C., Manocha, D.: DAB: interactive haptic painting with 3D virtual brushes. In: Proceedings of ACM Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), Los Angeles, CA, pp. 461–468 (2001)Google Scholar
  13. 13.
    Blakowski, G., Steinmetz, R.: A media synchronization survey: reference model, specification, and case studies. IEEE J. Sel. Areas Commun. 1, 5–35 (1996)CrossRefGoogle Scholar
  14. 14.
    Blesser, B.: Digitization of audio: a comprehensive examination of theory, implementation, and current practice. AES J. Audio Eng. Soc. 26(10), 739–771 (1978)Google Scholar
  15. 15.
    Boronat, F., Cebollada, J.C.G., Mauri, J.L.: An RTP/RTCP based approach for multimedia group and inter-stream synchronization. Springer J. Multimedia Tools Appl. 40(2), 285–319 (2008)CrossRefGoogle Scholar
  16. 16.
    Boronat, F., Lloret, J., Garcia, M.: Multimedia group and inter-stream synchronization techniques: a comparative study. Elsevier Inf. Syst. 34(1), 108–131 (2009)CrossRefGoogle Scholar
  17. 17.
    Bulterman, D.: Specification and support of adaptable networked multimedia. Springer Multimedia Syst. 1(2), 68–76 (1993)CrossRefGoogle Scholar
  18. 18.
    Campbell, A., Coulson, G., Garcła, F., Hutchison, D.: Orchestration services for distributed multimedia synchronisation. In: Proceedings of IFIP International Conference on High Performance Networking (HPN), Liegel, Belgium (1992)Google Scholar
  19. 19.
    Chung, S.M., Pereira, A.L.: Timed petri net representation of SMIL. IEEE Multimedia 12(1), 64–72 (2005)CrossRefGoogle Scholar
  20. 20.
    Courtiat, J., de Oliveira, R.C.: Proving temporal consistency in a new multimedia synchronization model. In: Proceedings of ACM International Conference on Multimedia (MM), Boston, USA, pp. 141–152 (1996)Google Scholar
  21. 21.
    Curcio, I., Lundan, M.: Human perception of lip synchronization in mobile environment. In: Proceedings of IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM), Espoo, Finland, pp. 1–7 (2007)Google Scholar
  22. 22.
    Damer, B.: Avatars: Exploring and Building Virtual Worlds on the Internet, pp. 383–386 . Peachpit Press (1998)Google Scholar
  23. 23.
    Dannenberg, R., Stern, R.: Experiments concerning the allowable skew of two audio channels operating in the stereo mode. Pers. Commun. (1993)Google Scholar
  24. 24.
    Deventer, M., Stokking, H., Hammond, M., Cesar, P.: Standards for multi-stream and multi-device media synchronization. IEEE Commun. Mag. 54(3), 16–21 (2016)CrossRefGoogle Scholar
  25. 25.
    Ehley, L., Furth, B., Ilyas, M.: Evaluation of multimedia synchronization techniques. In: Proceedings of IEEE International Conference on Multimedia Computing and Systems (ICMCS), Boston, USA, pp. 110–119 (1994)Google Scholar
  26. 26.
    Fujimoto, T., Ishibashi, Y., Sugawara, S.: Influences of inter-stream synchronization error on collaborative work in haptic and visual environments. In: Proceedings of IEEE Symposium on Haptic Interfaces for Virtual Environment and Teleoperator System (HAPTICS), Reno, USA, pp. 113–119 (2008)Google Scholar
  27. 27.
    Gardner, B.: A realtime multichannel room simulator. In: Proceedings of 124th Meeting of the Acoustical Society of America, New Orleans, USA (1992)Google Scholar
  28. 28.
    Geerts, D., Vaishnavi, I., Mekuria, R., van Deventer, O., Cesar, P.: (2011) Are we in sync? Synchronization requirements for watching online video together. In: Proceedings of the 29th ACM Conference on Human Factors in Computing Systems (SIGCHI), Vancouver, Canada, pp. 311–314Google Scholar
  29. 29.
    Ghinea, G., Ademoye, O.A.: Perceived synchronization of olfactory multimedia. IEEE Trans. Syst. Man Cybern. 40(4), 657–663 (2010)CrossRefGoogle Scholar
  30. 30.
    Goldmann, L., Lee, J.S., Ebrahimi, T.: Temporal synchronization in stereoscopic video: Influence on quality of experience and automatic asynchrony detection, hong kong, china. In: Proceedings of IEEE International Conference on Image Processing (ICIP), pp. 3241–3244 (2010)Google Scholar
  31. 31.
    Hodges, M., Sasnett, R., Ackerman, M.: Athena Muse: a construction set for multimedia applications. IEEE Softw. 6(1), 37–43 (1989)CrossRefGoogle Scholar
  32. 32.
    Hoshino, S., Ishibashi, Y., Fukushima, N., Sugawara, S.: Qoe assessment in olfactory and haptic media transmission: Influence of inter-stream synchronization error. In: Proceedings of IEEE International Workshop on Communications Quality and Reliability (CQR), Naples, FL, USA, pp. 1–6 (2011)Google Scholar
  33. 33.
    Hsu, P., Chen, Y., Chang, Y.: STRPN: a petri-net approach for modeling spatial-temporal relations between moving multimedia objects. IEEE Trans. Softw. Eng. 29(1), 63–76 (2003)CrossRefGoogle Scholar
  34. 34.
    Hu, N., Steenkiste, P.: Estimating available bandwidth using packet pair probing. Carnegie Mellon University Techical Report, CMU-CS-02-166 (2002)Google Scholar
  35. 35.
    Huang, Z., Nahrstedt, K.: Perception-based media packet scheduling for high-quality tele-immersion. In: Proceedings of IEEE International Conference on Computer Communications (INFOCOM), Orlando, USA, pp. 29–34 (2012)Google Scholar
  36. 36.
    Huang, Z., Wu, W., Nahrstedt, K., Arefin, A., Rivas, R.: TSync: A new synchronization framework for multi-site 3D tele-immersion. In: Proceedings of ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV), Amsterdam, the Netherlands, pp. 39–44 (2010)Google Scholar
  37. 37.
    Huang, Z., Mei, C., Li, L., Woo, T.: CloudStream: delivering high-quality streaming video through a cloud-based H.264/SVC proxy. In: Proceedsings of IEEE International Conference on Computer Communications (INFOCOM), Shanghai, China, pp. 201–205 (2011)Google Scholar
  38. 38.
    Huang, Z., Wu, W., Nahrstedt, K., Rivas, R., Arefin, A.: Synccast: synchronized dissemination in multi-site interactive 3D tele-immersion. In: Proceedings of ACM Multimedia Systems Conference (MMSYS), San Jose, USA, pp. 69–80 (2011)Google Scholar
  39. 39.
    Huang, Z., Nahrstedt, K., Liang, K.: Human-centric multi-layer synchronization scheme with inter-sender synchronization skew control. In: Proceedings of IEEE International Workshop on Quality of Multimedia Experience (QoMEX), Singapore, pp. 25–30 (2014)Google Scholar
  40. 40.
    Iimura, T.: Zoned federation of game servers: A peer-to-peer approach to scalable multi-player online games. In: Proceedings of ACM Proceedings of 3rd ACM SIGCOMM workshop on Network and system support for games (NetGames), Portland, Oregon, pp. 116–120 (2004)Google Scholar
  41. 41.
    Ishibashi, Y., Tasaka, S.: A distributed control scheme for group synchronization in multicast communications. In: Proceedings of International Symposium Communications (ISCOM), Japan, pp. 317–323 (1999)Google Scholar
  42. 42.
    Ishibashi, Y., Tasaka, S.: A comparative survey of synchronization algorithms for continuous media in network environments. In: Proceedings of IEEE Conference on Local Computer Networks (LCN), Tampa, FL, USA, pp. 337–348 (2000)Google Scholar
  43. 43.
    Ishibashi, Y., Tsuji, A., Tasaka, S.: A group synchronization mechanism for stored media in multicast communications. In: Proceedings of Annual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM), Kobe, Japan, pp. 692–700 (1997)Google Scholar
  44. 44.
    ISO International Standard: Information technology hypermedia/time-based structuring language (HyTime). https://www.iso.org/standard/18834.html (1992). Accessed 28 Apr 2017
  45. 45.
    ITU-H263: Video coding for low bit rate communication. http://www.itu.int/rec/T-REC-H.263/en/ (2005). Accessed 28 Apr 2017
  46. 46.
    ITU-H323: Packet-based multimedia communications systems. http://www.itu.int/rec/T-REC-H.323/en/ (2009). Accessed 28 Apr 2017
  47. 47.
    King, P.: Towards a temporal logic based formalism for expressing temporal constraints in multimedia documents. Technical Report, 942, LRI, Universite de Paris-Sud, Orsay, France (1994)Google Scholar
  48. 48.
    Leroux, P., Verstraete, V., De Turck, F., Demeester, P.: Synchronized interactive services for mobile devices over IPDC/DVB-H and UMTS. In: Proceedings of IEEE/IFIP International Workshop on Broadband Convergence Networks (BCN), Munich, Germany, pp. 1–12 (2007)Google Scholar
  49. 49.
    Little, T.: A framework for synchronous delivery of time-depdent multimedia data. Springer Multimedia Syst. 1(2), 87–94 (1993)CrossRefGoogle Scholar
  50. 50.
    Little, T., Ghafoor, A.: Synchronization and storage models for multimedia objects. IEEE J. Sel. Areas Commun. 8(3), 413–427 (1990)CrossRefGoogle Scholar
  51. 51.
    Little, T., Ghafoor, A.: Spatio-temporal composition of distributed multimedia objects for value-added networks. IEEE Comput. 24(10), 42–50 (1991)CrossRefGoogle Scholar
  52. 52.
    Lo, B., Thiemjarus, S., King, R., Yang, G.: Body sensor network - a wireless sensor platform for pervasive healthcare monitoring. In: Proceedings of IEEE International Conference on Pervasive Computing (PERCOM), pp. 77–80 (2005)Google Scholar
  53. 53.
    Marcheschi, S., Portillo ,O., Raspolli, M., Avizzano, C., Bergamasco, M.: The haptic desktop: a novel 2D multimodal device. In: Proceedings of IEEE International Conference on Robot and Human Interactive Communication (ROMAN), Kurashiki, Okayama, Japan, pp. 521–526 (2004)Google Scholar
  54. 54.
    Mayagoitia, R.E., Nene, A.V., Veltink, P.H.: Accelerometer and rate gyroscope measurement of kinematics: an inexpensive alternative to optical motion analysis systems. Elsevier J. Biomech. 35(4), 537–542 (2002)CrossRefGoogle Scholar
  55. 55.
    Meyer, T., Effelsberg, W., Steinmetz, R.: A taxonomy on multimedia synchronization. In: Proceedings of IEEE Workshop on Future Trends of Distributed Computing Systems, Lisbon, Portugal, pp. 97–103 (1994)Google Scholar
  56. 56.
    Michel, U.: History of acoustic beamforming. In: Proceedings of Berlin Beamforming Conference (BeBeC), Berlin, Germany (2006)Google Scholar
  57. 57.
    Montagud, M., Boronat, F.: On the use of adaptive media playout for inter-destination synchronization. IEEE Commun. Lett. 15(8), 863–865 (2011)CrossRefGoogle Scholar
  58. 58.
    Montagud, M., Boronat, F., Stokking, H., van Brandenburg, R.: Inter-destination multimedia synchronization: schemes, use cases and standardization. Springer Multimedia Syst. 18(6), 459–482 (2012)CrossRefGoogle Scholar
  59. 59.
    Montagud, M., Boronat, F., Stokking, H., César, P.: Design, development and assessment of control schemes for IDMS in a standardized RTCP-based solution. Elsevier Comput. Netw. 70(1), 240–259 (2014)CrossRefGoogle Scholar
  60. 60.
    Montagud, M., Boronat, F., Roig, B., Sapena, A.: How to perform AMP? Cubic adjustments for improving the QoE. Elsevier Comput. Commun. 103, 61–73 (2017)CrossRefGoogle Scholar
  61. 61.
    Montagud Climent, M.A., Jansen, AJ., Cesar Garcia, PS., Boronat, F.: Review of media sync reference models: Advances and open issues. Media Synchronization Workshop (MediaSync), Brussels, Belgium (2015)Google Scholar
  62. 62.
    Murray, N., Lee, B., Qiao, Y., Muntean, G.:The influence of human factors on olfaction based mulsemedia quality of experience. In: Proceedings of IEEE International Conference on Quality of Multimedia Experience (QoMEX), Lisbon, Portugal, pp. 1–6 (2016)Google Scholar
  63. 63.
    PictureTel: Picturetel In Project With I.B.M., New York Times. http://www.nytimes.com/1991/10/22/business/company-news-picturetel-in-project-with-ibm.html (1991). Accessed 28 Apr 2017
  64. 64.
    Rainer, B., Timmerer, C.: A quality of experience model for adaptive media playout. In: Proceedings of IEEE International Workshop on Quality of Multimedia Experience (QoMEX), Singapore, pp. 1–4 (2014)Google Scholar
  65. 65.
    Rainer, B., Petscharnig, S., Timmerer, C.: Is one second enough? Evaluating QoE for inter-destination multimedia schronization using human computation and crowdsourcing. In: Seventh International Workshop on Quality of Multimedia Experience, pp. 1–6 (2015)Google Scholar
  66. 66.
    Ramanathan, S., Rangan, P.: Feedback techniques for intra-media continuity and inter-media synchronization in distributed media systems. Comput. J. Oxford Univ. Press 36(1), 19–31 (1993)Google Scholar
  67. 67.
    Ramanathan, S., Rangan, P.V.: Continuous media synchronization in distributed multimedia systems. In: Proceedings of ACM International Workshop on Network and Operating System Support for Digital Audio and Video (NOSSDAV), La Jolla, CA, USA, pp. 289–296 (1992)Google Scholar
  68. 68.
    Ravindran, K., Bansal, V.: Delay compensation protocols for synchronization of multimedia data streams. IEEE Trans. Knowl. Data Eng. 4(5), 574–589 (1993)CrossRefGoogle Scholar
  69. 69.
    RFC-1889: Obsolete version—RTP: a transport protocol for real-time applications. http://tools.ietf.org/html/rfc1889/ (1996). Accessed 28 Apr 2017
  70. 70.
    RFC-3550: RTP: a transport protocol for real-time applications. http://tools.ietf.org/html/rfc3550/ (2003). Accessed 28 Apr 2017
  71. 71.
    RFC-3611: RTP control protocol extended reports (RTCP XR). http://tools.ietf.org/html/rfc3611/ (2003). Accessed 28 Apr 2017
  72. 72.
    RFC-7272: Inter-destination media synchronization (IDMS) using the RTP control protocol (RTCP). http://tools.ietf.org/html/rfc7272 (2014). Accessed 28 Apr 2017
  73. 73.
    Rothermel, K., Helbig, T.: An adaptive stream synchronization protocol. In: Proceedings of ACM International Workshop on Network and Operating System Support for Digital Audio and Video (NOSSDAV), Durham, NH, USA, pp. 189–202 (1995)Google Scholar
  74. 74.
    Rouskas, G.N., Baldine, I.: Multicast routing with end-to-end delay and delay variation constraints. IEEE J. Sel. Areas in Commun. 15(3), 346–356 (1997)CrossRefGoogle Scholar
  75. 75.
    Shi, S.Y., Turner, J.S., Waldvogel, M.: Dimensioning server access bandwidth and multicast routing in overlay networks. In: Proceedings of ACM International Workshop on Network and Operating System Support for Digital Audio and Video (NOSSDAV), Danfords on the Sound, NY, USA, pp. 83–92 (2001)Google Scholar
  76. 76.
    Steinmetz, R.: Analyse von synchronisation mechanismen mit anwendung im multimedia-bereich. In: Proceedings of GI ITG Workshop Sprachen und System zur Parallelverarbeitung, Arnoldshain, Germany, pp. 39–47 (1990)Google Scholar
  77. 77.
    Steinmetz, R.: Human perception of jitter and media synchronation. IEEE J. Sel. Areas Commun. 14(1), 61–72 (1996)CrossRefGoogle Scholar
  78. 78.
    Steinmetz, R., Nahrstedt, K.: Multimedia Computing, Communications and Applications. Prentice Hall (2015)Google Scholar
  79. 79.
    Stockham, T.: A/D and D/A converters: their effect on digital audio fidelity. IEEE Digital Signal Process. 55–66 (1972)Google Scholar
  80. 80.
    Tov, S.Y.: Happy 10th birthday, VoIP, The Marker. http://archive.li/TqIrI (2005). Accessed 28 Apr 2017
  81. 81.
    Vaishnavi, I., Cesar, P., Bulterman, D., Friedrich, O., Gunkel, S., Geerts, D.: From IPTV to synchronous shared experiences: challenges in design: distributed media synchronization. Signal Process. Image Commun. 26, 370–377 (2011)CrossRefGoogle Scholar
  82. 82.
    Wahl, T., Rothernel, K.: Representing time in multimedia systems. In: Proceedings of IEEE International Conference on Multimedia Computing and Systems (ICMCS), Boston, USA, pp. 538–543 (1994)Google Scholar
  83. 83.
    Woo, M., Qazi, N., Ghafoor, A.: A synchronization framework for communication of pre-orchestrated multimedia information. IEEE Netw. 1(8), 52–61 (1994)Google Scholar
  84. 84.
    Yavatkar, R.: MCP: A protocol for coordination and temporal synchronization in collaborative applications. In: Proceedings of the IEEE International Conference Distributed Computing Systems (ICDCS), Yokohama, Japan, pp. 606–613 (1992)Google Scholar
  85. 85.
    Zhang, X., Liu, J., Li, B., shing Peter Yum T.: CoolStreaming/DONet: A data-driven overlay network for peer-to-peer live media streaming. In: Proceedings of IEEE International Conference on Computer Communications (INFOCOM), Miami, USA, pp. 2102–2111 (2005)Google Scholar
  86. 86.
    Zimmermann R, Liang K.: Spatialized audio streaming for networked virtual environments. In: Proceedings of ACM International Conference on Multimedia (MM), Vancouver, Canada, pp. 299–308 (2008)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Google Inc.Mountain ViewUSA
  2. 2.University of Illinois at Urbana-ChampaignChampaignUSA
  3. 3.Technische Universitat DarmstadtDarmstadtGermany

Personalised recommendations