MediaSync pp 73-104 | Cite as

Theoretical Foundations: Formalized Temporal Models for Hyperlinked Multimedia Documents

  • Britta Meixner


Consistent linking and accurate synchronization of multimedia elements in hypervideos or multimedia documents are essential to provide a good quality of experience to viewers. Temporal models are needed to define relationships and constraints between multimedia elements and create an appealing presentation. However, no commonly used description language for temporal models exists. This makes existing temporal models harder to understand, compare, and transform from one to another temporal model. Using a formal description is more accurate than commonly used textual descriptions or figures of temporal models. This abstract representation makes it is easier to precisely define algorithms and constraints for delivery and buffering, as well as behavior of user and/or multimedia document. The use of a common formalism for all temporal models makes it possible to define synchronization constraints and media management. The same variables and terminology can then be used for describing algorithms that are applied to the documents, for example, to implement pre-fetching or download and cache management in order to increase the quality of experience for users. In this chapter, we give an overview of different existing temporal models for linked and temporally synchronized multimedia documents, like point-based, event-based, or interval-based temporal models. We analyze their common features and formally define their elementary components. We then give formal definitions for each temporal model covering essential features. These can then be used to computationally solve existing problems. We show this by defining basic functions that can be used in algorithms. We also show how user interaction and resulting video behavior can be precisely defined.


Multimedia document Hypervideo Temporal model Formal definition 


  1. 1.
    AFNOR Expert Group: Multimedia synchronization: Definitions and model, input contribution on time variant aspects and synchronization in oda-extensions. ISO IE JTC 1/SC 18/WG3 (1989)Google Scholar
  2. 2.
    Allen, J.F.: Maintaining knowledge about temporal intervals. Commun. ACM 26(11), 832–843 (1983)Google Scholar
  3. 3.
    Appelt, W., Scheller, A.: HyperODA–going beyond traditional document structures. Comput. Stand. Interfaces 17(1), 13–21 (1995)CrossRefGoogle Scholar
  4. 4.
    Benbernou, S., Makhoul, A., Hacid, M.S., Mostefaoui, A.: A spatio-temporal adaptation model for multimedia presentations. In 7th IEEE International Symposium on Multimedia (ISM’05), pp. 8–pp. Dec (2005)Google Scholar
  5. 5.
    Blakowski, G., Steinmetz, R.: A media synchronization survey: reference model, specification, and case studies. IEEE J. Sel. Areas Commun. 14(1), 5–35 (1996)CrossRefGoogle Scholar
  6. 6.
    Blakowski, G., Hübel, J., Langrehr, U., Mühlhäuser, M.: Tool support for the synchronization and presentation of distributed multimedia. Comput. Commun. 15(10), 611–618 (1992)CrossRefGoogle Scholar
  7. 7.
    Boll, S., Klas, W., Westermann, U.: A comparison of multimedia document models concerning advanced requirements. Technical Report UIB-1999–01, DBIS (1999)Google Scholar
  8. 8.
    Boll, S., Klas, W.: ZYX—A Semantic Model for Multimedia Documents and Presentations, pp. 189–209. Springer, US, Boston, MA (1999)Google Scholar
  9. 9.
    Buchanan, M.C., Zellweger, P.T.: Automatic temporal layout mechanisms. In: Proceedings of the First ACM International Conference on Multimedia, MULTIMEDIA ’93, pp. 341–350. ACM, New York, NY, USA (1993)Google Scholar
  10. 10.
    Buchanan, M.C., Zellweger, P.T.: Automatic temporal layout mechanisms revisited. ACM Trans. Multimedia Comput. Commun. Appl. 1(1), 60–88 (2005)CrossRefGoogle Scholar
  11. 11.
    Campbell, and A. N. Habermann. The specification of process synchronization by path expressions. In: Operating Systems, Proceedings of an International Symposium, pp. 89–102. Springer, London, UK (1974)Google Scholar
  12. 12.
    Chen, J.-J., Hang, H.-M.: Source model for transform video coder and its application ii. variable frame rate coding. IEEE Trans. Circuits Syst. Video Technol. 7(2), 299–311 (1997)CrossRefGoogle Scholar
  13. 13.
    Courtiat, J.P., De Oliveira, R.C.: Proving temporal consistency in a new multimedia synchronization model. In: Proceedings of the 4th ACM International Conference on Multimedia, MULTIMEDIA ’93, pp. 141–152. ACM, New York, NY, USA (1996)Google Scholar
  14. 14.
    Drapeau, G.D.: Synchronization in the maestro multimedia authoring environment. In: Proceedings of the First ACM International Conference on Multimedia, MULTIMEDIA ’93, pp. 331–339. ACM, New York, NY, USA (1993)Google Scholar
  15. 15.
    Duda, A., Keramane, C.: Structured temporal composition of multimedia data. In: 1995 Proceedings International Workshop on Multi-Media Database Management Systems, pp. 136. Aug 1995Google Scholar
  16. 16.
    Eijk, P.V., Diaz, M.: Formal Description Technique Lotos: Results of the Esprit Sedos Project. Elsevier Science Inc., New York, NY, USA (1989)zbMATHGoogle Scholar
  17. 17.
    Euzenat, J., Layaïda, N., Dias, V.: A semantic framework for multimedia document adaptation. In: Proceedings of the 18th International Joint Conference on Artificial Intelligence IJCAI’2003, pp. 31–36. Morgan Kauffman, Acapulco, Mexico, Aug 2003Google Scholar
  18. 18.
    Fiume, E., Tsichritzis, D., Dami, L.: A temporal scripting language for object-oriented animation. In: EG 1987-Technical Papers Eurographics Association (1987)Google Scholar
  19. 19.
    Fujikawa, K., Shimojo, S., Matsuura, T., Nishio, S., Miyahara, H.: Multimedia presentation system “harmony” with temporal and activemedia. In: Multimedia for Now and the Future, USENIX (1991)Google Scholar
  20. 20.
    Gibbs, S.J., Breiteneder, C., Tsichritzis, D.: Audio/video databases: an object-oriented approach. In: Proceedings of the 9th International Conference on Data Engineering, pp. 381–390. IEEE Computer Society, Washington, DC, USA (1993)Google Scholar
  21. 21.
    Gotthardt, K.: Grundlagen der Informationstechnik. LIT Verlag Münster, Einführungen–Informatik (2001)Google Scholar
  22. 22.
    Hirzalla, N., Falchuk, B., Karmouch, A.: A temporal model for interactive multimedia scenarios. IEEE Multimedia 2(3), 24–31 (1995)CrossRefGoogle Scholar
  23. 23.
    Hoepner, P.: Synchronisation der Präsentation von Multimedia-Objekten, pp. 455–464. Springer, Berlin Heidelberg (1991)Google Scholar
  24. 24.
    Illik, J.: Formale Methoden der Informatik: Von der Automatentheorie zu Algorithmen und Datenstrukturen. Expert, Reihe Technik (2009)Google Scholar
  25. 25.
    ISO/EIC: Iso/iec 13522-6:1998(en) information technology—coding of multimedia and hypermedia information. Website (1998)Google Scholar
  26. 26.
    ISO/IEC: Information technology—hypermedia/time-based structuring language (hytime) (iso/iec jtc 1/sc 34) (1997)Google Scholar
  27. 27.
    Kim, W., Kenchammana-hosekote, D., Lim, E.P., Srivastava, J.: Synchronization relation tree: a model for temporal synchronization in multimedia presentations. Technical Report (1992)Google Scholar
  28. 28.
    Kim, J.W., Kim, Y.-G., Song, H., Kuo, T.-Y., Chung, Y.J., Kuo, C.-C.J.: Tcp-friendly internet video streaming employing variable frame-rate encoding and interpolation. IEEE Trans. Circuits Syst. Video Technol. 10(7), 1164–1177 (2000)CrossRefGoogle Scholar
  29. 29.
    Kretz, F., Colaitis, F.: Standardizing hypermedia information objects. Comm. Mag. 30(5), 60–70 (1992)Google Scholar
  30. 30.
    Little, T.D.C., Ghafoor, A.: Interval-based conceptual models for time-dependent multimedia data. IEEE Trans. Knowl. Data Eng. 5(4), 551–563 (1993)Google Scholar
  31. 31.
    Little, T.D.C., Ghafoor, A.: Scheduling of bandwidth-constrained multimedia traffic. In: Proceedings of the 2nd International Workshop on Network and Operating System Support for Digital Audio and Video, pp. 120–131. Springer, London, UK (1992)Google Scholar
  32. 32.
    Little, T.D.C., Ghafoor, A.: Synchronization and storage models for multimedia objects. IEEE J. Sel. Areas Commun. 8(3), 413–427 (1990)CrossRefGoogle Scholar
  33. 33.
    Little, T.D.C., Ghafoor, A.: Spatio-temporal composition of distributed multimedia objects for value-added networks. Computer 24(10), 42–50 (1991)CrossRefGoogle Scholar
  34. 34.
    Macromind Director: Version 3.0: Overview Manual. MacroMind (1991)Google Scholar
  35. 35.
    Meixner, B., Einsiedler, C.: Download and cache management for HTML5 hypervideo players. In: Proceedings of the 27th ACM Conference on Hypertext and Social Media HT ’16, pp. 125–136. ACM, New York, NY, USA (2016)Google Scholar
  36. 36.
    Meixner, B., John, S., Handschigl, C.: Siva suite: framework for hypervideo creation, playback and management. In: Proceedings of the 23rd ACM International Conference on Multimedia, MM ’15, pp. 713–716. ACM, New York, NY, USA (2015)Google Scholar
  37. 37.
    Meixner, B., Kosch, H.: Interactive non-linear video: definition and xml structure. In: Proceedings of the 2012 ACM Symposium on Document Engineering, DocEng ’12, pp. 49–58. ACM, New York, NY, USA (2012)Google Scholar
  38. 38.
    Meixner, B.: Annotated interactive non-linear video–software suite, download and cache management. Ph.D. thesis, Universität Passau (2014)Google Scholar
  39. 39.
    Meixner, B.: Hypervideos and interactive multimedia presentations. ACM Comput. Surv. 50(1), 9:1–9:34 (2017)Google Scholar
  40. 40.
    Ogawa, R., Harada, H., Kaneko, A.: Scenario-based hypermedia: a model and a system. In: Rizk, A., Streitz, N., Andre, J. (eds.) Hypertext: Concepts, Systems and Applications–Proceeding the First European Conference on Hypertext, pp. 38–51. Cambridge University Press, Cambridge (1990)Google Scholar
  41. 41.
    Oxford University Press: British & world english. Website (2017). Accessed 03 May 2017
  42. 42.
    Pan, F., Lin, X., Rahardja, S., Lim, K.P., Li, Z.G., Wu, D.J., Wu, S.: Proactive frame-skipping decision scheme for variable frame rate video coding. In: 2004 IEEE International Conference on Multimedia and Expo 2004. ICME ’04, Vol. 3, pp. 1903–1906. IEEE (2004)Google Scholar
  43. 43.
    Poole, L.: Quicktime in motion. MacWorld. 8(9), 154–159 (1991)Google Scholar
  44. 44.
    Rousseau, F., Duda, A.: An execution architecture for synchronized multimedia presentations, pp. 42–55. Springer, Berlin, Heidelberg (1998)Google Scholar
  45. 45.
    Shepherd, D., Salmony, M.: Extending osi to support synchronization required by multimedia applications. Comput. Commun. 13(7), 399–406 (1990)CrossRefGoogle Scholar
  46. 46.
    Shue, J.-S., Hsieh, C.-H., Tsai, H.-S., Wang, C.-C.: Variable-rate video codec using frame adaptive finite-state vector quantization. In: 1993 IEEE International Symposium on Circuits and Systems, 1993 ISCAS ’93, vol. 1, pp. 28–31. IEEE (1993)Google Scholar
  47. 47.
    Steinmetz, R.: Synchronization properties in multimedia systems. IEEE J. Sel. A. Commun. 8(3), 401–412 (1990)CrossRefGoogle Scholar
  48. 48.
    Vazirgiannis, M., Kostalas, I., Sellis, T.: Specifying and authoring multimedia scenarios. IEEE Multimedia 6(3), 24–37 (1999)CrossRefGoogle Scholar
  49. 49.
    Wahl, T., Rothermel, K.: Representing time in multimedia systems. In: Proceedings of the International Conference on Multimedia Computing and Systems, pp. 538–543. May 1994Google Scholar
  50. 50.
    Wahl, T., Wirag, S., Rothermel, K.: Tiempo: temporal modeling and authoring of interactive multimedia. In: Proceedings of the International Conference on Multimedia Computing and Systems, pp. 274–277. May 1995Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.CWI, Science Park 123AmsterdamNetherlands

Personalised recommendations