Skip to main content
SpringerLink
Log in

A formal model for output multimodal HCI

An Event-B formalization

  • Published:
Computing Aims and scope Submit manuscript

Abstract

Multimodal human–computer interaction (HCI) combine modalities at an abstract specification level in order to get information from the user (input multimodality) and to return information to the user (output multimodality). These multimodal interfaces use two mechanisms: first, the fusion of information transmitted by the user on different modalities during input interaction and second, the fission or decomposition of information produced by the functional core in order to distribute the composite information on the different modalities during output interaction. In this paper, we present a generic approach to design output multimodal interfaces. This approach is based on a formal model, composed of two models: semantic fission model for information decomposition process and allocation model for modalities and media allocation to composite information. An Event-B formalization has been proposed for the fission model and for allocation model. This Event-B formalization extends the generic model and support the verification of some relevant properties such as safety or liveness. An example of collision freeness property verification is presented in this paper.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

Notes

  1. Backus Naur Form.

References

  1. Abrial JR (1996) The B-book: assigning programs to meanings. Cambridge University Press, New York

    Book  MATH  Google Scholar 

  2. Abrial JR (2010) Modeling in Event-B: system and software engineering. Cambridge University Press, New York

    Book  Google Scholar 

  3. Ait-Ameur Y, Ait-Sadoune I, Baron M (2006a) Etude et comparaison de scénarios de développements formels d’interfaces multi-modales fondés sur la preuve et le raffinement. In: MOSIM 2006, 6ème Conférence Francophone de Modélisation et Simulation. Modélisation, Optimisation et Simulation des Systèmes: Défis et Opportunités, Rabat

  4. Ait-Ameur Y, Ait-Sadoune I, Mota JM, Baron M (2006b) Validation et vérification formelles de systèmes interactifs multi-modaux fondées sur la preuve. In: Proceedings of the 18th International Conference of the Association Francophone d’Interaction Homme-Machine. ACM, Montréal, pp 123–130

  5. Ait-Ameur Y, Baron M, Kamel N, Mota JM (2009) Encoding a process algebra using the Event B method: application to the validation of human computer interactions. Int J Softw Tools Technol Transf 11(3):239–253

    Article  Google Scholar 

  6. Ait-Ameur Y, Ait-Sadoune I, Baron M, Mota JM (2010) Vérification et validation formelles de systèmes interactifs fondées sur la preuve : application aux systèmes multi-modaux. Journal d’Interaction Personne-Système 1(1):1–30. http://www.journal-interaction-personne-systeme.fr/articles/80-articles/85-vol-1-2010-num-1-art-3

  7. Bernsen O (1994) Foundations of multimodal representations. A taxonomy of representational modalities. Interact Comput 6(4):347–371

    Article  Google Scholar 

  8. Bordegoni M, Faconti G, Maybury M, Rist T, Ruggieri S, Trahanias P, Wilson M (1997) A standard reference model for intelligent multimedia presentation systems. Comput Stand Interface 6(4):477–496

    Article  Google Scholar 

  9. Bouchet J, Madani L, Nigay L, Oriat C, Parissis I (2008) Formal testing of multimodal interactive systems. In: Gulliksen J, Harning M, Palanque P, van der Veer G, Wesson J (eds) Engineering Interactive Systems, vol 4940., Lecture Notes in Computer ScienceSpringer, Berlin Heidelberg, pp 36–52

    Chapter  Google Scholar 

  10. Bourguet ML (2003) Designing and prototyping multimodal commands. INTERACT, vol 3., LNCSSpringer, Berlin, pp 717–720

    Google Scholar 

  11. Cohen PR, Johnston M, McGee D, Oviatt S, Pittman J, Smith I, Chen L, Clow J (1997) Quickset: Multimodal interaction for distributed applications. In: Proceedings of the Fifth ACM International Conference on Multimedia, ACM, New York, MULTIMEDIA ’97, pp 31–40

  12. Coutaz J, Nigay L (1994) Les propriétés CARE dans les interfaces multimodales. Actes de la Conférence IHM’94. Lille, France, pp 7–14

    Google Scholar 

  13. Dijkstra EW (1977) A discipline of programming, 1st edn. Prentice Hall PTR, Upper Saddle River

    Google Scholar 

  14. Duarte C, Carriço L (2006) A conceptual framework for developing adaptive multimodal applications. In: Proceedings of the 11th international conference on Intelligent user interfaces. ACM, Sydney, pp 132–139

  15. Dumas B, Lalanne D, Oviatt S (2009) Multimodal interfaces: a survey of principles, models and frameworks. In: Lalanne D, Kohlas J (eds) Human Machine Interaction, vol 5440., Lecture Notes in Computer ScienceSpringer, Berlin Heidelberg, pp 3–26

    Chapter  Google Scholar 

  16. Flippo F, Krebs A, Marsic I (2003) A framework for rapid development of multimodal interfaces. In: Proceedings of the 5th International Conference on Multimodal Interfaces, ACM, New York, ICMI ’03, pp 109–116

  17. Glass J, Weinstein E, Cyphers S, Polifroni J, Chung G, Nakano M (2005) A framework for developing conversational user interfaces. In: Jacob RJ, Limbourg Q, Vanderdonckt J (eds) Computer-aided design of user interfaces IV. Springer, Amsterdam, pp 349–360

    Chapter  Google Scholar 

  18. Gram C, Cockton G (eds) (1997) Design principles for interactive software. Chapman & Hall Ltd, London

    Google Scholar 

  19. Jourde F, Nigay L, Parissis I (2006) Test formel de systèmes interactifs multimodaux : couplage ICARE - Lutess. In: ICSSEA’2006, 19ème journées Internationales “génie logiciel & Ingénierie de Systèmes et leurs Applications” Globalisation des services et des systèmes, Paris

  20. Kamel N (2004) Utilisation de SMV pour la vérification de propriétés d’IHM multimodales. 16ème Conférence Francophone sur l’Interaction Homme-Machine (IHM’2004). ACM Press, Namur, Belgique, pp 219–222

    Google Scholar 

  21. Kamel N, Ait-Ameur Y (2007) A formal model for CARE usability properties verification in multimodal HCI. In: IEEE International Conference on Pervasive Services. IEEE, Istanbul, pp 341–348

  22. Krahnstoever N, Kettebekov S, Yeasin M, Sharma R (2002) A real-time framework for natural multimodal interaction with large screen displays. In: Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, IEEE Computer Society, Washington, ICMI ’02, p 349

  23. Larson JA, Raman T, Raggett D, Bodell M, Johnston M, Kumar S, Potter S, Waters K (2003) W3C multimodal interaction framework. W3C NOTE 6

  24. MacColl I, Carrington D (1998) Testing MATIS: a case study on specification-based testing of interactive systems. In: Formal Aspects of Human Computer Interaction Workshop (FAHCI98), pp 57–69

  25. Madani L, Parissis I (2009) Automatically testing interactive applications using extended task trees. J Log Algebr Program 78(6):454–471

    Article  MATH  MathSciNet  Google Scholar 

  26. Mohand-Oussaid L, Ait-Ameur Y, Ahmed-Nacer M (2009) A generic formal model for fission of modalities in output multi-modal interactive systems. In: International Workshop on Verification and Evaluation of Computer and Communication Systems, Rabat

  27. Mohand-Oussaid L, Ait-Sadoune I, Ait-Ameur Y (2011) Modelling information fission in output multi-modal interactive systems using Event-B. MEDI: model and data engineering. Springer, Obidos, pp 200–213

    Chapter  Google Scholar 

  28. Mohand-Oussaid L, Kamel N, Ait-Sadoune I, Ait-Ameur Y, Ahmed-Nacer M (2011) Human computer interaction in transport, ISTE Ltd and John Wiley and Sons Inc, chap A formal framework for design and validation of multimodal interactive systems in transport domain, pp 93–108

  29. Mohand-Oussaid L, Ait-Sadoune I, Ait-Ameur Y, Ahmed-Nacer M (2014) Formal modelling of output multi-modal HCI in Event-B: Modalities and media allocation. In: AAAI Symposium: modeling in human-machine systems: challenges for formal verification, Palo Alto

  30. Mohand-Oussaid L, Ait-Sadoune I, Ait-Ameur Y, Ahmed-Nacer M (2014) Modélisation formelle d’IHM multi-modales en sortie avec B Événementiel. In: Approches Formelles dans l’Assistance au Dveloppement de Logiciels AFADL 2014, Paris, p 76

  31. Monin JF, Hinchey MG (2003) Understanding formal methods. Springer, New York

    Book  MATH  Google Scholar 

  32. Navarre D, Palanque P, Bastide R, Schyn A, Winckler M, Nedel L, Freitas C (2005) A formal description of multimodal interaction techniques for immersive virtual reality applications. INTERACT 2005., LNCSSpringer, Roma, pp 25–28

    Google Scholar 

  33. Nigay L, Coutaz J (1995) A generic platform for addressing the multimodal challenge. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM Press/Addison-Wesley Publishing Co., New York, CHI ’95, pp 98–105

  34. Oviatt S, Cohen P, Wu L, Duncan L, Suhm B, Bers J, Holzman T, Winograd T, Landay J, Larson J et al (2000) Designing the user interface for multimodal speech and pen-based gesture applications: state-of-the-art systems and future research directions. Hum Comput Interact 15(4):263–322

    Article  Google Scholar 

  35. Palanque PA, Schyn A (2003) A model-based approach for engineering multimodal interactive systems. INTERACT., LNCSSpringer, Berlin Heidelberg, pp 03–05

    Google Scholar 

  36. Rodin (2007) User Manual of the RODIN Platform. http://deploy-eprints.ecs.soton.ac.uk/11/1/manual-2.3.pdf

  37. Rousseau C (2006) Présentation multimodale et contextuelle de l’information. PhD thesis, Université Paris sud XI-Orsay, France

  38. Song K, Lee KH (2008) Generating multimodal user interfaces for Web services. Interact Comput 20(4–5):480–490

    Article  Google Scholar 

  39. Westeyn T, Brashear H, Atrash A, Starner T (2003) Georgia tech gesture toolkit: Supporting experiments in gesture recognition. In: Proceedings of the 5th International Conference on Multimodal Interfaces, ACM, New York, ICMI ’03, pp 85–92

Download references

Author information

Authors and Affiliations

  1. CERIST, Algiers, Algeria

    Linda Mohand-Oussaid

  2. LRI, CENTRALE SUPELEC, Gif-Sur-Yvette, France

    Idir Ait-Sadoune

  3. IRIT, ENSEEIHT, Toulouse, France

    Yamine Ait-Ameur

  4. LSI, USTHB, Algiers, Algeria

    Mohamed Ahmed-Nacer

Authors
  1. Linda Mohand-Oussaid
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Idir Ait-Sadoune
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yamine Ait-Ameur
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mohamed Ahmed-Nacer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Linda Mohand-Oussaid.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mohand-Oussaid, L., Ait-Sadoune, I., Ait-Ameur, Y. et al. A formal model for output multimodal HCI. Computing 97, 713–740 (2015). https://doi.org/10.1007/s00607-015-0460-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00607-015-0460-y

Keywords

Mathematics Subject Classification

Search

Navigation