Formal Aspects of Computing

, 21:613 | Cite as

Physigrams: modelling devices for natural interaction

  • Alan DixEmail author
  • Masitah Ghazali
  • Steve Gill
  • Joanna Hare
  • Devina Ramduny-Ellis
Original Article


This paper explores the formal specification of the physical behaviour of devices ‘unplugged’ from their digital effects. By doing this we seek to better understand the nature of physical interaction and the way this can be exploited to improve the design of hybrid devices with both physical and digital features. We use modified state transition networks of the physical behaviour, which we call physiograms, and link these to parallel diagrams of the digital state. These are used to describe a number of features of physical interaction exposed by previous work and relevant properties expressed using a formal semantics of the diagrams. As well as being an analytic tool, the physigrams have been used in a case study where product designers used and adapted them as part of the design process.


Physicality Interaction modelling Affordance Natural interaction Physical devices Product design Physigrams 


  1. Ans92.
    Anson E (1992) The device model of interaction. SIGGRAPH Comput Graph 16(3): 107–114CrossRefGoogle Scholar
  2. AvH02.
    Avrahami D, Hudson S (2002) Forming interactivity: a tool for rapid prototyping of physical interactive products. In: Proc. of the 4th Conf. on Designing interactive Systems: Processes, Practices, Methods, and Techniques (DIS ’02). ACM, New York, pp 141–146.
  3. BSK03.
    Benford S, Schnadelbach H, Koleva B, Gaver B, Schmidt A, Boucher A, Steed A, Anastasi R, Greenhalgh C, Rodden T, Gellersen H (2003) Sensible, sensable and desirable: a framework for designing physical interfaces, Technical Report Equator-03-003, Equator, 2003.
  4. BoV90.
    booker S, Vertelney L (1990) Designing the whole-product user interface. In: Laural B (eds) The art of computer interface design. Addison-Wesley, Reading, pp 57–63Google Scholar
  5. Bux86.
    Buxton W (1986) There’s more to interaction than meets the eye: some issues in manual input. In: Norman D, Draper S (eds) User centered system design: new perspectives on human–computer interaction. Lawrence Erlbaum Associates, Hillsdale, pp 319–337Google Scholar
  6. Bux90.
    Buxton W (1990) A three-state model of graphical input. In: Proc. of human–computer interaction—INTERACT ’90. Elsevier, Amsterdam, pp 449–456Google Scholar
  7. CMR90.
    Card S, Mackinlay J, Robertson G (1990) The design space of input devices. In: Proc. of CHI’90. ACM Press, New York, pp 117–124Google Scholar
  8. CMR91.
    Card S, Mackinlay J, Robertson G (1991) A morphological analysis of the design space of input devices. ACM Trans Inf Syst 9(2): 99–122CrossRefGoogle Scholar
  9. Car94.
    Carr D (1994) Specification of interface interaction objects. In: Proc. of CHI ’94. ACM, New York, pp 372–378Google Scholar
  10. CoN06.
    Coutrix C, Nigay L (2006) Mixed reality: a model of mixed interaction. In: Proc. of AVI’06. ACM Press, New York, pp 43–50Google Scholar
  11. CoN08.
    Coutrix C, Nigay L (2008) Balancing physical and digital properties in mixed objects. In: Proc. of AVI’08, the Working Conf. on Advanced Visual interfaces. ACM Press, New York, pp 305–308Google Scholar
  12. CuR07.
    Curzon P, Rukšėnas R, Blandford A (2007) An approach to formal verification of human-computer interaction. Formal Aspects Comput 19(4): 513–550zbMATHCrossRefGoogle Scholar
  13. Deg04.
    Degani A (2004) Taming HAL: designing interfaces beyond 2001. Palgrave Macmillan, New YorkGoogle Scholar
  14. DiR85.
    Dix A, Runciman C (1985) Abstract models of interactive systems. People and computers: designing the interface. Cambridge University Press, Cambridge, pp 13–22Google Scholar
  15. Dix91.
    Dix A (1991) Formal methods for interactive systems. Academic Press, New York.
  16. Dix91b.
    Dix A (1991) Status and events: static and dynamic properties of interactive systems. In: Proc. of the Eurographics Seminar: Formal Methods in Computer Graphics.
  17. DiA96.
    Dix A, Abowd G (1996) Modelling status and event behaviour of interactive systems. Softw Eng J 11(6):334–346 (1996).
  18. DiA96b.
    Dix A, Abowd G (1996) Delays and temporal incoherence due to the mediated status–status mappings. SIGCHI Bull 28(2): 47–49CrossRefGoogle Scholar
  19. Dix03.
    Dix A (2003) Getting physical, keynote at: OZCHI 2003, Brisbane, Australia.
  20. DFA04.
    Dix A, Finlay J, Abowd G, Beale R (2004) Human–computer interaction, 3rd edn. Prentice Hall, Englewood Cliffs.
  21. Dix07.
    Dix A (2007) Designing for appropriation. In: Procedings of BCS HCI 2007, People and Computers XXI, vol 2, BCS eWiC.
  22. DGR07.
    Dix A, Ghazali M, Ramduny-Ellis D (2007) Modelling devices for natural interaction. In: Proc. of Second Intnl. Workshop on Formal Methods for Interactive Systems, FMIS2007, ENTCS. Elsevier, AmsterdamGoogle Scholar
  23. DLF07.
    Dix A, Leite J, Friday A (2008) XSED—XML-based description of status–event components and systems. In: Proc. of Engineering Interactive Systems 2007 (EIS 2007). Lecture notes in computer science, vol 4940. Springer, BerlinGoogle Scholar
  24. DSG02.
    Dubois E, Silva P, Gray P (2002) Notational support for the design of augmented reality systems. In: Proc. of the 9th International Workshop on interactive Systems. Design, Specification, and Verification, DSVIS2002. Lecture notes in computer science, vol 2545. Springer, Berlin, pp 74–88Google Scholar
  25. DuG07a.
    Dubois E, Gray P (2008) A design-oriented information-flow refinement of the ASUR interaction model. In: Engineering interactive systems (incorporating EHCI, HCSE, DSV-IS). Lecture notes in computer science, vol 4940. Springer, BerlinGoogle Scholar
  26. DGR07b.
    Dubois E, Gray P, Ramsay A (2007) A model-based approach to describing and reasoning about the physicality of interaction. In: Proc. of Physicality 2007. UWIC Press, Cardiff, pp 77–82Google Scholar
  27. E06.
    Eslambolchilar P (2006) Making sense of interaction using a model-based approach. Ph D thesis, Hamilton Institute, National University of Ireland, NUIM, IrelandGoogle Scholar
  28. EvG06.
    Evans M, Gill S (2006) Rapid development of information appliances. In: Proc. of International Design Conf. Design 2006, (Croatia, 15–18 May 2006)Google Scholar
  29. Gav91.
    Gaver W (1991) Technology affordances. In: Proc. of CHI ’91. ACM Press, New York, pp 79–84Google Scholar
  30. GhD03.
    Ghazali M, Dix A (2003) Aladdin’s lamp: understanding new from old. In: Proc. of 1st UK-UbiNet Workshop, Imperial College London.
  31. GhD05.
    Ghazali M, Dix A (2005) Visceral interaction. In: Proc. of the 10th British HCI Conf., vol 2, pp 68–72.
  32. GhD06.
    Ghazali M, Dix A (2006) Natural inverse: physicality, interaction & meaning. In: Let’s Get Physical: Tangible Interaction and Rapid Prototyping in, for, and about Design Workshop at 2nd International Conf. on Design Computing & Cognition 2006Google Scholar
  33. Gib86.
    Gibson J (1986) The ecological approach to visual perception. Houghton Mifflin, USAGoogle Scholar
  34. GLH05.
    Gill S, Loudon G, Hewett B, Barham G (2005) How to design and prototype an information appliance in 24 hours—integrating product & interface design processes. In: Proc. of the 6th International Conf. on Computer Aided Industrial Design and Concept Design, University of Delft, The NetherlandsGoogle Scholar
  35. GrP96.
    Green T, Petri M (1996) Usability analysis of visual programming environments: a ‘cognitive dimensions’ framework. J Vis Languages Comput 7: 131–174CrossRefGoogle Scholar
  36. GrF01.
    Greenberg S, Fitchett C (2001) Phidgets: easy development of physical interfaces through physical widgets. In: Proc. of the 14th Annual ACM Symposium on User interface Software and Technology (UIST ’01), pp 209–218.
  37. Har87.
    Harel D (1987) Statecharts: a visual formalism for complex systems. Sci Comput Program 8(3): 231–274zbMATHCrossRefMathSciNetGoogle Scholar
  38. HKC07.
    Harrison M, Kray C, Campos J (2007/2008) Exploring an option space to engineer a ubiquitous computing system. In: 2nd International Workshop on Formal Methods for Interactive Systems (FMIS 2007), Electronic Notes in Theorectical Computer Science, vol 208. Elsevier, Amsterdam, pp 41–55Google Scholar
  39. HKB05.
    Hartman B, Klemmer S, Bernstein M, Mehta N (2005) Visually Prototyping Physical UIs through Statecharts. In: Extended Abstracts of UIST 2005. ACM, New YorkGoogle Scholar
  40. Har03.
    Hartson H (2003) Cognitive, physical, sensory, and functional affordances in interaction design. Behav Inform Technol 22(5): 315–338CrossRefGoogle Scholar
  41. Ish08.
    Ishii H (2008) Tangible user interfaces. In: Sears A, Jacko J (eds) The human–computer interaction handbook fundamentals, evolving technologies, and emerging applications, Chapter 24, 2nd edn.. Laurence Earlbaum, London, pp 469–487Google Scholar
  42. JDM99.
    Jacob J, Deligiannidis L, Morrison S (1999) A software model and specification language for non-WIMP user interfaces. ACM Trans Comput Hum Interact 6(1): 1–46CrossRefGoogle Scholar
  43. Joh96.
    Johnson C (1996) The evaluation of user interface design notations. In: Proc. of Design, Specification and Verification of Interactive Systems ’96. Springer, Berlin, pp 188–206.
  44. LaM95.
    Landay J, Myers B (1995) Interactive sketching for the early stages of user interface design. In: Proc. of CHI’95. ACM Press/Addison-Wesley, New York, pp 43–50.
  45. LoH02.
    Loer K, Harrison M (2002) Towards usable and relevant model checking techniques for the analysis of dependable interactive systems. In: Proc. 17th International Conf. on Automated Software Engineering. IEEE Computer Society, New York, pp 223–226Google Scholar
  46. MDS99.
    Massink M, Duke D, Smith S (1999) Towards hybrid interface specification for virtual environments. In: DSV-IS 1999 Design, Specification and Verification of Interactive Systems. Springer, Berlin, pp 30–51Google Scholar
  47. Mil88.
    Milner N (1988) A review of human performance and preferences with different input devices to computer systems. In: Proc. of HCI88, People and Computers IV. Cambridge University Press, Cambridge, pp 341–362Google Scholar
  48. MDB96.
    Moher T, Dirda V, Bastide R, Palanque P (1996) Monolingual, articulated modelling of users, devices and interfaces. In: 3rd EUROGRAPHICS workshop on design, specification and verification of Interactive systems. Springer, Berlin, pp 312–329Google Scholar
  49. Mon86.
    Monk A (1986) Mode errors: a user-centered analysis and some preventative measures using keying-contingent sound. Int J Man Mach Stud 24(4): 313–327CrossRefGoogle Scholar
  50. NiM94.
    Nielsen J, Mack R (1994) Usability inspection methods. Wiley, New YorkGoogle Scholar
  51. NCo91.
    Nigay L, Coutaz J (1991) Building user interfaces: organizing software agents. In: ESPRIT ‘91 Conf., pp 707–719Google Scholar
  52. NCo95.
    Nigay L, Coutaz J (1995) A generic platform for addressing the multimodal challenge. In: Proc. of CHI’95. ACM, New York, pp 98–105Google Scholar
  53. Nor99.
    Norman D (1999) Affordance, conventions, and design. Interactions 6(3): 38–43CrossRefGoogle Scholar
  54. PaP97.
    Palanque P, Paterno F (1997) (eds) Formal methods in human–computer interaction. Springer, BerlinGoogle Scholar
  55. Pa69.
    Parnas D (1969) On the use of transition diagrams in the design of a user interface for an interactive computer system. In: Proc. of the 1969 24th National Conf.. ACM, New York, pp 379–385Google Scholar
  56. PaG86.
    Payne S, Green T (1986) Task–action grammars: a model of mental representation of task languages.. Hum Comput Interact 2(2): 93–133CrossRefGoogle Scholar
  57. PfH85.
    Pfaff G, Hagen P (1985) (eds) Seeheim workshop on user interface management systems. Springer, BerlinGoogle Scholar
  58. Phi08.
    Phidgets Inc., 2008.
  59. PSP99.
    Pierce J, Stearns B, Pausch R (1999) Voodoo dolls: seamless interaction at multiple scales in virtual environments. In: Proc. of the 1999 Symposium on Interactive 3D Graphics, pp 141–145Google Scholar
  60. RDR05.
    Ramduny-Ellis D, Dix A, Rayson P, Onditi V, Sommerville I, Ransom J (2005) Artefacts as designed, Artefacts as used: resources for uncovering activity dynamics. Cogn Technol Work 7(2): 76–87CrossRefGoogle Scholar
  61. Rei81.
    Reisner P (1981) Formal grammar and human factors design of an interactive graphics system. IEEE Trans Softw Eng SE-7(2):229–240Google Scholar
  62. Shn83.
    Shneiderman B (1983) Direct manipulation: a step beyond programming languages. IEEE Comput 16(8): 57–69Google Scholar
  63. Shn84.
    Shneiderman B (1984) Response time and display rate in human performance with computers. ACM Comput Surv 16(3): 265–285CrossRefGoogle Scholar
  64. Smi06.
    Smith S (2007) Exploring the specification of haptic interaction. In: Interactive systems: design, specification and verification (DSVIS 2006). Lecture notes in computer science, vol 4323. Springer, Berlin, pp 171–184Google Scholar
  65. Suf82.
    Sufrin B (1982) Formal specification of a display editor. Sci Comput Program 1: 157–202zbMATHCrossRefGoogle Scholar
  66. ThH90.
    Thimbleby H, Harrison M (1990) Formal methods in human–computer interaction. Cambridge University Press, CambridgeGoogle Scholar
  67. Thi07.
    Thimbleby H (2007) Press On: principles of interaction programming. MIT Press, CambridgeGoogle Scholar
  68. Thi07.
    Thimbleby H (2007) Using the Fitts law with state transition systems to find optimal task timings. In: Pre-Proc. of Second Intnl. Workshop on Formal Methods for Interactive Systems, FMIS2007.
  69. UIM92.
    (1992) A metamodel for the runtime architecture of an interactive system: the UIMS tool developers workshop. SIGCHI Bull 24(1): 32–37CrossRefGoogle Scholar
  70. UIJ05.
    Ullmer B, Ishii H, Jacob R (2005) Token+constraint systems for tangible interaction with digital information. ACM Trans Comput Hum Interact 12(1): 81–118CrossRefGoogle Scholar
  71. WDO04.
    Wensveen S, Djajadiningrat J, Overbeeke C (2004) Interaction frogger: a design framework to couple action and function. In: Proc. of the DIS’04. ACM, New York, pp 177–184Google Scholar
  72. WiH00.
    Willans J, Harrison M (2001) Verifying the behaviour of virtual world objects. In: Palanque P, Paternó F (eds) Proc. of DSV-IS’2000. Springer, Berlin, pp 65–77Google Scholar
  73. Wüt99.
    Wüthrich C (1999) An analysis and model of 3D interaction methods and devices for virtual reality. In: Proc. of DSV-IS’99. Springer, Berlin, pp 18–29Google Scholar
  74. YGS89.
    Young R, Green T, Simon T (1989) Programmable user models for predictive evaluation of interface design. In: Proc. of CHI’89: Human Factors in Computing Systems. ACM Press, New YorkGoogle Scholar

Copyright information

© British Computer Society 2008

Authors and Affiliations

  • Alan Dix
    • 1
    Email author
  • Masitah Ghazali
    • 2
  • Steve Gill
    • 3
  • Joanna Hare
    • 3
  • Devina Ramduny-Ellis
    • 1
  1. 1.Computing Department, InfoLab21Lancaster UniversityLancasterUK
  2. 2.Department of Software EngineeringUniversiti Teknologi MalaysiaSkudai, JohorMalaysia
  3. 3.Cardiff School of Art and Design, UWICCardiff, WalesUK

Personalised recommendations