Personal and Ubiquitous Computing

, Volume 10, Issue 5, pp 269–283 | Cite as

Can we do without GUIs? Gesture and speech interaction with a patient information system

  • Eamonn O’Neill
  • Manasawee Kaenampornpan
  • Vassilis Kostakos
  • Andrew Warr
  • Dawn Woodgate
Original Article


We have developed a gesture input system that provides a common interaction technique across mobile, wearable and ubiquitous computing devices of diverse form factors. In this paper, we combine our gestural input technique with speech output and test whether or not the absence of a visual display impairs usability in this kind of multimodal interaction. This is of particular relevance to mobile, wearable and ubiquitous systems where visual displays may be restricted or unavailable. We conducted the evaluation using a prototype for a system combining gesture input and speech output to provide information to patients in a hospital Accident and Emergency Department. A group of participants was instructed to access various services using gestural inputs. The services were delivered by automated speech output. Throughout their tasks, these participants could see a visual display on which a GUI presented the available services and their corresponding gestures. Another group of participants performed the same tasks but without this visual display. It was predicted that the participants without the visual display would make more incorrect gestures and take longer to perform correct gestures than the participants with the visual display. We found no significant difference in the number of incorrect gestures made. We also found that participants with the visual display took longer than participants without it. It was suggested that for a small set of semantically distinct services with memorable and distinct gestures, the absence of a GUI visual display does not impair the usability of a system with gesture input and speech output.


Multimodal interaction Gesture input Speech output GUI Mobile Ubiquitous 



The research reported here is part of a UK EPSRC-funded research project ‘Designing for common ground in mobile distributed collaborative systems’, award number GR/R24562/01. We thank Hilary Johnson and Leon Watts for their advice and insightful comments.


  1. 1.
    Brewster SA (2002) Overcoming the lack of screen space on mobile computers. Personal and Ubiquitous Computing 6(3):188–205CrossRefGoogle Scholar
  2. 2.
    Brewster SA, Lumsden J, Bell M, Hall M, Tasker S (2003) Multi-modal ‘eyes free’ interaction techniques for wearable devices. Proc. CHI’03 Conference on Human Factors in Computing Systems, CHI Letters, ACM Press, 5(1):473–480Google Scholar
  3. 3.
    Kostakos V, O’Neill E (2003) A directional stroke recognition technique for mobile interaction in a pervasive computing world. In People and Computer XVII, Proc. HCI 2003: Designing for Society. Bath, UK, 197–206Google Scholar
  4. 4.
    O’Neill E, Woodgate D, Kostakos V (2004) Easing the wait in the Emergency Room: building a theory of public information systems. Proc. DIS’04 Conference on Designing Interactive Systems, ACM Press, pp 17–25Google Scholar
  5. 5.
    James F, Roelands J (2002) Voice over Workplace (VoWP): voice navigation in a complex business GUI. Proc. Fifth International Conference on Assistive Technologies, ACM Press, pp 197–204Google Scholar
  6. 6.
    Amar R, Dow S, Gordon R, Hamid MR, Sellers C (2003) Mobile ADVICE:an accessible device for visually impaired capability enhancement. Extended Abstracts CHI’03 Conference on Human Factors in Computing Systems, CHI Letters, ACM Press 5(1):918–919Google Scholar
  7. 7.
    Xiao Y, Lasome C, Moss, J, Mackenzie C, Faraj S (2001) Cognitive properties of a whiteboard: a case study in a trauma centre. Proc. ECSCW 2001 Seventh European Conference on Computer Supported Cooperative Work Kluwer, pp 259–278Google Scholar
  8. 8.
    Clarke, K., Hughes, J. and Rouncefield, M (2002) When a bed is not a bed: the situated display of knowledge on a hospital ward. In Workshop on Public, Community and Situated Displays, at CSCW 2002, ACM Conference on Computer Supported Cooperative Work. Available at Last accessed December 2004
  9. 9.
    Naumann S, Miles JA, (2001) Managing waiting patients’ perceptions the role of process control. Journal of Management in Medicine 15(5):376–386CrossRefPubMedGoogle Scholar
  10. 10.
    Maister DH (1988) The psychology of waiting lines. In: Lovelock J (ed) Managing services: marketing, operations and human resources. Prentice-Hall, pp 176–183Google Scholar
  11. 11.
    Dansky KH, Miles JA (1997) Patient satisfaction with ambulatory healthcare services: waiting time and filling time. Hospital and Health Services Administration 42:165–177Google Scholar
  12. 12.
    Goldstein M, Book R, Alsio G, Tessa S (1999) Non-keyboard QWERTY touch typing: a portable input interface for the mobile user. Proc. CHI’99 Conference on Human Factors in Computing Systems, ACM Press, pp 32–39Google Scholar
  13. 13.
    Wigdor D, Balakrishnan R (2003) TiltText:using tilt for text input to mobile phones. Proc. 16th Annual ACM Symposium on User Interface Software and Technology, ACM press, pp 81–90Google Scholar
  14. 14.
    Pirhonen A, Brewster S, Holguin C (2002) Gestural and audio metaphors as a means of control for mobile devices. Proc. CHI’99 Conference on Human Factors in Computing Systems, ACM Press, pp 291–298Google Scholar
  15. 15.
    Lines L, Hone KS (2002) Older adults’ evaluations of speech output. Proc. Fifth International Conference on Assistive Technologies, ACM Press, pp 170–177Google Scholar
  16. 16.
    Igarashi T, Edwards WK, LaMarca A, Mynatt ED, (2000) An architecture for pen-based interaction on electronic whiteboards. Proc. Working Conference on Advanced Visual Interfaces, ACM Press, pp 68–75Google Scholar
  17. 17.
    Kostakos V, O’Neill E (2004) Pervasive computing in emergency situations, Proc. Thirty-Seventh Annual Hawaii International Conference on System Sciences. IEEE Computer Society Press, p.30081bGoogle Scholar
  18. 18.
    Kostakos V (2004) A design framework for pervasive information systems. PhD thesis. Department of Computer Science, University of Bath. Technical Report CSBU-2005-02, Technical Report Series ISSN-1740-9497 Google Scholar
  19. 19.
    Sawhney N, Schmandt C (2000) Nomadic radio: speech and audio interaction for contextual messaging in nomadic environments. ACM Transactions on Human-Computer Interaction 7(3):353–383CrossRefGoogle Scholar
  20. 20.
    Manzke JM (1998) Adaptation of a cash dispenser to the needs of blind and visually impaired people. Proc. Third International Conference on Assistive Technologies, ACM Press, pp 116–123Google Scholar
  21. 21.
    Ross DA, Blasch BB (2002) Development of a wearable computer orientation system. Personal and Ubiquitous Computing 6(1):49–63CrossRefGoogle Scholar
  22. 22.
    Miller GA (1956) The magical number seven, plus or minus two: some limits on our capacity for processing information. The Psychological Review 63:81–97CrossRefGoogle Scholar
  23. 23.
    Cicero MT, De Oratore, II lxxxvi. Translated by H. Rackham, pp 350–353Google Scholar
  24. 24.
    Hoc J-M (2001) Towards ecological validity of research in cognitive ergonomics. Theoretical Issues in Ergonomics Science 2(3):278–288CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2005

Authors and Affiliations

  • Eamonn O’Neill
    • 1
  • Manasawee Kaenampornpan
    • 1
  • Vassilis Kostakos
    • 1
  • Andrew Warr
    • 1
  • Dawn Woodgate
    • 1
  1. 1.Human-Computer Interaction Group, Department of Computer ScienceUniversity of BathBathUK

Personalised recommendations