Personal and Ubiquitous Computing

, Volume 17, Issue 4, pp 783–799 | Cite as

Configuring and controlling ubiquitous computing infrastructure with semantic connections: a tangible and an AR approach

  • Bram van der Vlist
  • Gerrit Niezen
  • Stefan Rapp
  • Jun Hu
  • Loe Feijs
Original Article


In the transition from a device-oriented paradigm toward a more task-oriented paradigm with increased interoperability, people are struggling with inappropriate user interfaces, competing standards, technical incompatibilities, and other difficulties. The current handles for users to explore, make, and break connections between devices seem to disappear in overly complex menu structures displayed on small screens. This paper tackles the problem of establishing connections between devices in a smart home environment, by introducing an interaction model that we call semantic connections. Two prototypes are demonstrated that introduce both a tangible and an augmented reality approach toward exploring, making, and breaking connections. In the augmented reality approach, connections between real-world objects are visualized by displaying visible lines and icons from a mobile device containing a pico projector. In the tangible approach, objects in the environment are tagged and can be scanned and interacted with, to explore connection possibilities, and manipulate the connections. We discuss the technical implementation of a pilot study setup used to evaluate both our interaction approaches. We conclude the paper with the results of a user study that shows how the interaction approaches influence the mental models users construct after interacting with our setup.


Personal projection Semantic web Ontologies Smart home Interaction design 



We would like to thank Aly Syed, Riccardo Trevisan, Sriram Srinivasan, Hans van Amstel, Jettie Hoonhout, and Jolijn Teunisse for their contributions to the smart home pilot. This work is supported by Smart Objects For Intelligent Applications (SOFIA), funded by the European Artemis programme under the subprogramme SP3 Smart environments and scalable digital service.


  1. 1.
    Weiser M (1991) The computer for the 21st century. Scientific AmericanGoogle Scholar
  2. 2.
    OWL web ontology language use cases and requirements. (2004)
  3. 3.
    Niezen G, van der Vlist B, Hu J, Feijs L (2010) From events to goals: Supporting semantic interaction in smart environments. In: 2010 IEEE symposium on computers and communications (ISCC), pp 1029–1034. doi: 10.1109/ISCC.2010.5546634
  4. 4.
    Rapp S, Michelitsch G, Osen M, Williams J, Barbisch M, Bohan R, Valsan Z, Emele M (2004) Spotlight navigation: interaction with a handheld projection device. In: Advances in pervasive computing: a collection of contributions presented at PERVASIVE 2004. Oesterreichische Computer Gesellschaft, pp 397–400Google Scholar
  5. 5.
    Rapp S (2010) Spotlight navigation: a pioneering user interface for mobile projection. In: Proceedings of ubiprojection, HelsinkiGoogle Scholar
  6. 6.
    van der Vlist B, Niezen G, Rapp S, Hu J, Feijs L (2012) Controlling smart home environments with semantic connections: a tangible and an AR approach. In: Design and semantics of form and movement (DeSForM) 2012. Wellington, New Zealand, pp 160–169Google Scholar
  7. 7.
    Siio I, Masui T, Fukuchi K (1999) Real-world interaction using the FieldMouse. In: Proceedings of the 12th annual ACM symposium on user interface software and technology—UIST’99, vol 1. ACM Press, pp 113–119. doi: 10.1145/320719.322592.
  8. 8.
    Ayatsuka Y, Rekimoto J (2005) tranSticks: physically manipulatable virtual connections. In: CHI’05: proceedings of the SIGCHI conference on human factors in computing systems. ACM, Portland, OR, USA, pp 251–260. doi: 10.1145/1054972.1055008
  9. 9.
    Want R, Fishkin KP, Gujar A, Harrison BL (1999) Bridging physical and virtual worlds with electronic tags. In: CHI’99: proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, NY, USA, pp 370–377. doi: 10.1145/302979.303111
  10. 10.
    Holmquist LE, Redström J, Ljungstrand P (1999) Token-based acces to digital information. In: HUC’99: proceedings of the 1st international symposium on handheld and ubiquitous computing. Springer, London, UK, pp 234–245Google Scholar
  11. 11.
    Rekimoto J (1997) Pick-and-drop: a direct manipulation technique for multiple computer environments. In: Proceedings of the 10th annual ACM symposium on user interface software and technology. ACM, Banff, AB, Canada, pp 31–39. doi: 10.1145/263407.263505
  12. 12.
    Lee H, Jeong H, Lee J, Yeom K, Shin H, Park J (2008) Select-and-point: a novel interface for multi-device connection and control based on simple hand gestures. In: CHI’08: extended abstracts on Human factors in computing systems. ACM, Florence, Italy, pp 3357–3362. doi: 10.1145/1358628.1358857
  13. 13.
    Rekimoto J, Ayatsuka Y, Kohno M, Oba H (2003) Proximal interactions: a direct manipulation technique for wireless networking. In: Proceedings of human–computer interaction; INTERACT’03. IOS Press, Amsterdam, The Netherlands, pp 511–518Google Scholar
  14. 14.
    Fitzmaurice GW (1993) Situated information spaces and spatially aware palmtop computers. Commun ACM 36:39–49. doi: 10.1145/159544.159566. Google Scholar
  15. 15.
    Rukzio E, Holleis P (2010) Projector phone interactions: design space and survey. In: Workshop on coupled display visual interfaces at AVI 2010. Rome, ItalyGoogle Scholar
  16. 16.
    Ballagas R, Borchers J, Rohs M, Sheridan JG (2006) The smart phone: a ubiquitous input device. IEEE Pervasive Comput 5(1):70–77. doi: 10.1109/MPRV.2006.18 CrossRefGoogle Scholar
  17. 17.
    Foley JD, Wallace VL, Chan P (1984) The human factors of computer graphics interaction techniques. IEEE Comput Graph Appl 4(11):13–48Google Scholar
  18. 18.
    Rukzio E, Leichtenstern K, Callaghan V, Holleis P, Schmidt A, Chin J (2006) An experimental comparison of physical mobile interaction techniques: touching, pointing and scanning. In: 8th international conference on ubiquitous computing (UbiComp’06). Springer, pp 87–104.
  19. 19.
    Hinckley K, Ramos G, Guimbretiere F, Baudisch P, Smith M (2004) Stitching: pen gestures that span multiple displays. In: Proceedings of the working conference on advanced visual interfaces. ACM, pp 23–31.
  20. 20.
    Molyneaux D, Gellersen H, Kortuem G, Schiele B (2007) Cooperative augmentation of smart objects with projector–camera systems. In: Proceedings of the 9th international conference on ubiquitous computing. Springer, pp 501–518.
  21. 21.
    Beardsley P, van Baar J, Raskar R, Forlines C (2005) Interaction using a handheld projector. IEEE Comput Graph Appl 25(1):39–43. Google Scholar
  22. 22.
    Heider T, Kirste T (2002) Supporting goal-based interaction with dynamic intelligent environments. In: Proceedings of the 15th European conference on artificial intelligence. Lyon, FranceGoogle Scholar
  23. 23.
    Torge S, Rapp S, Kompe R (2002) The planning component of an intelligent human–machine interface in changing environments. In: Multi-modal dialogue in mobile environments (IDS-02), ISCA Tutorial and Research Workshop (ITRW)Google Scholar
  24. 24.
    Newman MW, Sedivy JZ, Neuwirth CM, Edwards WK, Hong JI, Izadi S, Marcelo K, Smith TF (2002) Designing for serendipity: supporting end-user configuration of ubiquitous computing environments. In: Proceedings of the 4th conference on designing interactive systems: processes, practices, methods, and techniques, DIS’02. ACM, New York, NY, USA, pp 147–156. doi: 10.1145/778712.778736
  25. 25.
    Oliver I, Honkola J (2008) Personal semantic web through a space based computing environment. In: Middleware for the semantic web, second IEEE international conference on semantic computing. Santa Clara, CA, USA.
  26. 26.
    van der Vlist B, Niezen G, Hu J, Feijs L (2010) Semantic connections: exploring and manipulating connections in smart spaces. In: 2010 IEEE symposium on computers and communications (ISCC), pp 1–4. doi: 10.1109/ISCC.2010.5546636
  27. 27.
    Poole ES, Chetty M, Grinter RE, Edwards WK (2008) More than meets the eye: transforming the user experience of home network management. In: Proceedings of the 7th ACM conference on designing interactive systems, DIS’08. ACM, New York, NY, USA, pp 455–464. doi: 10.1145/1394445.1394494
  28. 28.
    Wensveen SAG, Djajadiningrat JP, Overbeeke CJ (2004) Interaction frogger: a design framework to couple action and function through feedback and feedforward. In: Proceedings of the 5th conference on designing interactive systems: processes, practices, methods, and techniques, pp 177–184Google Scholar
  29. 29.
    van der Vlist B, Niezen G, Hu J, Feijs L (2010) Design semantics of connections in a smart home environment. In: Chen LL, Djajadiningrat T, Feijs L, Kyffin S, Steffen D, Young B (eds) Proceedings of design and semantics of form and movement (DeSForM) 2010. Koninklijke Philips Electronics N.V., Lucerne, Switzerland, pp 48–56Google Scholar
  30. 30.
    Kwak M, Niezen G, van der Vlist B, Hu J, Feijs L (2011) Tangible interfaces to digital connections, centralized versus decentralized. In: Pan Z, Cheok A, Müller W, Yang X (eds) Transactions on edutainment V, Lecture Notes in Computer Science, vol 6530. Springer, Berlin, pp 132–146Google Scholar
  31. 31.
    Sutherland IE (1968) A head-mounted three dimensional display. In: AFIPS’68 (fall, part I): proceedings of the December 9–11, 1968, fall joint computer conference, part I. ACM, New York, NY, USA, pp 757–764Google Scholar
  32. 32.
    Honkola J, Laine H, Brown R, Tyrkko O (2010) Smart-M3 information sharing platform. In: 2010 IEEE symposium on computers and communications (ISCC). Riccione, Italy, pp 1041–1046Google Scholar
  33. 33.
    Miller RB (1968) Response time in man–computer conversational transactions. In: Proceedings of the fall joint computer conference. ACM Press, p 267. doi: 10.1145/1476589.1476628
  34. 34.
    Niezen G, van der Vlist BJJ, Bhardwaj S, Ozcelebi T (2012) Performance evaluation of a semantic smart space deployment. In: IEEE international conference on pervasive computing and communications workshops, pp 835–841. doi: 10.1109/PerComW.2012.6197628
  35. 35.
    van der Heijden H, Sorensen LS (2003) Measuring attitudes towards mobile information services: an emperical validation of the hed/ut scale. In: European conference on information systems (ECIS). Napels, ItalyGoogle Scholar
  36. 36.
    Hinds P (1998) User control and its many facets: a study of perceived control in human-computer interaction. Technical report, Hewlett Packard Laboratories.
  37. 37.
    van der Veer GC, del Carmen Puerta Melguizo M (2003) The human–computer interaction handbook. In: Mental models. Lawrence Erlbaum Associates Inc., Hillsdale, NJ, USA, pp 52–80Google Scholar
  38. 38.
    Want R, Hopper A, Falcão V, Gibbons J (1992) The active badge location system. ACM Trans Inf Syst 10(1):91–102CrossRefGoogle Scholar
  39. 39.
    Miyaoku K, Higashino S, Tonomura Y (2004) C-blink: a hue-difference-based light signal marker for large screen interaction via any mobile terminal. In: UIST’04: proceedings of the 17th annual ACM symposium on user interface software and technology. ACM, New York, NY, USA, pp 147–156Google Scholar

Copyright information

© Springer-Verlag London 2012

Authors and Affiliations

  • Bram van der Vlist
    • 1
  • Gerrit Niezen
    • 1
  • Stefan Rapp
    • 2
  • Jun Hu
    • 1
  • Loe Feijs
    • 1
  1. 1.Department of Industrial DesignTechnische Universiteit EindhovenEindhovenThe Netherlands
  2. 2.ConanteEindhovenThe Netherlands

Personalised recommendations