Skip to main content
SpringerLink
Log in

Configuring and controlling ubiquitous computing infrastructure with semantic connections: a tangible and an AR approach

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

In the transition from a device-oriented paradigm toward a more task-oriented paradigm with increased interoperability, people are struggling with inappropriate user interfaces, competing standards, technical incompatibilities, and other difficulties. The current handles for users to explore, make, and break connections between devices seem to disappear in overly complex menu structures displayed on small screens. This paper tackles the problem of establishing connections between devices in a smart home environment, by introducing an interaction model that we call semantic connections. Two prototypes are demonstrated that introduce both a tangible and an augmented reality approach toward exploring, making, and breaking connections. In the augmented reality approach, connections between real-world objects are visualized by displaying visible lines and icons from a mobile device containing a pico projector. In the tangible approach, objects in the environment are tagged and can be scanned and interacted with, to explore connection possibilities, and manipulate the connections. We discuss the technical implementation of a pilot study setup used to evaluate both our interaction approaches. We conclude the paper with the results of a user study that shows how the interaction approaches influence the mental models users construct after interacting with our setup.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. http://www.sofia-project.eu/.

  2. As used in semantic web technologies, and in the construction of natural language user interfaces or speech dialogue systems.

  3. Available from http://sourceforge.net/projects/smart-m3/.

  4. A symmetric property is its own inverse, which means that if we indicate a connectedTo relationship from device A to device B, device B will also have a connectedTo relationship to device A. An irreflexive property is a property that never relates an individual to itself. This allows us to restrict our model by not allowing a connectedTo relationship from a device to itself.

  5. http://jena.sourceforge.net/.

  6. http://topbraid.org/spin/api/.

  7. http://www.spinrdf.org.

References

  1. Weiser M (1991) The computer for the 21st century. Scientific American

  2. OWL web ontology language use cases and requirements. http://www.w3.org/TR/webont-req/ (2004)

  3. Niezen G, van der Vlist B, Hu J, Feijs L (2010) From events to goals: Supporting semantic interaction in smart environments. In: 2010 IEEE symposium on computers and communications (ISCC), pp 1029–1034. doi:10.1109/ISCC.2010.5546634

  4. Rapp S, Michelitsch G, Osen M, Williams J, Barbisch M, Bohan R, Valsan Z, Emele M (2004) Spotlight navigation: interaction with a handheld projection device. In: Advances in pervasive computing: a collection of contributions presented at PERVASIVE 2004. Oesterreichische Computer Gesellschaft, pp 397–400

  5. Rapp S (2010) Spotlight navigation: a pioneering user interface for mobile projection. In: Proceedings of ubiprojection, Helsinki

  6. van der Vlist B, Niezen G, Rapp S, Hu J, Feijs L (2012) Controlling smart home environments with semantic connections: a tangible and an AR approach. In: Design and semantics of form and movement (DeSForM) 2012. Wellington, New Zealand, pp 160–169

  7. Siio I, Masui T, Fukuchi K (1999) Real-world interaction using the FieldMouse. In: Proceedings of the 12th annual ACM symposium on user interface software and technology—UIST’99, vol 1. ACM Press, pp 113–119. doi:10.1145/320719.322592. http://portal.acm.org/citation.cfm?doid=320719.322592

  8. Ayatsuka Y, Rekimoto J (2005) tranSticks: physically manipulatable virtual connections. In: CHI’05: proceedings of the SIGCHI conference on human factors in computing systems. ACM, Portland, OR, USA, pp 251–260. doi:10.1145/1054972.1055008

  9. Want R, Fishkin KP, Gujar A, Harrison BL (1999) Bridging physical and virtual worlds with electronic tags. In: CHI’99: proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, NY, USA, pp 370–377. doi:10.1145/302979.303111

  10. Holmquist LE, Redström J, Ljungstrand P (1999) Token-based acces to digital information. In: HUC’99: proceedings of the 1st international symposium on handheld and ubiquitous computing. Springer, London, UK, pp 234–245

  11. Rekimoto J (1997) Pick-and-drop: a direct manipulation technique for multiple computer environments. In: Proceedings of the 10th annual ACM symposium on user interface software and technology. ACM, Banff, AB, Canada, pp 31–39. doi:10.1145/263407.263505

  12. Lee H, Jeong H, Lee J, Yeom K, Shin H, Park J (2008) Select-and-point: a novel interface for multi-device connection and control based on simple hand gestures. In: CHI’08: extended abstracts on Human factors in computing systems. ACM, Florence, Italy, pp 3357–3362. doi:10.1145/1358628.1358857

  13. Rekimoto J, Ayatsuka Y, Kohno M, Oba H (2003) Proximal interactions: a direct manipulation technique for wireless networking. In: Proceedings of human–computer interaction; INTERACT’03. IOS Press, Amsterdam, The Netherlands, pp 511–518

  14. Fitzmaurice GW (1993) Situated information spaces and spatially aware palmtop computers. Commun ACM 36:39–49. doi:10.1145/159544.159566.

    Google Scholar 

  15. Rukzio E, Holleis P (2010) Projector phone interactions: design space and survey. In: Workshop on coupled display visual interfaces at AVI 2010. Rome, Italy

  16. Ballagas R, Borchers J, Rohs M, Sheridan JG (2006) The smart phone: a ubiquitous input device. IEEE Pervasive Comput 5(1):70–77. doi:10.1109/MPRV.2006.18

    Article  Google Scholar 

  17. Foley JD, Wallace VL, Chan P (1984) The human factors of computer graphics interaction techniques. IEEE Comput Graph Appl 4(11):13–48

    Google Scholar 

  18. Rukzio E, Leichtenstern K, Callaghan V, Holleis P, Schmidt A, Chin J (2006) An experimental comparison of physical mobile interaction techniques: touching, pointing and scanning. In: 8th international conference on ubiquitous computing (UbiComp’06). Springer, pp 87–104. http://www.springerlink.com/index/h6jut558g604x083.pdf

  19. Hinckley K, Ramos G, Guimbretiere F, Baudisch P, Smith M (2004) Stitching: pen gestures that span multiple displays. In: Proceedings of the working conference on advanced visual interfaces. ACM, pp 23–31. http://portal.acm.org/citation.cfm?id=989866

  20. Molyneaux D, Gellersen H, Kortuem G, Schiele B (2007) Cooperative augmentation of smart objects with projector–camera systems. In: Proceedings of the 9th international conference on ubiquitous computing. Springer, pp 501–518. http://portal.acm.org/citation.cfm?id=1771592.1771621

  21. Beardsley P, van Baar J, Raskar R, Forlines C (2005) Interaction using a handheld projector. IEEE Comput Graph Appl 25(1):39–43. http://www.ncbi.nlm.nih.gov/pubmed/15691170

    Google Scholar 

  22. Heider T, Kirste T (2002) Supporting goal-based interaction with dynamic intelligent environments. In: Proceedings of the 15th European conference on artificial intelligence. Lyon, France

  23. Torge S, Rapp S, Kompe R (2002) The planning component of an intelligent human–machine interface in changing environments. In: Multi-modal dialogue in mobile environments (IDS-02), ISCA Tutorial and Research Workshop (ITRW)

  24. Newman MW, Sedivy JZ, Neuwirth CM, Edwards WK, Hong JI, Izadi S, Marcelo K, Smith TF (2002) Designing for serendipity: supporting end-user configuration of ubiquitous computing environments. In: Proceedings of the 4th conference on designing interactive systems: processes, practices, methods, and techniques, DIS’02. ACM, New York, NY, USA, pp 147–156. doi:10.1145/778712.778736

  25. Oliver I, Honkola J (2008) Personal semantic web through a space based computing environment. In: Middleware for the semantic web, second IEEE international conference on semantic computing. Santa Clara, CA, USA. http://arxiv.org/pdf/0808.1455

  26. van der Vlist B, Niezen G, Hu J, Feijs L (2010) Semantic connections: exploring and manipulating connections in smart spaces. In: 2010 IEEE symposium on computers and communications (ISCC), pp 1–4. doi:10.1109/ISCC.2010.5546636

  27. Poole ES, Chetty M, Grinter RE, Edwards WK (2008) More than meets the eye: transforming the user experience of home network management. In: Proceedings of the 7th ACM conference on designing interactive systems, DIS’08. ACM, New York, NY, USA, pp 455–464. doi:10.1145/1394445.1394494

  28. Wensveen SAG, Djajadiningrat JP, Overbeeke CJ (2004) Interaction frogger: a design framework to couple action and function through feedback and feedforward. In: Proceedings of the 5th conference on designing interactive systems: processes, practices, methods, and techniques, pp 177–184

  29. van der Vlist B, Niezen G, Hu J, Feijs L (2010) Design semantics of connections in a smart home environment. In: Chen LL, Djajadiningrat T, Feijs L, Kyffin S, Steffen D, Young B (eds) Proceedings of design and semantics of form and movement (DeSForM) 2010. Koninklijke Philips Electronics N.V., Lucerne, Switzerland, pp 48–56

  30. Kwak M, Niezen G, van der Vlist B, Hu J, Feijs L (2011) Tangible interfaces to digital connections, centralized versus decentralized. In: Pan Z, Cheok A, Müller W, Yang X (eds) Transactions on edutainment V, Lecture Notes in Computer Science, vol 6530. Springer, Berlin, pp 132–146

  31. Sutherland IE (1968) A head-mounted three dimensional display. In: AFIPS’68 (fall, part I): proceedings of the December 9–11, 1968, fall joint computer conference, part I. ACM, New York, NY, USA, pp 757–764

  32. Honkola J, Laine H, Brown R, Tyrkko O (2010) Smart-M3 information sharing platform. In: 2010 IEEE symposium on computers and communications (ISCC). Riccione, Italy, pp 1041–1046

  33. Miller RB (1968) Response time in man–computer conversational transactions. In: Proceedings of the fall joint computer conference. ACM Press, p 267. doi:10.1145/1476589.1476628

  34. Niezen G, van der Vlist BJJ, Bhardwaj S, Ozcelebi T (2012) Performance evaluation of a semantic smart space deployment. In: IEEE international conference on pervasive computing and communications workshops, pp 835–841. doi:10.1109/PerComW.2012.6197628

  35. van der Heijden H, Sorensen LS (2003) Measuring attitudes towards mobile information services: an emperical validation of the hed/ut scale. In: European conference on information systems (ECIS). Napels, Italy

  36. Hinds P (1998) User control and its many facets: a study of perceived control in human-computer interaction. Technical report, Hewlett Packard Laboratories. http://www.hpl.hp.com/techreports/98/HPL-98-154.html

  37. van der Veer GC, del Carmen Puerta Melguizo M (2003) The human–computer interaction handbook. In: Mental models. Lawrence Erlbaum Associates Inc., Hillsdale, NJ, USA, pp 52–80

  38. Want R, Hopper A, Falcão V, Gibbons J (1992) The active badge location system. ACM Trans Inf Syst 10(1):91–102

    Article  Google Scholar 

  39. Miyaoku K, Higashino S, Tonomura Y (2004) C-blink: a hue-difference-based light signal marker for large screen interaction via any mobile terminal. In: UIST’04: proceedings of the 17th annual ACM symposium on user interface software and technology. ACM, New York, NY, USA, pp 147–156

Download references

Acknowledgments

We would like to thank Aly Syed, Riccardo Trevisan, Sriram Srinivasan, Hans van Amstel, Jettie Hoonhout, and Jolijn Teunisse for their contributions to the smart home pilot. This work is supported by Smart Objects For Intelligent Applications (SOFIA), funded by the European Artemis programme under the subprogramme SP3 Smart environments and scalable digital service.

Author information

Authors and Affiliations

  1. Department of Industrial Design, Technische Universiteit Eindhoven, Eindhoven, The Netherlands

    Bram van der Vlist, Gerrit Niezen, Jun Hu & Loe Feijs

  2. Conante, Fellenoord 130, 5611 ZB, Eindhoven, The Netherlands

    Stefan Rapp

Authors
  1. Bram van der Vlist
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gerrit Niezen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefan Rapp
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jun Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Loe Feijs
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bram van der Vlist.

Rights and permissions

Reprints and permissions

About this article

Cite this article

van der Vlist, B., Niezen, G., Rapp, S. et al. Configuring and controlling ubiquitous computing infrastructure with semantic connections: a tangible and an AR approach. Pers Ubiquit Comput 17, 783–799 (2013). https://doi.org/10.1007/s00779-012-0627-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-012-0627-x

Keywords

Search

Navigation