Skip to main content

Select & Apply: understanding how users act upon objects across devices

Personal and Ubiquitous Computing volume 19, pages 881–896 (2015)Cite this article

Abstract

As our interactions increasingly cut across diverse devices, we often encounter situations where we find information on one device but wish to use it on another device, for instance, a phone number spotted on a public display but wanted on a mobile. We conceptualize this problem as Select & Apply and contribute two user studies where we presented participants with eight different scenarios involving different device combinations, applications and data types. In the first, we used a think-aloud methodology to gain insights on how users currently accomplish such tasks and how they ideally would like to accomplish them. In the second, we conducted a focus group study to investigate which factors influence their actions. Results indicate shortcomings in present support for Select & Apply and contribute a better understanding of which factors affect cross-device interaction.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

References

  1. Ballendat T, Marquardt N, Greenberg S (2010) Proxemic interaction: designing for a proximity and orientation-aware environment. In: ACM international conference on interactive tabletops and surfaces, ITS’10. ACM, New York, NY, USA, pp 121–130

  2. Baudisch P, Cutrell E, Robbins D, Czerwinski M, Tandler P, Bederson B, Zierlinger A (2003) Drag-and-pop and drag-and-pick: techniques for accessing remote screen content on touch and pen-operated systems. In: Proceedings of interact’03. IOS Press, pp 57–64

  3. Beigl M (1999) Point and click: interaction in smart environments. In: Proceedings of Handheld and Ubiquitous Computing, HUC’99. Springer, London, UK pp 311–313

  4. Boring S, Baur D, Butz A, Gustafson S, Baudisch P (2010) Touch projector: mobile interaction through video. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI’10. ACM, New York, NY, USA, pp 2287–2296

  5. Bragdon A, DeLine R, Hinckley K, Morris MR (2011) Code space: touch + air gesture hybrid interactions for supporting developer meetings. In: Proceedings of interactive tabletops and surfaces, ITS’11. ACM, pp 212–221

  6. Chang T-H, Li Y (2011) Deep shot: a framework for migrating tasks across devices using mobile phone cameras. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI’11. ACM, pp 2163–2172

  7. Dearman D, Pierce JS (2008) It’s on my other computer!: computing with multiple devices. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI’08. ACM New York, NY, USA, pp 767–776

  8. Fitzmaurice GW, Ishii H, Buxton WA (1995) Bricks: laying the foundations for graspable user interfaces. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI’95. ACM Press/Addison-Wesley Publishing Co., pp 442–449

  9. Hardy R, Rukzio E (2008) Touch and interact: touch-based interaction of mobile phones with displays. In: Proceedings of Mobile HCI’08, pp 245–254

  10. Hinckley K (2003) Synchronous gestures for multiple persons and computers. In: Proceedings of the ACM symposium on user interface software and technology, UIST’03. ACM, pp 149–158

  11. Hinckley K, Ramos G, Guimbretiere F, Baudisch P, Smith M (2004) Stitching: pen gestures that span multiple displays. In: Proceedings of advanced visual interfaces, AVI’04. ACM, pp 23–31

  12. Holman D, Vertegaal R, Altosaar M, Troje N, Johns D (2005) Paper windows: interaction techniques for digital paper. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI’05. ACM, New York, NY, USA, pp 591–599

  13. Kray C, Nesbitt D, Dawson J, Rohs M (2010) User-defined gestures for connecting mobile phones, public displays, and tabletops. In: Proceedings of mobile HCI’10. ACM, pp 239–248

  14. Kujala S (2003) User involvement: a review of the benefits and challenges. Behav Inf Technol 22(1):1–16

    Article  Google Scholar 

  15. Kujala S (2008) Effective user involvement in product development by improving the analysis of user needs. Behav Inf Technol 27(6):457–473

    Article  Google Scholar 

  16. Kurdyukova E, Redlin M, André E Studying user-defined ipad gestures for interaction in multi-display environment. In: Proceedings of intelligent user interfaces, IUI’12. ACM, pp 93–96

  17. Lakoff G, Johnson M (1999) Philosophy in the flesh: the embodied mind and interactive tabletops and surfaces challenge to western thought. Basic books , New York

  18. Lakoff G, Turner M (1989) More than cool reason: a field guide to poetic metaphor. University of Chicago Press, Chicago

    Book  Google Scholar 

  19. Lucero A, Holopainen J, Jokela T (2011) Pass-them-around: collaborative use of mobile phones for photo sharing. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI’11. ACM, pp 1787–1796

  20. Marquardt N, Hinckley K, Greenberg S (2012) Cross-device interaction via micro-mobility and f-formations. In: Proceedings of the ACM symposium on user interface software and technology, UIST’12. ACM, pp 13–22

  21. Miller RC, Myers BA et al (1999) Synchronizing clipboards of multiple computers. In: Proceedings of the ACM symposium on user interface software and technology, vol 7 of UIST’99. pp 65–66

  22. Mistry P, Nanayakkara S, Maes P Touch and copy, touch and paste. In: CHI extended Abstract’11. ACM, pp 1095–1098

  23. Morris MR, Wobbrock JO, Wilson AD (2010) Understanding users’ preferences for surface gestures. In: Proceedings of graphics interface, GI’10. Canadian Information Processing Society, pp 261–268

  24. Rekimoto J (1997) Pick-and-drop: a direct manipulation technique for multiple computer environments. In: Proceedings of the ACM symposium on user interface software and technology, UIST’97. ACM, pp 31–39

  25. Rekimoto J, Ayatsuka Y, Kohno M, Oba H (2003) Proximal interactions: a direct manipulation technique for wireless networking. In: Proceedings of INTERACT’03. IOS Press, pp 511–518

  26. Rekimoto J, Saitoh M (1999) Augmented surfaces: a spatially continuous work space for hybrid computing environments. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI’99. ACM, pp 378–385

  27. Scharf F, Wolters C, Cassens J, Herczeg M (2013) Cross-device interaction. In: AMBIENT 2013, the 3rd international conference on ambient computing, applications, services and technologies, pp 35–41

  28. Schmidt D, Chehimi F, Rukzio E, Gellersen H (2010) PhoneTouch: a technique for direct phone interaction on surfaces. In: Proceedings of the ACM symposium on user interface software and technology, UIST’10. ACM, pp 13–16

  29. Simeone AL, Seifert J, Schmidt D, Holleis P, Rukzio E, Gellersen H (2013) A cross-device drag-and-drop technique. In: Proceedings of the 12th international conference on mobile and ubiquitous multimedia, MUM’13. ACM, New York, NY, USA, pp 10:1–10:4

  30. Swindells C, Inkpen KM, Dill JC, Tory M (2002) That one there! pointing to establish device identity. In: Proceedings of the ACM symposium on user interface software and technology, UIST’02. ACM, pp 151–160

  31. Turner J, Bulling A, Alexander J, Gellersen H (2014) Cross-device gaze-supported point-to-point content transfer. In: Proceedings of eye tracking research and applications symposium, ETRA’14

  32. Wobbrock JO, Morris MR, Wilson AD (2009) User-defined gestures for surface computing. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI’09. ACM, pp 1083–1092

Download references

Acknowledgments

The first author was supported by a Marie Curie Intra European Fellowship within the 7th European Community Framework Programme.

Author information

Authors and Affiliations

  1. School of Computing and Communications, Lancaster University, Lancaster, LA1 4WA, UK

    Adalberto L. Simeone, Ming Ki Chong, Corina Sas & Hans Gellersen

  2. School of Creative Technologies, University of Portsmouth, Portsmouth, PO1 2DJ, UK

    Adalberto L. Simeone

Authors
  1. Adalberto L. Simeone
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ming Ki Chong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Corina Sas
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hans Gellersen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adalberto L. Simeone.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Simeone, A.L., Chong, M.K., Sas, C. et al. Select & Apply: understanding how users act upon objects across devices. Pers Ubiquit Comput 19, 881–896 (2015). https://doi.org/10.1007/s00779-015-0836-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-015-0836-1

Keywords

  • Mobile Device
  • Interaction Technique
  • Direct Manipulation
  • Public Display
  • Apply Task