PointerPhone: Using Mobile Phones for Direct Pointing Interactions with Remote Displays

  • Julian Seifert
  • Andreas Bayer
  • Enrico Rukzio
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8119)


Large screens or projections in public and private settings have become part of our daily lives, as they enable the collaboration and presentation of information in many diverse ways. When discussing the shown information with other persons, we often point to a displayed object with our index finger or a laser pointer in order to talk about it. Although mobile phone-based interactions with remote screens have been investigated intensively in the last decade, none of them considered such direct pointing interactions for application in everyday tasks. In this paper, we present the concept and design space of PointerPhone which enables users to directly point at objects on a remote screen with their mobile phone and interact with them in a natural and seamless way. We detail the design space and distinguish three categories of interactions including low-level interactions using the mobile phone as a precise and fast pointing device, as well as an input and output device. We detail the category of widget-level interactions. Further, we demonstrate versatile high-level interaction techniques and show their application in a collaborative presentation scenario. Based on the results of a qualitative study, we provide design implications for application designs.


Mobile phone pointing interaction collaboration 


  1. 1.
    Alt, F., Kubitza, T., Bial, D., Zaidan, F., Ortel, M., Zurmaar, B., Lewen, T., Shirazi, A.S., Schmidt, A.: Digifieds: insights into deploying digital public notice areas in the wild. In: MUM 2011, pp. 165–174. ACM, New York (2011)Google Scholar
  2. 2.
    Ballagas, R., Borchers, J., Rohs, M., Sheridan, J.: The smart phone: a ubiquitous input device. IEEE Pervasive Computing 5(1), 70–77 (2006)CrossRefGoogle Scholar
  3. 3.
    Baur, D., Boring, S., Feiner, S.: Virtual projection: Exploring optical projection as a metaphor for multi-device interaction. In: CHI 2012. ACM, Austin (2012)Google Scholar
  4. 4.
    Boring, S., Altendorfer, M., Broll, G., Hilliges, O., Butz, A.: Shoot & Copy: Phonecam-based information transfer from public displays onto mobile phones. In: Mobility 2007, pp. 24–31. ACM, New York (2007)Google Scholar
  5. 5.
    Boring, S., Baur, D., Butz, A., Gustafson, S., Baudisch, P.: Touch projector: mobile interaction through video. In: CHI 2010, pp. 2287–2296. ACM, New York (2010)Google Scholar
  6. 6.
    Boring, S., Jurmu, M., Butz, A.: Scroll, tilt or move it: using mobile phones to continuously control pointers on large public displays. In: OZCHI 2009, pp. 161–168. ACM, New York (2009)Google Scholar
  7. 7.
    Cesar, P., Bulterman, D.C., Jansen, A.J.: Usages of the secondary screen in an interactive television environment: Control, enrich, share, and transfer television content. In: Tscheligi, M., Obrist, M., Lugmayr, A. (eds.) EuroITV 2008. LNCS, vol. 5066, pp. 168–177. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  8. 8.
    Chehimi, F., Rukzio, E.: Throw your photos: an intuitive approach for sharing between mobile phones and interactive tables. In: Ubicomp 2010 Adjunct, pp. 443–444. ACM, New York (2010)Google Scholar
  9. 9.
    Cruickshank, L., Tsekleves, E., Whitham, R., Hill, A., Kondo, K.: Making inter- active tv easier to use: Interface design for a second screen approach. The Design Journal 10(3), 41–53 (2007)CrossRefGoogle Scholar
  10. 10.
    Dachselt, R., Buchholz, R.: Natural throw and tilt interaction between mobile phones and distant displays. In: CHI EA 2009, pp. 3253–3258. ACM, New York (2009)Google Scholar
  11. 11.
    Google Inc. Android developers (2013), (last access January 15, 2013)
  12. 12.
    Google Inc. LG TV remote 2011 - Android Apps on Google Play (2013), (last access January 15, 2013)
  13. 13.
    Jeon, S., Hwang, J., Kim, G.J., Billinghurst, M.: Interaction with large ubiquitous displays using camera-equipped mobile phones. Personal Ubiquitous Comput. 14(2), 83–94 (2010)CrossRefGoogle Scholar
  14. 14.
    Microsoft. Welcome to Microsoft Pixelsense (2013), (last access January 15, 2013)
  15. 15.
    Morris, M.R.: Web on the wall: insights from a multimodal interaction elicitation study. In: ITS 2012, pp. 95–104. ACM, New York (2012)Google Scholar
  16. 16.
    Myers, B.A., Bhatnagar, R., Nichols, J., Peck, C.H., Kong, D., Miller, R., Long, A.C.: Interacting at a distance: measuring the performance of laser pointers and other devices. In: CHI 2002, pp. 33–40. ACM, New York (2002)Google Scholar
  17. 17.
    Myers, B.A., Stiel, H., Gargiulo, R.: Collaboration using multiple PDAs connected to a pc. In: CSCW 1998, pp. 285–294. ACM, New York (1998)Google Scholar
  18. 18.
    Nintendo. Wii u official site at Nintendo (2012), (last access December 18, 2012)
  19. 19.
    Olsen Jr., D.R., Nielsen, T.: Laser pointer interaction. In: CHI 2001, pp. 17–22. ACM, New York (2001)Google Scholar
  20. 20.
    Pan, G., Ren, H., Hua, W., Zheng, Q., Li, S.: Easypointer: what you pointing at is what you get. In: CHI EA 2011, pp. 499–502. ACM, New York (2011)Google Scholar
  21. 21.
    Pawar, U.S., Pal, J., Gupta, R., Toyama, K.: Multiple mice for retention tasks in disadvantaged schools. In: CHI 2007, pp. 1581–1590. ACM, New York (2007)Google Scholar
  22. 22.
    Rashid, U., Kauko, J., Häkkilä, J., Quigley, A.: Proximal and distal selection of widgets: designing distributed ui for mobile interaction with large display. In: MobileHCI 2011, pp. 495–498. ACM, New York (2011)Google Scholar
  23. 23.
    Rekimoto, J., Saitoh, M.: Augmented surfaces: a spatially continuous work space for hybrid computing environments. In: CHI 1999, pp. 378–385. ACM, New York (1999)Google Scholar
  24. 24.
    Rukzio, E., Leichtenstern, K., Callaghan, V., Holleis, P., Schmidt, A., Chin, J.: An experimental comparison of physical mobile interaction techniques: touching, pointing and scanning. In: Dourish, P., Friday, A. (eds.) UbiComp 2006. LNCS, vol. 4206, pp. 87–104. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  25. 25.
    Schmidt, D., Molyneaux, D., Cao, X.: Picontrol: using a handheld projector for direct control of physical devices through visible light. In: UIST 2012, pp. 379–388. ACM, New York (2012)Google Scholar
  26. 26.
    Schmidt, D., Seifert, J., Rukzio, E., Gellersen, H.: A cross-device interaction style for mobiles and surfaces. In: DIS 2012, pp. 318–327. ACM, New York (2012)Google Scholar
  27. 27.
    Shizuki, B., Hisamatsu, T., Takahashi, S., Tanaka, J.: Laser pointer interaction techniques using peripheral areas of screens. In: Proceedings of the Working Conference on Advanced Visual Interfaces, AVI 2006, pp. 95–98. ACM, New York (2006)CrossRefGoogle Scholar
  28. 28.
    Sugimoto, M., Hosoi, K., Hashizume, H.: Caretta: a system for supporting face- to-face collaboration by integrating personal and shared spaces. In: CHI 2004, pp. 41–48. ACM, New York (2004)Google Scholar
  29. 29.
    Tangram. Tangram - Smart Dot Pro (2013), (last access January 15, 2013)
  30. 30.
    Tse, E., Histon, J., Scott, S.D., Greenberg, S.: Avoiding interference: how people use spatial separation and partitioning in SDG workspaces. In: CSCW 2004, pp. 252–261. ACM, New York (2004)Google Scholar
  31. 31.
    Vogt, F., Wong, J., Fels, S.S., Cavens, D.: Tracking multiple laser pointers for large screen interaction. In: Extended Abstracts of ACM UIST 2003, pp. 95–96 (2003)Google Scholar
  32. 32.
    Vogt, F., Wong, J., Po, B.A., Argue, R., Fels, S.S., Booth, K.S.: Exploring collaboration with group pointer interaction. In: CGI 2004, pp. 636–639 (June 2004)Google Scholar
  33. 33.
    Wilson, A.D., Shafer, S.: XWand: UI for intelligent spaces. In: CHI 2003, pp. 545–552 (2003)Google Scholar
  34. 34.
    Zizka, J., Olwal, A., Raskar, R.: Specklesense: fast, precise, low-costandcompact motion sensing using laser speckle. In: UIST 2011, pp. 489–498. ACM, New York (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Julian Seifert
    • 1
  • Andreas Bayer
    • 2
  • Enrico Rukzio
    • 1
  1. 1.Institute of Media InformaticsUlm UniversityUlmGermany
  2. 2.University of Duisburg-EssenEssenGermany

Personalised recommendations