Development of Multiple Device Collaboration System Using Built-in Camera Image

  • Kazuki TadaEmail author
  • Jiro Tanaka
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9732)


In this paper, we introduce a multi-device collaboration system using the image obtained from the built-in camera. Users can use applications that utilize multiple devices in cooperation without needing special devices such as a touch panel sensor. Our system enables accurate position tracking of a smartphone any screen by obtaining an image from the device’s front camera to recognize the device and measure the device’s position by template matching [1]. In many multi-device collaboration approaches, it was necessary to overlap the screens. However, our proposed method is capable of multi-device collaboration without overlapping the screens of the devices, and it can be applied using a wide range of off-screens. We implemented some of the applications using this technique. In addition, we showed the usefulness of this approach by evaluation experiments.


Multiple device Image processing Image recognition Smartphone Template matching 


  1. 1.
    Hiroki, K., Sugimoto, M., Yatani, K., Tamura, K., Hashizume, H.: Toss-it: intuitive information transfer techniques for mobile devices. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2005, pp. 1881–1884 (2005)Google Scholar
  2. 2.
    Kamo, H., Tanaka, J.: Interlocked surfaces: a dynamic multi-device collaboration system. In: The 15th International Conference on Human-Computer Interaction, HCII 2013, pp. 317–325 (2013)Google Scholar
  3. 3.
    Elstner, S., Hahne, U., Schild, J., Alexa, M.: Multi-touch focus+context sketch-based interaction. In: Proceedings of the 6th Eurographics Symposiumn Sketch-Based Interfaces and Modeling, SMIB 2009, pp. 77–83 (2009)Google Scholar
  4. 4.
    Dill, J.C., Swindells, C., Inkpen, K.M., Tory, M.: That one there! pointing to establish device identity. In: Proceedings of the 15th Annual ACM symposium on User Interface Software and Technology, UIST 2002, pp. 151–160 (2002)Google Scholar
  5. 5.
    Olwal, A., Lightsense: enabling spatially aware handheld interaction devices. In: Proceedings of the 5th IEEE and ACM International Symposium on Mixed and Augmented Reality, ISMAR 2006, pp. 119–122 (2006)Google Scholar
  6. 6.
    Sun, S.W., Cheng, W.H., Liu, K.W., Lin, I.P., Hsu, X.S.C.: G-spacing: a gyro sensor based relative 3d space positioning scheme. In: Proceeding of ACM SIGGRAPH 2015 Posters, SIGGRAPH 2015, Article No. 35 (2015)Google Scholar
  7. 7.
    Momeni, A., Xu, D., Brockmeyer, E.: Magpad: a near surface augmented reading system for physical paper and smartphone coupling. In: Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology, UIST 2015, pp. 103–104 (2015)Google Scholar
  8. 8.
    Strohmeier, P.: Displaypointers- seamless cross-device interactions. In: Proceedings of the 12th International Conference on Advances in Computer Entertainment Technology, ACE 2015, pp. 86–93 (2015)Google Scholar
  9. 9.
    Yasumoto, M., Teraoka, T.: Vistouch: dynamic three-dimensional connection between multiple mobile devices. In: Proceedings of the 6th Augmented Human International Conference, AH 2015, pp. 89–92 (2015)Google Scholar
  10. 10.
    Roudaut, A., Chan, L., Muller, S., Baudisch, P.: Capstones and zebrawidgets: sensing stacks of building blocks, dials and sliders on capacitive touch screens. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2012, pp. 2189–2192 (2012)Google Scholar
  11. 11.
    Chan, L.W., Wu, H.T., Kao, H.S., Ko, J.C., Lin, H.R., Chen, M.Y., Hsu, J., Hung, Y.P.: Enabling beyond-surface interactions for interactive surface with an invisible projection. In: Proceedings of the 23th Annual ACM Symposium on User Interface Software and Technology, UIST 2010, pp. 263–272 (2010)Google Scholar
  12. 12.
    Vanaken, C., Reeth, F.V., Cuypers, T., Francken, Y., Bekaert, P.: Smartphone localization on interactive surfaces using the built-in camera. In: Proceedings of the IEEE International Workshop on Project-Camera Systems, Procam 2009, pp. 61–68 (2009)Google Scholar
  13. 13.
    Raubal, M., Essl, G., Rohs, M., Schoning, H., Kruger, A.: Map navigation with mobile devices: virtual versus physical movement with and without visual context. In: Proceedings of the 9th International Conference on Multimodal Interfaces, ICMI 2007, pp. 146–153 (2007)Google Scholar
  14. 14.
    Heibeck, E., Maes, P., Leigh, S., Schoessler, P., Ishii, H.: Tangible interaction with see-through augmentation for smartphones on computer screens. In: Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST 2014, pp. 55–56 (2014)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.University of TsukubaTsukubaJapan

Personalised recommendations