Learning Instead of Markers: Flexible Recognition of Mobile Devices on Interactive Surfaces

  • Philipp MockEmail author
  • Jörg Edelmann
  • Wolfgang Rosenstiel
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9189)


We propose an approach for recognition of mobile devices on interactive surfaces that do not support optical markers. Our system only requires an axis aligned bounding box of the object placed on the touchscreen in combination with position data from the mobile devices integrated inertia measurement unit (IMU). We put special emphasis on maximum flexibility in terms of compatibility with varying multi-touch sensor techniques and different kinds of mobile devices. A new device can be added to the system with a short training phase, during which the device is moved across the interactive surface. A device model is automatically created from the recorded data using support vector machines. Different devices of the same size are identified by analyzing their IMU data streams for transitions into a horizontally resting state. The system has been tested in a museum environment.


Human computer interaction Tabletop interaction Markerless object tracking Machine learning 



Philipp Mock was a doctoral student of the LEAD Graduate School [GSC1028], funded by the Excellence Initiative of the German federal and state governments.


  1. 1.
    Albarelli, A., Bergamasco, F., Celentano, A., Cosmo, L., Torsello, A.: Using multiple sensors for reliable markerless identification through supervised learning. Mach. Vision Appl. 24(7), 1539–1554 (2013)CrossRefGoogle Scholar
  2. 2.
    Hesselmann, T., Henze, N., Boll, S.: Flashlight: Optical communication between mobile phones and interactive tabletops. In: Proceedings of the ITS 2010, pp. 135–138. ACM (2010)Google Scholar
  3. 3.
    Hodges, S., Izadi, S., Butler, A., Rrustemi, A., Buxton, B.: Thinsight: Versatile multi-touch sensing for thin form-factor displays. In: Proceedings of the UIST 2007, pp. 259–268. ACM (2007)Google Scholar
  4. 4.
    Jovanovic, N., Korst, J., Pronk, V.: Object detection in flatland. In: Proceedings of the ADVCOMP 2009, pp. 95–100. IEEE Computer Society (2009)Google Scholar
  5. 5.
    Kaltenbrunner, M., Bencina, R.: Reactivision: a computer-vision framework for table-based tangible interaction. In: Proceedings of the TEI 2007, pp. 69–74. ACM (2007)Google Scholar
  6. 6.
    Kaltenbrunner, M., Bovermann, T., Bencina, R., Costanza, E.: Tuio: a protocol for table-top tangible user interfaces. In: Proceedings of the 2nd Interactive Sonification Workshop (2005)Google Scholar
  7. 7.
    Marquardt, N., Kiemer, J., Greenberg, S.: What caused that touch?: Expressive interaction with a surface through fiduciary-tagged gloves. In: Proceedings of the ITS 2010, pp. 139–142. ACM (2010)Google Scholar
  8. 8.
    Miyaoku, K., Higashino, S., Tonomura, Y.: C-blink: a hue-difference-based light signal marker for large screen interaction via any mobile terminal. In: Proceedings of the UIST 2004, pp. 147–156. ACM (2004)Google Scholar
  9. 9.
    Moeller, J., Kerne, A.: Zerotouch: an optical multi-touch and free-air interaction architecture. In: Proceedings of the CHI 2012, pp. 2165–2174. ACM (2012)Google Scholar
  10. 10.
    MultiTouch Ltd., Multitaction codice.
  11. 11.
    PQ Labs. G4 multi-touch screen.
  12. 12.
    Schmidt, D., Chehimi, F., Rukzio, E., Gellersen, H.: Phonetouch: a technique for direct phone interaction on surfaces. In: Proceedings of the UIST 2010, pp. 13–16. ACM (2010)Google Scholar
  13. 13.
    Schöning, J., Rohs, M., Krüger, A.: Using mobile phones to spontaneously authenticate and interact with multi-touch surfaces. In: Proceedings of the Workshop on Designing Multitouch Interaction Techniques for Coupled Public and Private Displays (2008)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Philipp Mock
    • 1
    Email author
  • Jörg Edelmann
    • 2
  • Wolfgang Rosenstiel
    • 1
  1. 1.University of TübingenTübingenGermany
  2. 2.Knowledge Media Research CenterTübingenGermany

Personalised recommendations