KI - Künstliche Intelligenz

, Volume 30, Issue 1, pp 37–44 | Cite as

How Companion-Technology can Enhance a Multi-Screen Television Experience: A Test Bed for Adaptive Multimodal Interaction in Domestic Environments

  • Jan GugenheimerEmail author
  • Frank Honold
  • Dennis Wolf
  • Felix Schüssel
  • Julian Seifert
  • Michael Weber
  • Enrico Rukzio
Technical Contribution


This article deals with a novel multi-screen interactive TV setup (smarTVision) and its enhancement through Companion-Technology. Due to their flexibility and the variety of interaction options, such multi-screen scenarios are hardly intuitive for the user. While research known so far focuses on technology and features, the user itself is often not considered adequately. Companion-Technology has the potential of making such interfaces really user-friendly. Building upon smarTVision, it’s extension via concepts of Companion-Technology is envisioned. This combination represents a versatile test bed that not only can be used for evaluating usefulness of Companion-Technology in a TV scenario, but can also serve to evaluate Companion-Systems in general.


Multimodal Interaction Depth Camera Interaction Option Leap Motion Secondary Screen 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work was supported by the Transregional Collaborative Research Center SFB/TRR 62 “Companion-Technology for Cognitive Technical Systems”, which is funded by the German Research Foundation (DFG).


  1. 1.
    Basapur S, Harboe G, Mandalia H, Novak A, Vuong V, Metcalf C (2011) Field Trial of a Dual Device User Experience for iTV. In: Proceedings of the EuroITV ’11. doi: 10.1145/2000119.2000145
  2. 2.
    Basapur S, Mandalia H, Chaysinh S, Lee Y, Venkitaraman N, Metcalf C (20102) FANFEEDS: Evaluation of Socially Generated Information Feed on Second Screen As a TV Show Companion. In: Proceedings of the EuroiTV ’12. doi: 10.1145/2325616.2325636
  3. 3.
    Cauchard JR, Fraser M, Han T, Subramanian S (2012) Steerable projection: exploring alignment in interactive mobile displays. Pers Ubiquitous Comput. doi: 10.1007/s00779-011-0375-3
  4. 4.
    Cesar P, Bulterman DCA, Jansen AJ (2008) Usages of the secondary screen in an interactive television environment: control, enrich, share, and transfer television content. In: Changing television environments. LNCS. Springer, Heidelberg . DOI 10.1007/978-3-540-69478-6\_22Google Scholar
  5. 5.
    Courtois C, D’heer E (2012) Second screen applications and tablet users: constellation, awareness, experience, and interest. In: Proceedings of the EuroiTV ’12. doi: 10.1145/2325616.2325646
  6. 6.
    Courtois C, Schuurman D, De Marez L (2011) Triple screen viewing practices: diversification or compartmentalization? In: Proceedings of the EuroITV ’11. ACM, New York, pp 75–78. doi: 10.1145/2000119.2000132
  7. 7.
    Coutaz J (2007) Meta-user interfaces for ambient spaces. In: Coninx K, Luyten K, Schneider K (eds) Task models and diagrams for UI design. LNCS, vol 4385. Springer, Heidelberg, pp 1–15. doi: 10.1007/978-3-540-70816-2_1
  8. 8.
    Coutaz J, Nigay L, Salber D, Blandford A, May J, Young RM (1995) Four easy pieces for assessing the usability of multimodal interaction: the CARE properties. In: Proceedings of INTERACT95, pp 115–120Google Scholar
  9. 9.
    Dey AK, Abowd GD (1999) Towards a better understanding of context and context-awareness. In: HUC ’99: Proceedings of the 1st int. symposium on Handheld and Ubiquitous Computing. Springer, Heidelberg, pp 304–307Google Scholar
  10. 10.
    Glodek M, Honold F, Geier T, Krell G, Nothdurft F, Reuter S, Schüssel F, Hörnle T, Dietmayer K, Minker W, Biundo S, Weber M, Palm G, Schwenker F (2015) Fusion paradigms in cognitive technical systems for human-computer interaction. Neurocomputing 161:17–37. doi: 10.1016/j.neucom.2015.01.076 CrossRefGoogle Scholar
  11. 11.
    Gugenheimer J, Knierim P, Seifert J, Rukzio E (2014) Ubibeam: An interactive projector-camera system for domestic deployment. In: Proceedings of the ITS ’14. ACM, New York, pp 305–310. doi: 10.1145/2669485.2669537
  12. 12.
    Häkkilä JR, Posti M, Schneegass S, Alt F, Gultekin K, Schmidt A (2014) Let me catch this! Experiencing interactive 3D cinema through collecting content with a mobile phone. In: Proceedings of the CHI ’14). doi: 10.1145/2556288.2557187
  13. 13.
    Harboe G, Metcalf CJ, Bentley F, Tullio J, Massey N, Romano G (2008) Ambient social Tv: drawing people into a shared experience. In: Proceedings of the CHI ’08. doi: 10.1145/1357054.1357056
  14. 14.
    Hardy J, Alexander J (2012) Toolkit support for interactive projected displays. In: Proceedings of the MUM ’12. doi: 10.1145/2406367.2406419
  15. 15.
    Honold F, Bercher P, Richter F, Nothdurft F, Geier T, Barth R, Hörnle T, Schüssel F, Reuter S, Rau M, Bertrand G, Seegebarth B, Kurzok P, Schattenberg B, Minker W, Weber M, Biundo S (2014) Companion-technology: towards user- and situation-adaptive functionality of technical systems. In: IE’14: Proceedings of the 10th international conference on intelligent environments, pp 378–381. IEEE (2014). doi: 10.1109/IE.2014.60. Video:
  16. 16.
    Honold F, Schüssel F, Weber M (2012) Adaptive probabilistic fission for multimodal systems. In: Proceedings of the 24th Australian computer-human interaction conference, OzCHI ’12. ACM, New York, pp 222–231. doi: 10.1145/2414536.2414575
  17. 17.
    Honold F, Schüssel F, Weber M (2014) The automated interplay of multimodal fission and fusion in adaptive HCI. In: IE’14: Proceedings of the 10th international conf.erenceon intelligent environments, pp 170–177. doi: 10.1109/IE.2014.32
  18. 18.
    Jones B, Sodhi R, Murdock M, Mehra R, Benko H, Wilson A, Ofek E, MacIntyre B, Raghuvanshi N, Shapira L (2014) Roomalive: magical experiences enabled by scalable, adaptive projector-camera units. In: Proceedings of the UIST ’14. doi: 10.1145/2642918.2647383
  19. 19.
    Jones BR, Benko H, Ofek E, Wilson AD (2013) IllumiRoom: peripheral projected illusions for interactive experiences. In: ACM SIGGRAPH 2013 emerging technologies. doi: 10.1145/2503368.2503375
  20. 20.
    Joyent Inc (2014) Node js. Access:ed 29 May 2015
  21. 21.
    Kane SK, Avrahami D, Wobbrock JO, Harrison B, Rea AD, Philipose M, LaMarca A (2009) Bonfire: a nomadic system for hybrid laptop-tabletop interaction. In: Proceedings of the UIST ’09. doi: 10.1145/1622176.1622202
  22. 22.
    Lalanne D, Nigay L, Palanque P, Robinson P, Vanderdonckt J, Ladry JF (2009) Fusion engines for multimodal input: a survey. In: Proceedings of the 2009 international conference on multimodal interfaces, ICMI-MLMI ’09. ACM, New York, pp.153–160. doi: 10.1145/1647314.1647343
  23. 23.
    aViola JJ Jr, Buchanan S, Pittman C (2014) Multimodal input for perceptual user interfaces, chap. 9. Wiley, New York, pp 285–312. doi: 10.1002/9781118706237.ch9 Google Scholar
  24. 24.
    Miller LJ (1995) Family togetherness and the suburban ideal. Sociol Forum 10(3):393–418CrossRefGoogle Scholar
  25. 25.
    Myers BA (2001) Using handhelds and pcs together. Commun ACM. doi: 10.1145/384150.384159
  26. 26.
    Nigay L, Coutaz J (1995) A generic platform for addressing the multimodal challenge. In: CHI ’95: Proceedings of the SIGCHI conference on human factors in computing systems. ACM Press/Addison-Wesley Publishing Co., New York, pp 98–105. doi: 10.1145/223904.223917
  27. 27.
    Pinhanez C (2001) The everywhere displays projector: a device to create ubiquitous graphical interfaces. In: Ubicomp 2001: ubiquitous computing. LNCS. Springer, New YorkGoogle Scholar
  28. 28.
    Robertson S, Wharton C, Ashworth C, Franzke M (1996) Dual Device User Interface Design: PDAs and Interactive Television. In: Proceedings of the CHI ’96. doi: 10.1145/238386.238408
  29. 29.
    Roscher D, Blumendorf M, Albayrak S (2009) Using meta user interfaces to control multimodal interaction in smart environments. In: Meixner G, Görlich D, Breiner K, Hußmann H, Pleuß A, Sauer S, den Bergh JV (eds) Proceedings of the IUI’09 workshop on model driven development of advanced user interfaces, CEUR Workshop Proceedings ISSN 1613-0073, vol 439Google Scholar
  30. 30.
    Schmidt D, Ramakers R, Pedersen EW, Jasper J, Köhler S, Pohl A., Rantzsch H, Rau A, Schmidt P, Sterz C, Yurchenko Y, Baudisch P (2014) Kickables: tangibles for feet. In: Proceedings of the CHI ’14. doi: 10.1145/2556288.2557016
  31. 31.
    Schüssel F, Honold F, Weber M (2013) Using the transferable belief model for multimodal input fusion in companion systems. In: Schwenker F, Scherer S, Morency LP (eds) Multimodal pattern recognition of social signals in HCI. LNCS, vol 7742. Springer, New York, pp 100–115. doi: 10.1007/978-3-642-37081-6_12
  32. 32.
    Schüssel F, Honold F, Weber M, Schmidt M, Bubalo N, Huckauf A (2014) Multimodal interaction history and its use in error detection and recovery. In: Proceedings of the 16th ACM international conference on multimodal interaction, ICMI ’14. ACM, New York, pp 164–171. doi: 10.1145/2663204.2663255
  33. 33.
    Seifert J (2015) Mobile mediated interaction with pervasive displays. Ph.D. thesis, Ulm University,UlmGoogle Scholar
  34. 34.
    Sugimoto M, Hosoi K, Hashizume H (2004) Caretta: a system for supporting face-to-face collaboration by integrating personal and shared spaces. In: Proceedings of the CHI ’04. doi 10.1145/985692.985698
  35. 35.
    Vatavu RD (2013) Designing Gestural Interfaces for the Interactive TV. In: Proceedings of the EuroITV ’13. doi: 10.1145/2465958.2465981
  36. 36.
    Wendemuth A, Biundo S (2012) A companion technology for cognitive technical systems. In: Esposito A, Esposito AM, Vinciarelli A, Hoffmann R, Müller VC (eds) Cognitive behavioural systems. LNCS, vol 7403. Springer, New York, pp 89–103. doi: 10.1007/978-3-642-34584-5_7
  37. 37.
    Winkler C, Löchtefeld M, Dobbelstein D, Krüger A, Rukzio E (2014) SurfacePhone: a mobile projection device for single- and multiuser everywhere tabletop interaction. In: Proceedings of the CHI ’14. doi: 10.1145/2556288.2557075

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • Jan Gugenheimer
    • 1
    Email author
  • Frank Honold
    • 1
  • Dennis Wolf
    • 1
  • Felix Schüssel
    • 1
  • Julian Seifert
    • 1
  • Michael Weber
    • 1
  • Enrico Rukzio
    • 1
  1. 1.Institute of Media InformaticsUlm UniversityUlmGermany

Personalised recommendations