Multimedia Tools and Applications

, Volume 50, Issue 3, pp 587–607 | Cite as

Taking advantage of contextualized interactions while users watch TV

  • Cesar A. C. TeixeiraEmail author
  • Erick Lazaro Melo
  • Renan G. Cattelan
  • Maria da Graça C. Pimentel


While watching TV, viewers use the remote control to turn the TV set on and off, change channel and volume, to adjust the image and audio settings, etc. Worldwide, research institutes collect information about audience measurement, which can also be used to provide personalization and recommendation services, among others. The interactive digital TV offers viewers the opportunity to interact with interactive applications associated with the broadcast program. Interactive TV infrastructure supports the capture of the user–TV interaction at fine-grained levels. In this paper we propose the capture of all the user interaction with a TV remote control—including short term and instant interactions: we argue that the corresponding captured information can be used to create content pervasively and automatically, and that this content can be used by a wide variety of services, such as audience measurement, personalization and recommendation services. The capture of fine grained data about instant and interval-based interactions also allows the underlying infrastructure to offer services at the same scale, such as annotation services and adaptative applications. We present the main modules of an infrastructure for TV-based services, along with a detailed example of a document used to record the user–remote control interaction. Our approach is evaluated by means of a proof-of-concept prototype which uses the Brazilian Digital TV System, the Ginga-NCL middleware.


Digital Interactive TV Capture and access multimedia applications User-media interaction 



We thank the following organizations for their support: FINEP, FAPESP, CAPES and CNPq. We thank Dick Bulterman for great discussions on this topic. We thank Luis F.G. Soares for inspiring us with the SBTVD. Erick Melo is a MSc candidate supported by CAPES. During the research reported in this paper, Renan Cattelan was a PhD candidate supported by FAPESP.


  1. 1.
    Abowd GD, Mynatt ED, Rodden T (2002) The human experience. IEEE Pervasive Comput 1(1):48–57CrossRefGoogle Scholar
  2. 2.
    Adar E, Weld DS, Bershad BN, Gribble SS (2007) Why we search: visualizing and predicting user behavior. In: WWW ’07: proceedings of the 16th international conference on world wide web. ACM, New York, pp 161–170CrossRefGoogle Scholar
  3. 3.
    Álvarez F, Martin C, Alliez D, Roc P, Steckel P, Menendez J, Cisneros G, Jones S (2009) Audience measurement modeling for convergent broadcasting and IPTV networks. IEEE Trans Broadcast 55(2, Part 2):502–515CrossRefGoogle Scholar
  4. 4.
    Baluja S, Seth R, Sivakumar D, Jing Y, Yagnik J, Kumar S, Ravichandran D, Aly M (2008) Video suggestion and discovery for youtube: taking random walks through the view graph. In: WWW ’08: proceeding of the 17th international conference on world wide web. ACM, New York, pp 895–904CrossRefGoogle Scholar
  5. 5.
    BARBE (2009) Broadcaster’s audience research board ltd: television measurement service.
  6. 6.
    Blanco-Fernández Y, Arias JJP, Gil-Solla A, Cabrer MR, Nores ML, Duque JG, Vilas AF, Redondo RPD, Muñoz JB (2008) An MHP framework to provide intelligent personalized recommendations about digital TV contents. Softw Pract Exp 38(9):925–960CrossRefGoogle Scholar
  7. 7.
    Bulterman DCA (2003) Using SMIL to encode interactive, peer-level multimedia annotations. In: ACM symposium on document engineering (DocEng’03). ACM, New York, pp 32–41Google Scholar
  8. 8.
    Cattelan RG, Santos FS, Goularte R, Teixeira CAC, Pimentel MdGC (2008) Watch-and-comment as a paradigm toward ubiquitous interactive video editing. ACM Trans Multimed Comput Commun Appl 4(4):28Google Scholar
  9. 9.
    César P, Bulterman DCA, Obrenovic Z, Ducret J, Cruz-Lara S (2007) An architecture for non-intrusive user interfaces for interactive digital television. In: EuroITV’07—European interactive TV conference, pp 11–20Google Scholar
  10. 10.
    César P, Bulterman DCA, Jansen AJ (2008) Usages of the secondary screen in an interactive television environment: control, enrich, share, and transfer television content. In: European interactive TV conference. Lecture notes in computer science, vol 5066, pp 168–177Google Scholar
  11. 11.
    Cesar P, Bulterman DC, Jansen J (2009) Leveraging user impact: an architecture for secondary screens usage in interactive television. Multimed Syst J 15(3):127–142CrossRefGoogle Scholar
  12. 12.
    Chen W-Y, Chu J-C, Luan J, Bai H, Wang Y, Chang EY (2009) Collaborative filtering for orkut communities: discovery of user latent behavior. In: WWW ’09: proceedings of the 18th international conference on world wide web. ACM, New York, pp 681–690CrossRefGoogle Scholar
  13. 13.
    Das AS, Datar M, Garg A, Rajaram S (2007) Google news personalization: scalable online collaborative filtering. In: WWW ’07: proceedings of the 16th international conference on world wide web. ACM, New York, pp 271–280CrossRefGoogle Scholar
  14. 14.
    De Pessemier T, Deryckere T, Martens L (2009) Context aware recommendations for user-generated content on a social network site. In: EuroITV ’09: proceedings of the seventh European conference on European interactive television. ACM, New York, pp 133–136CrossRefGoogle Scholar
  15. 15.
    Digitaltvnews (2008) Orca Interactive and ruwido introduce finger print recognition to personalized TV.
  16. 16.
    Engebretson J (2006) Fingerprint-based remote control for IPTV debuts.
  17. 17.
    Fagá Jr R, Furtado B, Maximino R, Cattelan R, Pimentel MdGC (2009) Context information exchange and sharing in a peer-to-peer community: a video annotation scenario. In: The 27th ACM international conference on design of communication. ACM, New York, pp 1–8Google Scholar
  18. 18.
    Ferscha A, Vogl S, Emsenhuber B, Wally B (2007) Physical shortcuts for media remote controls. In: Proceedings international conference on intelligent technologies for interactive entertainment. ICST, Brussels, pp 1–8Google Scholar
  19. 19.
    Fuxman A, Tsaparas P, Achan K, Agrawal R (2008) Using the wisdom of the crowds for keyword generation. In: WWW ’08: proceeding of the 17th international conference on world wide web. ACM, New York, pp 61–70CrossRefGoogle Scholar
  20. 20.
    Hara Y, Tomomune Y, Shigemori M (2004) Categorization of Japanese TV viewers based on program genres they watch. User Model User-Adapt Interact J 14(1):87–117CrossRefGoogle Scholar
  21. 21.
    Harada T, Okubo WT (2001) Fingerprint activated remote input device for personal id recognition and access authentication. World intellectual property organization, patent (WO/2001/091057)Google Scholar
  22. 22.
    Hauglid JO, Heggland J (2008) Savanta-search, analysis, visualisation and navigation of temporal annotations. Multimed Tools Appl J 40(2):183–210CrossRefGoogle Scholar
  23. 23.
    Hofmann C, Hollender N, Fellner DW (2009) Workflow-based architecture for collaborative video annotation. In: Ozok AA, Zaphiris P (eds) HCI (12). Lecture notes in computer science, vol 5621. Springer, New York, pp 33–42Google Scholar
  24. 24.
    Hsu SH, Wen M-H, Lin H-C, Lee C-C, Lee C-H (2007) AIMED—a personalized TV recommendation system. In: Proceedings of the EuroITV’2007 conference, pp 166–174Google Scholar
  25. 25.
    Hwang M-C, Ha LT, Kim N-H, Park C-S, Ko S-J (2007) Person identification system for future digital TV with intelligence. IEEE Trans Consum Electron 53(1):218–226CrossRefGoogle Scholar
  26. 26.
  27. 27.
    IBOPE (2007) Methodology: TV (in Portuguese).
  28. 28.
    IEEE (2004) Air interface for fixed broadband wireless access systems. In: IEEE standard for local and metropolitan area networks—part 16Google Scholar
  29. 29.
    ITEG (2009) International television expert group: international comparison of telemetric TV research, peoplemeter, institutes & boards.
  30. 30.
    Jansen J, Bulterman D (2009) SMIL state: an architecture and implementation for adaptive time-based web applications. Multimed Tools Appl J 43(3):203–224CrossRefGoogle Scholar
  31. 31.
    Kientz JA, Hayes GR, Westeyn T, Starner T, Abowd GD (2007) Pervasive computing and autism: assisting caregivers of children with special needs. IEEE Pervasive Comput 6(1):28–35CrossRefGoogle Scholar
  32. 32.
    Kim S-H, Ok J, Kang HJ, Kim M-C, Kim M (2004) An interaction and product design of gesture based TV remote control. In: Companion CHI’04: extended abstracts on human factors in computing systems, pp 1548–1548Google Scholar
  33. 33.
    Kim S-K, Zhang D, Kim D-H, Park S-H, Ko S-J (2009) Real-time viewer detection algorithm for smart epg in the intelligent DTV. In: International conference on computers in education, pp 1–2Google Scholar
  34. 34.
    Macedo AA, Baldochi L, Guerrero JAC, Cattelan RG, Pimentel MdGC (2008) Automatically linking live experiences captured with a ubiquitous infrastructure. Multimed Tools Appl 37(2):93–115CrossRefGoogle Scholar
  35. 35.
    Mankovitz RJ (1996) Eletronic program guide: United States patent 5541738.
  36. 36.
    Masthoff J (2004) Group modeling: selecting a sequence of television items to suit a group of viewers. User Model User-Adapt Interact J 14(1):37–85CrossRefGoogle Scholar
  37. 37.
    Maybury MT, Greiff WR, Boykin S, Ponte JM, McHenry C, Ferro L (2004) Personalcasting: tailored broadcast news. User Model User-Adapt Interact J 14(1):119–144CrossRefGoogle Scholar
  38. 38.
  39. 39.
    Nores ML, Blanco-Fernández Y, Arias JJP, Duque JG, Cabrer MR, Gil-Solla A, Redondo RPD, Vilas AF (2009) Receiver-side semantic reasoning for digital TV personalization in the absence of return channels. Multimed Tools Appl 41(3):407–436CrossRefGoogle Scholar
  40. 40.
    O’Sullivan D, Smyth B, Wilson DC, McDonald K, Smeaton AF (2004) Improving the quality of the personalized electronic program guide. User Model User-Adapt Interact J 14(1):5–36CrossRefGoogle Scholar
  41. 41.
    Patel M, Gossweiler R, Sahami M, Blackburn J, Brown D, Knight A (2008) Google TV search: dual-wielding search and discovery in a large-scale product. In: UXTV ’08: proceeding of the 1st international conference on designing interactive user experiences for TV and video. ACM, New York, pp 95–104CrossRefGoogle Scholar
  42. 42.
    Philomin V, Trajkovic M, Gutta S (2001) Remote control with the fingerprint recognition capability. World intellectual property organization, patent (WO/2003/050770)Google Scholar
  43. 43.
    Pimentel MdGC, Baldochi L, Cattelan RG (2007) Prototyping applications to document human experiences. IEEE Pervasive Comput 6(2):93–100CrossRefGoogle Scholar
  44. 44.
    Pimentel MdGC, Cattelan RG, Freitas G, Melo EL, Teixeira CAC (2010) Watch-and-comment as an approach to collaborative annotate points of interest in video and interactive-TV programs. In: Marcus A, Roibás AC, Sala R (eds) Mobile TV: customizing content and experience. Springer, New York, pp 349–366. CrossRefGoogle Scholar
  45. 45.
    Pimentel MdGC, Cattelan RG, Melo EL, Teixeira CAC (2008) Pervasive user-generated content? Watch TV. IEEE Pervasive Comput 6(2):60–61Google Scholar
  46. 46.
    Richardson M (2008) Learning about the world through long-term query logs. ACM Trans Web 2(4):1–27CrossRefGoogle Scholar
  47. 47.
    SBTVD (2006) Brazilian digital TV system reference model.
  48. 48.
    Schmitz P, Shafton PL, Shaw R, Tripodi S, Williams B, Yang J (2006) International remix: video editing for the web. In: Nahrstedt K, Turk M, Rui Y, Klas W, Mayer-Patel K (eds) ACM multimedia. ACM, New York, pp 797–798Google Scholar
  49. 49.
    Shamma DA, Bastéa-Forte M, Joubert N, Liu Y (2008) Enhancing online personal connections through the synchronized sharing of online video. In: Czerwinski M, Lund AM, Tan DS (eds) CHI extended abstracts. ACM, New York, pp 2931–2936Google Scholar
  50. 50.
    Soares LFG, Rodrigues RF, Moreno MF (2007) Ginga-NCL: the declarative environment of the Brazilian digital TV system. J Braz Chem Soc 12(4):37–46Google Scholar
  51. 51.
    Spertus E, Sahami M, Buyukkokten O (2005) Evaluating similarity measures: a large-scale study in the orkut social network. In: KDD ’05: proceedings of the 11th ACM SIGKDD international conference on knowledge discovery in data mining. ACM, New York, pp 678–684Google Scholar
  52. 52.
    Teixeira CAC, Freitas G, Pimentel MdGC (2009) Distributed discrimination of moments and segments as an exploration of the watch-and-comment paradigm. In: SAC’10: proceedings of the 2010 ACM symposium on applied computing. ACM, New York (to appear in March 2010)Google Scholar
  53. 53.
    Teixeira CAC, Melo EL, Cattelan RG, Pimentel MdGC (2009) User–media interaction with interactive TV. In: SAC ’09: proceedings of the 2009 ACM symposium on applied computing. ACM, New York, pp 1829–1833CrossRefGoogle Scholar
  54. 54.
    Telekom-Austria (2009) Remote control via fingerprint.
  55. 55.
    Truong KN, Abowd GD, Brotherton JA (2001) Who, what, when, where, how: design issues of capture & access applications. In: Abowd GD, Brumitt B, Shafer SA (eds) Ubicomp. Lecture notes in computer science, vol 2201. Springer, New York, pp 209–224Google Scholar
  56. 56.
    Vildjiounaite E, Kyllönen V, Hannula T, Alahuhta P (2008) Unobtrusive dynamic modelling of TV program preferences in a household. In: Proceedings of the EuroITV’2008 conference, pp 82–91Google Scholar
  57. 57.
    Vildjiounaite E, Kyllönen V, Hannula T, Alahuhta P (2009) Unobtrusive dynamic modelling of TV programme preferences in a finnish household. Multimed Syst J 15(3):143–157CrossRefGoogle Scholar
  58. 58.
    Wittenburg K, Lanning T, Schwenke D, Shubin H, Vetro A (2006) The prospects for unrestricted speech input for TV content search. In: AVI’06: proceedings of the working conference on advanced visual interfaces. ACM, New York, pp 352–359CrossRefGoogle Scholar
  59. 59.
    Yu Z, Zhou X, Hao Y, Gu J (2006) TV program recommendation for multiple viewers based on user profile merging. User Model User-Adapt Interact J 16(1):63–82CrossRefGoogle Scholar
  60. 60.
    Zhang H, Zheng S, Yuan J (2005) A personalized TV guide system compliant with MHP. IEEE Trans Consum Electron 51(2):731–737CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Cesar A. C. Teixeira
    • 1
    Email author
  • Erick Lazaro Melo
    • 1
  • Renan G. Cattelan
    • 2
  • Maria da Graça C. Pimentel
    • 3
  1. 1.Departamento de ComputaçãoUniversidade Federal de São CarlosSao CarlosBrazil
  2. 2.Faculdade de ComputaçãoUniversidade Federal de UberlândiaUberlandiaBrazil
  3. 3.Instituto de Ciências Matemáticas e de ComputaçãoUniversidade de São PauloSão CarlosBrazil

Personalised recommendations