Mining TV Broadcasts 24/7 for Recurring Video Sequences

  • Ina Döhring
  • Rainer Lienhart
Part of the Studies in Computational Intelligence book series (SCI, volume 287)


Monitoring and analyzing TV broadcasts is an important task in the media as well as the advertising business. An important subtask is the frame-accurate detection of recurring video sequences. Examples of recurring video sequences are commercials, channel advertisements, channel intros, and newscast intros. Most of these different kinds of repeating video clips can automatically be classified by further analyzing their temporal and visual properties. In this work we introduce an algorithm and a real-time system for recognizing recurring video sequences frameaccurately in a highly effective and efficient manner. The algorithm does not require any temporal pre-segmentation by shot detection and can thus, in principle, be applied to any kind of temporal signal. It is frame-accurate, meaning that it exactly identifies with which frame a repeating sequence starts and ends. Thus, the temporal accuracy is 40 milliseconds for PAL and 33 milliseconds for NTSC videos. On a standard PC desktop a 24-hour live-stream can be processed in about 4 hours including the computational expensive video decoding. To achieve this efficiency the algorithmexploits an inverted index for identifying similar frames rapidly.Gradientbased image features are mapped to the index by means of a hash function. The search algorithm consists of two steps: firstly searching for recurring short segments of 1 second duration and secondly assembling these small segments into the set of repeated video clips. In our experiments we investigate the sensitivity of the algorithm concerning all system parameters and apply it to the detection of unknown commercials within 24 and 48 hours of various TV channels. It is shown that the method is an excellent technique for searching for unknown commercials. Currently the system is used 24 hours a day, 7 days a week in various countries to log all commercials broadcast without manual intervention.


Execution Time Video Sequence Hash Function Video Stream Inverted Index 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Half minute media,
  2. 2.
    Council directive 89/552/eec of 3 October 1989 on the coordination of certain provisions laid down by law, regulation or administrative action in member states concerning the pursuit of television broadcasting activities (1989)Google Scholar
  3. 3.
    Staatsvertag für Rundfunk- und Telemedien (Rundfunkstaatsvertrag – RStV –) (2008)Google Scholar
  4. 4.
    Agnihotri, L., Dimitrova, N., McGee, T., Jeannin, S., Schaffer, D., Nesvadba, J.: Evolvable visual commercial detetctor. In: Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2003), IEEE Computer Society, Madison (2003)Google Scholar
  5. 5.
    Albiol, A., Fullà, M.J.C., Albiol, A., Torres, L.: Detection of tv commercials. In: Proceedings of the International Conference on Acoustics, Speech and Signal Processing, Montreal, Canada, vol. 3, pp. 541–544 (2004)Google Scholar
  6. 6.
    Andoni, A., Indyk, P.: Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. Commun. ACM 51(1), 117–122 (2008), CrossRefGoogle Scholar
  7. 7.
    Can, T., Duygulu, P.: Searching for repeated video sequences. In: MIR 2007: Proceedings of the international workshop on multimedia information retrieval, pp. 207–216. ACM, New York (2007), CrossRefGoogle Scholar
  8. 8.
    Chen, J.C., Yeh, J.H., Chu, W.T., Kuo, J.H., Wu, J.: Improvement of commercial boundary detection using audiovisual features. In: Ho, Y.-S., Kim, H.-J. (eds.) PCM 2005. LNCS, vol. 3767, pp. 776–786. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  9. 9.
    Dimitrova, N., Jeannin, S., Nesvadba, J., McGee, T., Agnihotri, L., Mekenkamp, G.: Real time commercial detection using mpeg features. In: Proceedings of the 9th International Conference on Information Processing and Management of Uncertainty in Knowlwdge-based Systems (IPMU 2002), Annecey, France, pp. 481–486 (2002)Google Scholar
  10. 10.
    Döhring, I., Lienhart, R.: Fast and effective features for recognizing recurring video clips in very large databases. In: International Workshop on Video and Multimedia Digital Library (VMDL 2007), Modena, Italy, pp. 65–70 (2007)Google Scholar
  11. 11.
    Döhring, I., Lienhart, R.: Fast frame-accurate mining for repeating video clips. Technical Report Institut für Informatik, Universität Augsburg (2008),
  12. 12.
    Duan, L.Y., Wang, J., Zheng, Y., Jin, J.S., Lu, H., Xu, C.: Segmentation, categorization, and identification of commercial clips from tv streams using multimodal analysis. In: MULTIMEDIA 2006: Proceedings of the 14th annual ACM international conference on Multimedia, pp. 201–210. ACM, New York (2006), CrossRefGoogle Scholar
  13. 13.
    Duygulu, P., yu Chen, M., Hauptmann, A.: Comparison and combination of two novel commercial detection methods. In: Proceedings of the 2004 IEEE International Conference on Multimedia and Expo., Taipei, Taiwan (2004)Google Scholar
  14. 14.
    Gauch, J.M., Shivadas, A.: Identification of new commercials using repeated video sequence detection. In: Proceedings of the 2005 International Conference on Image Processing, Genoa, Italy, pp. 1252–1255 (2005)Google Scholar
  15. 15.
    Gionis, A., Indyk, P., Motwani, R.: Similarity search in high dimensions via hashing. In: VLDB 1999: Proceedings of the 25th International Conference on Very Large Data Bases, pp. 518–529. Morgan Kaufmann Publishers Inc., San Francisco (1999)Google Scholar
  16. 16.
    Glasberg, R., Tas, C., Sikora, T.: Recognizing commercials in real-time using three visual descriptors and a decision tree. In: Proceedings of the 2006 IEEE International Conference on Multimedia and Expo., pp. 1481–1484 (2006)Google Scholar
  17. 17.
    Hampapur, A., Bolle, R.: Feature based indexing for media tracking. In: Proceedings of International Conference on Multimedia and Expo., New York (2000)Google Scholar
  18. 18.
    Hampapur, A., Bolle, R.: Videogrep: Video copy detection using inverted file indices. Technical report on some unpublished work, IBM Research (2001),
  19. 19.
    Hampapur, A., Bolle, R.M.: Comparision of distance measures for video copy detection. Computer Science RC 22056, IBM Research (2001)Google Scholar
  20. 20.
    Hansen, A.: Bioinformatik: Ein Leitfaden für Naturwissenschaftler, 2nd edn. Birkhäuser, Basel (2006)Google Scholar
  21. 21.
    Hoad, T.C., Zobel, J.: Fast video matching with signature alignment. In: MIR 2003: Proceedings of the 5th ACM SIGMM international workshop on Multimedia information retrieval, pp. 262–269. ACM, New York (2003), CrossRefGoogle Scholar
  22. 22.
    Hoad, T.C., Zobel, J.: Video similarity detection for digital rights management. In: ACSC 2003: Proceedings of the 26th Australasian computer science conference, pp. 237–245. Australian Computer Society, Inc., Darlinghurst (2003)Google Scholar
  23. 23.
    Indyk, P., Motwani, R.: Approximate nearest neighbors: towards removing the curse of dimensionality. In: STOC 1998: Proceedings of the thirtieth annual ACM symposium on Theory of computing, pp. 604–613. ACM, New York (1998), CrossRefGoogle Scholar
  24. 24.
    Knuth, D.E.: Sorting and Searching. In: The Art of Computer Programming, 2nd edn., vol. 3. Addison-Wesley, Reading (1998)Google Scholar
  25. 25.
    Lienhart, R., Kuhmünch, C., Effelsberg, W.: On the detection and recognition of television commercials. In: Proc. IEEE Conf. on Multimedia Computing and Systems, Ottawa, Canada, pp. 509–516 (1997)Google Scholar
  26. 26.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2) (2004)Google Scholar
  27. 27.
    Pass, G., Zabih, R., Miller, J.: Comparing images using color coherence vectors. In: ACM Conference on Multimedia, Boston, Massachusetts, pp. 65–74 (1996)Google Scholar
  28. 28.
    Pua, K.M., Gauch, J.M., Gauch, S.E., Miadowicz, J.Z.: Real time repeated video sequence identification. Comput. Vis. Image Underst. 93(3), 310–327 (2004), CrossRefGoogle Scholar
  29. 29.
    Sánchez, J.M., Binefa, X.: Audicom: A video analysis system for auditing commercial broadcasts. In: International Conference on Multimedia Computing and Systems, vol. 2, p. 272 (1999),
  30. 30.
    Shivadas, A., Gauch, J.M.: Real-time commercial recognition using color moments and hashing. In: CRV 2007: Proceedings of the Fourth Canadian Conference on Computer and Robot Vision, pp. 465–472. IEEE Computer Society, Washington (2007), CrossRefGoogle Scholar
  31. 31.
    Squire, D.M., Müller, H., Müller, W.: Improving response time by search pruning in a content-based image retrieval system, using inverted file techniques. In: CBAIVL 1999: Proceedings of the IEEE Workshop on Content-Based Access of Image and Video Libraries, p. 45. IEEE Computer Society, Washington (1999)CrossRefGoogle Scholar
  32. 32.
    Yuan, J., Wang, W., Meng, J., Wu, Y., Li, D.: Mining repetitive clips through finding continuous paths. In: MULTIMEDIA 2007: Proceedings of the 15th international conference on Multimedia, pp. 289–292. ACM, New York (2007), CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Ina Döhring
    • 1
  • Rainer Lienhart
    • 1
  1. 1.Multimedia Computing LabUniversity of AugsburgGermany

Personalised recommendations