Multimedia Tools and Applications

, Volume 74, Issue 24, pp 11837–11865 | Cite as

Automatic pan-and-scan algorithm for heterogeneous displays

  • Paween Khoenkaw
  • Punpiti Piamsa-ngaEmail author


This paper presents a fully automatic algorithm for selecting the most appropriate areas of every video frame to show on heterogeneous display devices. The algorithm is used to analyze cinematic features in video and identify important parts of each frame. The proposed method is a client-server model. An importance map of video is created on the server and transmitted along with the video stream. The client uses this map to determine the cropping area of video frames to fit its display screen. An experimental comparison involving users’ satisfaction of 160 participants who compared the results of our algorithm with others, such as letterbox, and commercially cropped editions of well-known motion pictures was performed. The results show that the audience preferred the video processed by our algorithm as the algorithm requires little additional preprocessing and little extra information embedded in the video stream. The results suggest that this algorithm could improve the video viewing experience for mobile users.


Video cropping Image cropping Importance estimation Pan-and-scan Mobile device Cinematic Video aspect ratio adaption Video analysis 



The authors would like to thank the reviewers for their comments that help us improve the manuscript. We also would like to thank Dr. James E. Brucker and Dr. Bussba Tonthong for assistances on proofreading. This research is partially supported by research grant from Kasetsart University Research and Development Institute.


  1. 1.
    Abrams J (2009) Star trekGoogle Scholar
  2. 2. (2012) Iphone 3gs technical specification.
  3. 3.
    Baina J, Dublet J (1995) Automatic focus and iris control for video cameras. In: 5th international conference on image processing and its applications, 1995, pp 232–235Google Scholar
  4. 4.
    Bay M (2009) Transformers: Revenge of the fallenGoogle Scholar
  5. 5.
    Becker H (1998) Mercury risingGoogle Scholar
  6. 6.
    Bhat DN, Nayar SK (1998) Ordinal measures for image correspondence. IEEE Trans Pattern Anal Mach Intell 20(4):415–423Google Scholar
  7. 7. (2012) Blackberry bold 9900 technical specification
  8. 8.
    Byers Z, Dixon M, Goodier K, Grimm CM, Smart WD (2003) An autonomous robot photographer. In: Proceedings of the 2003 IEEE/RSJ international conference on Intelligent Robots and Systems, 2003 (IROS 2003), vol 3, pp 2636–2641Google Scholar
  9. 9.
    Chee Hwa C, Ying Z, Changyun W (2003) Visual servo auto-focusing using recursive weighted least-squares for machine vision inspection. In: Electronics Packaging Technology, 2003 5th Conference (EPTC 2003), pp 777–780Google Scholar
  10. 10.
    Chen DY, Luo YS (2011) Content-aware video resizing based on salient visual cubes. J Vis Commun Image Represent 22(3):226–236Google Scholar
  11. 11.
    Chen LQ, Xie X, Fan X, Ma WY, Zhang HJ, Zhou HQ (2003) A visual attention model for adapting images on small displays. Multimedia Systems 9(4):353–364CrossRefGoogle Scholar
  12. 12.
    Cheng WH, Wang CW, Wu JL (2007) Video adaptation for small display based on content recomposition. IEEE Trans Circ Syst Video Technol 17(1):43–58Google Scholar
  13. 13.
    Cheol-Hee P, Jong-Ho P, Young-Hwan Y, Hyoung-Kyu S, Yong-Soo C (2000) Auto focus filter design and implementation using correlation between filter and auto focus criterion. In: International Conference on Consumer Electronics, 2000 (ICCE, 2000). Digest of Technical Papers, pp 250–251Google Scholar
  14. 14.
    Chun-Hung S, Chen HH (2006) Robust focus measure for low-contrast images. In: International Conference on Consumer Electronics, 2006, ICCE 06. Digest of Technical Papers, pp 69–70Google Scholar
  15. 15.
    Chung Nam C, Jung Hun J, Joon Shik Y, Jeong Ho S, Joon Ki P (1999) Region selectable auto-focusing system by digital psf estimation. In: International Conference on Consumer Electronics, 1999, ICCE, pp 92–93Google Scholar
  16. 16.
    Dalton CJ (1996) Wide-screen production engineering-an overview from source to display. In: Broadcasting convention, international (Conf. Publ. No. 428), pp 366–372Google Scholar
  17. 17.
    Evans-Pughe C (2005) Movies on the move [mobile phone video content reception]. IEE Rev 51(10):44–47CrossRefGoogle Scholar
  18. 18.
    Fan X, Xie X, Zhou HQ, Ma WY (2003) L247-25ooking into video frames on small displays. In: Proceedings of the eleventh ACM international conference on Multimedia. ACM, pp 247–250Google Scholar
  19. 19.
    Frintrop S, Rome E, Christensen HI (2010) Computational visual attention systems and their cognitive foundations: a survey. ACM Trans Appl Percept 7(1):1–39Google Scholar
  20. 20.
    Grinias I, Tziritas G (2002) Robust pan, tilt and zoom estimation. In: 2002 14th international conference on Digital Signal Processing, 2002, DSP 2002, vol 2, pp 679–682Google Scholar
  21. 21. (2012) Nokia n80 technical specification.
  22. 22. (2012) Palm treo pro technical specification.
  23. 23.
    Harel J, Koch C, Perona P (2007) Graph-based visual saliency. In: Advances in neural information processing system, vol 19. MIT Press, pp 545–552Google Scholar
  24. 24.
    He LW, Cohen MF, Salesin DH (1996) The virtual cinematographer: a paradigm for automatic real-time camera control and directing. In: Proceedings of the 23rd annual conference on computer graphics and interactive techniques, pp 217–224. ACMGoogle Scholar
  25. 25.
    Holzinger A (2005) Usability engineering methods for software developers. Commun ACM 48(1):71–74CrossRefGoogle Scholar
  26. 26.
    Hyeong-Min N, Keun-Yung B, Jae-Yun J, Kang-Sun C, Sung-Jea K (2010) Low complexity content-aware video retargeting for mobile devices. IEEE Trans Consum Electron 56(1):182–189Google Scholar
  27. 27.
    Jianzhou Y, Lin S, Sing Bing K, Xiaoou T (2013) Learning the change for automatic image cropping. In: 2013 IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp 971–978Google Scholar
  28. 28.
    Jie H, Rongzhen Z, Zhiliang H (2003) Modified fast climbing search auto-focus algorithm with adaptive step size searching technique for digital camera. IEEE Trans Consum Electron 49(2):257–262Google Scholar
  29. 29.
    Jordan N, Schatz R (2006) Broadcast television services suited for mobile handheld devices. In: International Conference on Digital Telecommunications, 2006, ICDT ’06, pp 55–55Google Scholar
  30. 30.
    Jun W, Reinders MJT, Lagendijk RL, Lindenberg J, Kankanhalli MS (2004) Video content representation on tiny devices. In: 2004 IEEE International Conference on Multimedia and Expo, 2004, ICME ’04, vol 3, pp 1711–1714Google Scholar
  31. 31.
    Katz SD (1991) Film directing shot by shot : visualizing from concept to screen. Michael Wiese Productions in conjunction with Focal Press, Studio CityGoogle Scholar
  32. 32.
    Kopf S, Haenselmann T, Kiess J, Guthier B, Effelsberg W (2011) Algorithms for video retargeting. Multimedia Tools Appl 51 (2):819–861CrossRefGoogle Scholar
  33. 33.
    Kopf S, Lampi F, King T, Effelsberg W (2006) Automatic scaling and cropping of videos for devices with limited screen resolution. In: Proceedings of the 14th annual ACM international conference on multimedia. ACM, pp 957–958Google Scholar
  34. 34.
    Lee SY, Park SS, Kim CS, Kumar Y, Kim SW (2006) Low-power auto focus algorithm using modified dct for the mobile phones. In: International Conference on Consumer Electronics, 2006, ICCE ’06. 2006 Digest of Technical Papers, pp 67–68Google Scholar
  35. 35.
    Liu F, Gleicher M (2005) Automatic image retargeting with fisheye-view warping. In: Proceedings of the 18th annual ACM symposium on user interface software and technology. ACM, pp 153–162Google Scholar
  36. 36.
    Liu F, GleicherM(2006) Video retargeting: automating pan and scan. In: Proceedings of the 14th annual ACM international conference on multimedia. ACM, pp 241–250Google Scholar
  37. 37.
    Liu H, Xie X, Ma WY, Zhang HJ (2003) Automatic browsing of large pictures on mobile devices. In: Proceedings of the eleventh ACM international conference on multimedia. ACM, pp 148–155Google Scholar
  38. 38.
    Luming Z, Mingli S, Qi Z, Xiao L, Jiajun B, Chun C (2013) Probabilistic graphlet transfer for photo cropping. IEEE Trans Image Process 22(2):802–815Google Scholar
  39. 39.
    Luming Z, Mingli S, Yi Y, Qi Z, Chen Z, Sebe N (2014) Weakly supervised photo cropping. IEEE Trans Multimedia 16(1):94–107Google Scholar
  40. 40.
    Ma YF, Zhang HJ (2003) Contrast-based image attention analysis by using fuzzy growing. In: Proceedings of the 11th ACM international conference on multimedia, pp 374–381 ACMGoogle Scholar
  41. 41.
    Manchel F (1990) Film study : an analytical bibliography. Fairleigh Dickinson University Press, Rutherford, LondonGoogle Scholar
  42. 42.
    Mann M (1986) ManhunterGoogle Scholar
  43. 43.
    Ming-Hsuan Y, Kriegman DJ, Ahuja N (2002) Detecting faces in images: a survey. IEEE Trans Pattern Anal Mach Intell 24(1):34–58Google Scholar
  44. 44.
    Ming-Sui L, Mei-Yin S, Kuo CCJ, Akio Y (2007) Techniques for flexible image/video resolution conversion with heterogeneous terminals. IEEE Commun Mag 45(1):61–67CrossRefGoogle Scholar
  45. 45.
    Mingju Z, Lei Z, Yanfeng S, Lin F, Weiying M (2005) Auto cropping for digital photographs. In: IEEE International Conference on Multimedia and Expo, 2005, ICME 2005, p 4Google Scholar
  46. 46.
    Opensubtitles. (2014)
  47. 47.
    Piamsa-nga P, Babaguchi N (2004) Motion estimation and detection of complex object by analyzing resampled movements of parts. In: International Conference on Image Processing, 2004, ICIP ’04, vol 1, pp 365–368Google Scholar
  48. 48.
    Prince S (2004) Movies and meaning : an introduction to film, 3rd edn. Pearson/Allyn and Bacon, BostonGoogle Scholar
  49. 49.
    Sang Ku K, Sang Rae P, Joon Ki P (1998) Simultaneous out-of-focus blur estimation and restoration for a digital auto-focusing system. In: International Conference on Consumer Electronics, 1998, ICCE. Digest of Technical Papers, pp 430–432Google Scholar
  50. 50.
    Santella A, Agrawala M, DeCarlo D, Salesin D, Cohen M (2006) Gaze-based interaction for semiautomatic photo cropping. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 771–780Google Scholar
  51. 51.
    Setlur V, Takagi S, Raskar R, GleicherM, Gooch B (2005) Automatic image retargeting. In: Proceedings of the 4th international conference on mobile and ubiquitous multimedia. ACM, pp 59–68Google Scholar
  52. 52.
    (1994) SMPTE: Specifications for an operational test pattern for checking jitter,weave and travel ghost in television projectorsGoogle Scholar
  53. 53.
    (2003) SMPTE: Method for determining the degree of jump and weave in 70-, 35- and 16-mm motionpicture projected imagesGoogle Scholar
  54. 54.
    Spielberg S (1981) Raiders of the lost arkGoogle Scholar
  55. 55.
    Spielberg S (1984) Indiana jones and the temple of doomGoogle Scholar
  56. 56.
    Spielberg S (1989) Indiana jones and the last crusadeGoogle Scholar
  57. 57.
    Subbarao M, Choi T, Nikzad A (1993) Focusing techniques. J Opt Eng 32:2824–2836CrossRefGoogle Scholar
  58. 58.
    Suh B, Ling H, Bederson BB, Jacobs DW (2003) Automatic thumbnail cropping and its effectiveness. ACMGoogle Scholar
  59. 59.
    Sung-Hyun Y, Sung-Min S, Kuk-Tae H, Hyoung-Soo L, Bo-Ik S (2005) On-chip voice-coil motor driver for mobile auto-focus camera applications. In: Asian solid-state circuits conference, 2005, pp 101–104Google Scholar
  60. 60.
    Tae-Kyu K, Jae-Sik S, Suk-Hwan L, Ki-Ryong K, Duh-Gyoo K (2006) Fast auto-focus control algorithm using the vcm hysteresis compensation in the mobile phone camera. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 2006, ICASSP 2006, vol 3, p 3Google Scholar
  61. 61.
    Union IT (2012) Recommendation itu-r bt.500-13 methodology for the subjective assessment of the quality of television picturesGoogle Scholar
  62. 62.
    Walther D, Koch C (2006) Modeling attention to salient proto-objects. Neural Netw 19 (9):1395–1407CrossRefzbMATHGoogle Scholar
  63. 63.
    Watkinson J (2001) Convergence in broadcast and communications media : the fundamentals of audio, video, data processing, and communications technologies. Focal Press, OxfordGoogle Scholar
  64. 64.
    Wheeler LJ (1969) Principles of cinematography : a handbook of motion picture technology, 4th edn. Fountain press, LondonGoogle Scholar
  65. 65.
    Wright S (2006) Digital compositing for film and video, 2nd edn. Focal Press, Burlington, Mass, OxfordGoogle Scholar
  66. 66.
    Xue YZ, Du H, Liu Z, Zhang ZY (2011) Video retargeting using optimized crop-and-scale. J Shanghai Univ (Engl Ed) 15(4):331–334CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.Department of Computer Engineering, Faculty of EngineeringKasetsart UniversityChatuchakThailand

Personalised recommendations