Advertisement

Multimedia Tools and Applications

, Volume 77, Issue 17, pp 22173–22184 | Cite as

Adaptive recommendation for photo pose via deep learning

  • Tong Hao
  • Qian Wang
  • Dan Wu
  • Jin-Sheng Sun
Article
  • 107 Downloads

Abstract

With the development of image acquisition devices and the popularity of smart phones, more and more people would like to upload their photos to diverse social networks. It is hard to guarantee the quality and artistry of these photos because of not everyone is a professional photographer. In order to handle this problem and further help each common user to improve the beauty of photos, we propose an intelligent photo pose recommendation method to recommended professional photo pose according to everyone’s posture in viewfinder. Firstly, the CNN model (VGG-16) is utilized to extract the global features for each photo. Secondly, the salient region detection method is leveraged to extract the regions of interest in each photo. To represent the edge distribution in the local regions, we extract the histogram of oriented gradients. Finally, we propose an effective feature fusion method based on CCA to generate the global visual features for each photo. We implement the Euclidean distance to handle the similarity measure between uploaded photos and the professional photos. The most similar professional photo will be utilized to guide user photo composition. In order to evaluate the performance of the proposed method, we collected a set of professional photos form some professional photography websites. The comparison experiments and user study demonstrate the superiority of the proposed approach.

Keywords

Photo pose recommendation Deep learning Feature fusion 

Notes

Acknowledgments

This work was funded by National High-Tech Research and Development Program of China (863 programs, 2012AA10A401), Grants of the Major State Basic Research Development Program of China (973 programs, 2012CB114405), National Natural Science Foundation of China (31770904,21106095), National Key Technology R & D Program (2011BAD13B07, 2011BAD13B04), Tianjin Applied Basic and Advanced Technology Research Program (15JCYBJC30700), Project of introducing one thousand high level talents in three years(5KQM110003), Tianjin Normal University Academic Innovation Promotion Program for Young Teachers (52XC1403) and Tianjin Innovative Talent Training Program (ZX110170).

References

  1. 1.
    Achanta R, Hemami S, Estrada F, Susstrunk S (2009) Frequency-tuned salient region detection. In: IEEE conference on Computer vision and pattern recognition, 2009. cvpr 2009. IEEE, pp 1597–1604Google Scholar
  2. 2.
    Babenko A, Slesarev A, Chigorin A, Lempitsky V (2014) Neural codes for image retrieval. In: European conference on computer vision, Springer, pp 584–599Google Scholar
  3. 3.
    Bai X, Liu C, Ren P, Zhou J, Zhao H, Su Y (2015) Object classification via feature fusion based marginalized kernels. IEEE Geosci Remote Sens Lett 12(1):8–12CrossRefGoogle Scholar
  4. 4.
    Chandrasekhar V, Lin J, Liao Q, Morere O, Veillard A, Duan L, Poggio T Compression of deep neural networks for image instance retrieval. arXiv:1701.04923
  5. 5.
    Chen J, Chen Z, Chi Z, Fu H (2014) Emotion recognition in the wild with feature fusion and multiple kernel learning. In: Proceedings of the 16th International Conference on Multimodal Interaction, ACM, pp 508–513Google Scholar
  6. 6.
    Cheng Z, Shen J (2016) On very large scale test collection for landmark image search benchmarking. Signal Process 124:13–26CrossRefGoogle Scholar
  7. 7.
    Deng J, Dong W, Socher R, Li L, Li K, Feifei L (2009) Imagenet: A large-scale hierarchical image databaseGoogle Scholar
  8. 8.
    Fei H, Huan J (2008) Structure feature selection for graph classification. In: Proceedings of the 17th ACM conference on Information and knowledge management, ACM, pp 991–1000Google Scholar
  9. 9.
    Gao Y, Zhen Y, Li H, Chua T-S (2016) Filtering of brand-related microblogs using social-smooth multiview embedding. IEEE Trans Multimedia 18(10):2115–2126CrossRefGoogle Scholar
  10. 10.
    Gao Z, Li SH, Zhu YJ, Wang C, Zhang H (2017) Collaborative sparse representation leaning model for RGBD action recognition. J Vis Commun Image Represent 48:442–452CrossRefGoogle Scholar
  11. 11.
    Gao Z, Zhang H, Xu GP, Xue YB, Hauptmann AG (2015) Multi-view discriminative and structured dictionary learning with group sparsity for human action recognition. Signal Processs 112:83–97CrossRefGoogle Scholar
  12. 12.
    Gao Z, Zhang L, Chen M, Hauptmann AG, Zhang H, Cai A (2014) Enhanced and hierarchical structure algorithm for data imbalance problem in semantic extraction under massive video dataset. Multimedia Tools Appl 68(3):641–657CrossRefGoogle Scholar
  13. 13.
    Gens R, Domingos PM (2013) Learning the structure of sum-product networks, pp 873–880Google Scholar
  14. 14.
    Gonde AB, Murala S, Vipparthi SK, Maheshwari R, Balasubramanian R (2017) 3d local transform patterns: A new feature descriptor for image retrieval. In: Proceedings of International Conference on Computer Vision and Image Processing, Springer, pp 495–507Google Scholar
  15. 15.
    Gordo A, Almazán J., Revaud J, Larlus D (2016) Deep image retrieval: Learning global representations for image search. In: European Conference on Computer Vision, Springer, pp 241–257Google Scholar
  16. 16.
    Guo J, Ren T, Bei J (2016) Salient object detection in RGB-D image via saliency evolution. In: IEEE International Conference on Multimedia and Expo, IEEE, pp 1–6Google Scholar
  17. 17.
    Huang X, Sun L, Guo H, Liu S (2016) Discriminative extreme learning machine to content-based image retrieval with relevance feedback. In: 2016 12th World Congress on Intelligent Control and Automation (WCICA), IEEE, pp 3056–3060Google Scholar
  18. 18.
    Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: Convolutional architecture for fast feature embeddingGoogle Scholar
  19. 19.
    Krissinel E, Henrick K (2004) Secondary-structure matching (ssm), a new tool for fast protein structure alignment in three dimensions. Acta Crystallogr Sec D Biol Crystallogr 60(12):2256–2268CrossRefGoogle Scholar
  20. 20.
    Lai H, Yan P, Shu X, Wei Y, Yan S (2016) Instance-aware hashing for multi-label image retrieval. IEEE Trans Image Process 25(6):2469–2479MathSciNetCrossRefGoogle Scholar
  21. 21.
    Li A, Morariu VI, Davis LS (2014) Planar structure matching under projective uncertainty for geolocation. In: European Conference on Computer Vision, Springer, pp 265–280Google Scholar
  22. 22.
    Liu A, Su Y, Jia P, Gao Z, Hao T, Yang Z (2015) Multipe/single-view human action recognition via part-induced multitask structural learning. IEEE Trans Cybern 45(6):1194–1208CrossRefGoogle Scholar
  23. 23.
    Liu A, Nie W, Gao Y, Su Y (2016) Multi-modal clique-graph matching for view-based 3d model retrieval. IEEE Trans Image Process 25(5):2103–2116MathSciNetCrossRefGoogle Scholar
  24. 24.
    Liu A-A, Su Y-T, Nie W-Z, Kankanhalli M (2017) Hierarchical clustering multi-task learning for joint human action grouping and recognition. IEEE Trans Pattern Anal Mach Intell 39(1):102–114CrossRefGoogle Scholar
  25. 25.
    Liu G-H, Yang J-Y (2013) Content-based image retrieval using color difference histogram. Pattern Recogn 46(1):188–198CrossRefGoogle Scholar
  26. 26.
    Pong K-H, Lam K-M (2014) Multi-resolution feature fusion for face recognition. Pattern Recogn 47(2):556–567CrossRefGoogle Scholar
  27. 27.
    Qian X, Tan X, Zhang Y, Hong R, Wang M (2016) Enhancing sketch-based image retrieval by re-ranking and relevance feedback. IEEE Trans Image Process 25(1):195–208MathSciNetCrossRefGoogle Scholar
  28. 28.
    Vogelstein JT, Park Y, Ohyama T, Kerr RA, Truman JW, Priebe CE, Zlatic M (2014) Discovery of brainwide neural-behavioral maps via multiscale unsupervised structure learning. Science 344(6182):386–392CrossRefGoogle Scholar
  29. 29.
    Wu S, Chen Y-C, Li X, Wu A-C, You J-J, Zheng W-S (2016) An enhanced deep feature representation for person re-identification. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, pp 1–8Google Scholar
  30. 30.
    Xia Z, Wang X, Zhang L, Qin Z, Sun X, Ren K (2016) A privacy-preserving and copy-deterrence content-based image retrieval scheme in cloud computing. IEEE Trans Inf Forensics Secur 11(11):2594–2608CrossRefGoogle Scholar
  31. 31.
    Yu J, Tao D, Wang M, Rui Y (2015) Learning to rank using user clicks and visual features for image retrieval. IEEE Trans Cybern 45(4):767–779CrossRefGoogle Scholar
  32. 32.
    Zhang H, Shang X, Luan H, Wang M, Chua T (2016) Learning from collective intelligence: Feature learning using social images and tags. TOMCCAP 13 (1):1:1–1:23CrossRefGoogle Scholar
  33. 33.
    Zhang S, Yang M, Cour T, Yu K, Metaxas DN (2015) Query specific rank fusion for image retrieval. IEEE Trans Pattern Anal Mach Intell 37(4):803–815CrossRefGoogle Scholar
  34. 34.
    Zhao S, Yao H, Gao Y, Ji R, Ding G (2017) Continuous probability distribution prediction of image emotions via multitask shared sparse regression. IEEE Trans Multimedia 19(3):632–645CrossRefGoogle Scholar
  35. 35.
    Zhou W, Yang M, Wang X, Li H, Lin Y, Tian Q (2016) Scalable feature matching by dual cascaded scalar quantization for image retrieval. IEEE Trans Pattern Anal Mach Intell 38(1):159–171CrossRefGoogle Scholar
  36. 36.
    Zhu L, Shen J, Xie L (2016) Unsupervised visual hashing with semantic assistant for content-based image retrieval. IEEE Trans Knowl Data Eng 29(2):472–486CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Tianjin Key Laboratory of Animal and Plant Resistance/College of Life ScienceTianjin Normal UniversityTianjinChina
  2. 2.Tianjin Aquatic Animal Infectious Disease Control and Prevention CenterTianjinChina

Personalised recommendations