Skip to main content
SpringerLink
Log in

Real-time shape tracking of facial landmarks

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Detection of facial landmarks and accurate tracking of their shape are essential in real-time applications such as virtual makeup, where users can see the makeup’s effect by moving their face in diverse directions. Typical face tracking techniques detect facial landmarks and track them using a point tracker such as the Kanade-Lucas-Tomasi (KLT) point tracker. Typically, 5 or 64 points are used for tracking a face. Even though these points are enough to track the approximate locations of facial landmarks, they are not sufficient to track the exact shape of facial landmarks. In this paper, we propose a method that can track the exact shape of facial landmarks in real-time by combining a deep learning technique and a point tracker. We detect facial landmarks accurately using SegNet, which performs semantic segmentation based on deep learning. Edge points of detected landmarks are tracked using the KLT point tracker. In spite of its popularity, the KLT point tracker suffers from the point loss problem. We solve this problem by executing SegNet periodically to recalculate the shape of facial landmarks. That is, by combining the two techniques, we can avoid the computational overhead of SegNet and the point loss problem of the KLT point tracker, which leads to accurate real-time shape tracking. We performed several experiments to evaluate the performance of our method and report some of the results herein.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Badrinarayanan V, Kendall A, Cipolla R (2017) Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans Pattern Anal Mach Intell 39(12):2481–2495

    Article  Google Scholar 

  2. Barber CB et al (1996) The quickhull algorithm for convex hulls. ACM Trans Math Softw (TOMS) 22(4):469–483

    Article  MathSciNet  Google Scholar 

  3. Belaroussi R, Prevost L, Milgram M (2005) Combining model-based classifiers for face localization. Ninth IAPR Conference on Machine Vision Applications, pp 290–293

  4. Canny J (1986) A computational approach to edge detection. IEEE Trans Pattern Anal Mach Intell 8:679–698

    Article  Google Scholar 

  5. Danelljan M et al (2017) Discriminative scale space tracking. IEEE Trans Pattern Anal Mach Intell 39(8):1561–1575

    Article  Google Scholar 

  6. DEST v.08 software library (2016) http://github.com/cheind/dest/releases (visited on 15-12-16)

  7. Girshick R (2015) Fast r-cnn. ArXiv preprint arXiv:1504.08083

    Google Scholar 

  8. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition

  9. Hare S, Golodetz S, Saffari A, Vineet V, Cheng MM, Hicks SL, Torr PH (2016) Struck: structured output tracking with kernels. IEEE Trans Pattern Anal Mach Intell 38(10):2096–2109

    Article  Google Scholar 

  10. Henriques JF, Caseiro R, Martins P, Batista J (2015) High-speed tracking with Kernelized correlation filters. IEEE Trans Pattern Anal Mach Intell 37(3):583–596

    Article  Google Scholar 

  11. Kim K-S, Jang D-S, Choi, H-I (2007) Real time face tracking with pyramidal Lucas-Kanade feature tracker. International Conference on Computational Science and Its Applications

  12. Kim H, Park J, Kim H, Hwang E (2018) Facial landmark extraction scheme based on semantic segmentation. 2018 International Conference on Platform Technology and Service (PlatCon-18)

  13. Kim H et al (2018) Robust facial landmark extraction scheme using multiple convolutional neural networks. Multimedia Tools and Applications 2018:1–18

    Google Scholar 

  14. King DE (2009) Dlib-ml: a machine learning toolkit. J Mach Learn Res 10:1755–1758

    Google Scholar 

  15. Geonzalez RC, Woods RE (2008) Digital image processing. 3rd edn. Pearson Prentice Hall, Upper Saddle River

  16. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. CVPR, pp 3431–3440

  17. Lucas BD, Kanade T (1981) An iterative image registration technique with an application to stereo vision. Proceedings of the 7th international joint conference on artificial intelligence, vol 2, pp 674–679

  18. Noh H, Hong S, Han B (2015) Learning deconvolution network for semantic segmentation. ICCV, pp 1520–1528

  19. Park J, Kim H, Ji S, Hwang E (2018) An automatic virtual makeup scheme based on personal color analysis. International Conference on Ubiquitous Information Management and Communication (IMCOM 2018)

  20. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. The IEEE conference on computer vision and pattern recognition

  21. Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: towards real-time object detection with region proposal networks. Advances in neural information processing system

  22. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. MICCAI, pp 234–241

  23. Shaik Z, Asari V (2007) A robust method for multiple face tracking using Kalman filter. Applied Imagery Pattern Recognition Workshop, 36th IEEE

  24. Soille (2013) Morphological image analysis: principles and applications. Springer Science & Business Media

  25. Tomasi C, Kanade T (1991) Detection and tracking of point features. Carnegie Mellon University Technical Report CMU-CS-91-132

  26. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

    Article  Google Scholar 

  27. Wettum YC (2017) Facial landmark tracking on a mobile device, BS thesis. University of Twente

  28. Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters 23(10):1499–1503

    Article  Google Scholar 

Download references

Acknowledgments

This work was partly supported by Institute for Information & communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (No. R0190-16-2012, High Performance Big Data Analytics Platform Performance Acceleration Technologies Development) and Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2016R1D1A1A09919590).

Author information

Authors and Affiliations

  1. School of Electrical Engineering, Korea University, Seoul, Republic of Korea

    Hyungjoon Kim, HyeonWoo Kim & Eenjun Hwang

Authors
  1. Hyungjoon Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. HyeonWoo Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eenjun Hwang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Eenjun Hwang.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, H., Kim, H. & Hwang, E. Real-time shape tracking of facial landmarks. Multimed Tools Appl 79, 15945–15963 (2020). https://doi.org/10.1007/s11042-018-6814-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6814-7

Keywords

Search

Navigation