Robust Lip Segmentation Based on Complexion Mixture Model

  • Yangyang Hu
  • Hong Lu
  • Jinhua Cheng
  • Wenqiang ZhangEmail author
  • Fufeng Li
  • Weifei Zhang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9916)


Lip image analysis plays a vital role in Traditional Chinese Medicine (TCM) and other visual and speech recognition applications. However, if the lip images contain weak color difference with background parts or the background is complicated, most of the current methods are difficult to robustly and accurately segment the lip regions. In this paper, we propose a lip segmentation method based on complexion mixture model to resolve this problem. Specifically, we use the pixels’ color of the upper (lip-free) part of the face as training data to build a corresponding complexion Gaussian Mixture Model (GMM) for each face image in Lab color space. Then by iteratively removing the complexion pixels not belonging to the lip region in the lower part of the face based on the GMM, an initial lip can be obtained. We further build GMMs on the initial lip and non-lip regions, respectively. The background probability map can be obtained based on the GMMs. Finally, we extract the optimal lip contour via a smooth operation. Experiments are performed on our dataset with 1000 face images. Experimental results demonstrate the efficacy of the proposed method compared with the state-of-art lip segmentation methods.


Lip segmentation Complexion mixture model 



This work was supported in part by the National Natural Science Foundation of China (No. 81373555), and Shanghai Committee of Science and Technology (14JC1402202 and 14441904403).


  1. 1.
    Shin, J., Jun, B., Kim, D.: Robust two-stage lip tracker. In: IEEE International Symposium on Signal Processing and Information Technology, pp. 265–270 (2009)Google Scholar
  2. 2.
    Zhang, Y., Levinson, S., Huang, T.: Speaker independent audio-visual speech recognition. In: IEEE International Conference on Multimedia and Expo, vol. 2, pp. 1073–1076 (2000)Google Scholar
  3. 3.
    Lucey, S., Sridharan, S., Chandran, V.: Initialised eigenlip estimator for fast lip tracking using linear regression. In: IEEE International Conference on Pattern Recognition, vol. 3, pp. 178–181 (2000)Google Scholar
  4. 4.
    Rabi, G., Lu, S.: Visual speech recognition by recurrent neural networks. In: Engineering Innovation: Voyage of Discovery Electrical and Computer Engineering, vol. 1, pp. 55–58. IEEE (1997)Google Scholar
  5. 5.
    Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., Ng, A.Y.: Multimodal deep learning. In: Proceedings of the 28th International Conference on Machine Learning, pp. 689–696 (2011)Google Scholar
  6. 6.
    Sun, C., Lu, H., Zhang, W., Qiu, X., Li, F., Zhang, H.: Lip segmentation based on facial complexion template. In: Ooi, W.T., Snoek, C.G.M., Tan, H.K., Ho, C.-K., Huet, B., Ngo, C.-W. (eds.) PCM 2014. LNCS, vol. 8879, pp. 193–202. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-13168-9_20 Google Scholar
  7. 7.
    Ghaleh, V.E.C., Behrad, A.: Lip contour extraction using RGB color space and fuzzy c-means clustering. In: IEEE International Conference on Cybernetic Intelligent Systems, pp. 1–4 (2010)Google Scholar
  8. 8.
    Rahman, F.S., Nath, R., Nath, S., Basak, S., Audin, S.I., Fattah, S.A.: Lip contour extraction scheme based on k-means clustering in different color planes. In: IEEE International Conference on Informatics, Electronics & Vision, pp. 1–4 (2014)Google Scholar
  9. 9.
    Wang, S.-L., Lau, W.-H., Liew, A.W.-C., Leung, S.-H.: Robust lip region segmentation for lip images with complex background. Pattern Recognit. 40(12), 3481–3491 (2007)CrossRefzbMATHGoogle Scholar
  10. 10.
    Kalbkhani, H., Amirani, M.C.: An efficient algorithm for lip segmentation in color face images based on local information. J. World Elect. Eng. Technol. 1(1), 12–16 (2012)Google Scholar
  11. 11.
    Li, M., Cheung, Y.: Automatic segmentation of color lip images based on morphological filter. In: Diamantaras, K., Duch, W., Iliadis, L.S. (eds.) ICANN 2010. LNCS, vol. 6352, pp. 384–387. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-15819-3_51 CrossRefGoogle Scholar
  12. 12.
    Rother, C., Kolmogorov, V., Blake, A.: “Grabcut”: interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23(3), 309–314 (2004)CrossRefGoogle Scholar
  13. 13.
    Bai, X., Wang, J., David, S., Sapiro, G.: Video snapcut: robust video object cutout using localized classifiers. ACM Trans. Graph. (TOG) 28(3) (2009)Google Scholar
  14. 14.
    Wang, J., Agrawala, M., Cohen, M.F.: Soft scissors: an interactive tool for realtime high quality matting. In: ACM Transactions on Graphics (TOG), vol. 26. ACM (2007)Google Scholar
  15. 15.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2001, vol. 1, pp. I–511 (2001)Google Scholar
  16. 16.
    Otsu, N.: A threshold selection method from gray-level histograms. Automatica 11(285–296), 23–27 (1975)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Yangyang Hu
    • 1
  • Hong Lu
    • 1
  • Jinhua Cheng
    • 2
  • Wenqiang Zhang
    • 2
    Email author
  • Fufeng Li
    • 3
  • Weifei Zhang
    • 3
  1. 1.Shanghai Key Lab of Intelligent Information Processing, School of Computer ScienceFudan UniversityShanghaiPeople’s Republic of China
  2. 2.School of Computer Science, Shanghai Engineering Research Center for Video Technology and SystemFudan UniversityShanghaiPeople’s Republic of China
  3. 3.Shanghai University of Traditional Chinese MedicineShanghaiChina

Personalised recommendations