Synthesizing Imagined Faces Based on Relevance Feedback

  • Caie Xu
  • Shota Fushimi
  • Masahiro Toyoura
  • Jiayi Xu
  • Xiaoyang MaoEmail author
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10830)


In this paper, we propose a user-friendly system that can create a facial image from a corresponding image in the user’s mind. Unlike most of the existing methods, which require a sketch as input or the tedious work of selecting similar facial components from an example database, our method can synthesise a satisfying result without questioning the user on the explicit features of the face in his or her mind. Through a dialogic approach based on a relevance feedback strategy to translate facial features into input, the user only needs to look at several candidate face images and judge whether each image resembles the face that he or she is imagining. A set of sample face images that are based on users’ feedbacks are used to dynamically train an Optimum-Path Forest algorithm to classify the relevance of face images. Based on the trained Optimum-Path Forest classifier, candidate face images that best reflect the user’s feedback are retrieved and interpolated to synthesise new face images that are similar to those the user had imagined. The experimental results show that the proposed technique succeeded in generating images resembling a face a user had imagined or memorised.


Face image synthesis Relevance feedback Optimum-Path Forest 



This work was supported by JSPS KAKENHI (Grant No. 17H00737) and the Public Projects of Zhejiang Natural Science Foundation Province, China (Grant No. LGF18F020015).


  1. 1.
  2. 2.
    Wang, X., Tang, X.: Face photo-sketch synthesis and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31(11), 1955–1967 (2009)CrossRefGoogle Scholar
  3. 3.
    Wu, D., Dai, Q.: Sketch realizing: lifelike portrait synthesis from sketch. In: Computer Graphics International Conference, pp. 13–20 (2009)Google Scholar
  4. 4.
    Cootes, T., Edwards, G., Taylor, C.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001)CrossRefGoogle Scholar
  5. 5.
    Liu, C., Shum, H., Freeman, W.: Face hallucination: theory and practice. Int. J. Comput. Vision 75(1), 115–134 (2007)CrossRefGoogle Scholar
  6. 6.
    Gao, X., Yang, J., Lai, Z., Huang, P., Jiang, J., Gao, H., Yue, D.: Nuclear norm regularized coding with local position-patch and nonlocal similarity for face hallucination. Digit. Signal Proc. 64, 107–120 (2017)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Zhu, S., Liu, S., Loy, C., Tang, X.: Deep cascaded bi-network for face hallucination. In: Computer Vision and Pattern Recognition, pp. 614–630 (2016)Google Scholar
  8. 8.
    Liu, R., Wang, Y., Baba, T., Masumoto, D., Nagata, S.: SVM-based active feedback in image retrieval using clustering and unlabeled data. Pattern Recogn. 41(8), 2645–2655 (2008)CrossRefzbMATHGoogle Scholar
  9. 9.
    Xiang, Y., Yang, H., Li, Y., Chen, J.: A new SVM-based active feedback scheme for image retrieval. Eng. Appl. Artif. Intell. 37, 43–53 (2015)CrossRefGoogle Scholar
  10. 10.
    Wang, B., Zhang, X., Li, N.: Relevance feedback technique for content-based image retrieval using neural network learning. In: International Conference on Machine Learning and Cybernetics (2006)Google Scholar
  11. 11.
    Fu, H., Qiu, G.: Fast semantic image retrieval based on random forest. In: International Conference on Multimedia, pp. 909–912 (2012)Google Scholar
  12. 12.
    Sun, Y., Chen, Y., Wang, X., Tang, X.: Deep learning face representation by joint identification-verification. In: Advances in Neural Information Processing Systems, pp. 1988–1996 (2014)Google Scholar
  13. 13.
    Sun, Y., Wang, X., Tang, X.: Deep learning face representation from predicting 10,000 classes. In: Computer Vision and Pattern Recognition, pp. 1891–1898 (2014)Google Scholar
  14. 14.
    Ruthven, I., Lalmas, M.: A survey on the use of relevance feedback for information access systems. Knowl. Eng. Rev. 18(2), 95–145 (2003)CrossRefGoogle Scholar
  15. 15.
    Li, H., Toyoura, M., Shimizu, K., Yang, W., Mao, X.: Retrieval of clothing images based on relevance feedback with focus on collar designs. Vis. Comput. 32(10), 1351–1363 (2016)CrossRefGoogle Scholar
  16. 16.
    Silva, A., Falcao, A., Magalhaes, L.: Active learning paradigms for CRIR systems based on optimum-path forest classification. J. WSCG 18(1–3), 73–80 (2010)Google Scholar
  17. 17.
    Papa, J., Falca, A.: Optimum-path forest: a novel and powerful framework for supervised graph-based pattern recognition techniques, pp. 41–48. Institute of Computing University of Campinas (2010)Google Scholar
  18. 18.
    Papa, J., Falcao, A., Suzuki, C.: Supervised pattern classification based on optimum-path forest. Imaging Syst. Technol. 19(2), 120–131 (2009)CrossRefGoogle Scholar
  19. 19.
    Li, H., Liu, G., Ngan, K.: Guided face cartoon synthesis. IEEE Trans. Multimedia 13(6), 1230–1239 (2001)CrossRefGoogle Scholar
  20. 20.
    Gao, W., Gao, B., Shan, S., Chen, X., Zhou, D., Zhang, X., Zhao, D.: The CAS-PEAL large-scale Chinese face database and baseline evaluations. IEEE Trans. Syst. Man Cybern. Syst. 38(1), 149–161 (2008)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  • Caie Xu
    • 1
  • Shota Fushimi
    • 1
  • Masahiro Toyoura
    • 1
  • Jiayi Xu
    • 2
  • Xiaoyang Mao
    • 1
    Email author
  1. 1.University of YamanashiYamanashiJapan
  2. 2.Hangzhou Dianzi UniversityHangzhouChina

Personalised recommendations