Abstract
Face de-occlusion is essential to improve the accuracy of face-related tasks. However, most existing methods only focus on single occlusion scenarios, rendering them sub-optimal for multiple occlusions. To alleviate this problem, we propose a novel framework for face de-occlusion called FRNet, which is based on feature reconstruction. The proposed FRNet can automatically detect and remove single or multiple occlusions through the predict-extract-inpaint approach, making it a universal solution to deal with multiple occlusions. In this paper, we propose a two-stage occlusion extractor and a two-stage face generator. The former utilizes the predicted occlusion positions to get coarse occlusion masks which are subsequently fine-tuned by the refinement module to tackle complex occlusion scenarios in the real world. The latter utilizes the predicted face structures to reconstruct global structures, and then uses information from neighboring areas and corresponding features to refine important areas, so as to address the issues of structural deficiencies and feature disharmony in the generated face images. We also introduce a gender-consistency loss and an identity loss to improve the attribute recovery accuracy of images. Furthermore, to address the limitations of existing datasets for face de-occlusion, we introduce a new synthetic face dataset including both single and multiple occlusions, which effectively facilitates the model training. Extensive experimental results demonstrate the superiority of the proposed FRNet compared to state-of-the-art methods.
This work was supported in part by the National Natural Science Foundation of China under Grant 62172212, in part by the Natural Science Foundation of Jiangsu Province under Grant BK20230031.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Barnes, C., et al.: The patchmatch randomized matching algorithm for image manipulation. Commun. ACM 54(11), 103–110 (2011)
Dong, J., et al.: Occlusion-aware GAN for face de-occlusion in the wild. In: ICME, pp. 1–6. IEEE (2020)
Guo, J., Zhu, X., Lei, Z., Li, S.Z.: Face synthesis for eyeglass-robust face recognition. In: Zhou, J., et al. (eds.) CCBR 2018. LNCS, vol. 10996, pp. 275–284. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-97909-0_30
Guo, X., et al.: Image inpainting via conditional texture and structure dual generation. In: ICCV, pp. 14134–14143 (2021)
He, K., et al.: Computing nearest-neighbor fields via propagation-assisted KD-trees. In: CVPR, pp. 111–118. IEEE (2012)
Hertz, A., et al.: Blind visual motif removal from a single image. In: CVPR, pp. 6858–6867 (2019)
Hu, B., et al.: Unsupervised eyeglasses removal in the wild. TCYB 51(9), 4373–4385 (2020)
Ju, Y.J., et al.: Complete face recovery GAN: unsupervised joint face rotation and de-occlusion from a single-view image. In: WACV, pp. 3711–3721 (2022)
Karras, T., et al.: A style-based generator architecture for generative adversarial networks. In: CVPR, pp. 4401–4410 (2019)
Li, X., et al.: MISF: multi-level interactive siamese filtering for high-fidelity image inpainting. In: CVPR, pp. 1869–1878 (2022)
Li, X., et al.: Image-to-image translation via hierarchical style disentanglement. In: CVPR, pp. 8639–8648 (2021)
Liang, J., et al.: Visible watermark removal via self-calibrated localization and background refinement. In: ACM MM, pp. 4426–4434 (2021)
Liu, M., et al.: STGAN: a unified selective transfer network for arbitrary image attribute editing. In: CVPR, pp. 3673–3682 (2019)
Miyato, T., et al.: Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 (2018)
Modak, G., et al.: A deep learning framework to reconstruct face under mask. In: CDMA, pp. 200–205. IEEE (2022)
Nazeri, K., et al.: Edgeconnect: structure guided image inpainting using edge prediction. In: ICCV (2019)
Pathak, D., et al.: Context encoders: feature learning by inpainting. In: CVPR, pp. 2536–2544 (2016)
Peng, J., et al.: PP-LiteSeg: a superior real-time semantic segmentation model. arXiv preprint arXiv:2204.02681 (2022)
Quan, W., et al.: Image inpainting with local and global refinement. TIP 31, 2405–2420 (2022)
Wang, N., et al.: Dynamic selection network for image inpainting. TIP 30, 1784–1798 (2021)
Wang, T.C., et al.: High-resolution image synthesis and semantic manipulation with conditional GANs. In: CVPR, pp. 8798–8807 (2018)
Wang, Z., et al.: Masked face recognition dataset and application. arXiv preprint arXiv:2003.09093 (2020)
Yu, Y., et al.: Wavefill: a wavelet-based generation network for image inpainting. In: ICCV, pp. 14114–14123 (2021)
Zeng, Y., et al.: Aggregated contextual transformations for high-resolution image inpainting. TVCG (2022)
Zhu, J.Y., et al.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: ICCV, pp. 2223–2232 (2017)
Zhu, M., et al.: Image inpainting by end-to-end cascaded refinement with mask awareness. TIP 30, 4855–4866 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Du, S., Zhang, L. (2024). FRNet: Improving Face De-occlusion via Feature Reconstruction. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14435. Springer, Singapore. https://doi.org/10.1007/978-981-99-8552-4_25
Download citation
DOI: https://doi.org/10.1007/978-981-99-8552-4_25
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8551-7
Online ISBN: 978-981-99-8552-4
eBook Packages: Computer ScienceComputer Science (R0)