Abstract
Current face image retrieval methods achieve impressive results, but lack efficient ways to refine the search, particularly for geometric face attributes. Users cannot easily find faces with slightly more furrowed brows or specific leftward pose shifts, for example. This creates significant problems, especially for mobile users with small screens, low bandwidth, and awkward keyboard settings. To address this problem, we propose a new face search technique based on shape manipulation that is complementary to current search engines. Users drag one or a small number of contour points, like the bottom of the chin or the corner of an eyebrow, to search for faces similar in shape to the current face, but with updated geometric attributes specific to their edits. For example, the user can drag a mouth corner to find faces with wider smiles, or the tip of the nose to find faces with a specific pose. As part of our system, we propose (1) a novel confidence score for face alignment results that automatically constructs a contour-aligned face database with reasonable alignment accuracy, (2) a simple and straightforward extension of PCA with missing data to tensor analysis, and (3) a new regularized tensor model to compute shape feature vectors for each aligned face, all built upon previous work. Despite the powerful algorithms used in this application, we achieve real-time performance on Apple devices. To the best of our knowledge, our system demonstrates the first face retrieval approach based chiefly on shape manipulation. We show compelling results on a sizeable database of over 10,000 face images captured in uncontrolled environments.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
When selecting a query-target image pair with a similar expression but different poses, we first randomly pick the target image, then remove all the images whose Euclidean distance is among the nearest one-third. In the remaining images, we select the one whose expression is most similar to the target image by comparing \(\mathbf {c}_\mathtt {pose}\) and use it as the query image. We can select a pair with a similar pose but different expressions in a similar way. A pair with both dissimilar pose and dissimilar expressions can be selected randomly.
References
Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P.N., Nayar, S.K.: Face swapping: Automatically replacing faces in photographs. SIGGRAPH (2008)
Kemelmacher-Shlizerman, I., Sankar, A., Shechtman, E., Seitz, S.M.: Being John Malkovich. ECCV (2010)
Kumar, N., Belhumeur, P., Nayar, S.: Face tracer: A search engine for large collections of images with faces. ECCV (2008)
Wu, Z., Ke, Q., Sun, J., Shum, H.Y.: Scalable face image retrieval with identity-based quantization and multi-reference re-ranking. CVPR (2010)
Goldman, D.B., Gonterman, C., Curless, B., Salesin, D., Seitz, S.M.: Video Annotation, Navigation, and Composition. UIST (2008)
Lau, M., Chai, J., Xu, Y.Q., Shum, H.: Face poser: Interactive modeling of 3D facial expressions using facial priors. SIGGRAPH (2010)
Zhang, L., Snavely, N., Curless, B., Seitz, S.M.: Spacetime faces: high-resolution capture for modeling and animation. SIGGRAPH, pp. 548–558 (2004)
Sumner, R.W., Zwicker, M., Gotsman, C., Popović, J.: Mesh-based inverse kinematics. SIGGRAPH (2005)
Gu, L., Kanade, T.: A generative shape regularization model for robust face alignment. ECCV (2008)
Liu, X., Tong, Y., Weeler, F.W., Tu, P.H.: Facial contour labeling via congealing. ECCV (2010)
Lathauwer, L.D., Moor, B.D., Vandewalle, J.: On the best rank-1 and rank-(R1, R2,..., Rn) approximation of higher-order tensors. SIAM J. Matrix Anal. Appl. 21(4), 1324–1342 (2000)
Vasilescu, M.A.O., Terzopoulos, D.: TensorTextures: Multilinear image-based rendering. ACM Trans. Gr. 23(3): 336–342. ACM, New York (2004). http://doi.acm.org/10.1145/1015706.1015725
Wang, H., Wu, Q., Shi, L., Yu, Y., Ahuja, N.: Out-of-core tensor approximation of multi-dimensional matrices of visual data. ACM Trans. Gr. 24(3): 527–535. ACM, New York (2005). http://doi.acm.org/10.1145/1073204.1073224
Vasilescu, M.A.O., Terzopoulos, D.: Multilinear analysis of image ensembles: Tensor faces. ECCV (2002)
Vlasic, D., Brand, M., Pfister, H., Popović, J.: Face transfer with multilinear models. SIGGRAPH (2005)
Tao, D., Sun, J., Wu, X., Li, X., Shen, J., Maybank, S.J., Faloutsos, C.: Probabilistic tensor analysis with akaike and bayesian information criteria. In: Ishikawa, M., Doya, K. (eds.) Neural Information Processing. Springer, Berlin, Heidelberg (2008) http://dx.doi.org/10.1007/978-3-540-69158-7_82
Saragih, J., Lucey, S., Cohn, J.: Face alignment through subspace constrained mean-shifts. CVPR, pp. 1034–1041 (2009). doi:10.1109/ICCV.2009.5459377
Tong, Y., Liu, X., Wheeler, F.W., Tu, P.: Automatic facial landmark labeling with minimal supervision. CVPR (2009)
Zhu, J., Gool, L.V., Hoi, S.C.H.: Unsupervised face alignment by robust nonrigid mapping. ICCV (2009)
Gross, R., Matthews, I., Cohn, J., Kanade, T., Baker, S.: Multi-PIE. In: Proceedings of International Conference on Automation Face Gesture Recognition. 28(5): 807–813 (2010)
Kumar, N., Berg, A.C., Belhumeur, P.N., Nayar, S.K.: Attribute and simile classifiers for face verification. ICCV (2009)
Rowley, H., Baluja, S., Kanade, T.: Rotation invariant neural network-based face detection. CVPR (1998)
Kasiński, A., Florek, A., Schmidt, A.: The PUT face database. Tech. Rep., Poznan University of Technology, Poznan, Poland (2009)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Zhang, L., Smith, B.M., Zhu, S. (2015). FaceSimile: A Mobile Application for Face Image Search Based on Interactive Shape Manipulation. In: Hua, G., Hua, XS. (eds) Mobile Cloud Visual Media Computing. Springer, Cham. https://doi.org/10.1007/978-3-319-24702-1_6
Download citation
DOI: https://doi.org/10.1007/978-3-319-24702-1_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-24700-7
Online ISBN: 978-3-319-24702-1
eBook Packages: Computer ScienceComputer Science (R0)