Abstract
As image generation techniques mature, there is a growing interest in explainable representations that are easy to understand and intuitive to manipulate. In this work, we turn to co-occurrence statistics, which have long been used for texture analysis, to learn a controllable texture synthesis model. We propose a fully convolutional generative adversarial network, conditioned locally on co-occurrence statistics, to generate arbitrarily large images while having local, interpretable control over texture appearance. To encourage fidelity to the input condition, we introduce a novel differentiable co-occurrence loss that is integrated seamlessly into our framework in an end-to-end fashion. We demonstrate that our solution offers a stable, intuitive, and interpretable latent representation for texture synthesis, which can be used to generate smooth texture morphs between different textures. We further show an interactive texture tool that allows a user to adjust local characteristics of the synthesized texture by directly using the co-occurrence values.
Article PDF
Similar content being viewed by others
References
Radford, A.; Metz, L.; Chintala, S. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
Bau, D.; Zhu, J. Y.; Strobelt, H.; Zhou, B. L.; Tenenbaum, J. B.; Freeman, W. T.; Torralba, A. Visualizing and understanding generative adversarial networks. arXiv preprint arXiv:1901.09887, 2019.
Shen, Y. J.; Gu, J. J.; Tang, X. O.; Zhou, B. L. Interpreting the latent space of GANs for semantic face editing. arXiv preprint arXiv:1907.10786, 2019.
Julesz, B. Visual pattern discrimination. IEEE Transactions on Information Theory Vol. 8, No. 2, 84–92, 1962.
Haralick, R. M.; Shanmugam, K.; Dinstein, I. Textural features for image classification. IEEE Transactions on Systems, Man, and Cybernetics Vol. SMC-3, No. 6, 610–621, 1973.
Isola, P.; Zoran, D.; Krishnan, D.; Adelson, E. H. Crisp boundary detection using pointwise mutual information. In: Computer Vision-ECCV 2014. Lecture Notes in Computer Science, Vol. 8691. Fleet, D.; Pajdla, T.; Schiele, B.; Tuytelaars, T. Eds. Springer Cham, 799–814, 2014.
Heeger, D. J.; Bergen, J. R. Pyramid-based texture analysis/synthesis. In: Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, 229–238, 1995.
De Bonet, J. S. Multiresolution sampling procedure for analysis and synthesis of texture images. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, 361–368, 1997.
Portilla, J.; Simoncelli, E. P. A parametric texture model based on joint statistics of complex wavelet coefficients. International Journal of Computer Vision Vol. 40, No. 1, 49–70, 2000.
Efros, A. A.; Freeman, W. T. Image quilting for texture synthesis and transfer. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, 341–346, 2001.
Kwatra, V.; Schödl, A.; Essa, I.; Turk, G.; Bobick, A. Graphcut textures. ACM Transactions on Graphics Vol. 22, No. 3, 277–286, 2003.
Kwatra, V.; Essa, I.; Bobick, A.; Kwatra, N. Texture optimization for example-based synthesis. ACM Transactions on Graphics Vol. 24, No. 3, 795–802, 2005.
Wexler, Y.; Shechtman, E.; Irani, M. Space-time completion of video. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 29, No. 3, 463–476, 2007.
Simakov, D.; Caspi, Y.; Shechtman, E.; Irani, M. Summarizing visual data using bidirectional similarity. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1–8, 2008.
Barnes, C.; Zhang, F. L. A survey of the state-of-the-art in patch-based synthesis. Computational Visual Media Vol. 3, No. 1, 3–20, 2017.
Matusik, W.; Zwicker, M.; Durand, F. Texture design using a simplicial complex of morphable textures. ACM Transactions on Graphics Vol. 24, No. 3, 787–794, 2005.
Rabin, J.; Peyré, G.; Delon, J.; Bernot, M. Wasserstein barycenter and its application to texture mixing. In: Scale Space and Variational Methods in Computer Vision. Lecture Notes in Computer Science, Vol. 6667. Bruckstein, A. M.; ter Haar Romeny, B. M.; Bronstein, A. M.; Bronstein, M. M. Eds. Springer Berlin Heidelberg, 435–446, 2012.
Darabi, S.; Shechtman, E.; Barnes, C.; Goldman, D. B.; Sen, P. Image melding: Combining inconsistent images using patch-based synthesis. ACM Transactions on Graphics Vol. 31, No. 4, Article No. 82, 2012.
Gatys, L. A.; Ecker, A. S.; Bethge, M. Texture synthesis using convolutional neural networks. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, 262–270, 2015.
Ulyanov, D.; Lebedev, V.; Vedaldi, A.; Lempitsky, V. S. Texture networks: Feed-forward synthesis of textures and stylized images. In: Proceedings of the International Conference on Machine Learning, 1349–1357, 2016.
Sendik, O.; Cohen-Or, D. Deep correlations for texture synthesis. ACM Transactions on Graphics Vol. 36, No. 5, Article No. 161, 2017.
Li, C.; Wand, M. Precomputed real-time texture synthesis with Markovian generative adversarial networks. In: Computer Vision-ECCV 2016. Lecture Notes in Computer Science, Vol. 9907. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 702–716, 2016.
Gang, L.; Gousseau, Y.; Xia, G. S. Texture synthesis through convolutional neural networks and spectrum constraints. In: Proceedings of the 23rd International Conference on Pattern Recognition, 3234–3239, 2016.
Li, Y. J.; Fang, C.; Yang, J. M.; Wang, Z. W.; Lu, X.; Yang, M. H. Diversified texture synthesis with feed-forward networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 266–274, 2017.
Zhou, Y.; Zhu, Z.; Bai, X.; Lischinski, D.; Cohen-Or, D.; Huang, H. Non-stationary texture synthesis by adversarial expansion. ACM Transactions on Graphics Vol. 37, No. 4, Article No. 49, 2018.
Frühstück, A.; Alhashim, I.; Wonka, P. TileGAN: Synthesis of large-scale non-homogeneous textures. ACM Transactions on Graphics Vol. 38, No. 4, Article No. 58, 2019.
Jetchev, N.; Bergmann, U.; Vollgraf, R. Texture synthesis with spatial generative adversarial networks. In: Proceedings of the Workshop on Adversarial Training, 2016.
Bergmann, U.; Jetchev, N.; Vollgraf, R. Learning texture manifolds with the periodic spatial GAN. arXiv preprint arXiv:1705.06566, 2017.
Yu, N.; Barnes, C.; Shechtman, E.; Amirghodsi, S.; Lukác, M. Texture mixer: A network for controllable synthesis and interpolation of texture. arXiv preprint arXiv:1901.03447, 2019.
Yellott, J. I. Implications of triple correlation uniqueness for texture statistics and the Julesz conjecture. Journal of the Optical Society of America A Vol. 10, No. 5, 777–793, 1993.
Jevnisek, R. J.; Avidan, S. Co-occurrence filter. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3184–3192, 2017.
Kat, R.; Jevnisek, R.; Avidan, S. Matching pixels using co-occurrence statistics. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1751–1759, 2018.
Gulrajani, I.; Ahmed, F.; Arjovsky, M.; Dumoulin, V.; Courville, A. C. Improved training of Wasserstein GANs. In: Proceedings of the Annual Conference on Neural Information Processing Systems, 2017.
Cimpoi, M.; Maji, S.; Kokkinos, I.; Mohamed, S.; Vedaldi, A. Describing textures in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3606–3613, 2014.
Bellini, R.; Kleiman, Y.; Cohen-Or, D. Time-varying weathering in texture space. ACM Transactions on Graphics Vol. 35, No. 4, Article No. 141, 2016.
Wu, F. Z.; Dong, W. M.; Kong, Y.; Mei, X.; Yan, D. M.; Zhang, X. P.; Paul, J.-C. Feature-aware natural texture synthesis. The Visual Computer Vol. 32, No. 1, 43–55, 2016.
Kaspar, A.; Neubert, B.; Lischinski, D.; Pauly, M.; Kopf, J. Self tuning texture optimization. Computer Graphics Forum Vol. 34, No. 2, 349–359, 2015.
Author information
Authors and Affiliations
Corresponding author
Additional information
Anna Darzi received her M.Sc. degree in electrical engineering from Tel Aviv University in 2019, researcher in Avidan’s lab. She completed her B.Sc. degree in electrical engineering at the Technion in 2016. She is currently working as an algorithm developer in computer vision, deep learning, and object detection.
Itai Lang is a Ph.D. researcher at Tel Aviv University, working with Avidan. In addition, he is a senior algorithm engineer at Samsung Israeli Research Center. He received his B.Sc. degree in electrical engineering and his B.A. degree in physics from the Technion, Israel, in 2005. In 2013, he completed his M.Sc. degree in electrical engineering at Tel Aviv University. His research interests include computer vision and learning methods for 3D geometry and images.
Ashutosh Taklikar is a graduate researcher at Tel Aviv University working with Avidan in the field of computer vision. He completed his B.Sc. degree in electrical engineering at Tel Aviv University in 2019, where he is currently pursuing his M.Sc. degree. His research mainly focuses on image generation and super-resolution tasks.
Hadar Averbuch-Elor is a postdoctoral researcher at Cornell-Tech working with Snavely as part of the Cornell Graphics and Vision Lab. She completed her Ph.D. degree in electrical engineering at Tel Aviv University in 2017. She received her B.Sc. degree in electrical engineering from the Technion in 2012. Her research focuses on modeling and manipulating visual concepts by combining pixels with more structured modalities, including natural language and 3D geometry.
Shai Avidan received his Ph.D. degree from the School of Computer Science, Hebrew University, Jerusalem, Israel, in 1999. He is currently a professor in the Faculty of Engineering, Tel Aviv University. In between, he worked for Adobe, Mitsubishi Electric Research Labs, MobilEye, and Microsoft Research. He has published extensively in the fields of object tracking in video and 3D object modeling from images. He is also interested in Internet vision applications such as privacy-preserving image analysis, distributed algorithms for image analysis, and image retargeting.
Supplementary Material
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduc-tion in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Darzi, A., Lang, I., Taklikar, A. et al. Co-occurrence based texture synthesis. Comp. Visual Media 8, 289–302 (2022). https://doi.org/10.1007/s41095-021-0243-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-021-0243-7