Abstract
In the product design life-cycle, the conceptual design stage is an important and time-consuming phase where designers ideate in the form of sketches and 3D renderings of a product. Specifically, with 3D renderings, the choice of material texture and color is an important aspect that is often critiqued by designers because it impacts the product’s visual aesthetic and the impression it evokes in the customer when first viewing the product; thus, making material selection and the conceptual design stage, overall, challenging. In this study, we turn to deep texture synthesis for generating material textures and propose a novel method, TextureAda. TextureAda creates high-fidelity textures by performing adaptive instance normalization between multiple layers of a texture generator and a pre-trained image encoder. Our experiments show that our method beats previous methods in texture synthesis visually and quantitatively. Lastly, we show how TextureAda can be applied for ideation in product design conceptualization by material texturing 3D models of furniture.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ashby, M., Johnson, K.: The art of materials selection. Mater. Today 6(12), 24–35 (2003). https://doi.org/10.1016/S1369-7021(03)01223-9
Bergmann, U., Jetchev, N., Vollgraf, R.: Learning texture manifolds with the periodic spatial GAN. In: 34th International Conference on Machine Learning, ICML 2017, vol. 1, pp. 722–730 (2017)
Bi, S., Kalantari, N.K., Ramamoorthi, R.: Patch-based optimization for image-based texture mapping. ACM Trans. Graph. 36(4) (2017). https://doi.org/10.1145/3072959.3073610
Chang, A.X., et al.: ShapeNet: an information-rich 3D model repository. arXiv preprint arXiv:1512.03012 (2015)
Chen, W., et al.: Learning to predict 3D objects with an interpolation-based differentiable renderer, pp. 1–12 (2019). https://nv-tlabs.github.io/DIB-R/
Chen, Y., Chen, R., Lei, J., Zhang, Y., Jia, K.: TANGO: text-driven photorealistic and robust 3D stylization via lighting decomposition. In: NeurIPS, pp. 1–13 (2022). http://arxiv.org/abs/2210.11277
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., Vedaldi, A.: Describing textures in the wild. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3606–3613 (2014). https://doi.org/10.1109/CVPR.2014.461
Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2001 (August), pp. 341–346 (2001). https://doi.org/10.1145/383259.383296
Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: Proceedings of the IEEE International Conference on Computer Vision 2(September), 1033–1038 (1999). https://doi.org/10.1109/iccv.1999.790383
Gatys, L., Ecker, A.S., Bethge, M.: Texture synthesis using convolutional neural networks. Advances in Neural Information Processing Systems, vol. 28 (2015)
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Houdard, A., Leclaire, A., Papadakis, N., Rabin, J.: Wasserstein generative models for patch-based texture synthesis, pp. 269–280 (2021)
Houdard, A., Leclaire, A., Papadakis, N., Rabin, J.: A generative model for texture synthesis based on optimal transport between feature distributions. J. Math. Imaging Vis. (2022). https://doi.org/10.1007/s10851-022-01108-9
Hu, R., Su, X., Chen, X., Van Kaick, O., Huang, H.: Photo-to-shape material transfer for diverse structures. ACM Trans. Graph. 41(4) (2022). https://doi.org/10.1145/3528223.3530088
Huang, X., Belongie, S.J.: Arbitrary style transfer in real-time with adaptive instance normalization. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1510–1519 (2017)
Jetchev, N.: ClipMatrix: Text-controlled creation of 3D textured meshes (2021). http://arxiv.org/abs/2109.12922
Jin, B., Tian, B., Zhao, H., Zhou, G.: Language-guided semantic style transfer of 3D indoor scenes, pp. 11–17 (2022). https://doi.org/10.1145/3552482.3556555
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution, pp. 694–711 (2016)
Kato, H., Ushiku, Y., Harada, T.: Neural 3D mesh renderer, pp. 3907–3916 (2018). https://doi.org/10.1109/CVPR.2018.00411
Kingma, D.P., Ba, J.L.: Adam: a method for stochastic optimization. In: 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, pp. 1–15 (2015)
Kwatra, V., Schödl, A., Essa, I., Turk, G., Bobick, A.: Graphcut textures: image and video synthesis using graph cuts. ACM Trans. Graph. 22(3), 277–286 (2003). https://doi.org/10.1145/882262.882264
Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., Yang, M.H.: Diversified texture synthesis with feed-forward networks (2017)
Lin, J., Sharma, G., Pappas, T.N.: Towards universal texture synthesis by combining Texton broadcasting with noise injection in StyleGAN-2 (2022). http://arxiv.org/abs/2203.04221
Liu, S., Chen, W., Li, T., Li, H.: Soft rasterizer: a differentiable renderer for image-based 3D reasoning 2019-Octob, 7707–7716 (2019). https://doi.org/10.1109/ICCV.2019.00780
Michel, O., Bar-On, R., Liu, R., Benaim, S., Hanocka, R.: Text2Mesh: text-driven neural stylization for meshes, 13492–13502 (2022). https://arxiv.org/abs/2112.03221
Milton, A., Rodgers, P.: Product design. Laurence King Publishing (2011)
Mir, A., Alldieck, T., Pons-Moll, G.: Learning to transfer texture from clothing images to 3D humans. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7021–7032 (2020). https://doi.org/10.1109/CVPR42600.2020.00705
Mordvintsev, A., Niklasson, E., Randazzo, E.: Texture generation with neural cellular automata (2021). http://arxiv.org/abs/2105.07299
Perroni-Scharf, M., Sunkavalli, K., Eisenmann, J., Hold-Geoffroy, Y.: Material swapping for 3D scenes using a learnt material similarity measure. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 2033–2042 (2022). https://doi.org/10.1109/CVPRW56347.2022.00221
Radford, A., et al.: Learning transferable visual models from natural language supervision (2021). http://arxiv.org/abs/2103.00020
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. ArXiv abs/2204.06125 (2022)
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models, 10674–10685 (2022). https://doi.org/10.1109/cvpr52688.2022.01042
Saharia, C., et al.: Photorealistic text-to-image diffusion models with deep language understanding (2022). http://arxiv.org/abs/2205.11487
Schuhmann, C., et al.: LAION-400M: open dataset of CLIP-Filtered 400 million image-text Pairs, 1–5 (2021). http://arxiv.org/abs/2111.02114
Shaham, T.R., Dekel, T., Michaeli, T.: SinGAN: learning a generative model from a single natural image 2019-Octob, 4569–4579 (2019). https://doi.org/10.1109/ICCV.2019.00467
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2015)
Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)
Ulyanov, D., Lebedev, V., Vedaldi, A., Lempitsky, V.S.: Texture networks: feed-forward synthesis of textures and stylized images (2016)
Virtusio, J.J., Tan, D.S., Cheng, W.H., Tanveer, M., Hua, K.L.: Enabling artistic control over pattern density and stroke strength. In: IEEE Transactions on Multimedia (2020)
Wang, T.Y., Su, H., Huang, Q., Huang, J., Guibas, L., Mitra, N.J.: Unsupervised texture transfer from images to model collections. ACM Trans. Graph. 35(6) (2016). https://doi.org/10.1145/2980179.2982404
Wei, L.Y., Levoy, M.: Fast texture synthesis using tree-structured vector quantization. In: Proceedings of the ACM SIGGRAPH Conference on Computer Graphics, pp. 479–488 (2000). https://doi.org/10.1145/344779.345009
Yeh, Y.Y., et al.: PhotoScene: photorealistic material and lighting transfer for indoor scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18541–18550 (2022). https://doi.org/10.1109/cvpr52688.2022.01801
Yin, K., Gao, J., Shugrina, M., Khamis, S., Fidler, S.: 3DStyleNet: creating 3D shapes with geometric and texture style variations. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 12436–12445 (2021). https://doi.org/10.1109/ICCV48922.2021.01223
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Gallega, R.W., Azcarraga, A., Sumi, Y. (2023). TextureAda: Deep 3D Texture Transfer for Ideation in Product Design Conceptualization. In: Degen, H., Ntoa, S. (eds) Artificial Intelligence in HCI. HCII 2023. Lecture Notes in Computer Science(), vol 14050. Springer, Cham. https://doi.org/10.1007/978-3-031-35891-3_30
Download citation
DOI: https://doi.org/10.1007/978-3-031-35891-3_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-35890-6
Online ISBN: 978-3-031-35891-3
eBook Packages: Computer ScienceComputer Science (R0)