Abstract
Convolutional Neural Networks have revolutionized vision applications. There are image domains and representations, however, that cannot be handled by standard CNNs (e.g., spherical images, superpixels). Such data are usually processed using networks and algorithms specialized for each type. In this work, we show that it may not always be necessary to use specialized neural networks to operate on such spaces. Instead, we introduce a new structured graph convolution operator that can copy 2D convolution weights, transferring the capabilities of already trained traditional CNNs to our new graph network. This network can then operate on any data that can be represented as a positional graph. By converting non-rectilinear data to a graph, we can apply these convolutions on these irregular image domains without requiring training on large domain-specific datasets. Results of transferring pre-trained image networks for segmentation, stylization, and depth prediction are demonstrated for a variety of such data forms.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Süsstrunk, S.: SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2282 (2012). https://doi.org/10.1109/TPAMI.2012.120
Armeni, I., Sax, A., Zamir, A.R., Savarese, S.: Joint 2D-3D-Semantic Data for Indoor Scene Understanding. arXiv:1702.01105 (2017)
Ashikhmin, M.: Synthesizing natural textures. In: Symposium on Interactive 3D Graphics, pp. 217–226 (2001)
Bronstein, M.M., Bruna, J., Cohen, T., Velickovic, P.: Geometric deep learning: grids, groups, graphs, geodesics, and gauges. arXiv:2104.13478 (2021)
Chou, S.H., Sun, C., Wen-Yen, C., Hsu, W.T., Sun, M., Fu, J.: 360-indoor: towards learning real-world objects in 360\(^\circ \) indoor equirectangular images, pp. 834–842 (2020). https://doi.org/10.1109/WACV45572.2020.9093262
Defferrard, M., Bresson, X., Vandergheynst, P.: Convolutional neural networks on graphs with fast localized spectral filtering. In: Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 29. Curran Associates, Inc. (2016)
Eder, M., Shvets, M., Lim, J., Frahm, J.M.: Tangent images for mitigating spherical distortion. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12426–12434. IEEE (2020)
Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, pp. 341–346 (2001)
Fey, M., Lenssen, J.E.: Fast graph representation learning with PyTorch Geometric. In: ICLR Workshop on Representation Learning on Graphs and Manifolds (2019)
Fey, M., Lenssen, J.E., Weichert, F., Müller, H.: SplineCNN: fast geometric deep learning with continuous B-spline kernels. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 869–877. IEEE (2018)
Fišer, J., et al.: StyLit: illumination-guided example-based stylization of 3D renderings. ACM Trans. Graph. 35(4), 92:1–92:11 (2016). https://doi.org/10.1145/2897824.2925948
Frühstück, A., Alhashim, I., Wonka, P.: TileGAN: synthesis of large-scale non-homogeneous textures. ACM Trans. Graph. (TOG) 38(4), 1–11 (2019)
Gatys, L., Ecker, A.S., Bethge, M.: Texture synthesis using convolutional neural networks. Adv. Neural. Inf. Process. Syst. 28, 262–270 (2015)
Godard, C., Mac Aodha, O., Brostow, G.J.: Unsupervised monocular depth estimation with left-right consistency. In: Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)
Jampani, V., Sun, D., Liu, M.-Y., Yang, M.-H., Kautz, J.: Superpixel sampling networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 363–380. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_22
Jiang, C.M., Huang, J., Kashinath, K., Marcus, P., Niessner, M.: Spherical CNNs on unstructured grids. In: International Conference on Learning Representations (ICLR) (2019)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, 24–26 April 2017, Conference Track Proceedings (2017)
Krizhevsky, A., Nair, V., Hinton, G.: CIFAR-10 (Canadian institute for advanced research). http://www.cs.toronto.edu/~kriz/cifar.html
Li, X., Liu, S., Kautz, J., Yang, M.H.: Learning linear transformations for fast arbitrary style transfer. In: Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2019)
Li, Z., Yang, W., Peng, S., Liu, F.: A survey of convolutional neural networks: analysis, applications, and prospects (2020)
Lin, Q., Zhong, W., Lu, J.: Deep superpixel cut for unsupervised image segmentation. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 8870–8876 (2021)
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2015)
Lu, Y., et al.: Taskology: utilizing task relations at scale. In: Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2021)
Mao, J., Wang, X., Li, H.: Interpolated convolutional networks for 3D point cloud understanding. In: International Conference on Computer Vision (ICCV), pp. 1578–1587. IEEE (2019)
Monodepth. https://github.com/OniroAI/MonoDepth-PyTorch
Morris, C., et al.: Weisfeiler and leman go neural: higher-order graph neural networks. In: AAAI, vol. 33, no. 01, pp. 4602–4609 (2019)
PyTorch, Torchvision Models. https://pytorch.org/vision/stable/models.html
Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: deep hierarchical feature learning on point sets in a metric space. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 5105–5114. Curran Associates Inc., Red Hook (2017)
Ruder, M., Dosovitskiy, A., Brox, T.: Artistic style transfer for videos and spherical images. Int. J. Comput. Vision 126(11), 1199–1219 (2018)
Shi, W., Qiao, Y.: Fast texture synthesis via pseudo optimizer. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5498–5507. IEEE (2020)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (ICLR) (2015)
Snelgrove, X.: High-resolution multi-scale neural texture synthesis. In: SIGGRAPH Asia 2017 Technical Briefs, pp. 1–4 (2017)
Stutz, D., Hermans, A., Leibe, B.: Superpixels: an evaluation of the state-of-the-art. Comput. Vis. Image Underst. 166, 1–27 (2018). https://doi.org/10.1016/j.cviu.2017.03.007
Sýkora, D., et al.: StyleBlit: fast example-based stylization with local guidance. Comput. Graph. Forum 38(2), 83–91 (2019)
Tateno, K., Navab, N., Tombari, F.: Distortion-aware convolutional filters for dense prediction in panoramic images. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11220, pp. 732–750. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01270-0_43
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., Bengio, Y.: Graph attention networks. In: International Conference on Learning Representations (ICLR) (2018)
Verelst, T., Blaschko, M.B., Berman, M.: Generating superpixels using deep image representations. arXiv:1903.04586 (2019)
Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M.: Dynamic graph CNN for learning on point clouds. ACM Trans. Graph. 38(5), 1–12 (2019)
Wei, L.Y., Levoy, M.: Fast texture synthesis using tree-structured vector quantization. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 479–488 (2000)
Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., Yu, P.S.: A comprehensive survey on graph neural networks. IEEE Trans. Neural Netw. Learn. Syst. 1–21 (2020). https://doi.org/10.1109/tnnls.2020.2978386
Xu, M., Ding, R., Zhao, H., Qi, X.: Paconv: position adaptive convolution with dynamic kernel assembling on point clouds. In: Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2021)
Yang, F., Sun, Q., Jin, H., Zhou, Z.: Superpixel segmentation with fully convolutional networks. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13961–13970 (2020)
Yin, K., Gao, J., Shugrina, M., Khamis, S., Fidler, S.: 3DStyleNet: creating 3D shapes with geometric and texture style variations. In: International Conference on Computer Vision (ICCV). IEEE (2021)
Zamir, A.R., Sax, A., Shen, W.B., Guibas, L., Malik, J., Savarese, S.: Taskonomy: disentangling task transfer learning. In: Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2018)
Zhang, C., Liwicki, S., Smith, W., Cipolla, R.: Orientation-aware semantic segmentation on icosahedron spheres. In: International Conference on Computer Vision (ICCV). IEEE (2019)
Zhao, G., Ge, W., Yu, Y.: GraphFPN: graph feature pyramid network for object detection. In: International Conference on Computer Vision (ICCV), pp. 2763–2772. IEEE (2021)
Zhou, H., Feng, Y., Fang, M., Wei, M., Qin, J., Lu, T.: Adaptive graph convolution for point cloud analysis. In: International Conference on Computer Vision (ICCV), pp. 4945–4954 (2021)
Zhou, Y., Zhu, Z., Bai, X., Lischinski, D., Cohen-Or, D., Huang, H.: Non-stationary texture synthesis by adversarial expansion. ACM Trans. Graph. 37(4) (2018). https://doi.org/10.1145/3197517.3201285
Zhuang, F., et al.: A comprehensive survey on transfer learning. Proc. IEEE 109, 43–76 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hart, D., Whitney, M., Morse, B. (2022). SelectionConv: Convolutional Neural Networks for Non-rectilinear Image Data. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13667. Springer, Cham. https://doi.org/10.1007/978-3-031-20071-7_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-20071-7_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20070-0
Online ISBN: 978-3-031-20071-7
eBook Packages: Computer ScienceComputer Science (R0)