Abstract
The automatic colorization of anime line drawings is a challenging problem in production pipelines. Recent advances in deep neural networks have addressed this problem; however, collectingmany images of colorization targets in novel anime work before the colorization process starts leads to chicken-and-egg problems and has become an obstacle to using them in production pipelines. To overcome this obstacle, we propose a new patch-based learning method for few-shot anime-style colorization. The learning method adopts an efficient patch sampling technique with position embedding according to the characteristics of anime line drawings. We also present a continuous learning strategy that continuously updates our colorization model using new samples colorized by human artists. The advantage of our method is that it can learn our colorization model from scratch or pre-trained weights using only a few pre- and post-colorized line drawings that are created by artists in their usual colorization work. Therefore, our method can be easily incorporated within existing production pipelines. We quantitatively demonstrate that our colorizationmethod outperforms state-of-the-art methods.
![](http://media.springernature.com/lw685/springer-static/image/art%3A10.1007%2Fs41095-024-0414-4/MediaObjects/41095_2024_414_Fig1_HTML.jpg)
Article PDF
Similar content being viewed by others
Explore related subjects
Find the latest articles, discoveries, and news in related topics.Avoid common mistakes on your manuscript.
References
Kanamori, Y. Region matching with proxy ellipses for coloring hand-drawn animations. In: Proceedings of the SIGGRAPH Asia Technical Briefs, Article No. 4, 2012.
Sato, K.; Matsui, Y.; Yamasaki, T.; Aizawa, K. Reference-based manga colorization by graph correspondence using quadratic programming. In: Proceedings of the SIGGRAPH Asia Technical Briefs, Article No. 15, 2014.
Maejima, A.; Kubo, H.; Funatomi, T.; Yotsukura, T.; Nakamura, S.; Mukaigawa, Y. Graph matching based anime colorization with multiple references. In: Proceedings of the ACM SIGGRAPH Posters, Article No. 13, 2019.
Liu, S.; Wang, X.; Liu, X.; Wu, Z.; Seah, H. S. Shape correspondence for cel animation based on a shape association graph and spectral matching. Computational Visual Media Vol. 9, No. 3, 633–656, 2023.
Zhu, H.; Liu, X.; Wong, T. T.; Heng, P. A. Globally optimal toon tracking. ACM Transactions on Graphics Vol. 35, No. 4, Article No. 75, 2016.
Casey, E.; Pérez, V.; Li, Z. The animation transformer: Visual correspondence via segment matching. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 11323–11332, 2021.
Ishii, D.; Kubo, H.; Shinagawa, S.; Maejima, A.; Funatomi, T.; Nakamura, S.; Mukaigawa, Y. Confidence-aware practical anime-style colorization. In: Proceedings of the ACM SIGGRAPH Talks, Article No. 40, 2020.
Ramassamy, S.; Kubo, H.; Funatomi, T.; Ishii, D.; Maejima, A.; Nakamura, S.; Mukaigawa, Y. Preand post-processes for automatic colorization using a fully convolutional network. In: Proceedings of the SIGGRAPH Asia Posters, Article No. 70, 2018.
Texler, O.; Futschik, D.; Kucera, M.; Jamriška, O.; Sochorové, Š.; Chai, M.; Tulyakov, S.; Sýkora, D. Interactive video stylization using few-shot patch-based training. ACM Transactions on Graphics Vol. 39, No. 4, Article No. 73, 2020.
Maejima, A.; Kubo, H.; Shinagawa, S.; Funatomi, T.; Yotsukura, T.; Nakamura, S.; Mukaigawa, Y. Anime character colorization using few-shot learning. In: Proceedings of the SIGGRAPH Asia Technical Communications, Article No. 8, 2021.
Chen, S. Y.; Zhang, J. Q.; Gao, L.; He, Y.; Xia, S.; Shi, M.; Zhang, F. L. Active colorization for cartoon line drawings. IEEE Transactions on Visualization and Computer Graphics Vol. 28, No. 2, 1198–1208, 2022.
Sýkora, D.; Buriének, J.; Žéra, J. Unsupervised colorization of black-and-white cartoons. In: Proceedings of the 3rd International Symposium on Non-Photorealistic Animation and Rendering, 121–127, 2004.
Barnes, C.; Shechtman, E.; Finkelstein, A.; Goldman, D. B. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics Vol. 28, No. 3, Article No. 24, 2009.
Dang, T. D. Q.; Do, T. D.; Nguyen, A.; Pham, V.; Nguyen, Q.; Hoang, B.; Nguyen, G. Correspondence neural network for line art colorization. In: Proceedings of the ACM SIGGRAPH Posters, Article No. 44, 2020.
Hensman, P.; Aizawa, K. cGAN-based manga colorization using a single training image. In: Proceedings of the 14th IAPR International Conference on Document Analysis and Recognition, Vol. 3, 72–77, 2017.
Shi, M.; Zhang, J. Q.; Chen, S. Y.; Gao, L.; Lai, Y. K.; Zhang, F. L. Reference-based deep line art video colorization. IEEE Transactions on Visualization and Computer Graphics Vol. 29, No. 6, 2965–2979, 2023.
Li, Z.; Geng, Z.; Kang, Z.; Chen, W.; Yang, Y. Eliminating gradient conflict in reference-based lineart colorization. In: Proceedings of the European Conference on Computer Vision, 579–596, 2022.
Levin, A.; Lischinski, D.; Weiss, Y. Colorization using optimization. ACM Transactions on Graphics Vol. 23, No. 3, 689–694, 2004.
Orzan, A.; Bousseau, A.; Barla, P.; Winnemöller, H.; Thollot, J.; Salesin, D. Diffusion curves: A vector representation for smooth-shaded images. Communications of the ACM Vol. 56, No. 7, 101–109, 2013.
Qu, Y.; Wong, T. T.; Heng, P. A. Manga colorization. ACM Transactions on Graphics Vol. 25, No. 3, 1214–1220, 2006.
Zhang, R.; Zhu, J.-Y.; Isola, P.; Geng, X.; Lin, A. S.; Yu, T.; Efros, A. A. Real-time user-guided image colorization with learned deep priors. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 119, 2017.
Sangkloy, P.; Lu, J.; Fang, C.; Yu, F.; Hays, J. Scribbler: Controlling deep image synthesis with sketch and color. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6836–6845, 2017.
Furusawa, C.; Hiroshiba, K.; Ogaki, K.; Odagiri, Y. Comicolorization: Semi-automatic manga colorization. In: Proceedings of the SIGGRAPH Asia Technical Briefs, Article No. 12, 2017.
Zhang, L.; Li, C.; Wong, T. T.; Ji, Y.; Liu, C. Two-stage sketch colorization. ACM Transactions on Graphics Vol. 37, No. 6, Article No. 261, 2018.
Sýkora, D.; Dingliana, J.; Collins, S. LazyBrush: Flexible painting tool for hand-drawn cartoons. Computer Graphics Forum Vol. 28, No. 2, 599–608, 2009.
Zhang, L.; Li, C.; Simo-Serra, E.; Ji, Y.; Wong, T. T.; Liu, C. User-guided line art flat filling with split filling mechanism. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9884–9893, 2021.
Shi, J.; Tomasi. Good features to track. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 593–600, 1994.
Siyao, L.; Li, Y.; Li, B.; Dong, C.; Liu, Z.; Loy, C. C. AnimeRun: 2D animation visual correspondence from open source 3D movies. In: Proceedings of the 36th Conference on Neural Information Processing Systems, 18996–19007, 2022.
Yin, J.; Liu, C.; Lin, R.; Vining, N.; Rhodin, H.; Sheffer, A. Detecting viewer-perceived intended vector sketch connectivity. ACM Transactions on Graphics Vol. 41, No. 4, Article No. 87, 2022.
Acknowledgments
We would like to thank the anonymous reviewers for their constructive comments. We are grateful to Zekun Li and Prof. Fang-Lue Zhang for providing their code and data for comparison. We give thanks to Benjamin Allen for helping the development of leakproof segmentation, also thank Mohammad Shafiq Bin Md Shawal, Muhammad Mohamad Din Yati, Yap Fei, Raihanah Ayuna Faiz, Kiyoumi Agemura, and Shogo Sakurazawa for testing our colorization system and for providing valuable feedback from the production side. We thank Ken Anjyo, Marc Salvati, and Alexandre Derouet-Jourdan for reviewing the paper and providing feedback. Finally, we would like to give our thanks and appreciation to Junpei Inuzuka and IMAGICA INFOS for their permission for us to use images from Restaurant to Another World 2 for research purposes.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
The authors have no competing interests to declare that are relevant to the content of this article.
Additional information
Akinobu Maejima is an R&D lead at OLM Digital, Inc. and a senior researcher at IMAGICA GROUP Inc. concurrently. His main research interest is in computer graphics. He received his M.S. degree in engineering from Seikei University in 2004, and his Ph.D. degree in engineering from Waseda University in 2010. He is a member of Asiagraphics and ACM SIGGRAPH.
Seitaro Shinagawa received his B.E. and M.S. degrees in information science in 2013 and 2015 from Tohoku University, respectively. He received his Doctoral degree in engineering in 2020 from Nara Institute of Science and Technology. From 2020, he has been an assistant professor in Nara Institute of Science and Technology (NAIST). His working area is vision and language, especially in visually-grounded dialogue systems.
Hiroyuki Kubo has been an associate professor at Chiba University, Japan since 2022. His research interests include computer graphics and computer vision. He received his M.S. and Ph.D. degrees from Waseda University, in 2008 and 2012, respectively. He is a member of IEEE and ACM.
Takuya Funatomi has been an associate professor at the Division of Information Science, NAIST, Japan since 2015. He was an assistant professor at Kyoto University, Japan from 2007 to 2015, and a visiting assistant professor at Stanford University, USA in 2014. He received his B.E., M.S., and Ph.D. degrees in informatics from Kyoto University, Japan in 2002, 2004, and 2007, respectively. He is a senior member of IEEE.
Tatsuo Yotsukura is head R&D supervisor at OLM Digital and a research director at IMAGICA GROUP Inc., concurrently. Before joining OLM Digital in 2009, He worked as a computer graphics researcher at ATR (Advanced Telecommunications Research Institute International), Japan. He received his Ph.D. degree from Seikei University in 2003.
Satoshi Nakamura is a professor in the Graduate School of Science and Technology, NAIST, Japan, a visiting professor of the Chinese University of Hong Kong, Shenzhen, China, and an Honorary Professor of Karlsruhe Institute of Technology, Germany. He received his Ph.D. degree from Kyoto University in 1992. He was an associate professor of the Graduate School of Information Science at NAIST from 1994 to 2000, the director of the ATR Spoken Language Communication Research Laboratories in 2000–2008, and vice president of ATR in 2007–2008. He was director general of the Keihanna Research Laboratories, National Institute of Information and Communications Technology, Japan in 2009–2010. His interests include modeling and systems of speech-to-speech translation and speech recognition. He received the LREC Antonio Zampolli Award in 2012. He was an IEEE Signal Processing Magazine Editorial Board Member in 2012–2014. He is an IEEE Life Fellow, ISCA Fellow, IPSJ Fellow, and ATR Fellow.
Yasuhiro Mukaigawa received his M.E. and Ph.D. degrees from the University of Tsukuba in 1994 and 1997, respectively. He became a research associate at Okayama University in 1997, an assistant professor at University of Tsukuba in 2003, an associate professor at Osaka University in 2004, and a professor at NAIST in 2014. His current research interests include photometric analysis and computational photography. He is a member of IEEE.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Maejima, A., Shinagawa, S., Kubo, H. et al. Continual few-shot patch-based learning for anime-style colorization. Comp. Visual Media (2024). https://doi.org/10.1007/s41095-024-0414-4
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s41095-024-0414-4