Skip to main content

SickZil-Machine: A Deep Learning Based Script Text Isolation System for Comics Translation

  • Conference paper
  • First Online:
Document Analysis Systems (DAS 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12116))

Included in the following conference series:

Abstract

The translation of comics (and Manga) involves removing text from a foreign comic images and typesetting translated letters into it. The text in comics contain a variety of deformed letters drawn in arbitrary positions, in complex images or patterns. These letters have to be removed by experts, as computationally erasing these letters is very challenging. Although several classical image processing algorithms and tools have been developed, a completely automated method that could erase the text is still lacking. Therefore, we propose an image processing framework called ‘SickZil-Machine’ (SZMC) that automates the removal of text from comics. SZMC works through a two-step process. In the first step, the text areas are segmented at the pixel level. In the second step, the letters in the segmented areas are erased and inpainted naturally to match their surroundings. SZMC exhibited a notable performance, employing deep learning based image segmentation and image inpainting models. To train these models, we constructed 285 pairs of original comic pages, a text area-mask dataset, and a dataset of 31,497 comic pages. We identified the characteristics of the dataset that could improve SZMC performance. SZMC is available at: https://github.com/KUR-creative/SickZil-Machine.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Anonymous, The Danbooru Community, Branwen, G., Gokaslan, A.: Danbooru 2018: a large-scale crowdsourced and tagged anime illustration dataset, January 2019. Accessed 1 Jan 2020

    Google Scholar 

  2. Aramaki, Y., Matsui, Y., Yamasaki, T., Aizawa, K.: Text detection in manga by combining connected-component-based and region-based classifications. In: 2016 IEEE International Conference on Image Processing (ICIP). IEEE, September 2016

    Google Scholar 

  3. Augereau, O., Iwata, M., Kise, K.: A survey of comics research in computer science. J. Imaging 4(7), 87 (2018)

    Article  Google Scholar 

  4. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. 36(4), 1–14 (2017)

    Article  Google Scholar 

  5. Ito, K., Matsui, Y., Yamasaki, T., Aizawa, K.: Separation of Manga Line Drawings and Screentones, May 2015

    Google Scholar 

  6. Liu, G., et al.: Image inpainting for irregular holes using partial convolutions. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 89–105. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01252-6_6

    Chapter  Google Scholar 

  7. Liu, X., Li, C., Wong, T.-T.: Boundary-aware texture region segmentation from manga. Comput. Vis. Med. 3(1), 61–71 (2016). https://doi.org/10.1007/s41095-016-0069-x

    Article  Google Scholar 

  8. Matsui, Y., et al.: Sketch-based manga retrieval using manga109 dataset. Multimedia Tools Appl. 76(20), 21811–21838 (2016). https://doi.org/10.1007/s11042-016-4020-z

    Article  Google Scholar 

  9. Mirza, M., Osindero, S.: Conditional Generative Adversarial Nets. arXiv:1411.1784 [cs, stat], November 2014

  10. Nakamura, T., Zhu, A., Yanai, K., Uchida, S.: Scene Text Eraser. arXiv:1705.02772 [cs], May 2017

  11. Rigaud, C., Burie, J., Ogier, J.: Segmentation-free speech text recognition for comic books. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 03, pp. 29–34, November 2017

    Google Scholar 

  12. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  13. Tursun, O., et al.: MTRNet: a generic scene text eraser. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). IEEE, September 2019. https://doi.org/10.1109/icdar.2019.00016

  14. U-ram, K., Hwan-Gue, C.: A text script removal system for comics using deep learning (Korean). In: Proceedings of Korea Computer Congress, June 2019

    Google Scholar 

  15. Yu, J., et al.: Free-form image inpainting with gated convolution. In: The IEEE International Conference on Computer Vision (ICCV), October 2019

    Google Scholar 

  16. yu45020: yu45020/Text\_segmentation\_image\_inpainting, October 2019. https://github.com/yu45020/Text_Segmentation_Image_Inpainting. original-date: 2018–06-25T02:48:51Z

  17. Zhang, S., Liu, Y., Jin, L., Huang, Y., Lai, S.: EnsNet: Ensconce Text in the Wild. arXiv:1812.00723 [cs], December 2018

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Hwan-Gue Cho .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ko, UR., Cho, HG. (2020). SickZil-Machine: A Deep Learning Based Script Text Isolation System for Comics Translation. In: Bai, X., Karatzas, D., Lopresti, D. (eds) Document Analysis Systems. DAS 2020. Lecture Notes in Computer Science(), vol 12116. Springer, Cham. https://doi.org/10.1007/978-3-030-57058-3_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-57058-3_29

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-57057-6

  • Online ISBN: 978-3-030-57058-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics