Reading-Life Log as a New Paradigm of Utilizing Character and Document Media

  • Koichi Kise
  • Shinichiro Omachi
  • Seiichi Uchida
  • Masakazu Iwamura
  • Masahiko Inami
  • Kai Kunze
Chapter

Abstract

“You are what you read.” As this sentence implies, reading is important for building our minds. We are investing a huge amount of time for reading to input information. However the activity of “reading” is done only by each individual in an analog way and nothing is digitally recorded and reused. In order to solve this problem, we record reading activities as digital data and analyze them for various goals. We call this research “reading-life log.” In this chapter, we describe our achievements of the reading-life log. A target of the reading-life log is to analyze reading activities quantitatively and qualitatively: when, how much, what you read, and how you read in terms of your interests and understanding. Body-worn sensors including intelligent eyewear are employed for this purpose. Another target is to analyze the contents of documents based on the users’ reading activities: for example, which are the parts most people feel difficult/interesting. Materials to be read are not limited to books and documents. Scene texts are also important materials which guide human activities.

Keywords

Reading-life log Document image retrieval Scene character recognition Scene character dataset Font generation Eye-tracking Wordometer Smart eyewear Document annotation AffectiveWear Augmented narrative 

References

  1. 1.
    K. Kise, S. Omachi, S. Uchida, M. Iwamura, A trial for development of fundamental technologies for new usage of character and document media. J. Inst. Electron. Inf. Commun. Eng. 98(4), 311–327 (2015)Google Scholar
  2. 2.
    D. Karatzas, F. Shafait, S. Uchida, M. Iwamura, L. Gomez, S.R. Mestre, J. Mas, D.F. Mota, J.A. Almazan, L.P. de las Heras, ICDAR 2013 robust reading competition, in Proceedings of the 12th International Conference on Document Analysis and Recognition (ICDAR 2013) (2013), pp. 1484–1493Google Scholar
  3. 3.
    D. Karatzas, L. Gomez, A. Nicolaou, S. Ghosh, A. Bagdanov, M. Iwamura, J. Matas, L. Neumann, V.R. Chandrasekhar, S. Lu, F. Shafait, S. Uchida, E. Valveny, ICDAR 2015 robust reading competition, in Proceedings of the 13th International Conference on Document Analysis and Recognition (ICDAR 2015) (2015), pp. 1156–1160Google Scholar
  4. 4.
    S. Uchida, Text Localization and Recognition in Images and Video, Handbook of Document Image Processing and Recognition (Springer, 2014)Google Scholar
  5. 5.
    Q. Ye, D. Doermann, Text detection and recognition in imagery: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 37(7), 1480–1500 (2015)CrossRefGoogle Scholar
  6. 6.
    Y. Matsuda, S. Omachi, H. Aso, String detection from scene images by binarization and edge detection. IEICE Trans. Inf. Syst. (Japanese edition), J93-D (3), 336–344 (2010) (in Japanese)Google Scholar
  7. 7.
    R. Huang, P. Shivakumara, Y. Feng, S. Uchida, Scene character detection and recognition with cooperative multiple-hypothesis framework. IEICE Trans. Inf. Syst. E96-D (10), 2235–2244 (2013)Google Scholar
  8. 8.
    H. Takebe, S. Uchida, Scene character extraction by an optimal two-dimensional segmentation. IEICE Trans. Inf. Syst. (Japanese edition) (D), J97-D (3), 667–675 (2014) (in Japanese)Google Scholar
  9. 9.
    Y. Kunishige, Y. Feng, S. Uchida, Scenery character detection with environmental context, in Proceedings of the 11th International Conference on Document Analysis and Recognition (ICDAR 2011) (2011), pp. 1049–1053Google Scholar
  10. 10.
    A. Zhu, R. Gao, S. Uchida, Could scene context be beneficial for scene text detection?Pattern Recognit. 58C, 204–215 (2016)Google Scholar
  11. 11.
    A. Shahab, F. Shafait, A. Dengel, S. Uchida, How salient is scene text? in Proceedings of the 10th IAPR International Workshop on Document Analysis Systems (DAS2012) (2012), pp. 317–321Google Scholar
  12. 12.
    R. Gao, S. Uchida, A. Shahab, F. Shafait, V. Frinken, Visual Saliency Models for Text Detection in Real World. PLoS one 9(12), e114539 (2014)CrossRefGoogle Scholar
  13. 13.
    S. Wang, S. Uchida, M. Liwicki, Y. Feng, Part-based methods for handwritten digit recognition. Front. Comput. Sci. 7(4), 514–525 (2013)MathSciNetCrossRefGoogle Scholar
  14. 14.
    S. Toba, H. Kudo, T. Miyazaki, Y. Sugaya, S. Omachi, Ultra-low resolution character recognition system with pruning mutual subspace method, in Proceedings of the 2015 International Conference on Consumer Electronics—Taiwan (ICCE-TW) (2015), pp. 284–285Google Scholar
  15. 15.
    M. Goto, R. Ishida, Y. Feng, S. Uchida, Analyzing the distribution of a large-scale character pattern set using relative neighborhood graph, in Proceedings of the 12th International Conference on Document Analysis and Recognition (ICDAR 2013) (2013), pp. 3–7Google Scholar
  16. 16.
    M. Goldstein, S. Uchida, A comparative study on outlier removal from a large-scale dataset using unsupervised anomaly detection, in Proceedings of the 5th International Conference on Pattern Recognition Applications and Methods (ICPRAM2016) (2016), pp. 263–269Google Scholar
  17. 17.
    T. Saito, H. Yamada, K. Yamamoto, On the data base ETL9 of handprinted characters in JIS Chinese characters and its analysis. IEICE Trans. Inf. Syst. (Japanese edition) (D), J68-D(4), 757–764 (1985) (in Japanese)Google Scholar
  18. 18.
    T. Nakai, K. Kise, M. Iwamura, Use of affine invariants in locally likely arrangement hashing for camera-based document image retrieval, in Document Analysis Systems VII, vol. 3872, Lecture Notes in Computer Science, (2006), pp. 541–552Google Scholar
  19. 19.
    S. Ahmed, K. Kise, M. Iwamura, M. Liwicki, A. Dengel, Automatic ground truth generation of camera captured documents using document image retrieval, in Proceedings of the 12th International Conference on Document Analysis and Recognition (ICDAR 2013) (2013), pp. 528–532Google Scholar
  20. 20.
    S.M. Lucas, A. Panaretos, L. Sosa, A. Tang, S. Wong, R. Young, K. Ashida, H. Nagai, M. Okamoto, H. Yamamoto, H. Miyao, J. Zhu, W. Ou, C. Wolf, J.-M. Jolion, L. Todoran, M. Worring, X. Lin, ICDAR 2003 robust reading competitions: entries, results and future directions. Int. J. Doc. Anal. Recognit. (IJDAR) 7(2–3), 105–122 (2005)CrossRefGoogle Scholar
  21. 21.
    S. Lucas, ICDAR 2005 text locating competition results, in Proceedings of the 8th International Conference on Document Analysis and Recognition (ICDAR2005), 1, (2005), pp. 80–84Google Scholar
  22. 22.
    A. Shahab, F. Shafait, A. Dengel, ICDAR 2011 robust reading competition challenge 2: reading text in scene images, in Proceedings of the 11th International Conference on Document Analysis and Recognition (ICDAR 2011) (2011), pp. 1491–1496Google Scholar
  23. 23.
    K. Wang, S. Belongie, Word spotting in the wild, in Proceedings of the 11th European Conference on Computer Vision (ECCV2010), Part I (2010), pp. 591–604Google Scholar
  24. 24.
    K. Wang, B. Babenko, S. Belongie, End-to-end scene text recognition, in Proceedings of the 13th International Conference on Computer Vision (ICCV2011) (2011), pp. 1457–1464Google Scholar
  25. 25.
    R. Gao, F. Shafait, S. Uchida, Y. Feng, A hierarchical visual saliency model for character detection in natural scenes. Camera-Based Document Analysis and Recognition, LNCS 8357, 18–29 (2014)Google Scholar
  26. 26.
    M. Iwamura, T. Matsuda, N. Morimoto, H. Sato, Y. Ikeda, K. Kise, Downtown osaka scene text dataset, in Proceedings of the 2nd International Workshop on Robust Reading (IWRR2016) (2016) (in printing)Google Scholar
  27. 27.
    Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, A.Y. Ng, Reading digits in natural images with unsupervised feature learning, in Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning (2011), p. 9Google Scholar
  28. 28.
    M. Iwamura, M. Tsukada, K. Kise, Automatic labeling for scene text database, in Proceedings of the 12th International Conference on Document Analysis and Recognition (ICDAR 2013) (2013), pp. 1397–1401Google Scholar
  29. 29.
    H. Saito, Y. Sugaya, S. Omachi, S. Uchida, M. Iwamura, K. Kise, Generation of character patterns from sample character images, in IEICE Technical Report, PRMU2010-287 (2011) (in Japanese)Google Scholar
  30. 30.
    T.S. Cho, S. Avidan, W.T. Freeman, The patch transform. IEEE Trans. Pattern Anal. Mach. Intell. 32(8), 1489–1501 (2010)CrossRefGoogle Scholar
  31. 31.
    S. Belongie, J. Malik, and J. Puzicha, Shape context: a new descriptor for shape matching and object recognition, in Advances in Neural Information Processing Systems (2000), pp. 831–837Google Scholar
  32. 32.
    M. Iwamura, T. Sato, K. Kise, What is the most efficient way to select nearest neighbor candidates for fast approximate nearest neighbor search? in Proceedings of the 14th International Conference on Computer Vision (ICCV 2013) (2013), pp. 3535–3542Google Scholar
  33. 33.
    M. Iwamura, T. Tsuji, K. Kise, Memory-based recognition of camera-captured characters, in Proceedings of the 9th IAPR International Workshop on Document Analysis Systems (DAS2010) (2010), pp. 89–96Google Scholar
  34. 34.
    Y. Lamdan, H.J. Wolfson, Geometric hashing: a general and efficient model-based recognition scheme, in Proceedings of the 2nd International Conference on Computer Vision (ICCV1988) (1988), pp. 238–249Google Scholar
  35. 35.
    N. Asada, M. Iwamura, K. Kise, Improvement of word recognition accuracy with spellchecker based on tendency of recognition error of characters, IEICE Technical Report, 110(467), PRMU2010-268, pp. 183–188 (2011) (in Japanese)Google Scholar
  36. 36.
    M. Iwamura, T. Kobayashi, K. Kise, Recognition of multiple characters in a scene image using arrangement of local features, in Proceedings of the 11th International Conference on Document Analysis and Recognition (ICDAR 2011) (2011), pp. 1409–1413Google Scholar
  37. 37.
    D.G. Lowe, Object recognition from local scale-invariant features, in Proceedings of the International Conference on Computer Vision (1999), pp. 1150–1157Google Scholar
  38. 38.
    T. Kobayashi, M. Iwamura, T. Matsuda, K. Kise, An anytime algorithm for camera-based character recognition, in Proceedings of the 12th International Conference on Document Analysis and Recognition (ICDAR 2013) (2013), pp. 1172–1176Google Scholar
  39. 39.
    T. Matsuda, M. Iwamura, K. Kise, Performance improvement in local feature based camera-captured character recognition, in Proceedings of the 11th IAPR International Workshop on Document Analysis Systems (DAS2014) (2014), pp. 196–201Google Scholar
  40. 40.
    X. Liu, J. Samarabandu, An edge-based text region extraction algorithm for indoor mobile robot navigation, in Proceedings of the 2005 IEEE International Conference on Mechatronics and Automation, 2, (2005), pp. 701–706Google Scholar
  41. 41.
    K. Kunze, H. Kawaichi, K. Kise, K. Yoshimura, The wordometer—estimating the number of words read using document image retrieval and mobile eye tracking, in Proceedings of the 12th International Conference on Document Analysis and Recognition (ICDAR 2013) (2013), pp. 25–29Google Scholar
  42. 42.
    S. Ishimaru, J. Weppner, K. Kunze, A. Bulling, K. Kise, A. Dengel, P. Lukowicz, In the blink of an eye—combining head motion and eye blink frequency for activity recognition with Google Glass, in Proceedings of the 5th Augmented Human International Conference (2014), pp. 150–153Google Scholar
  43. 43.
    Y. Shiga, T. Toyama, Y. Utsumi, A. Dengel, K. Kise, Daily activity recognition combining gaze motion and visual features, in PETMEI 2014: 4th International Workshop on Pervasive Eye Tracking and Mobile Eye-based Interaction, Proceedings of the 16th International Conference on Ubiquitous Computing (2014), pp. 1103–1111Google Scholar
  44. 44.
    K. Kunze, Y. Shiga, S. Ishimaru, K. Kise, Reading activity recognition using an off-the-shelf EEG—detecting reading activities and distinguishing genres of documents, in Proceedings of the12th International Conference on Document Analysis and Recognition (ICDAR2013) (2013), pp. 96–100Google Scholar
  45. 45.
    Y. Utsumi, Y. Shiga, M. Iwamura, K. Kunze, K. Kise, Document type classification toward understanding reading habits, in Proceedings of the 20th Korea-Japan Joint Workshop on Frontiers of Computer Vision, 3, (2014), pp. 11–17Google Scholar
  46. 46.
    K. Kunze, Y. Utsumi, Y. Shiga, K. Kise, A. Bulling, I know what you are reading: recognition of document types using mobile eye tracking, in Proceedings of the 17th Annual International Symposium on Wearable Computers (2013), pp. 113–116Google Scholar
  47. 47.
    O. Augereau, K. Kise, K. Hoshika, A proposal of a document image reading-life log based on document image retrieval and eyetracking, in Proceedings of the 13th International Conference on Document Analysis and Recognition (ICDAR2015) (2015), pp. 246–250Google Scholar
  48. 48.
    K. Kunze, H. Kawaichi, K. Yoshimura, K. Kise, Towards inferring language expertise using eye tracking, in CHI’13 Extended Abstracts on Human Factors in Computing Systems (2013), p. 6Google Scholar
  49. 49.
    K. Yoshimura, K. Kunze, K. Kise, The eye as the window of the language ability: estimation of english skills by analyzing eye movement while reading documents, in Proceedings of the 13th International Conference on Document Analysis and Recognition (ICDAR2015) (2015), pp. 251–255Google Scholar
  50. 50.
    H. Fujiyoshi, K. Yoshimura, K. Kunze, K. Kise, A method of estimating English skills using eye gaze information of answering questions of English exercises, IEICE Technical Report, 115(24), PRMU2015-10, pp. 49–54 (2015) (in Japanese)Google Scholar
  51. 51.
    K. Kunze, K. Masai, M. Inami, Ö. Sacakli, M. Liwicki, A. Dengel, S. Ishimaru, K. Kise, Quantifying reading habits: counting how many words you read, in Presented at the UbiComp’15: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (2015), pp. 87–96Google Scholar
  52. 52.
    S. Ishimaru, K. Kunze, K. Tanaka, Y. Uema, K. Kise, M. Inami, Smart eyewear for interaction and activity recognition, in Presented at the CHI EA’15: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (2015), pp. 307–310Google Scholar
  53. 53.
    T. Kimura, R. Huang, S. Uchida, M. Iwamura, S. Omachi, K. Kise, The reading-life log—technologies to recognize texts that we read, in Proceedings of the 12th International Conference on Document Analysis and Recognition (ICDAR 2013) (2013), pp. 91–95Google Scholar
  54. 54.
    A. Okoso, K. Kunze, K. Kise, Implicit gaze based annotations to support second language learning, in Proceedings of the 2014 ACM Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp2014) (2014), pp. 143–146Google Scholar
  55. 55.
    R. Biedert, G. Buscher, S. Schwarz, J. Hees, A. Dengel, Text 2.0, in Proceedings of the 28th ACM Conference on Human Factors in Computing Systems (CHI2011) (2011)Google Scholar
  56. 56.
    T. Toyama, W. Suzuki, A. Dengel, K. Kise, User attention oriented augmented reality on documents with document dependent dynamic overlay, in Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR2013) (2013), pp. 299–300Google Scholar
  57. 57.
    K. Masai, Y. Sugiura, K. Suzuki, S. Shimamura, K. Kunze, M. Ogata, M. Inami, M. Sugimoto, AffectiveWear: towards recognizing affect in real life, in Presented at the UbiComp/ISWC’15 Adjunct: Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers (2015), pp. 357–360Google Scholar
  58. 58.
    S. Sanchez, T. Dingler, H. Gu, K. Kunze, Embodied reading: a multisensory experience, in Presented at the CHI EA’16: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (2016), pp. 1459–1466Google Scholar

Copyright information

© Springer Japan KK 2017

Authors and Affiliations

  • Koichi Kise
    • 1
  • Shinichiro Omachi
    • 2
  • Seiichi Uchida
    • 3
  • Masakazu Iwamura
    • 1
  • Masahiko Inami
    • 4
  • Kai Kunze
    • 5
  1. 1.Osaka Prefecture UniversityNaka, Sakai, OsakaJapan
  2. 2.Tohoku UniversitySendaiJapan
  3. 3.Kyushu UniversityFukuokaJapan
  4. 4.The University of TokyoTokyoJapan
  5. 5.Keio UniversityYokohamaJapan

Personalised recommendations