Reconstruction of 3D Surface and Restoration of Flat Document Image from Monocular Image Sequence

  • Hiroki Shibayama
  • Yoshihiro Watanabe
  • Masatoshi Ishikawa
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7727)


There is a strong demand for the digitization of books. To meet this demand, camera-based scanning systems are considered to be effective because they could work with the cameras built into mobile terminals. One promising technique proposed to speed up book digitization involves scanning a book while the user flips the pages. In this type of camera-based document image analysis, it is extremely important to rectify distorted images. In this paper, we propose a new method of reconstructing the 3D deformation and restoring a flat document image by utilizing a unique planar development property of a sheet of paper from a monocular image sequence captured while the paper is deformed. Our approach uses multiple input images and is based on the natural condition that a sheet of paper is a developable surface, enabling high-quality restoration without relying on the document structure. In the experiments, we tested the proposed method for the target application using images of different documents and different deformations, and demonstrated its effectiveness.


Input Image Document Image Optical Character Recognition Developable Surface Bezier Curve 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Nakashima, T., Watanabe, Y., Komuro, T., Ishikawa, M.: Book flipping scanning. In: Adjunct Proceedings of UIST, pp. 79–80 (2009)Google Scholar
  2. 2.
    Carmo, M.P.D.: Differential Geometry of Curves and Surfaces. Prentice Hall (1976)Google Scholar
  3. 3.
    Watanabe, Y., Nakashima, T., Komuro, T., Ishikawa, M.: Estimation of non-rigid surface deformation using developable surface model. In: Proceedings of ICPR, pp. 197–200 (2010)Google Scholar
  4. 4.
    Cao, H., Ding, X., Liu, C.: Rectifying the bound document image captured by the camera: A model based approach. In: Proceedings of ICDAR, pp. 71–75 (2003)Google Scholar
  5. 5.
    Liang, J., DeMenthon, D., Doermann, D.: Unwarping Images of Curved Documents Using Global Shape Optimization. In: Proceedings of CBDAR, pp. 25–29 (2005)Google Scholar
  6. 6.
    Yamashita, A., Kawarago, A., Kaneko, T., Miura, K.T.: Shape reconstruction and image restoration for non-flat surfaces of documents with a stereo vision system. In: Proceedings of ICPR, pp. 482–485 (2004)Google Scholar
  7. 7.
    Zhang, Z., Tan, C.L., Fan, L.: Estimation of 3D shape of warped document surface for image restoration. In: Proceedings of ICPR, pp. 486–489 (2004)Google Scholar
  8. 8.
    Prados, E., Camilli, F.: A unifying and rigorous shape from shading method adapted to realistic data and applications. Journal of Mathematical Imaging and Vision 25, 307–328 (2006)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Hase, H., Shinokawa, T., Yoneda, M., Suen, C.: Recognition of rotated characters by eigen-space. In: Proceedings of ICDAR, pp. 731–735 (2003)Google Scholar
  10. 10.
    Narita, R., Ohyama, W., Wakabayashi, T., Kimura, F.: Three dimensional rotation-free recognition of characters. In: Proceedings of ICDAR, pp. 824–828 (2011)Google Scholar
  11. 11.
    Liang, J., DeMenthon, D., Doermann, D.: Geometric rectification of camera-captured document images. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 591–605 (2008)CrossRefGoogle Scholar
  12. 12.
    Tian, Y., Narasimhan, S.G.: Rectification and 3D Reconstruction of Curved Document Images. In: Proceedings of ICCV, pp. 377–384 (2011)Google Scholar
  13. 13.
    Zhang, L., Tan, C.L.: Warped Image Restoration with Applications to Digital Libraries. In: Proceedings of ICDAR, pp. 192–196 (2005)Google Scholar
  14. 14.
    Gumerov, N.A., Zandifar, A., Duraiswami, R., Davis, L.S.: 3d structure recovery and unwarping of surfaces applicable to planes. International Journal of Computer Vision 66, 261–281 (2006)CrossRefGoogle Scholar
  15. 15.
    Crum, W.R., Hartkens, T., Hill, D.L.G.: Non-rigid image registration: theory and practice. The British Journal of Radiology 77, 140–153 (2004)CrossRefGoogle Scholar
  16. 16.
    Bartoli, A., Zisserman, A.: Direct estimation of non-rigid registrations. In: Proceedings of BMVC (2004)Google Scholar
  17. 17.
    Gay-Bellile, V., Bartoli, A., Sayd, P.: Direct estimation of nonrigid registrations with image-based self-occlusion reasoning. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 87–104 (2010)CrossRefGoogle Scholar
  18. 18.
    Salzmann, M., Urtasun, R., Fua, P.: Local deformation models for monocular 3D shape recovery. In: Proceedings of CVPR (2008)Google Scholar
  19. 19.
    Salzmann, M., Pilet, J., Ilic, S., Fua, P.: Surface deformation models for non-rigid 3D shape recovery. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 1481–1487 (2007)CrossRefGoogle Scholar
  20. 20.
    Brunet, F., Hartley, R., Bartoli, A., Navab, N., Malgouyres, R.: Monocular Template-Based Reconstruction of Smooth and Inextensible Surfaces. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010, Part III. LNCS, vol. 6494, pp. 52–66. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  21. 21.
    Varol, A., Salzmann, M., Tola, E., Fua, P.: Template-free monocular reconstruction of deformable surfaces. In: Proceedings of the ICCV, pp. 1811–1818 (2009)Google Scholar
  22. 22.
    Bay, H., Tuytelaars, T., Van Gool, L.: SURF: Speeded Up Robust Features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 404–417. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  23. 23.
    Sprengel, R., Rohr, K., Stiehl, H.S.: Thin-plate spline approximation for image registration. In: Proceedings of the IEEE Engineering in Medicine and Biology Society, pp. 1190–1191 (1996)Google Scholar
  24. 24.
    Hore, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: Proceedings of ICPR, pp. 2366–2369 (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Hiroki Shibayama
    • 1
  • Yoshihiro Watanabe
    • 1
  • Masatoshi Ishikawa
    • 1
  1. 1.Graduate School of Information Science and TechnologyUniversity of TokyoJapan

Personalised recommendations