Skip to main content
SpringerLink
Log in

A system for supporting paper-based augmented reality

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, we aim to implement augmented reality (AR) on distant text documents or books. For this purpose, we propose a new paper-based AR system that can detect text documents in real scenes, markerize and identify them, estimate their relative 3D poses to the camera, and augment them with virtual contents. Unlike the previous paper-based AR systems (applicable to only close documents), the proposed system not only requires no detection of words or characters, but allows partial occlusions like the previous systems. In our experiments, the proposed system worked at 24 fps and could consistently achieve high identification rates for both occluded and unoccluded pages.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Notes

  1. In preliminary experiments, we tested several approaches and the black pixel counting was extremely fast and accurate, robust to the occlusion, and thus the best for our purpose.

  2. Since our marker identification method was not rotation-invariant, each document was in-plane rotated by 0, 90, 180, and 270∘.

  3. Notice that the paper-based AR methods [22, 35, 41] or text detection/recognition methods [13, 18, 22, 28, 30, 31] are not available for distant text-based documents.

References

  1. http://www.marxentlabs.com/5-augmented-reality-books-that-delight-and-engage/. [Online; accessed 2-Sep-2014]

  2. http://virtual.vtt.fi/virtual/proj2/multimedia/alvar/. ALVAR [Online; accessed 2-Sep-2014]

  3. http://opencv.org. OpenCV [Online; accessed 2-Sep-2014]

  4. http://www.opengl.org/. OpenGL [Online; accessed 2-Sep-2014]

  5. https://code.google.com/p/glmetaseq/. GLMetaseq [Online; accessed 2-Sep-2014]

  6. Alahi A, Ortiz R, Vandergheynst P (2012) FREAK: Fast retina keypoint. In: Proceedings of CVPR, pp. 510–517. IEEE

  7. Back M, Cohen J, Gold R, Harrison SR, Minneman SL (2001) Listen Reader: an electronically augmented paper-based book. In: Proceedings of CHI, pp. 23–29. ACM

  8. Bay H, Ess A, Tuytelaars T, Gool LV (2008) SURF: speeded up robust features. Comp Vision Image Underst 110:346–359

    Article  Google Scholar 

  9. Billinghurst M, Dunser A (2012) Augmented reality in the classroom. Computer 45(7):56–63

    Article  Google Scholar 

  10. Billinghurst M, Kato H, Poupyrev I (2001) The MagicBook-moving seamlessly between reality and virtuality. IEEE Comput Graph Appl 21 (3):6–8

    Google Scholar 

  11. Clark AJ, Dnser A (2012) An interactive augmented reality coloring book. In: Proceedings of 3DUI, pp. 7–10. IEEE

  12. Clark P, Mirmehdi M (2002) Recognising text in real scenes. IJDAR 4(4):243–257

    Article  Google Scholar 

  13. Coates A, Carpenter B, Case C, Satheesh S, Suresh B, Wang T, Wu DJ, Ng AY (2011) Text detection and character recognition in scene images with unsupervised feature learning. In: Proceedings of ICDAR, pp. 440–445. IEEE

  14. Duda RO, Hart PE (1972) Use of the hough transformation to detect lines and curves in pictures. Commun ACM 15(1):11–15

    Article  MATH  Google Scholar 

  15. Fiala M (2005) ARTag, a fiducial marker system using digital techniques. In: Proceedings of CVPR, pp. 590–596. IEEE Computer Society

  16. Fragoso V, Gauglitz S, Zamora S, Kleban J, Turk M (2011) TranslatAR: a mobile augmented reality translator. In: Proceedings of WACV, pp. 497–502

  17. Gauglitz S, Höllerer T, Turk M (2011) Evaluation of interest point detectors and feature descriptors for visual tracking. Int J Comput Vis 94(3):335–360

    Article  MATH  Google Scholar 

  18. Gomez L, Karatzas D (2013) Multi-script text extraction from natural scenes. In: Proceedings of ICDAR, pp. 467–471. IEEE

  19. Grasset R, Dnser A, Seichter H, Billinghurst M (2007) The mixed reality book: a new multimedia reading experience. In: CHI Extended Abstracts, pp. 1953–1958. ACM

  20. Harris C, Stephens M (1988) A combined corner and edge detector. In: Proceedings of the 4th Alvey Vision Conference, pp. 147–151

  21. Hartley R, Zisserman A (2003) Multiple view geometry in computer vision, 2 edn. Cambridge University Press, New York

    MATH  Google Scholar 

  22. Hull JJ, Erol B, Graham J, Ke Q, Kishi H, Moraleda J, Olst DGV (2007) Paper-based augmented reality. In: Proceedings of ICAT, pp. 205–209

  23. Kato H, Billinghurst M (1999) Marker tracking and HMD calibration for a video-based augmented reality conferencing system. In: Proceedings of IWAR, pp. 85–94. IEEE Computer Society

  24. Kato H, Billinghurst M, Poupyrev I, Imamoto K, Tachibana K (2000) Virtual object manipulation on a table-top ar environment. In: Proceedings of ISAR, pp. 111–119

  25. Kato H, Tachibana K, Billinghurst M, Grafe M (2003) A registration method based on texture tracking using ARToolKit. Proceedings of ARToolkit Workshop, pp. 77–85. IEEE

  26. Kim K, Lepetit V, Woo W (2010) Scalable real-time planar targets tracking for digilog books. Vis Comput 26(6–8):1145–1154

    Article  Google Scholar 

  27. Leutenegger S, Chli M, Siegwart R (2011) BRISK: binary robust invariant scalable keypoints. In: ICCV, pp. 2548–2555. IEEE

  28. Li N (1993) An implementation of ocr system based on skeleton matching. University of Kent at Canterbury, Computing Laboratory

  29. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vision 60(2):91–110

    Article  Google Scholar 

  30. Nakai T, Kise K, Iwamura M (2007) Camera based document image retrieval with more time and memory efficient LLAH. In: Proc CBDAR

  31. Neumann L, Matas J (2012) Real-time scene text localization and recognition. In: Proceedings of CVPR, pp. 3538–3545

  32. Norrie MC, Palinginis A, Signer B (2005) Content publishing framework for interactive paper documents. In: Proceedings of ACM Symposium on Document Engineering, pp. 187–196. ACM

  33. Park H, Park JI (2004) Invisible marker tracking for AR. In: Proceedings of ISMAR, pp. 272–273. IEEE Computer Society

  34. Park J, Woo W (2012) Hybrid document matching method for page identification of digilog books. T. Edutainment 8:24–34

    Google Scholar 

  35. Peng H, Long F, Chi Z (2003) Document image recognition based on template matching of component block projections. IEEE Trans Pattern Anal Mach Intell 25(9):1188–1192

    Article  Google Scholar 

  36. Petter M, Fragoso V, Turk M, Baur C (2011) Automatic text detection for mobile augmented reality translation. In: Proceedings of ICCV Workshops, pp. 48–55. IEEE

  37. Rekimoto J, Ayatsuka Y (2000) CyberCode: designing augmented reality environments with visual tags. In: Proceedings of DARE, pp. 1–10

  38. Rublee E, Rabaud V, Konolige K, Bradski GR (2011) ORB: an efficient alternative to SIFT or SURF. In: Proceedings of ICCV, pp. 2564–2571

  39. Saso TI, Iguchi K, Inakage M (2003) Little red: storytelling in mixed reality. In: Proceedings of SIGGRAPH. ACM

  40. Scherrer C, Pilet J, Fua P, Lepetit V (2008) The haunted book. In: Proceedings of ISMAR, pp. 163–164. IEEE

  41. Uchiyama H, Saito H (2009) Augmenting text document by on-line learning of local arrangement of keypoints. In: Proc. of ISMAR, pp. 95–98

  42. Wagner D, Langlotz T, Schmalstieg D (2008) Robust and unobtrusive marker tracking on mobile phones. In: Proceedings of ISMAR, pp. 121–124. IEEE

  43. Wagner D, Reitmayr G, Mulloni A, Drummond T, Schmalstieg D (2008) Pose tracking from natural features on mobile phones. In: Proceedings of ISMAR, pp. 125–134. IEEE

  44. Willis KDD, Shiratori T, Mahler M (2013) HideOut: Mobile projector interaction with tangible objects and surfaces. In: Proceedings of TEI, pp. 331–338. ACM

  45. Zhang S, Tian Q, Huang Q, Gao W, Rui Y (2014) USB: ultrashort binary descriptor for fast visual matching and retrieval. In: Proceedings of CVPR. IEEE

Download references

Author information

Authors and Affiliations

  1. Department of Electronic Engineering, Pukyong National University, 45 Yongso-ro, Nam-gu, Busan, 608-737, Korea

    Ho-Sub Ryu & Hanhoon Park

Authors
  1. Ho-Sub Ryu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hanhoon Park
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hanhoon Park.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ryu, HS., Park, H. A system for supporting paper-based augmented reality. Multimed Tools Appl 75, 3375–3390 (2016). https://doi.org/10.1007/s11042-014-2439-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-014-2439-7

Keywords

Search

Navigation