Multimedia Tools and Applications

, Volume 76, Issue 5, pp 7067–7085 | Cite as

Robust handwriting extraction and lecture video summarization

  • Greg C. Lee
  • Fu-Hao Yeh
  • Ying-Ju Chen
  • Tao-Ku Chang


In e-Learning research, teachers can record lecture videos in e-class and upload these lecture videos to e-Learning system themselves. Once lecture videos and handouts can be generated automatically in traditional classroom, it can help students with self-learning and teacher with lecture content development for e-Learning services. This paper proposed a teaching assistant system based on computer vision that can help in content development for e-Learning services. Lecture videos are taken by using two cameras and merged on both sides so that students can see a clear and complete teaching content. The k-means segmentation is used to extract board area and then connected component technique helps refill the board area which is covered by lecturer’s body. Then we use adaptive threshold to extract handwritings in various light conditions and time-series denoising technique is designed to reduce noise. According to extracted handwritings, the lecture videos can be automatically structured with high level of semantics. The lecture videos are segmented into video clips and all key-frames are integrated as handouts of the education videos.


Video segmentation Video summarization Notes extraction Image processing 


  1. 1.
    Bay H, Tuytelaars T, Gool LV (2008) SURF: speeded up robust features. Comput Vis Image Underst 110(3):346–359CrossRefGoogle Scholar
  2. 2.
    Bhogal AK, Singla N, Kaur M (2010) Color image segmentation using K-means clustering algorithm. 1(2):18–20Google Scholar
  3. 3.
    Brown M, Lowe DG (2007) Automatic panoramic image stitching using invariant features. Int J Comput Vis 74(1):59–73CrossRefGoogle Scholar
  4. 4.
    Chang HS, Sull S, Lee SU (1999) Efficient video indexing scheme for content-based retrieval. IEEE Trans Circ Syst Video Technol 9(8):1269–1279CrossRefGoogle Scholar
  5. 5.
    Choudary C, Liu T (2007) Summarization of visual content in instructional videos. IEEE Trans Multimed 9(7):1443–1455CrossRefGoogle Scholar
  6. 6.
    Ferman AM, Tekalp AM, Mehrotra R (2002) Robust color histogram descriptors for video segment retrieval and identification. IEEE Trans Image Process 11(5):497–508CrossRefGoogle Scholar
  7. 7.
    Fink GA, Wienecke M, Plötz T (2005) Experiments in video-based whiteboard reading. Proceedings of International Workshop on Camera-Based Document Analysis and Recognition, pp. 95–100Google Scholar
  8. 8.
    Hartley R, Zisserman A (2000) Multiple view geometry in computer vision. Cambridge University Press, CambridgeMATHGoogle Scholar
  9. 9.
    He L, Zhang Z (2007) Real-time whiteboard capture and processing using a video camera for remote collaboration. IEEE Trans Multimed 9(1):198–206CrossRefGoogle Scholar
  10. 10.
    Hirzallah N, Nusir S, Al Sayyed A, Kayed A (2008) Notes extraction algorithm from traditional presentations without the use of e-boards. Proceedings of the International Conference on Computer and Communication Engineering, pp. 195–200Google Scholar
  11. 11.
    Imran AS, Cheikh FA (2011) Blackboard content classification for lecture videos. Proceedings of International Conference on Image Processing, pp. 2989–2992Google Scholar
  12. 12.
    Imran AS, Cheikh FA (2012) Lecture content classification tool. Proceedings of International Symposium on Communications Control and Signal Processing, pp. 1–6Google Scholar
  13. 13.
    Imran AS, Rahadianti L, Cheikh FA, Yayilgan SY (2012) Semantic tags for lecture videos. Proceedings of International Conference on Semantic Computing, pp. 117–120Google Scholar
  14. 14.
    Jain A (1986) Fundamentals of digital image processing. Prentice-HallGoogle Scholar
  15. 15.
    Lei Z, Chou W, Zhong J, Lee CH (2000) Video segmentation using spatial and temporal statistical analysis method. Proc Int Conf Multimed Expo 3:1527–1530CrossRefGoogle Scholar
  16. 16.
    Lin C, Sheu M, Chiang H, Liaw C, Tsai C (2005) An efficient video de-interlacing with scene change detection. Proceedings of the International Conference on Information, Communications and Signal Processing, pp. 36–40Google Scholar
  17. 17.
    Liu TT, Choudary C (2006) Content Extraction and Summarization of Instructional Videos”, Proceedings of International Conference on Image Processing, pp. 149–152Google Scholar
  18. 18.
    Okuni S, Tsuruoka S, Rayat GP, Kawanaka H, Shinogi T (2007) Video scene segmentation using the state recognition of blackboard for blended learning. Proceedings of International Conference on Convergence Information Technology, pp. 2437–2442Google Scholar
  19. 19.
    Onishi M, Izumi M, Fukunaga K (2000) Blackboard segmentation using video image of lecture and its applications. Proceedings of International Conference on Pattern Recognition, pp. 615–618Google Scholar
  20. 20.
    Saez E, Benavides JI, Guil N (2004) Reliable real time scene change detection in MPEG compressed video. Proc Int Conf Multimed Expo 1:567–570Google Scholar
  21. 21.
    Saund E (1999) Image mosaicing and a diagrammatic user interface for an office whiteboard scanner. Technical report, Xerox Palo Alto Research CenterGoogle Scholar
  22. 22.
    Zhang Z, He LW (2004) Notetaking with a camera: whiteboard scanning and image enhancement. Proc IEEE Int Conf Acoust Speech Signal Process 3:533–536Google Scholar
  23. 23.
    Zhang HJ, Kankanhalli A, Smoliar SW (1993) Automatic partitioning of full-motion video. Multimedia Systems 1(1):10–28CrossRefGoogle Scholar
  24. 24.
    Zhang D, Qi W, Zhang HJ (2001) A new shot boundary detection algorithm. Proceedings of the Second Pacific Rim Conference on Multimedia: Advances in Multimedia Information Processing, pp. 63–70Google Scholar
  25. 25.
    Zhao L, Qi W, Li SZ, Yang SQ, Zhang HJ (2001) Content-based retrieval of video shot using the-improved nearest feature line method. Proc Int Conf Acoust Speech Signal Process 3:1625–1628Google Scholar
  26. 26.
    Zhou J, Zhang XP (2004) A web-enabled video indexing system. Proceedings of the International Workshop on Multimedia Information Retrieval, pp. 307–314Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Greg C. Lee
    • 1
  • Fu-Hao Yeh
    • 2
  • Ying-Ju Chen
    • 1
  • Tao-Ku Chang
    • 3
  1. 1.Department of Computer Science and Information EngineeringNational Taiwan Normal UniversityTaipei CityChina
  2. 2.Program of Information TechnologyFooyin UniversityKaohsiungChina
  3. 3.Department of Computer Science and Information EngineeringNational Dong Hwa UniversityHualienChina

Personalised recommendations