Segmentation of Text and Non-text in On-Line Handwritten Patient Record Based on Spatio-Temporal Analysis

  • Rattapoom Waranusast
  • Peter Haddawy
  • Matthew Dailey
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5651)


Note taking is a common way for physicians to collect information from their patients in medical inquiries and diagnoses. Many times, when describing the pathology in medical records, a physician also draws diagrams and/or anatomical sketches along with the free-text narratives. The ability to understand unstructured handwritten texts and drawings in patient record could lead to implementation of automated patient record systems with more natural interfaces than current highly structured systems. The first and crucial step in automated processing of free-hand medical records is to segment the record into handwritten text and drawings, so that appropriate recognizers can be applied to different regions. This paper presents novel algorithms that separate text from non-text strokes in an on-line handwritten patient record. The algorithm is based on analyses of spatio-temporal graphs extracted from an on-line patient record and support vector machine (SVM) classification. Experiments demonstrate that the proposed approach is effective and robust.


Automated patient record Document segmentation Spatio-temporal analysis Online handwritten document 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    HIMSS, 19th Annual HIMSS Leadership Survey: CIO Results Final Report. Healthcare Information and Management Systems Society (HIMSS) (2008)Google Scholar
  2. 2.
    Fraser, H.S., Biondich, P., Moodley, D., Choi, S., Mamlin, B.W., Szolovits, P.: Implementing Electronic Medical Record Systems in Developing Countries. Inform. Prim. Care. 13, 83–95 (2005)PubMedGoogle Scholar
  3. 3.
    Connolly, C.: Cedars-Sinai Doctors Cling to Pen and Paper. The Washington Post, March 21 (2005)Google Scholar
  4. 4.
    Walsh, S.H.: The Clinician’s Perspective on Electronic Health Records and How They Can Affect Patient Care. Bmj. 328, 1184–1187 (2004)CrossRefPubMedPubMedCentralGoogle Scholar
  5. 5.
    Miller, R.H., Sim, I.: Physicians’ Use of Electronic Medical Records: Barriers and Solutions. Health Affairs 23 (2004) Google Scholar
  6. 6.
    Stahovich, T.F., Interpreting the Engineer’s Sketch: A Picture is Worth a Thousand Constraints. In: 1997 AAAI Symposium on Reasoning with Diagrammatic Representations II, Cambridge, Massachusetts, pp. 31–38 (1997) Google Scholar
  7. 7.
    Alvarado, C., Davis, R.: Intelligent Mechanical Engineering Design Environment: From Sketching to Simulation. MIT AI Laboratory Annual Abstracts. MIT, Cambridge, MA (2000) Google Scholar
  8. 8.
    Kara, L.B., Gennari, L., Stahovich, T.F.: A Sketch-based Tool for Analyzing Vibratory Mechanical Systems. Journal of Mechanical Design 130, (2008) Google Scholar
  9. 9.
    Leclercq, P.: Interpretative Tool for Architectural Sketches. In: 1st International Roundtable Conference on Visual and Spatial Reasoning in Design: Computational and Cognitive Approaches, Cambridge, MA, USA (1999) Google Scholar
  10. 10.
    Gross, M.D., Do, E.: Demonstrating the Electronic Cocktail Napkin: A Paper-like Interface for Early Design. In: Conference on Human Factors in Computing Systems, Vancouver, British Columbia, Canada, pp. 5–6 (1996) Google Scholar
  11. 11.
    Qian, D., Gross, M.: Collaborative Design with Netdraw. In: Computer Aided Architectural Design Futures Futures 1999, pp. 213–226 (1999) Google Scholar
  12. 12.
    Landay, J.A.: Interactive Sketching for the Early Stages of User Interface Design. Computer Science Dept., Vol. Ph.D. Carnegie Mellon University, Pittsburgh, Pa (1996) Google Scholar
  13. 13.
    Hammond, T., Davis, R.: Tahuti: A Sketch Recognition System for UML Class Diagrams - Extended Abstract. In: AAAI Spring Symposium on Sketch Understanding, pp. 59–68. AAAI Press, Stanford (2002)Google Scholar
  14. 14.
    Forbus, K.D., Usher, J., Chapman, V.: Sketching for Military Courses of Action Diagrams. In: 8th International Conference on Intelligent User Interfaces, pp. 61–68. ACM Press, Miami (2003)Google Scholar
  15. 15.
    Forbus, K., Usher, J.: Sketching for Knowledge Capture: A Progress Report. In: 7th International Conference on Intelligent User Interfaces, pp. 71–77. ACM Press, New York (2002)Google Scholar
  16. 16.
    Macé, S., Anquetil, E., Bossis, B.: Pen-Based Interaction for Intuitive Music Composition and Editing. In: Shen, J., Shepherd, J., Cui, B., Liu, L. (eds.) Intelligent Music Information Systems: Tools and Methodologies, pp. 261–288. Idea Group, USA (2007)Google Scholar
  17. 17.
    Fonseca, M., Barroso, B., Ribeiro, P., Jorge, J.: Sketch-Based Retrieval of ClipArt Drawings. In: Advanced Visual Interfaces (AVI 2004). ACM Press, Gallipoli (2004)Google Scholar
  18. 18.
    Leung, W.H., Chen, T.: Trademark Retrieval Using Contour-Skeleton Stroke Classification. In: IEEE Intl. Conf. on Multimedia and Expo (ICME 2002), Lausanne, Switzerland (2002)Google Scholar
  19. 19.
    Haddawy, P., Dailey, M., Kaewruen, P., Sarakhette, N.: Anatomical Sketch Understanding: Recognizing Explicit and Implicit Structure. In: Miksch, S., Hunter, J., Keravnou, E.T. (eds.) AIME 2005. LNCS, vol. 3581, pp. 343–352. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  20. 20.
    Suebnukarn, S., Haddawy, P.: A Collaborative Intelligent Tutoring System for Medical Problem-Based Learning. In: Int’l Conf. on Intelligent User Interfaces 2004, Madeira, Portugal, pp. 14–21 (2004)Google Scholar
  21. 21.
    Suebnukarn, S., Haddawy, P.: COMET: A Collaborative Tutoring System for Medical Problem-based Learning. IEEE Intelligent Systems 22, 70–77 (2007)CrossRefGoogle Scholar
  22. 22.
    Anquetil, E., Lorette, G.: New Advances and New Challenges in On-line Handwriting Recognition & Electronic Ink Management. In: Chaudhuri, B.B. (ed.) Digital Document Processing: Major Directions and Recent Advances (Advances in Pattern Recognition), pp. 143–164. Springer, Heidelberg (2006)Google Scholar
  23. 23.
    Shilman, M., Wei, Z., Raghupathy, S., Simard, P., Jones, D.: Discerning Structure from Freeform Handwritten Notes. In: Seventh International Conference on Document Analysis and Recognition. IEEE Computer Society, Los Alamitos (2003)Google Scholar
  24. 24.
    Jain, A.K., Namboodiri, A.M., Subrahmonia, J.: Structure in On-line Documents. In: ICDAR 2001, pp. 844–848 (2001)Google Scholar
  25. 25.
    Bishop, C.M., Svensen, M., Hinton, G.E.: Distinguishing Text from Graphics in On-Line Handwritten Ink. In: 9th International Workshop on Frontiers in Handwriting Recognition (IWFHR 2004). IEEE Computer Society, Los Alamitos (2004)Google Scholar
  26. 26.
    Namboodiri, A.M., Jain, A.K.: Robust Segmentation of Unconstrained Online Handwritten Documents. In: ICVGIP, pp. 165–170 (2004)Google Scholar
  27. 27.
    Fischler, M.A., Bolles, R.C.: Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Comm. of the ACM 24, 381–395 (1981)CrossRefGoogle Scholar
  28. 28.
    Rosenfeld, A., Troy, E.B.: Visual Texture Analysis. In: Conf. Rec. Symp. Feature Extraction and Selection in Pattern Recognition, p. 115 (1970)Google Scholar
  29. 29.
    Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques, 2nd edn. Morgan Kaufmann, San Francisco (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Rattapoom Waranusast
    • 1
  • Peter Haddawy
    • 1
  • Matthew Dailey
    • 1
  1. 1.Computer Science and Information Management program, School of Engineering and TechnologyAsian Institute of TechnologyPathumthaniThailand

Personalised recommendations