Online recognition of sketched arrow-connected diagrams

  • Martin Bresler
  • Daniel Průša
  • Václav Hlaváč
Original Paper


We introduce a new, online, stroke-based recognition system for hand-drawn diagrams which belong to a group of documents with an explicit structure obvious to humans but only loosely defined from the machine point of view. We propose a model for recognition by selection of symbol candidates, based on evaluation of relations between candidates using a set of predicates. It is suitable for simpler structures where the relations are explicitly given by symbols, arrows in the case of diagrams. Knowledge of a specific diagram domain is used—the two domains are flowcharts and finite automata. Although the individual pipeline steps are tailored for these, the system can readily be adapted for other domains. Our entire diagram recognition pipeline is outlined. Its core parts are text/non-text separation, symbol segmentation, their classification and structural analysis. Individual parts have been published by the authors previously and so are described briefly and referenced. Thorough evaluation on benchmark databases shows the accuracy of the system reaches the state of the art and is ready for practical use. The paper brings several contributions: (a) the entire system and its state-of-the-art performance; (b) the methodology exploring document structure when it is loosely defined; (c) the thorough experimental evaluation; (d) the new annotated database for online sketched flowcharts and finite automata diagrams.


Diagram recognition Online document analysis Max-sum problem Segmentation Text/non-text separation Flowcharts Finite automata 



The first author was supported by the Grant Agency of the CTU under the project SGS16/085/OHK3/1T/13. The second and the third authors were supported by the Czech Science Foundation under grant no. 15-04960S. The authors thank Truyen van Phan for his help with the text/non-text separation, Daniel Martín-Albo for creating the synthesized samples for the SVM classifiers and Roger Boyle for proofreading of the paper.


  1. 1.
    Alvarado, C., Davis, R.: SketchREAD: a multi-domain sketch recognition engine. In: UIST ’04: 17th Annual ACM Symposium on User Interface Software and Technology. UIST ’04, pp. 23–32. ACM, New York (2004)Google Scholar
  2. 2.
    Álvaro, F., Sánchez, J.A., Benedí, J.M.: Recognition of on-line handwritten mathematical expressions using 2d stochastic context-free grammars and hidden Markov models. Pattern Recogn. Lett. 35, 58–67 (2014)CrossRefGoogle Scholar
  3. 3.
    Arvo, J., Novins, K.: Appearance-preserving manipulation of hand-drawn graphs. In: 3rd International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia, GRAPHITE ’05, pp. 61–68. ACM (2005)Google Scholar
  4. 4.
    Awal, A.M., Feng, G., Mouchere, H., Viard-Gaudin, C.: First experiments on a new online handwritten flowchart database. In: DRR’11, pp. 1–10 (2011)Google Scholar
  5. 5.
    Blagojevic, R., Plimmer, B., Grundy, J., Wang, Y.: Using data mining for digital ink recognition: dividing text and shapes in sketched diagrams. Comput. Graph. 35(5), 976–991 (2011)CrossRefGoogle Scholar
  6. 6.
    Bresler, M., Průša, D., Hlaváč, V.: Detection of arrows in on-line sketched diagrams using relative stroke positioning. In: WACV ’15: IEEE Winter Conference on Applications of Computer Vision, pp. 610–617. IEEE Computer Society (2015)Google Scholar
  7. 7.
    Bresler, M., Průša, D., Hlaváč, V.: modeling flowchart structure recognition as a max-sum problem. In: O’Conner, L. (ed.) ICDAR ’13: 12th International Conference on Document Analysis and Recognition, pp. 1247–1251. IEEE Computer Society (2013)Google Scholar
  8. 8.
    Bresler, M., Průša, D., Hlaváč, V.: Using agglomerative clustering of strokes to perform symbols over-segmentation within a diagram recognition system. In: Paul Wohlhart, V.L. (ed.) CVWW ’15: Proceedings of the 20th Computer Vision Winter Workshop, pp. 67–74. Graz University of Technology (2015)Google Scholar
  9. 9.
    Bresler, M., Van Phan, T., Průša, D., Nakagawa, M., Hlaváč, V.: Recognition system for on-line sketched diagrams. In: Guerrero, J.E. (ed.) ICFHR ’14: 14th International Conference on Frontiers in Handwriting Recognition, pp. 563–568. IEEE Computer Society (2014)Google Scholar
  10. 10.
    Carton, C., Lemaitre, A., Couasnon, B.: Fusion of statistical and structural information for flowchart recognition. In: ICDAR ’13: 12th International Conference on Document Analysis and Recognition, pp. 1210–1214 (2013)Google Scholar
  11. 11.
    Delaye, A.: Structured prediction models for online sketch recognition (2014). Unpublished manuscript.
  12. 12.
    Delaye, A., Anquetil, E.: HBF49 feature set: a first unified baseline for online symbol recognition. Pattern Recogn. 46(1), 117–130 (2013)CrossRefGoogle Scholar
  13. 13.
    Delaye, A., Lee, K.: A flexible framework for online document segmentation by pairwise stroke distance learning. Pattern Recogn. 48(4), 1197–1210 (2015)CrossRefGoogle Scholar
  14. 14.
    Delaye, A., Liu, C.L.: Contextual text/non-text stroke classification in online handwritten notes with conditional random fields. Pattern Recogn. 47(3), 959–968 (2014)CrossRefGoogle Scholar
  15. 15.
    Feng, G., Viard-Gaudin, C., Sun, Z.: On-line hand-drawn electric circuit diagram recognition using 2D dynamic programming. Pattern Recogn. 42(12), 3215–3223 (2009)CrossRefzbMATHGoogle Scholar
  16. 16.
    Hammond, T., Davis, R.: LADDER, a sketching language for user interface developers. Comput. Graph. 29, 518–532 (2005)CrossRefGoogle Scholar
  17. 17.
    Hammond, T., Davis, R.: Tahuti: A geometrical sketch recognition system for UML class diagrams. In: ACM SIGGRAPH 2006 Courses, SIGGRAPH ’06. ACM, New York (2006)Google Scholar
  18. 18.
    Hammond, T., Paulson, B.: Recognizing sketched multistroke primitives. ACM Trans. Interact. Intell. Syst. 1(1), 4:1–4:34 (2011)CrossRefGoogle Scholar
  19. 19.
    Indermühle, E., Frinken, V., Bunke, H.: Mode detection in online handwritten documents using BLSTM neural networks. In: ICFHR ’12: 13th International Conference on Frontiers in Handwriting Recognition, pp. 302–307 (2012)Google Scholar
  20. 20.
    Kara, L.B., Stahovich, T.F.: Hierarchical parsing and recognition of hand-sketched diagrams. In: 17th Annual ACM Symposium on User Interface Software and Technology, UIST ’04, pp. 13–22. ACM (2004)Google Scholar
  21. 21.
    Le, A.D., Van Phan, T., Nakagawa, M.: A system for recognizing online handwritten mathematical expressions and improvement of structure analysis. In: DAS ’14: 11th IAPR International Workshop on Document Analysis Systems, pp. 51–55 (2014)Google Scholar
  22. 22.
    Lemaitre, A., Mouchére, H., Camillerapp, J., Coüasnon, B.: Interest of syntactic knowledge for on-line flowchart recognition. In: GREC ’11: 9th IAPR International Workshop on Graphics Recognition, pp. 85–88 (2011)Google Scholar
  23. 23.
    Liu, C.L., Zhou, X.D.: Online Japanese character recognition using trajectory-based normalization and direction feature extraction. In: Lorette, G. (ed.) Tenth International Workshop on Frontiers in Handwriting Recognition. Université de Rennes 1, Suvisoft (2006)Google Scholar
  24. 24.
    Martín-Albo, D., Plamondon, R., Vidal, E.: Training of on-line handwriting text recognizers with synthetic text generated using the kinematic theory of rapid human movements. In: Guerrero, J.E. (ed.) ICFHR ’14: 14th International Conference on Frontiers in Handwriting Recognition, pp. 543–548. IEEE Computer Society (2014)Google Scholar
  25. 25.
    Miyao, H., Maruyama, R.: On-line handwritten flowchart recognition, beautification and editing system. In: ICFHR ’12: 13th International Conference on Frontiers in Handwriting Recognition, pp. 83–88 (2012)Google Scholar
  26. 26.
    Mouchère, H., Viard-Gaudin, C., Zanibbi, R., Garain, U.: ICFHR 2014 competition on recognition of on-line handwritten mathematical expressions (CROHME 2014). In: J.E. Guerrero (ed.) ICFHR ’14: 14th International Conference on Frontiers in Handwriting Recognition, pp. 791–796. IEEE Computer Society (2014)Google Scholar
  27. 27.
    Otte, S., Krechel, D., Liwicki, M., Dengel, A.: Local feature based online mode detection with recurrent neural networks. In: ICFHR ’12: 13th International Conference on Frontiers in Handwriting Recognition, pp. 531–535 (2012)Google Scholar
  28. 28.
    Ouyang, T.Y., Davis, R.: Chemink: A natural real-time recognition system for chemical drawings. In: 16th International Conference on Intelligent User Interfaces, IUI ’11, pp. 267–276. ACM (2011)Google Scholar
  29. 29.
    Plimmer, B., Purchase, H.C., Yang, H.Y.: Sketchnode: intelligent sketching support and formal diagramming. In: 22nd Conference of the Computer-Human Interaction Special Interest Group of Australia on Computer-Human Interaction, OZCHI ’10, pp. 136–143. ACM (2010)Google Scholar
  30. 30.
    Qi, Y., Szummer, M., Minka, T.P.: Diagram structure recognition by Bayesian conditional random fields. In: Conference on Computer Vision and Pattern Recognition, pp. 191–196. IEEE Computer Society (2005)Google Scholar
  31. 31.
    Refaat, K., Helmy, W., Ali, A., AbdelGhany, M., Atiya, A.: A new approach for context-independent handwritten offline diagram recognition using support vector machines. In: IJCNN ’08: IEEE International Joint Conference on Neural Networks, pp. 177–182 (2008)Google Scholar
  32. 32.
    Sezgin, T.M., Davis, R.: HMM-based efficient sketch recognition. In: IUI ’05: 10th International Conference on Intelligent User Interfaces. IUI ’05, pp. 281–283. ACM, New York (2005)Google Scholar
  33. 33.
    Stoffel, A., Tapia, E., Rojas, R.: Recognition of on-line handwritten commutative diagrams. In: ICDAR ’09: 10th International Conference on Document Analysis and Recognition, pp. 1211–1215 (2009)Google Scholar
  34. 34.
    Szwoch, W., Mucha, M.: Recognition of Hand Drawn Flowcharts, Advances in Intelligent Systems and Computing, vol. 184. Springer, Berlin (2013)Google Scholar
  35. 35.
    Van Phan, T., Nakagawa, M.: Text/non-text classification in online handwritten documents with recurrent neural networks. In: J.E. Guerrero (ed.) ICFHR ’14: 14th International Conference on Frontiers in Handwriting Recognition, pp. 23–28. IEEE Computer Society (2014)Google Scholar
  36. 36.
    Werner, T.: A linear programming approach to max-sum problem: a review. IEEE Trans. Pattern Anal. Mach. Intell. 29(7), 1165–1179 (2007)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Martin Bresler
    • 1
  • Daniel Průša
    • 1
  • Václav Hlaváč
    • 2
  1. 1.Center for Machine Perception, Faculty of Electrical EngineeringCzech Technical University in PraguePrague 6Czech Republic
  2. 2.Czech Institute of Informatics, Robotics and CyberneticsCzech Technical University in PraguePrague 6Czech Republic

Personalised recommendations