Data Mining and Knowledge Discovery

, Volume 31, Issue 4, pp 934–971 | Cite as

Retrieving geometric information from images: the case of hand-drawn diagrams

  • Dan Song
  • Dongming Wang
  • Xiaoyu ChenEmail author


This paper addresses the problem of retrieving meaningful geometric information implied in image data. We outline a general algorithmic scheme to solve the problem in any geometric domain. The scheme, which depends on the domain, may lead to concrete algorithms when the domain is properly and formally specified. Taking plane Euclidean geometry \({\mathbb {E}}\) as an example of the domain, we show how to formally specify \({\mathbb {E}}\) and how to concretize the scheme to yield algorithms for the retrieval of meaningful geometric information in \({\mathbb {E}}\). For images of hand-drawn diagrams in \({\mathbb {E}}\), we present concrete algorithms to retrieve typical geometric objects and geometric relations, as well as their labels, and demonstrate the feasibility of our algorithms with experiments. An example is presented to illustrate how nontrivial geometric theorems can be generated from retrieved geometric objects and relations and thus how implied geometric knowledge may be discovered automatically from images.


Formal specification Geometric information Image data Knowledge discovery Pattern matching Shape recognition 



The authors wish to thank the referees for their helpful comments on an early version of the paper and acknowledge the support of the research funds from the State Key Laboratory of Software Development Environment under Grant Nos. SKLSDE-2015ZX-18 and SKLSDE-2016ZX-18 and from the Central Universities under Grant No. YWF-16-SXXY-01.


  1. An W, Chen X, Wang D (2016) Searching for geometric theorems using features retrieved from diagrams. In: Kotsireas IS, Rump SM, Yap CK (eds) Mathematical aspects of computer and information sciences, LNCS 9582. Springer, Heidelberg, pp 383–397CrossRefGoogle Scholar
  2. ADG (1996–2014) Proceedings of international workshop on automated deduction in geometry. Accessed 4 Aug 2016
  3. Balbiani P, Dugat V, Fariñas del Cerro L, Lopez A (1994) Eléments de géométrie mécanique. Hermès, PariszbMATHGoogle Scholar
  4. Ballard DH (1981) Generalizing the Hough transform to detect arbitrary shapes. Pattern Recogn 13(2):111–122CrossRefGoogle Scholar
  5. Banerjee B, Chandrasekaran B (2010) A constraint satisfaction framework for executing perceptions and actions in diagrammatic reasoning. J Artif Intell Res 39(1):373–427MathSciNetCrossRefGoogle Scholar
  6. Beeson M, Wos L (2014) OTTER proofs in Tarskian geometry. In: Demri S, Kapur D, Weidenbach C (eds) Automated reasoning, LNAI 8562. Springer, Heidelberg, pp 495–510Google Scholar
  7. Belongie S, Malik J, Puzicha J (2002) Shape matching and object recognition using shape contexts. IEEE Trans Pattern Anal Mach Intell 24(4):509–522CrossRefGoogle Scholar
  8. Bhatt M, Lee JH, Schultz CPL (2011) CLP(QS): a declarative spatial reasoning framework. In: Egenhofer M, Giudice N, Moratz R, Worboys M (eds) Spatial information theory, LNCS 6899. Springer, Heidelberg, pp 210–230CrossRefGoogle Scholar
  9. Buchberger B (1985) Gröbner bases: an algorithmic method in polynomial ideal theory. In: Bose NK (ed) Multidimensional systems theory. D. Reidel, Dordrecht, pp 184–232CrossRefGoogle Scholar
  10. Bulko WC (1988) Understanding text with an accompanying diagram. In: Ali M (ed) Proceedings of the 1st international conference on industrial and engineering applications of aritificial intelligence and expert systems, vol 2. ACM Press, New York, pp 894–898Google Scholar
  11. Chen TC, Chung KL (2001) A new randomized algorithm for detecting lines. Real-Time Imaging 7(6):473–481CrossRefGoogle Scholar
  12. Chen TC, Chung KL (2001) An efficient randomized algorithm for detecting circles. Comput Vis Image Underst 83(2):172–191MathSciNetCrossRefGoogle Scholar
  13. Chen X, Wang D (2012) Management of geometric knowledge in textbooks. Data Knowl Eng 73:43–57CrossRefGoogle Scholar
  14. Chen X, Wang D (2013) Formalization and specification of geometric knowledge objects. Math Comput Sci 7(4):439–454MathSciNetCrossRefGoogle Scholar
  15. Chen X (2014) Representation and automated transformation of geometric statements. J Syst Sci Complex 27(2):382–412MathSciNetCrossRefGoogle Scholar
  16. Chen X, Song D, Wang D (2015) Automated generation of geometric theorems from images of diagrams. Ann Math Artif Intell 74(3–4):333–358MathSciNetCrossRefGoogle Scholar
  17. Chou S-C, Gao X-S, Zhang J-Z (1994) Machine proofs in geometry: automated production of readable proofs for geometry theorems. World Scientific, SingaporeCrossRefGoogle Scholar
  18. Chou S-C, Gao X-S (2001) Automated reasoning in geometry. In: Handbook of automated reasoning, chap 10, vol I. MIT Press, Cambridge, pp 623–665CrossRefGoogle Scholar
  19. Cohn AG, Hazarika SM (2001) Qualitative spatial representation and reasoning: an overview. Fundam Inf 46(1–2):1–29MathSciNetzbMATHGoogle Scholar
  20. Dehlinger C, Dufourd J-F, Schreck P (2001) Higher-order intuitionistic formalization and proofs in Hilberts elementary geometry. In: Richter-Gebert J, Wang D (eds) Automated deduction in geometry, LNAI 2061. Springer, Heidelberg, pp 306–323CrossRefGoogle Scholar
  21. Duda RO, Hart PE (1972) Use of the Hough transformation to detect lines and curves in pictures. Commun ACM 15(1):11–15CrossRefGoogle Scholar
  22. Ferguson RW, Forbus KD (1998) Telling juxtapositions: using repetition and alignable difference in diagram understanding. In: Holyoak K, Gentner D, Kokinov B (eds) Advances in analogy research: integration of theory and data from the cognitive, computational, and neural sciences, NBU Series in Cognitive Science. New Bulgarian University, Sofia, pp 109–117Google Scholar
  23. Forbus K, Usher J, Lovett A, Lockwood K, Wetzel J (2011) CogSketch: sketch understanding for cognitive science research and for education. Top Cogn Sci 3(4):648–666CrossRefGoogle Scholar
  24. Gelernter H (1995) Realization of a geometry theorem proving machine. In: Feigenbaum EA, Feldman J (eds) Computers and thought. MIT Press, Cambridge, pp 134–152Google Scholar
  25. GeoGebra. Accessed 4 Aug 2016
  26. GEOTHER. Accessed 4 Aug 2016
  27. Ghimire D, Lee J (2013) Geometric feature-based facial expression recognition in image sequences using multi-class AdaBoost and support vector machines. Sensors 13(6):7714–7734CrossRefGoogle Scholar
  28. Giaquinto M (2007) Visual thinking in mathematics: an epistemological study. Oxford University Press, New YorkCrossRefGoogle Scholar
  29. Glasgow J, Narayanan NH, Chandrasekaran B (1995) Diagrammatic reasoning: cognitive and computational perspectives. MIT Press, CambridgeGoogle Scholar
  30. Gonthier G (2008) Formal proof—the four-color theorem. Not AMS 55(11):1382–1393MathSciNetzbMATHGoogle Scholar
  31. Goodrum AA (2000) Image information retrieval: an overview of current research. Inf Sci Res Spec Issue Inf Sci 3(2):63–66Google Scholar
  32. Gorman JW, Mitchell OR, Kuhl FP (1988) Partial shape recognition using dynamic programming. IEEE Trans Pattern Anal Mach Intell 10(2):257–266CrossRefGoogle Scholar
  33. Hales T, Adams M, Bauer G et al (2015) A formal proof of the Kepler conjecture. arXiv preprint arXiv:1501.02155
  34. Hammond T, Davis R (2002) Tahuti: a geometrical sketch recognition system for UML class diagrams. In: Proceeding of the special interest group on computer graphics 2006 courses, Article No. 25. ACM Press, New YorkGoogle Scholar
  35. Hong H, Wang D, Winkler F (eds) (1995) Algebraic approaches to geometric reasoning. Special issue of annals of mathematics and artificial intelligence, vol 13. Baltzer, Basel, pp 1–2Google Scholar
  36. Johnson LE (2015) Automated elementary geometry theorem discovery via inductive diagram manipulation. Master thesis, Massachusetts Institute of TechnologyGoogle Scholar
  37. Jorge JA, Fonseca MJ (2000) A simple approach to recognise geometric shapes interactively. In: Chhabra AK, Dori D (eds) Graphics recognition recent advances, LNCS 1941. Springer, Heidelberg, pp 266–274CrossRefGoogle Scholar
  38. Jung CR, Schramm R (2004) Rectangle detection based on a windowed Hough transform. In: Proceedings of the 17th Braizilian symposium on computer graphics and image processing. IEEE Computer Society, New York, pp 113–120Google Scholar
  39. Kapur D, Mundy JL (eds) (1989) Geometric reasoning. Special issue of artificial intelligence, vol 37. MIT Press, Cambridge, pp 1–3Google Scholar
  40. Lafarge F, Descombes X (2010) Geometric feature extraction by a multi-marked point process. IEEE Trans Pattern Anal Mach Intell 32(9):1597–1609CrossRefGoogle Scholar
  41. Li K, Lu X, Ling H, Liu L, Feng T, Tang Z (2013) Detection of overlapped quadrangles in plane geometric figures. In: Proceedings of the 12th international conference on document analysis and recognition. IEEE Computer Society, Washington, pp 260–264Google Scholar
  42. Matas J, Galambos C, Kittler J (2000) Robust detection of lines using the progressive probabilistic Hough transform. Comput Vis Image Underst 78(1):119–137CrossRefGoogle Scholar
  43. Materlik D (2003) Using sketch recognition to enhance the human-computer interface of geometry software. Diploma thesis, Free University of BerlinGoogle Scholar
  44. McCharen JD, Overbeek RA, Wos L (1976) Problems and experiments for and with automated theorem-proving programs. IEEE Trans Comput 25(8):773–782CrossRefGoogle Scholar
  45. Montalvo FS (1990) Diagram understanding: the symbolic descriptions behind the scenes. In: Ichikawa T, Jungert E, Korfhage RR (eds) Visual languages and applications, series of languages and information systems. Plenum Press, New York, pp 5–27Google Scholar
  46. Moon H, Chellappa R, Rosenfeld A (2002) Optimal edge-based shape detection. IEEE Trans Image Process 11(11):1209–1227MathSciNetCrossRefGoogle Scholar
  47. Mori S, Nishida H, Yamada H (1999) Optical character recognition. Wiley, New YorkGoogle Scholar
  48. Nixon MS, Aguado AS (2008) Feature extraction & image processing, 2nd edn. Academic Press, Oxford, pp 87–88Google Scholar
  49. Novak G (1995) Diagrams for solving physical problems. In: Glasgow J, Narayanan NH, Chandrasekaran B (eds) Diagrammatic reasoning: cognitive and computational perspectives. AAAI Press, Palo Alto, pp 753–774Google Scholar
  50. Quaife A (1992) Automated development of fundamental mathematical theories. Kluwer, DordrechtzbMATHGoogle Scholar
  51. Schultz C, Bhatt M (2015) Spatial symmetry driven pruning strategies for efficient declarative spatial reasoning. In: Fabrikant SI, Raubal M, Bertolotto M, Davies C, Freundschuh S, Bell S (eds) Spatial information theory, LNCS 9368. Springer, Heidelberg, pp 331–353CrossRefGoogle Scholar
  52. Seo MJ, Hajishirzi H, Farhadi A, Etzioni O (2014) Diagram understanding in geometry questions. In: Proceedings of the 28th AAAI conference on artificial intelligence. AAAI Press, Palo Alto, pp 2831–2838Google Scholar
  53. Song D, Wang D, Chen X (2015) Discovering geometric theorems from scanned and photographed images of diagrams. In: Botana F, Quaresma P (eds) Automated deduction in geometry, LNAI 9201. Springer, Heidelberg, pp 149–165CrossRefGoogle Scholar
  54. Song D, Chen X (2016) Automated generation of keywords from images for geometric information search. In: Prodeedings of the 11th international workshop on automated deduction in geometry (ADG 2016) (Strasbourg, France, June 27–29, 2016), pp 172–189Google Scholar
  55. Sonka M, Hlavac V, Boyle R (2014) Image processing, analysis, and machine vision, 4th edn. Cengage Learning, StamfordGoogle Scholar
  56. Takahashi K (2012) PLCA: a framework for qualitative spatial reasoning based on connection patterns of regions. In: Qualitative spatio-temporal representation and reasoning: trends and future directions, chap 2. IGI Global, Hershey, pp 63–96Google Scholar
  57. Tangelder JWH, Veltkamp RC (2008) A survey of content based 3D shape retrieval methods. Multimed Tools Appl 39(3):441–471CrossRefGoogle Scholar
  58. Tesseract-OCR. Accessed 4 Aug 2016
  59. Ulgen F, Flavell AC, Akamatsu N (1995) Geometric shape recognition with fuzzy filtered input to a backpropagation neural network. IEICE Trans Inf Syst 78(2):174–183Google Scholar
  60. Wang D (1996) Geometry machines: from AI to SMC. In: Calmet J, Campbell JA, Pfalzgraf J (eds) Artificial intelligence and symbolic mathematical computation, LNCS 1138. Springer, Heidelberg, pp 213–239CrossRefGoogle Scholar
  61. Wang D (2001) Elimination methods. Springer, New YorkCrossRefGoogle Scholar
  62. Wang D (2003) Automated generation of diagrams with Maple and Java. In: Joswig M, Takayama N (eds) Algebra, geometry, and software systems. Springer, Heidelberg, pp 277–287CrossRefGoogle Scholar
  63. Wu W-T (1994) Mechanical theorem proving in geometries: basic principles (translated from the Chinese by X. Jin and D. Wang). Springer, New YorkCrossRefGoogle Scholar
  64. Yuen HK, Princen J, Illingworth J, Kittler J (1990) Comparative study of Hough transform methods for circle finding. Image Vis Comput 8(1):71–77CrossRefGoogle Scholar
  65. Zhang TY, Suen CY (1984) A fast parallel algorithm for thinning digital patterns. Commun ACM 27(3):236–239CrossRefGoogle Scholar

Copyright information

© The Author(s) 2017

Authors and Affiliations

  1. 1.LMIB-SKLSDE, School of Mathematics and Systems ScienceBeihang UniversityBeijingChina
  2. 2.Centre National de la Recherche ScientifiqueParis Cedex 16France

Personalised recommendations