Retrieving geometric information from images: the case of hand-drawn diagrams

Abstract

This paper addresses the problem of retrieving meaningful geometric information implied in image data. We outline a general algorithmic scheme to solve the problem in any geometric domain. The scheme, which depends on the domain, may lead to concrete algorithms when the domain is properly and formally specified. Taking plane Euclidean geometry ${\mathbb {E}}$ as an example of the domain, we show how to formally specify ${\mathbb {E}}$ and how to concretize the scheme to yield algorithms for the retrieval of meaningful geometric information in ${\mathbb {E}}$. For images of hand-drawn diagrams in ${\mathbb {E}}$, we present concrete algorithms to retrieve typical geometric objects and geometric relations, as well as their labels, and demonstrate the feasibility of our algorithms with experiments. An example is presented to illustrate how nontrivial geometric theorems can be generated from retrieved geometric objects and relations and thus how implied geometric knowledge may be discovered automatically from images.

Appendices

Appendix 1: Algorithmic description of ${\texttt {recognizeCircleAndLines}}(\mathrm {I}_2)$

In Algorithm 4, the subalgorithm

• $\texttt {detectKeyPoints}(\mathrm {I}_2)$ is called to recognize key points in $\mathrm {I}_2$ according to Definition 2;

• $\texttt {combineKp2PntInt}(\mathrm {K}_p)$ is called to collect points of interest from clusters of key points in $\mathrm {K}_p$;

• $\texttt {assignPntOfInterest}(\mathrm {C}_{rv}, \mathrm {P}_I)$ is called to assign s to $c.\textit{spoint}$ and e to $c.\textit{epoint}$ of each $c\in \mathrm {C}_{rv}$, where s is a point of interest nearest to $c.\textit{spoint}$ and e is a point of interest nearest to $c.\textit{epoint}$;

• $\texttt {sortCurves}(\mathrm {C}_{rv})$ is called to sort the elements of $\mathrm {C}_{rv}$ in a descending order with respect to length.

The above subalgorithms are rather simple and we do not present them formally. The other subalgorithms $\texttt {detectCurves}(\mathrm {I}_2,\mathrm {K}_p)$, $\texttt {computeCrvAttributes}(\mathrm {C}_{rv})$, and $\texttt {determineEntityType}(\mathrm {C}_{rv})$ of Algorithm 4 are formulated as Algorithms 5, 6, and 7 respectively.

In Algorithm 5, $\texttt {getNextPoint}(S)$ is called to find the next point in the 8-neighbours of the last point of S which does not occur in S, which can be added to S, and which will lead to a key point.

In Algorithm 6,

• $\texttt {random}(n)$ generates a randomized integer in [1, n];

• $\texttt {getRadi}(T_c[1], T_c[p_{idx}], T_c[n])$ computes the radius of the circle c passing through three points $T_c[1], T_c[p_{idx}], T_c[n]$;

• $\texttt {distance}(T_c[j], \texttt {line}(T_c[1], T_c[n]))$ computes the distance from the point $T_c[j]$ to the line passing through $T_c[1]$ and $T_c[n]$;

• $\texttt {polyfit}(T_c[1,\ldots ,N_2],t)$ computes the coefficients of a polynomial of degree t which fits the sequence $T_c[1,\ldots ,N_2]$ of points;

• $\texttt {polySlope}(coef_s,P)$ computes the slope of a line tangent to the curve defined by the polynomial with coefficients $coef_s$ at the point P;

• $\texttt {tangentVecAngle}(slope,T_c[1,\ldots ,N_2])$ computes the directed angle from the vector (1, 0) to $(v_x,slope)$, where $v_x=1$ if the X-coordinate of $T_c[1]$ is bigger than $T_c[N_2]$, and $v_x=-1$ otherwise.

In Algorithm 7, $\texttt {combine2curves}(\mathrm {C}_{rv}[i],\mathrm {C}_{rv}[m_i])$ combines the two curves $\mathrm {C}_{rv}[i]$ and $\mathrm {C}_{rv}[m_i]$ of points into one curve and then updates the attributes of the new curve.

Appendix 2: Selected experimental results

In Table 5, basic geometric entities and basic geometric relations, listed in the subcolumns under “Basic geometric information”, are represented in the form of T: n, where T is the type of the geometric entity or relation and n is the number of instances of geometric entities or relations of type T retrieved from the image shown in the column “Image of diagram”; “Size” denotes the image size in pixels; “Time” denotes the running time in seconds for retrieving the geometric information from the image.

Table 5 Test results