Multi-layer Ontologies for Integrated 3D Shape Segmentation and Annotation

Abstract

Mesh segmentation and semantic annotation are used as preprocessing steps for many applications, including shape retrieval, mesh abstraction, and adaptive simplification. In current practice, these two steps are done sequentially: a purely geometrical analysis is employed to extract the relevant parts, and then these parts are annotated. We introduce an original framework where annotation and segmentation are performed simultaneously, so that each of the two steps can take advantage of the other. Inspired by existing methods used in image processing, we employ an expert’s knowledge of the context to drive the process while minimizing the use of geometric analysis. For each specific context a multi-layer ontology can be designed on top of a basic knowledge layer which conceptualizes 3D object features from the point of view of their geometry, topology, and possible attributes. Each feature is associated with an elementary algorithm for its detection. An expert can define the upper layers of the ontology to conceptualize a specific domain without the need to reconsider the elementary algorithms. This approach has a twofold advantage: on one hand it allows to leverage domain knowledge from experts even if they have limited or no skills in geometry processing and computer programming; on the other hand, it provides a solid ground to be easily extended in different contexts with a limited effort.

Appendix: Details of the Segmentation Algorithms

In Sect. 4 we introduced a series of elementary algorithms to segment and identify regions of a requested object. We already introduced in Sect. 3.4.1 the two families of algorithms: “Is a” algorithm to label an already segmented region, and “Find all” algorithms to extract regions corresponding to a specific property. In this section, we present the implementation details of the core algorithms we’ve introduced to handle the experiments on Furnitures (see Sect. 5).

1.1 “Is a” Algorithms

Shape properties are associated with dedicated “is a” algorithms that follow a common pipeline. First we compute an approximation of the minimal volume oriented bounding box using ApproxMVBB,⁶ a C++ extension of the work of (Barequet and Har-Peled 2001) with many efficient preprocessing steps.

Using the three lengths $l_0 \ge l_1 \ge l_2$ of this bounding box, we wrote fuzzy rules to express the following descriptions:

• in a cube, $l_1$ is almost equivalent to $l_0$ and $l_2$,

• in a board, $l_0$ and $l_1$ are obviously longer than $l_2$,

• in a stick, $l_0$ is obviously longer than $l_2$, and $l_1$ is almost equivalent to $l_2$.

Vertical and horizontal orientations are well defined for sticks and boards, using the main directions of the minimal volume oriented bounding box. Let $v_0$ be the axis associated to the largest side of the box (w.r.t. its area), and let $v_1$ be the axis associated to the smallest side of the box.

We wrote fuzzy rules to express the following descriptions if the region is a board:

• if $v_0$ is almost parallel to the up-down axis, the region is horizontal,

• if $v_0$ is almost orthogonal to the up-down axis, the region is vertical.

If the region as been identified as a stick, we introduce the following descriptions:

• if $v_1$ is almost parallel to the up-down axis, the stick is vertical,

• if $v_1$ is almost orthogonal to the up-down axis, the stick is horizontal.

The compactness of a shape is defined by comparing the two first lengths $l_0$ and $l_1$ of the bounding box. If $l_0$ is sufficiently longer than $l_1$ the shape is elongated. If these two lengths are almost equivalent, the region has a compact shape.

In this work, we assume that the input mesh has as correct scale (in our case, 1 unit corresponding to 1 meter) and is correctly oriented. Existing methods such as (Fu et al. 2008) are available to automatically estimate the orientation of a manufactured object.

This property has been used to define two kinds of properties relative to the height of the regions. First we used the relative vertical positions comparing the highest, the central and the lowest points of the region with respect to the equivalent points of the object, to define algorithms able to identify position properties: up position, down position and central position.

To define the global position of a region, we compared the lowest point of the object with the highest point of the region. In his work Le Corbusier defined standard sizes for furnitures and buildings, based on the golden ratio (Corbusier 2000). In this work, we defined two Modulor properties: M27 and M43, corresponding respectively to bench and chair classical height.

1.2 “Find all” Algorithms

We introduced in Sect. 4.2.2 the motivations to design a board segmentation algorithm. To achieve this goal, we designed an original approach that performs the fitting of two parallel primitives at the same time, then it defines the shape by adding the lateral surfaces.

Figure 15 illustrates our board segmentation algorithm, defined as follows:

• let t be a triangle of a board (Fig. 15a), and $\overrightarrow{n}$ its outward normal,

• find the opposite triangle $t'$ using a ray in the opposite direction of $\overrightarrow{n}$ (Fig. 15b),

• fit two parallel planes to these triangles (Fig. 15c),

• starting from t and $t'$, grow iteratively the two regions by selecting only triangles if they fit with one of the two planes (Fig. 15d), and readjust the model at each step,

• when the growing process is finished, we stop the process if the two sides are not similar enough (areas significantly different, or too small overlapping),

• otherwise, we consider the adjacent triangles as initial triangles of the lateral surface growing process (Fig. 15e),

• for each new triangle, find the closest edge e in the boundary of the parallel surfaces (Fig. 15f),

• consider the virtual facet orthogonal to the planes starting from e, and compare it with the new candidate, using the distance between barycenters and the angle between normals.

The initial triangles are selected by first computing a simplified version of the Shape Diameter Function (SDF) (Shapira et al. 2008) where only one ray is used, since we are manipulating CAD models and meshes that represent manufactured objects. Then we sort triangles by ascending SDF value. For each non-visited triangle, we run our board segmentation algorithm. Finally, we run the “is a” algorithms to decide if it is as board shape, a stick shape or a cube shape.

Fig. 15

Details of the “find all” board algorithm, starting from a single triangle (a), growing a region by fitting two parallel planes (a–d), then adding the lateral surface (f–g)

Fig. 16

Result of the segmentation: boards in blue and green, stick in pink, cube in yellow. Light regions corresponds to bad fitting scores. Left original shape. Middle and right two different level of noise

Figure 16 left illustrates the detection of board, stick and cube in a basic shape composed of 75 520 triangles, with a computation time of 3.4 s on a Intel i-7 2.00GHz. We applied a multiscale random noise on the original shape with a root mean square deviation of 0.5 % (respectivly 1 %) of the object’s bounding box diagonal length.

Table 4 Result of the segmentation: semantic labelling and score

During the fitting process, a fitting score is computed for each triangle. The final score of the region (Table 4) is estimated using the mean of these scores, multiplied by the fuzzy result of the corresponding “is a” algorithm.