1 Introduction

Masonry buildings together with reinforced concrete structures account for the largest number of earthquake-related fatalities (D’Ayala 2013) because they are among the most vulnerable structures for various types of loads, including the dynamic loads of earthquakes as well as quasi-static loads such as differential settlements. This highlights the importance of understanding the mechanical response of these type of structures. To study the response of masonry elements at various scales, numerous experimental campaigns have been conducted over the years (e.g., (Morandi et al. 2018; Vasconcelos and Lourenço 2009; Beyer and Dazio 2012; Senaldi et al. 2020; Rezaie et al. 2020; Ghezelbash et al. 2020; Wilding et al. 2018; Parisse et al. 2021)), but experimental studies are limited due to economic restraints and the availability of appropriate testing infrastructure. As a cheaper and more accessible alternative, numerical modeling is thus used to up-scale experimental findings on individual components to entire buildings and to generalize conclusions (e.g., (Celano et al. 2021; Ma et al. 2022; D’Altri et al. 2018; Kouris et al. 2022; Pereira et al. 2021; Zhang and Beyer 2019; Zhang et al. 2017; Soti et al. 2020)). However, much work is still needed to improve the accuracy of models of masonry structures due to their complex behavior, which is caused by the heterogeneous and anisotropic nature of its material as well as and various sources of uncertainties (Preciado and Orduna 2018).

For modeling the mechanical behavior of masonry material, different approaches, scales, and strategies have been proposed, which can be grouped into four categories: block-based, continuum, geometry-based, and macro-element models (D’Altri et al. 2020). Block-based models (Serpieri et al. 2017; Portioli et al. 2014; Lourenço and Rots 1997; Baraldi and Cecchi 2017) represent the behavior of masonry at the size of the material’s primary variability, which is defined by mortar or dry joints connecting blocks. This method may account for the masonry’s texture, which has a substantial impact on its anisotropy and failure pattern. Continuum models (Massart et al. 2007; Berto et al. 2002; Milani et al. 2007; Petracca et al. 2016), on the other hand, consider masonry as a deformable body, enabling wider discretization dimensions and reduced computational cost. Nonetheless, constitutive laws for masonry must be defined adequately, which can be accomplished by either direct approaches or homogenization procedures. Geometry-based models (Marmo and Rosati 2017; Fraternali 2010; Chiozzi et al. 2017; Block and Lachauer 2014) depict the structure as a rigid body based solely on its geometry and loading conditions. Typically, solutions based on limit analysis are employed to assess structural equilibrium and/or collapse. Macroelement models (Vanin et al. 2020; Lagomarsino et al. 2013; Belmouden and Lestuzzi 2009; Caliò et al. 2012; Rinaldin et al. 2016), which are commonly used for seismic assessment, idealize the structure into panel-scale components with nonlinear responses.

In this study, we focus on solid element models and on macro element models (to which equivalent frame models (EFM) belong). In EFMs, walls are represented by frames, which are composed of deformable elements connected through rigid nodes. The deformable vertical elements are called piers and the deformable horizontal elements are called spandrels (Lagomarsino et al. 2013). The simplification of the walls as a frame model is based on post-earthquake observations, which showed that damage tends to concentrate in piers and spandrels (D’Altri et al. 2020). EFMs are mainly used for analyzing the global building response, which is associated with the in-plane capacity of walls (e.g., (Penna et al. 2014)), though EFM approaches have recently been proposed that can directly or indirectly consider the out-of-plane response (Vanin et al. 2020; Angiolilli et al. 2021). When compared to solid-element models, the number of degrees of freedom of an EFM of a masonry building is typically three orders of magnitudes lower, which reduces the computational costs and makes EFMs appealing for engineering practice. Of course, however, the equivalent-frame simplification comes at a cost. For example, in addition the lack of an out-of-plane response, which has since been overcome, D’Altri et al. (2020) also lists the toothing between orthogonal walls as an example of a structural detail that cannot be explicitly considered in EFMs.

Modeling masonry buildings with EFMs requires the discretization of walls into piers, spandrels, and nodes, which can be particularly challenging for walls with an irregular layout of openings. Developed based on damage observations in buildings, Quagliarini et al. (2017) present a collection of rules for discretizing walls (Augenti 2006; Bracchi et al. 2015; Dolce 1991; Lagomarsino et al. 2013), and the most recent work on this topic was presented by Morandini et al. (2022). As a general rule, openings occur above and below spandrels and to the left and right of piers. The length of a spandrel element is typically taken as the mean width of the openings above and below the spandrel. If the openings are not aligned, non-overlapping areas between them are considerate to be rigid. To determine the length of the pier element, there are numerous proposals that differ in their consideration of openings to the left and right of the pier, the direction of the seismic load, and numerical observations. These varying sets of rules produce similar discretizations for regular opening layouts, but the results differ significantly for irregular opening layouts. Figure 1 presents example discretizations of walls with regular and irregular opening distributions, as presented by Quagliarini et al. (2017).

Fig. 1
figure 1

Example approaches for discretizing facades as piers, spandrels, and rigid nodes (from Quagliarini et al. (2017))

Full size image

Multiple works have used EFMs to study the seismic performance of specific buildings (e.g., (Tomić et al. 2021, 2021; Fenu et al. 2022; Miglietta et al. 2021; Vanin et al. 2020; Angiolilli et al. 2021)). In these studies, the finite element geometry was typically drawn manually using computer-aided design (CAD) tools, which can be cumbersome, particularly when the opening layout is irregular. Because several sets of rules exist, the irregular layouts are typically discretized either by the expert selection of the most appropriate method from several or by modifying a discretization after evaluating the numerical results. As such, automating the discretization will accelerate the modeling process. Automation would also make EFM simulations more feasible in post-earthquake assessments when a large number of buildings need to be modeled in a short amount of time. This automation has been done using tools from computer vision and machine learning. For example, 3D point clouds from LiDAR or laser scan devices have been used previously for the generation of finite element models of masonry buildings (Lucidi et al. 2021; Kujawa et al. 2020; Bassier et al. 2019; Funari et al. 2021; Shabani et al. 2021). Additionally, RGB images and photogrammetry techniques have also been explored previously for generating models for finite or discrete element methods (Shabani et al. 2021; Pantoja-Rosero et al. 2023; Abu-Haifa and Lee 2022; Kassotakis and Sarhosis 2021).

Our work aims to contribute to the automatic generation of finite element models of the facade walls of historical masonry buildings. We assume for this purpose that the outer walls are the main load-bearing elements, and we do not model slabs or ring beams, limiting this work to historical masonry elements. We present in this study the first image-based framework that automatically generates finite-element geometries for solid-element models and EFMs for masonry buildings. Because of the subjectivity of the discretization process, our intention is not to produce EFM models that give the most accurate simulation results, but instead provide a generic EFM that meets the criteria of experienced design engineers, which can then be refined to suit specific needs. Available refinements include modifying the discretization or including elements such as roofs, floors, and gables for more accurate results. Overall, we believe that our presented framework can be useful for modeling numerous buildings in a short period of time, such as in post-earthquake assessments.

In this paper, we outline our methodology and apply it to various historical masonry buildings. Section 2 describes the framework and its components, including the construction of a simplified 3D model that captures the information required to generate EFMs, the generation of the EFMs from the geometrical model, and the metrics used to assess the performance of our method. Section 3 presents the results for several example buildings with various degrees of irregularity in the floor plan and layout of the openings. The models are tested using modal analyses. In the final Sect. 5, we present the conclusions of this study and outline future developments.

2 Methodology

Our end-to-end pipeline for automatically generating solid FE models and EFMs is depicted in Fig. 2. Our work builds on state-of-the-art computer vision and machine learning technologies. As input, our framework uses multiple-view RGB images, which are processed using structure-from-motion (SfM) to encode the building scene as 3D point clouds (structure) and the camera poses where the images were taken (motion). To extract the geometrical features for the finite element models, we post-process the 3D point cloud to produce a polygonal surface model that represents a simplified geometry of the building, as detailed in Pantoja-Rosero et al. (2022). This produces a level of detail (LOD) model, specifically LOD3, which consists of a polygonal surface model that details extrusions and openings (Verdie et al. 2015). Such models use simple primitives (e.g., planes) to approximate the actual geometry (Luebke et al. 2003), and we generate them by post-processing the 3D point clouds, segmenting the semantic information in the 2D images (e.g., openings) and projecting it to 3D. Here, we adapted this previous work by performing extra regularization and scaling it to the real dimensions. We used our generated LOD3 model for the FE models of the outer walls of the masonry buildings, specifically solid finite element models and EFMs. For the solid finite element models, we used the Gmsh (Geuzaine and Remacle 2009) open source finite element mesh generator. For the EFM, we proposed a simple rule-based algorithm based on suggestions from the literature (Quagliarini et al. 2017). As output, we provide ready-to-use geometry files in the format required for the finite element software Amaru (Durand and da Silva 2021) (solid FEmodels), Tremuri (Lagomarsino et al. 2013) (EFMs), and OpenSees (McKenna 2011) (EFM with the macro-element by Vanin et al. (2020)).

Fig. 2
figure 2

Pipeline for generating finite element models of freestanding masonry buildings. From top-left to bottom-right: acquire images to generate structure-from-motion (SfM) and regularized LOD2 models; segment and triangulate the openings to 3D; post-process geometric information to generate finite element models with solid- and macro-element approaches

Full size image

2.1 Structure from motion (SfM)

The core of our methodology is photogrammetry, specifically the SfM framework. SfM uses multiple-view images of a scene (i.e., building) to compute a 3D point cloud that shows both point features (structure) and the camera poses where the images where taken (motion) (Szeliski 2021; Hartley and Zisserman 2001). This method’s advantages include its low cost (it does not require expensive hardware), accessibility (cameras and software tools are widely available), versatility (it may be used for a variety of applications), and high precision (when appropriate images are taken). On the other hand, some limitations have been noticed, including limited coverage (objects must be seen on photos), limited resolution (model quality is dependent on the quality of the photographs), time-consuming (for huge amounts of image data), and lighting conditions that must be fulfilled.For an example building scene, Fig. 3 shows the images used to generate the SfM model and structure and motion outputs, which were generated using the same open source library meshroom (Griwodz et al. 2021) that is also used later to generate the solid finite element models.

Fig. 3
figure 3

SfM modeling. a Some images used for SfM processing. b Reconstructed 3D point clouds and camera poses

Full size image

2.2 LOD2: regularized polygonal surface model

The SfM point cloud is first translated to a polygonal surface model without openings (LOD2 model), wherein the exterior geometry of the building is simplified as a set of planes comprising its facades, its roof planes, and the ground level. To generate this model, we use the Polyfit framework proposed by Nan and Wonka (2017) presented in Fig. 4. The input for this methodology is point clouds clustered as plane primitives whose intersections generate candidate faces that might be part of the final model. The faces that will compose the resultant LOD2 model are selected by solving a binary linear optimization problem that accounts for the coverage and fitting of the point cloud over the faces as well as the complexity of the geometry. This problem is formulated using hard constraints to generate a manifold and watertight model. Although this method has proven robustness in the generation of simplified geometries of objects whose main geometry can be represented by plane primitives, it does require the input point cloud to be complete and noise-free, necessitating preprocessing in some cases (e.g., delete points that belong to foreign objects).

Fig. 4
figure 4

Polygonal surface models generated using Polyfit. a Input point cloud. b Clustered planar primitives. c Candidate faces from plane intersections. (d) Polygonal surface model after optimization process

Full size image

Modeling the exterior structural walls of a building with an EFM requires that the walls be vertical. This is not guaranteed when applying the Polyfit approach, so we regularized the models with the assumption that the buildings were constructed as per the Manhattan world scenes (Coughlan and Yuille 1999), which assumes most civil infrastructure is built following three main perpendicular directions. Note that this does not always apply to historical masonry buildings, so we plan to improve this aspect for future work. The building regularization first defines the three main directions by randomly sampling the LOD2 model as a point cloud and performing a principal component analysis (PCA) (Jolliffe 2002). PCA yields the three main directions, which are approximately aligned with the vertical direction and the normal directions of the walls. The three main directions of the building are then computed as the two mean vectors of normal vectors of the planes identified as facades in the perpendicular directions and their cross product. Once the three main building directions are defined, the building is regularized by: 1) modifying the facade planes such that their normal vector is aligned with one of the two horizontal components of the main directions; 2) replacing the ground plane with a plane that passes through the lowest LOD2 vertex and has a vertical normal vector (note that facade vertices in contact with the ground plane are displaced accordingly). This regularization procedure is depicted in Fig. 5.

Fig. 5
figure 5

LOD2 regularization. a Irregular LOD2 and global coordinate system. b Sampled point cloud and main PCA directions. c Regularized LOD2 (filled) overlaying the irregular LOD2 (wireframe)—a regular LOD2 guarantees perpendicularity for the ground and facade planes

Full size image

2.3 LOD3: mapping opening information to 3D

Analyzing the building via a solid finite element model or EFM requires information on the geometry of the openings. Based on our work presented in Pantoja-Rosero et al. (2022), we first segment openings in the facades in the 2D images inputted to the SfM analysis and then map these openings to the LOD2 using the camera poses. Different from the previous work (Pantoja-Rosero et al. 2022), here we first regularize the polygonal surface model and then map the openings using ray-casting instead of two-view triangulation. For the later, we segment the openings and their corners in the 2D images using convolutional neural networks (CNNs). Each opening is represented on the image as a set of four points indicating their vertices \(\textbf{x}\) that are then mapped over the LOD2 model (finding their 3D corresponding \(\textbf{X}\)) using ray-casting. This ray-casting was already used in (Pantoja-Rosero et al. 2023) to project damage features (e.g., cracks) onto an LOD3 model. In general, the ray-casting algorithm finds the 3D correspondence \(\textbf{X}\) of a 2D image point \(\textbf{x}\) by locating the intersection of a projected ray that passes by the camera center C of the view (output of SfM) and \(\textbf{x}\) with the 3D model (LOD2). This information about the 3D opening can be merged with the LOD2 model to obtain an LOD3 model, which is the basis for the finite element models. This procedure is illustrated in Fig. 6.

Fig. 6
figure 6

LOD3 model—mapping information about openings to 3D. a Segmentation of openings and their corners using CNNs. b Ray-casting the points of the 2D image \(\textbf{x}\) that correspond to opening corners to obtain 3D correspondences \(\textbf{X}\). c Openings in 3D formed by their points \(\textbf{X}\). d LOD3 model with surface and opening information

Full size image

Since this method relies on the deep learning technique to segment the requisite 3D mapping openings, it inherits the benefits and drawbacks of such techniques. Despite the fact that the results in general demonstrate excellent accuracy, flexibility, speed, and automation, it is necessary for training to utilize computationally intensive huge data sets.

2.4 Model initialization

The SfM pipeline produces similarity reconstructions (typically called Euclidean reconstructions) (Hartley and Zisserman 2001), meaning that relative distances and angles are preserved but do not represent the absolute dimensions. Furthermore, the orientation and position of the coordinate system is aleatory. Therefore, before generating FE models, the LOD3 model is initialized by applying a similarity transformation that rotates, translates, and scales the model to place it at the origin in alignment with the main directions of the building and scaled to the real dimensions. In homogeneous coordinates, this transformation can be represented as:

$$\begin{aligned} \textbf{V}_{init} = {\textbf {T}}(\textbf{R}, \textbf{t}, s)\textbf{V}, \end{aligned}$$
(1)

where \(\textbf{V}\) and \(\textbf{V}_{init}\) are the coordinates of the vertices of the model before and after initialization, and \({\textbf {T}}(\textbf{R}, \textbf{t}, s)\) is the 4x4 similarity transformation matrix defined as

$$\begin{aligned} {\textbf {T}}(\textbf{R}, \textbf{t}, s) = \begin{bmatrix} s\textbf{R} &{} \textbf{t} \\ \textbf{0}^\intercal &{} 1 \end{bmatrix}, \end{aligned}$$
(2)

with \(\textbf{R}\), \(\textbf{t}\), and s respectively representing a 3x3 rotation matrix, a 3x1 translation vector, and a scaling factor. In our algorithm, we first use the main LOD2 directions located using PCA to find the transformation matrix that represents rotation and translation and its lowest ground point (sect.  2.2). This transformation matrix, which places the LOD model at the origin and aligns it with the main directions of the building, is computed solving the equation:

$$\begin{aligned} {\textbf {T}}(\textbf{R}, \textbf{t}) = \textbf{P}_{init} \textbf{P}^{-1}, \end{aligned}$$
(3)

where \(\textbf{P}\) and \(\textbf{P}_{init}\) are 4x4 matrices wherein each column represents a point (in homogeneous coordinates) of the LOD model and the origin of the coordinate system. Specifically, \(\textbf{P}\) columns are formed by: \(\textbf{P}[:,1]\) representing the lowest LOD2 vertex of the ground plane; \(\textbf{P}[:,2]\) representing a point placed at unit distance from \(\textbf{P}[:,1]\) in the same direction as the main vertical LOD2 direction; and \(\textbf{P}[:,3]\) and \(\textbf{P}[:,4]\) points that are placed at a unit distance from \(\textbf{P}[:,1]\) in the same direction as the other two main LOD2 directions. At the same time, the \(\textbf{P}_{init}\) columns represent the four points where the \(\textbf{P}\) points should be placed after the transformation. These points are: \(\textbf{P}_{init}[:,1] = [0,0,0,1]^\intercal\) representing the origin; \(\textbf{P}_{init}[:,2] = [0,0,1,1]^\intercal\) representing the point at a unit distance from the origin in the z direction; \(\textbf{P}_{init}[:,3] = [1,0,0,1]^\intercal\) representing a point at a unit distance from the origin in the x direction; and \(\textbf{P}_{init}[:,4] = [0,1,0,1]^\intercal\) representing a point at a unit distance from the origin in the x direction. To compute the scaling factor, the user needs to provide two image points \(\textbf{x}\) and the distance d between them. The ray-casting algorithm finds the corresponding \(\textbf{X}\) over the LOD2 model and then computes their distance D. With this information, the scaling factor is obtained as \(s=d/D\). The initialization of the model is presented in Fig. 7.

Fig. 7
figure 7

Model initialization. LOD models and opening coordinates are initialized by placing them at the origin, aligning them with the main building directions, and scaling them to the real dimension. a LOD3 models before and after rotation and translation (\(\textbf{P}\) and \(\textbf{P}_{init}\) points shown in magenta and orange, respectively). b Initial (small wireframe) and final LOD3 (filled) models after full initialization (rotation, translation, and scaling)

Full size image

2.5 Generating a finite element model with solid elements

One of the outputs produced by our framework is a finite element model that uses solid elements. This is produced by applying the Gmsh finite element mesh generator (Geuzaine and Remacle 2009) to the geometrical information encapsuled in the initialized LOD models and 3D openings. As a first step in this process, all the planes labeled as facades (sect.  2.2) are built as surface entities of dimension 2 by joining their vertices (entities of dimension 0) with lines (entities of dimension 1). To next generate volume entities of dimension 3, an extrusion operation is applied to these facade surfaces to simulate the wall thickness th (user input). This same technique is also used to generate volume entities for all the building openings. Finally, the opening volumes are subtracted from the facade volumes using a Boolean difference operation to generate a physical group that the mesh generator discretizes into solid elements. The type of element, its order, and its size are defined by the user. Figure 8 presents the procedure for generating a finite element model for a building using first-order tetrahedral elements.

Fig. 8
figure 8

Finite element model with solid elements generated using GmshAPI (Geuzaine and Remacle 2009). a Facades and openings as surface entities. b Facades and openings as volume entities after extruding the wall width. c Physical group used to generate the solid-element mesh after performing binary operations. d Finite element model generated using tetrahedral solid elements

Full size image

2.6 Generating an EFM

As outlined in the introduction, EFM requires the discretization of each building facade into deformable parts (piers or spandrels) connected by rigid nodes. To achieve this aim, we coded for regular openings using the rule by Dolce (1991). To adapt the code for irregular opening layouts, we designed a general first step for defining the layout of the cells, which we then label as openings, piers, spandrels, or nodes (Fig. 9). The first part of the algorithm generates the geometry of the polygonal cells in each facade (in 2D local coordinates) as follows:

  • Step 0: Find the lines that represent the facade contour (contour lines) and openings (opening lines).

  • Step 1: Compute all of the intersections of the facade lines.

  • Step 2: Filter the intersections: (1) leave those that lay inside the facade or on the contour; (2) remove intersections produced by a contour line and an opening line that lay inside the facade; (3) remove intersections produced by two contour lines that are not part of the facade contour vertices.

  • Step 3: Create line segments by connecting the consecutive points that lay in each of the facade lines.

  • Step 4: Produce a graph in which the nodes are formed by the filtered intersections and the edges are formed by the line segments.

  • Step 5: Find the polygonal cells formed by the graph using the graph–cycles-based algorithm proposed by Ferreira et al. (2003).

Fig. 9
figure 9

Generation of polygonal cells in the building facades that will be labeled as openings, piers, spandrels, and nodes. a Lines representing the facade contour and openings. b Intersections of facade lines, where red points correspond to filtered intersections. c Graph formed by filtered intersections and line segments. (d) Polygonal cells to be labeled

Full size image

The second part of the algorithm is a rule-based procedure that labels the polygonal cells as openings (o), piers (p), spandrels (s), and nodes (n). Here, we offer the user two options, which yield the same results for a regular layout of openings but different results for irregular layouts. Approach A leads to longer piers while Approach B leads to deeper spandrels. The process applied to one building facade is presented in Fig. 10, and the results for all facades of the example building are presented in Fig. 11. The labeling algorithm is as follows:

  • Step 0: For each cell, determine the eight neighboring cells: top (t), bottom (b), left (l), right (r), top-left (tl), top-right (tr), bottom-left (bl), bottom-right (br).

  • Step 1: Label as openings all the cells whose mean coordinate lays inside the LOD openings.

  • Step 2: Label as piers the cells to the left (l) and right (r) of the labeled openings. Label as spandrels the cells above (t) and below (b) the labeled openings.

  • Step 3: Label as nodes the neighboring cells of the labeled openings in the tl, tr, bl, and br positions.

  • Step 4: Grow horizontally the cells labeled as piers and grow vertically the cells labeled as spandrels. This item has two approaches. Approach A: longer piers; approach B: deeper spandrels.

    • Approach A Loop through all the polygonal cells. If a cell is labeled as a pier, also sequentially label as a pier all the unlabeled cells to its left and right until an already-labeled cell is found. Loop again through all the polygonal cells. If a cell is labeled as a spandrel, also sequentially label as a spandrel all the unlabeled cells above and below until an already-labeled cell is found.

    • Approach B Loop through all the polygonal cells. If a cell is labeled as a pier, also sequentially label as a pier all the unlabeled cells to its left and right until the t or b neighbor of the unlabeled cell is a spandrel (without labeling this cell). Loop again through all the polygonal cells. If a cell is labeled as a spandrel, also sequentially label as a spandrel all the unlabeled cells above and below until the l and r neighbors of the unlabeled cell are piers (labeling this cell as pier).

  • Step 5: Loop through all the polygonal cells. If a cell is labeled as a node, also sequentially label as a node all the unlabeled cells to its left and right until reaching an already-labeled cell. Loop again through all the polygonal cells. If a cell is labeled as a node, also sequentially label as a node all the unlabeled cells above and below until reaching an already-labeled cell.

Fig. 10
figure 10

Facade discretization process. a Facade with unlabeled polygonal cells. b Facade polygonal cells after labeling step 1. c Facade polygonal cells after labeling step 2. d Facade polygonal cells after labeling step 3. e Approach A: Facade polygonal cells after labeling steps 4 and 5. f Approach B: Facade polygonal cells after labeling steps 4 and 5

Full size image
Fig. 11
figure 11

3D results of the facade discretization as seen from two views

Full size image

Note that our framework was designed to model only piers and spandrels as macro-elements. The use of gable elements, such as the one used by Vanin et al. (2020), should improve the discretization and will be considered in future work. Similarly, we will consider implementing variable pier lengths depending on the loading direction, as has been suggested by several research groups (e.g., Quagliarini et al. (2017)).

2.7 Meshing the EFM

The last part of our end-to-end pipeline is an algorithm that transforms the discretized facade into a mesh for an EFM. The EFM consists of a wire frame composed of special frame elements (macro-elements) defined by their end points (structural nodes), center point, and geometric dimensions. We assume that the cross sections of all piers and spandrels are rectangular. The finite element mesh we generate here matches the rules set by Tremuri Lagomarsino et al. (2013), which can be used either with the original nonlinear beam formulation or the macro-element by Penna et al. (2014). The same mesh is also adopted when using the macro-element by Vanin et al. (2020) in OpenSees McKenna (2011). In our mesh, structural nodes are defined based on the polygonal cells labeled as nodes and on the connectivity between each pair of walls. The macro-elements (piers and spandrels) are then defined based on their connection with neighboring nodes. The process followed by the algorithm is shown in Fig. 12, and the results for all the facades are shown in Fig. 13. The algorithm follows these steps:

  • Step 0: Cluster the macro-element cells. Neighboring polygonal cells that have the same label are assigned to a single element (pier, spandrel, or node). If the cells that compose the element produce a non-rectangular shape, the neighboring cells that have different labels are re-labeled to guarantee rectangular elements.

  • Step 1: Label facade elements (nodes, spandrels, and piers) as contour or \(non-contour\). Assign an extra label to those placed at the contour as top (t), bottom (b), left (l), right (r), top-left (tl), top-right (tr), bottom-left (bl), bottom right (br).

  • Step 2: Define whether nodes are 2D or 3D and find their structural node coordinates. Because Tremuri (Lagomarsino et al. 2013) captures only the in-plane and not out-of-plane behavior of the piers and spandrels, the elements representing the piers and spandrels only have degrees of freedom in the plane of the façade. As a result, Tremuri  distinguishes between 3D nodes, which are those that are on the vertical edges of the contour and therefore in contact with other facades, and all other nodes, which are considered 2D nodes. The structural nodes are placed on the bounding box (bbx) that surrounds each node element and its type. \(non-contour\) node elements: middle point of bbx. t: middle-top point of bbx. b: middle-bottom point of bbx. l: middle-left point of bbx. r: middle-right point of bbx. tl: top-left point of bbx. tr: top-right point of bbx. bl: bottom-left point of bbx. br: bottom-right point of bbx.

  • Step 3: Determine the center point of the pier and spandrel elements.

  • Step 4: Place extra structural nodes (without mass) at the ground level of the model beneath each pier element. The new nodes are placed below contour bl, b, and br piers. b: middle-bottom point of bbx. bl: bottom-left point of bbx. br: bottom-right point of bbx.

  • Step 5a: Define pier and spandrel end points \(n_i\) and \(n_j\) for facades with openings. For each pier, \(n_i\) and \(n_j\) are the neighboring nodes to the bottom and top, respectively. If there are multiple neighboring nodes at the top and/or bottom, the pier is split into several piers according to the number of nodes. For each spandrel, \(n_i\) and \(n_j\) are the neighboring nodes to the left and right, respectively. If there are multiple neighboring nodes to the left and/or right, the spandrel is split into several spandrels according to the number of nodes.

  • Step 5b: Define piers for facades without openings. When there are no openings in a facade, the pier element is split horizontally into two. To do this, two additional structural nodes (without mass) are created at the top-left and top-right of the bbx surrounding the pier element. Then, \(n_i\) and \(n_j\) represent the bottom-left and top-left nodes for the first pier, and bottom-right and top-right for the second. If the facade is connected to another facade containing more than one row of openings in its layout, the pier corresponding to this connection is split according to the number of extra nodes in the neighboring facade. Every time a pier is split, rigid nodes are generated to link the new piers to guarantee a proper connectivity within the wall.

  • Step 6: Merge corresponding 3D nodes between facade pairs. Two facades are connected through their 3D nodes by merging the closest 3D nodes to the two facades into one new coordinate equal to the mean of their individual coordinates. If the new coordinate lays outside one of the node element regions, the new coordinate is changed to the largest or lowest point of that node element. If the number of opening rows from one facade and the next do not match, there will be more 3D nodes in the facade with the highest number of row openings. In this case, the extra nodes are shared between the facades, and a rigid link is created to connect each extra node to its closest node. When one of the facades does not have openings, as described in the previous step, the piers of that facade are split at the position of the 3D nodes of a neighboring facade that has more than one row of openings in its layout.

  • Step 7: Create finite element geometry files for Tremuri (Lagomarsino et al. 2013) and OpenSees (McKenna 2011) considering rectangular geometry for the elements, structural node placements, and frame element connections. As boundary conditions, all the nodes placed at the ground level are constrained to have no displacements on all degrees of freedom (Fig. 14).

Fig. 12
figure 12

Generating the geometric components of an equivalent frame model (EFM). a Step 0: Neighboring polygonal cells with same label (piers, spandrels, and nodes) are clustered into a single facade element. b Step 1: Facade elements labeled as contour or \(non-contour\) elements at t, b, l, r, tl, tr, bl, br positions. c Steps 2–4: 3D and 2D nodes and the position of the structural node point. d Steps 5,6: Representation of the finite macro-element as wire elements that connect to the final position of the structural nodes

Full size image
Fig. 13
figure 13

Geometric components of EFM for the four facades that comprise the building. a Facade 1. b Facade 2. c Facade 3. d Facade 4

Full size image
Fig. 14
figure 14

3D results for the EFM seen from two views

Full size image

2.8 Metrics and performance assessment

A general rule that works well for regular openings states that wall regions with openings above and below correspond to spandrels and wall regions with openings left and right correspond to piers. For facades with irregular opening layouts, this procedure is subjective, and the discretization relies on the stated criteria as well as the experience of the engineer. Often, a group of engineers discuss the definition of a model that could best represent the expected behavior of the building. We attempted to reproduce the discretization of experienced engineers with the algorithm that we introduced in this section, but the proposed solution remains only one of many. To quantitatively assess the performance of our framework for modeling masonry buildings using macro-elements in light of this subjectivity, we propose two metrics for its evaluation by experienced engineers (evaluator). The two metrics are as follows:

  • Metric A (MA): For this metric, the evaluator was given images of building facades containing unlabeled polygonal cells (Fig. 15a). The evaluator assigned each unlabeled cell as what she/he considered to be the most reasonable label. The cell labels given by the evaluator were compared with the labels automatically assigned by our algorithm. Metric A is the percentage of cells that were labeled in the same manner.

  • Metric B (MB): For this metric, the evaluator was asked to give a grade between 1.0 to 4.0 to the mesh of the EFM produced by our framework (Fig. 15b). The grades were assigned as follows:

    1. 1.

      Poor: the model makes no sense. Totally misaligned with the evaluator's criteria.

    2. 2.

      2 Sufficient: the model might produce acceptable results and can be used for numerical analyses. Partially misaligned with the evaluator's criteria.

    3. 3.

      3 Good: the model might produce accurate results and can be used for numerical analyses. Almost completely aligned with the evaluator's criteria.

    4. 5.

      4 Excellent: the model should produce accurate results and can be used for numerical analyses. Completely aligned with the evaluator's criteria.

Fig. 15
figure 15

Images given to the evaluator for assessing our methodology. a Unlabeled polygonal cells of a facade. b Building facades with the areas defined as elements (piers, spandrels, and nodes) with wire-frames that represent the geometrical components of the EFM

Full size image

3 Experiments

In this section, we present the results of applying our framework to 12 example free-standing historical masonry buildings in Switzerland or Croatia, as published by Pantoja-Rosero et al. (2022), Pantoja-Rosero et al. Pantoja-Rosero et al. (2023). The buildings in Croatia were damaged in the 2020 Zagreb or Petrinja earthquakes (Atalić et al. 2021; Miranda et al. 2021), which fits well with a proposed application of this method for the numerical modeling of earthquake-damaged buildings. For each building, the input data set consisted of multiple-view images. We first present various models generated within the pipeline, starting with the SfM up to the finite element model with solid elements and the EFM. For the EFMs, we also show the scores assigned by four engineers with experience in modeling masonry buildings. To perform modal analyses of the solid element models and the EFMs, we used the Amaru (Durand and da Silva 2021) and OpenSees (McKenna 2011) finite element libraries, respectively. Some of the deformed shapes of this modal analysis are presented in the last section.

3.1 Finite element geometry

Figures 16 and 17 present some of the models obtained from our framework, focusing on non-damaged and damaged buildings, respectively. Across both images, column A shows the photogrammetry output from the meshroom software, which is a textured mesh. Column B shows the SfM model, which consists of a sparse point cloud and registered camera poses. Next, the polygonal surface model is displayed as an LOD3 model in column C. The finite element geometries using solid elements and EFMs are split by discretization approach, with Approach A presented Fig. 16 and Approach B presented in Fig. 17. In addition to the visual results, the four scores computed for each building and discretization approach are presented in Table 1 (\(MA-A\): Metric A; discretization approach A. \(MA-B\): Metric A; discretization approach B. \(MB-A\): Metric B; discretization approach A. \(MB-B\): Metric B; discretization approach B). The metrics were computed based on the assessments of four engineers, who are experienced in modeling masonry buildings using EFMs as described in Sect. 2.8.

Fig. 16
figure 16

Models generated by our framework for non-damaged buildings. a Photogrammetry textured mesh. b SfM model as point cloud and camera poses. c LOD3 models containing simplified surface and opening information. d Finite element model using solid-elements (tetrahedral elements of the first order). e Finite element model using macro-elements with discretization Approach A

Full size image
Fig. 17
figure 17

Models generated by our framework for damaged buildings. a Photogrammetry textured mesh. b SfM model as point cloud and camera poses. c LOD3 models containing simplified surface and opening information. d Finite element model using solid-elements (tetrahedral elements of the first order). e Finite element model using macro-elements with discretization Approach B

Full size image
Table 1 Metric values for assessing the generation of EFMs using our framework (MA-A: Metric A%; discretization approach A. MA-B%: Metric A; discretization approach B. MB-A: Metric B; discretization approach A. MB-B; Metric B, discretization approach B)
Full size table

Table 1 shows that the MA scores average over 90% (metric mean values - \(MA-A: 92.7\%\); \(MA-B: 90.6\%\)) and MB scores, with one exception, are equal to or larger than 2.0 (metric mean values - \(MB-A: 3.2\); \(MB-B: 3.1\)). In general, the experts were satisfied with the performance of the methodology for all example buildings studied, with the framework producing appropriate discretizations and definition in the EFMs for numerical simulations of the building. The only disagreement with the experts stemmed from personal criteria in the discretization process, which they considered to be expected and normal. Their most relevant comments for improvement were: 1) the systematic methodology sometimes produces overly large and rigid nodes—their size should be reduced and adjacent piers or spandrels should be increased; 2) small openings should be disregarded to avoid creating too many rigid nodes. Improvements along these directions will be considered in the future (see Conclusions). Overall, these results show the clear potential for the use of image-based approaches in automatically generating finite element geometries of the facades of historical masonry buildings.

3.2 Modal analysis

To evaluate the performance of the models in the elastic domain, we performed modal analyses using two different open source software, Amaru (Durand and da Silva 2021) and OpenSees (McKenna 2011), for the solid-element model and the EFM, respectively. For the material properties, we chose values representative of stone masonry. As our intention with this analysis is only to present the functionality of the models, details of the analysis, such as material properties and input and output information, are not discussed in detail, and we refer the reader to the dataset published together with this work. The results consisting of some of the deformed shapes obtained after performing such simulations are presented in Figs. 18 and 19.

Fig. 18
figure 18

Deformed shapes output by modal analyses using the FE models generated by our framework as the input geometry—non-damaged buildings. a Two deformed shapes obtained using solid-elements in Amaru Durand and da Silva (2021). b Two deformed shapes using macro-elements in OpenSees McKenna (2011)

Full size image
Fig. 19
figure 19

Deformed shapes output by modal analyses using the FE models generated by our framework as the input geometry—damaged buildings. a Two deformed shapes obtained using solid-elements in Amaru Durand and da Silva (2021). b Two deformed shapes using macro-elements in OpenSees McKenna (2011)

Full size image

4 Discussion

Our findings have significant implications for future structural engineering and construction research. Using image-based methodologies to generate finite element geometries offers the potential to automate the structural modeling procedure. This could result in substantial time and cost savings when analyzing existing buildings. Moreover, the methodologies described in this study could be combined with other developing technologies, including machine learning and artificial intelligence, to enhance the precision and efficiency of structural modeling of damaged structures. For instance, the addition of damage observed on images can be associated to the mechanical properties of damaged material, which can then be incorporated into numerical models.

Advanced technologies enable faster and more accurate data collection and analysis, allowing the identification of potential structural issues before they become critical. The use of drones, remote sensors, and other cutting-edge technologies, for instance, can provide important details about a structure’s condition, including any damage or deterioration. This data can be used to generate more precise structural models, which can be used to simulate various scenarios and identify potential vulnerabilities. This information can be used to guide decisions on maintenance, repair, and retrofitting, and to ensure that structures are better prepared to withstand natural disasters.

5 Conclusions

The objective of this study was to contribute to the numerical modeling of historical masonry buildings by providing an end-to-end framework that automatically generates finite element meshes of facade walls for solid-element models and EFMs. This automated approach is an alternative to the manual operation, in which engineers draw the geometry of the building using CAD software. The only input required by our methodology is multiple-view images of the building, which are processed using SfM to encode the geometrical information of the building as a point cloud with camera poses. Convolutional neural networks extract information about openings and their corners, which together with the point clouds can produce the LOD3 model.

Though the solid element model of the facades can be obtained simply by extruding the facades along the wall thickness, generating EFMs is more challenging. The discretization of a facade into an EFM is still a subjective process that requires the expertise of experienced engineers on numerical modeling of masonry buildings. Our algorithms were design to aid this process by capturing the overall concepts applied by engineers as an initial generic model that the user can modify. Even so, the results obtained during our experiments were satisfactory to four expert engineers, which motivates further development of our work. To this end, we plan to perform additional investigations in various directions, including the addition of gable elements and other building elements such as roof and floor levels and varying the length of the pier element when the openings to the left and right have different heights or if the pier is at the edge of a facade. Additionally, we will increase the capabilities of the discretization process by allowing the consideration of different literature criteria with available pier and spandrel dimensions. We will also include information from interior point clouds, from which we can retrieve wall thicknesses, interior wall geometries, and spanning directions of timber slabs. Finally, to develop inputs for rapid post-earthquake damage assessments, we plan to develop techniques for assessing the response of damaged buildings after earthquakes for several aftershock scenarios, including uncertainty analyses.

This study demonstrated that automated image-based approaches may produce accurate finite element models for masonry buildings, which can be used for estimating their structural response. This method can considerably reduce the time and effort necessary to build finite element models, which is now a difficult manual procedure using CAD software. By automating this procedure, our system can rapidly simulate several buildings, which is required for some engineering applications such as post-earthquake assessments. Overall, our research emphasizes the capability of automated image-based techniques in creating finite element models for masonry structures. We are confident that our approach can be integrated into the software utilized by civil engineering practitioners.