Digital 3D models for medieval heritage: diachronic analysis and documentation of its architecture and paintings

Abstract

In this paper, we discuss the requirements and technical challenges within the EHEM project, Enhancement of Heritage Experiences: The Middle Ages, an ongoing research program for the acquisition, analysis, documentation, interpretation, digital restoration, and communication of medieval artistic heritage. The project involves multidisciplinary teams comprising art historians and visual computing experts. Despite the vast literature on digital 3D models in support of Cultural Heritage, the field is so rich and diverse that specific projects often imply distinct, unique requirements which often challenge the computational technologies and suggest new research opportunities. As good representatives of such diversity, we describe the three monuments that serve as test cases for the project, all of them with a rich history of architecture and paintings. We discuss the art historians’ view of how digital models can support their research, the expertise and technological solutions adopted so far, as well as the technical challenges in multiple areas spanning geometry and appearance acquisition, color analysis and digital restitution, as well as the representation of the profound transformations due to the alterations suffered over the centuries.

1 Introduction

Visual computing technologies such as 3D modeling, photogrammetry, and laser scanning have been extensively used to digitize historical monuments. These technologies allow for the 3D reconstruction of the surfaces of monuments and their later visualization. Yet, although there is a vast literature on the use of these technologies in Cultural Heritage, it does not reflect much interaction between art historians and engineering experts in order to get proper insights into the creation and curation of these models. This leads, on one hand, to virtual reconstructions, made only by technical teams, that soon fade away as they do not satisfy art historian requirements in terms of quality, context, or added value, and, on the other hand, to Cultural Heritage projects that do not benefit of ICT technologies and run aground in manual processes, disconnected data, or the need of models where hypotheses can be tested. This gap in the communication between art historians and engineering experts results in simplified superficial representations of the monuments that lack proper context, documentation, and historical interpretation.

Fig. 1

A Digital Twin is a digital representation of a physical monument that integrates different types of data from different sources such as sensors, layouts, and cameras. This allows for the creation of novel applications such as advanced interactive visualizations in VR, dynamic timelines of change, and analytics/machine learning to get a better understanding of the monument by experts and anyone interested

Fig. 2

In the context of the EHEM project, we explore three significant sites of medieval paintings, each exemplifying paintings’ qualities, intricate study aspects, and diverse conservation challenges. From left to right: Enkleistra of Neophytos (Cyprus), Santa Maria Antiqua (Italy), and Sant Quirze de Pedret (Spain)

In this paper, we discuss the requirements from art historians and technical challenges within the EHEM project,¹ an ongoing research project for the analysis, documentation, interpretation, digital restoration, dissemination, and communication of medieval artistic heritage. This project is developed by a multidisciplinary team that includes experts from both areas: art history and visual computing. We discuss this work in the context of Digital Twins (DT) [1]; digital representations of physical entities that integrate data that typically come from a variety of sources (Fig. 1). These data include operation data, live data from sensors, or other data such as drawings and documents. A realistic 3D DT of a monument allows for the development of several novel tools that include technologies such as Virtual Reality, timelines of change, simulations, and analytics; in our opinion, these tools can lead to better interpretations of the history of physical monuments, better dissemination and communication between stakeholders, and better maintenance, management, and operations.

2 Requirements from art historians

As Otto Demus [2] convincingly stated in 1970, monumental paintings of medieval churches were an integral part of a three-dimensional liturgical device that “echoed the text of the liturgy in pictorial terms.” The efficiency of this mural decoration in churches and monastic complexes as a spiritual model and devotional aid justifies its repeated repairs throughout Medieval times. However, medieval monumental painting has survived to the current day in incomplete form, with considerable lacunae and in a state of decay, especially of the vivid colors they once had, which makes its understanding extremely difficult. Since there is no justification whatsoever for the restoration of these medieval pictorial decorations, the only sustainable way to restore their integrity and advance how the public views, comprehends, and enjoys them is by generating enhanced virtual 3D models, like a Digital Twin.

A digital twin for cultural heritage monuments caters to a diverse audience, including researchers, historians, preservationists, architects, urban planners, tourism agencies, and the general public. The primary goals of this technology are preservation and documentation, capturing the monument’s physical features and historical significance in a digital format. It facilitates in-depth research, analysis, and comparative studies, enabling interdisciplinary investigations and architectural exploration. Moreover, the digital twin serves as an educational tool, offering immersive experiences for public engagement and promoting wider access to cultural heritage. Additionally, it assists in conservation and restoration efforts by providing a comprehensive understanding of the monument’s condition and aiding in strategic decision-making for its long-term preservation.

Table 1 Comparison between the paintings of the different sites studied in the EHEM project: Enkleistra of Agios Neophitos (EAN), Santa Maria Antiqua (SMA), and Sant Quirze de Pedret (SQP)

The EHEM project proposes to recover the medieval buildings and decorations of three ensembles as illustrative examples of different histories and different stages of degradation and fragmentation (Fig. 2). Thus, in the case of the Enkleistra of Agios Neophytos (EAN) (Cyprus), the main objective is to show compellingly the research hypotheses on the successive architectural and decorative phases from the twelfth to the sixteenth centuries. Santa Maria Antiqua (SMA) (Rome, Italy) presents as a major challenge the coexistence in its walls up to ten layers of pictorial decorations. As Sant Quirze de Pedret (SQP) (Catalonia, Spain) is concerned, one of the biggest defies to achieving a comprehensive understanding of the Pedret’s two layers of frescoes is that they no longer decorate the walls for which they were painted, because they have been removed and split up, and they are now on display in two different museums, the Museu Nacional d’Art de Catalunya (MNAC), in Barcelona, and the Museu Diocesá i Comarcal de Solsona (MDCS). All these monuments have superimposed layers of pictorial decorations that should be displayed in a chronological sequence to facilitate a better understanding of them.

Fig. 3

The four hypothetical layouts of the first level of the Enkleistra

Fig. 4

Current situation of the interior of St. Quirze de Pedret (central nave and central and north apse, left) and the exhibition of its paintings in MDCS (central nave and apse, center) and MNAC (north and south apse, right). The fragmentation of the artworks in three different locations considerably complicates its understanding

In the following sections, we give some more details about the sites, the requirements, as defined by the art history experts of the EHEM project, and the process we followed to define them.

2.1 Case studies and previous digitizations

In this section, we explore our three significant historical sites: Enkleistra of Agios Neophytos, Santa Maria Antiqua, and Sant Quirze de Pedret. We provide a detailed depiction of their architectural features and historical importance. Additionally, we review previous digitization efforts undertaken to document and preserve these sites. Table 1 compares the three sites highlighting similarities and differences.

Enkleistra of Neophytos Located near the coastal city of Paphos on the island of Cyprus, the Enkleistra of Agios Neophytos at Tala is considered one of the major Byzantine and Medieval monuments of the Mediterranean, mostly due to its extraordinary Byzantine-style wall paintings—still surviving in situ—and its founder Neophytos the Recluse (d. ca. 1215) who was a prolific writer. The site initially functioned as the hermitage of Neophytos who found the cave on June 24, 1159, when the island was still part of the Byzantine Empire. Over this first year of his stay, he transformed it into his ascetic retreat, naming it Enkleistra in Greek (Fig. 3a.) The Enkleistra evolved into a monastery with a small number of monks, supported by the bishop of Paphos. During this period, a partition wall was likely built, confining Neophytos to his cell. Theodoros Apseudes conducted a mural campaign in the bema of the Enkleistra (Fig. 3b). During the Latin rule of the island, the Enkleistra expanded including the cave refectory north of Neophytos’ cell, the cave church of the Holy Cross south of his cell, and the adornment of the Orthodox church with Byzantine-style wall paintings (Fig. 3c). The Enkleistra complex also included an upper storey Hagiasterion, connected to the church through a hagioscope shaft. Besides that, a new cell and a chapel were carved higher up the cliff. Additional structures were hewn to the north of the refectory and west of the occasional torrent. These developments accommodated the growing monastic community, which reached a maximum of eighteen monks by 1214. The first major restoration of the Enkleistra and its wall paintings took place in 1502/3 (Fig. 3d).

In recent years, the site has been the focus of various campaigns of study and conservation. The main restoration campaign of 1963 at the monument was undertaken by the Dumbarton Oaks research institute, Washington DC, in collaboration with the Department of Antiquities of the Republic of Cyprus [3]. Besides that, there was a digitization of the monument, following a “holistic” approach, that documented its cultural landscape and was published by the project’s interdisciplinary team [4]. Nevertheless, within the EHEM project, an experiential approach is sought in order to reconstruct the appearance of the site at different times across the Middle Ages.

Santa Maria Antiqua Situated in the Roman Forum at the foot of Palatine Hill, the church of Santa Maria Antiqua holds a significant place in the Christian world from the sixth to the eleventh centuries. Discovered through archaeological excavations in 1900, its surviving wall paintings are a remarkable heritage. Originally established within Roman structures dating back to the age of Domitian, the church fell into abandonment after an earthquake in 847. However, the Atrium and the Oratory of the Forty Martyrs remained in use until the eleventh century. Revered as the Sistine Chapel of the early Middle Ages, Santa Maria Antiqua’s murals play a crucial role in understanding the evolution of early medieval painting in both Rome and Byzantium [5, 6]. The apsidal wall in the sanctuary, with its ten layers of painted plaster, is widely regarded as a milestone in the history of art. The intricate and multilayered pictorial decoration of Santa Maria Antiqua, characterized by over twenty phases of artwork throughout the complex (see Fig. 5), presents a captivating challenge for scholars, conservators, and visitors alike. The palimpsest wall and the coexistence of murals from different periods invite dynamic engagement, encouraging exploration of the complex interplay of obliterations, reuses, and cohabitations within this extraordinary site [7].

The first graphic virtual reconstructions of Santa Maria Antiqua were created in 1901 by Antonio Petrignani, a talented draughtsman of the early twentieth century. From 2003 to 2016, the Colosseum’s archaeological park commissioned a meticulous survey of the monument, including plans and sections. In 2007, Elizabeth Louden from Texas Tech University and Michele Chiuini from Ball State University conducted laser scanning of S. Maria Antiqua within the Roman Forum. This survey data enabled the creation of a digital model that accurately mapped the ancient masonry walls, documenting the construction materials and the precise locations of the frescoes [8]. A significant development occurred in 2016 with the exhibition Santa Maria Antiqua. Between Rome and Byzantium, a collaborative effort with Katatexilux, that brought multimedia elements like video mapping and light mapping to the Roman Forum. Through captivating videos and an emotional journey, visitors were able to explore the history and complexity of the monument’s pictorial phases. The video projection inside the church allowed viewers to experience the extraordinary pictorial palimpsest preserved on the apse wall, with dynamic images and virtual reconstructions illustrating the transformations undergone by the paintings over five centuries. Utilizing 3D reconstructions, the video mapping further projected the complete pictorial fabric and opus sectile decoration onto the walls of the chapels, presenting line drawings and virtual restorations of the painted decoration in an immersive manner.

Sant Quirze de Pedret a medieval church located in Catalonia, holds great significance in the study of Medieval art in the Pyrenean region. Its Romanesque-style frescoes, particularly the remarkable apocalyptic cycle found in the main apse, have captured the attention of esteemed art historians and the general public [9,10,11]. However, understanding these frescoes presents a challenge as they are no longer in their original location but instead housed in two separate museums, namely MNAC in Barcelona and MDCS in Solsona, both located in Spain (Fig. 4). Throughout its history, the church underwent various architectural transformations, including expansions and renovations. Today, it is composed of a central nave, a central apse, two apsidioles, and a lateral nave. These changes resulted in parts of the frescoes being concealed or hidden from view since the thirteenth century. It was not until the late nineteenth century that these remarkable frescoes were recognized and brought to the attention of the wider public. Unfortunately, the newfound recognition of the frescoes led to their removal in two different phases using the strappo technique. In 1921, to prevent their sale to international art dealers, the Catalan Museums Board acquired frescoes from both apsidioles and preserved them in the Barcelona Art Museum, which is now known as MNAC. The second phase of removal, which included frescoes from the central nave and apse, took place during the tumultuous period of the Spanish Civil War. This phase also uncovered additional pre-Romanesque frescoes, which were discovered during the extraction process. After a bizarre sequence of events, these frescoes found their home in the collections of the Solsona Museum.

In the case of Pedret, there has been no digitization conducted for either the church’s architecture or the mural paintings. However, the archaeological campaigns conducted in the 1990 s have provided valuable information for creating two-dimensional architectural plans, enabling the reconstruction of the church in its different historical phases. Additionally, there exist two replicas of the paintings from the south apsidiole, each varying in fidelity and quality. One of the replicas is currently displayed in the Museu de Berga, while the other can be found within the premises of Pedret’s church.

2.2 Requirements definition process

In the design of the requirements for our project, we have focused primarily on functional requirements, while also considering some non-functional requirements. Our chosen approach follows an authority-based methodology, drawing upon the expertise and guidance of our esteemed art historian collaborators (some of them authors of this manuscript) who are regarded as the most important experts in these sites. The requirements were defined through an iterative process involving multiple meetings, where experts actively proposed new requirements, while simultaneously refining and enhancing the previously proposed ones. By incorporating their authoritative insights, we ensure the defined requirements align with the authentic representation and preservation of these sites. In the subsequent paragraphs, we will review the expertise and contributions of our art historian collaborators, highlighting their influence on the design of requirements and the resulting project outcomes.

Maria Paschali, a historian of Byzantine and Medieval art, has joined the CYENS Centre of Excellence as a Research Associate for the EHEM project. With a focus on the Enkleistra of Agios Neophytos, she collaborates with the EHEM team and has conducted extensive research on Byzantine and Medieval monumental art in the Eastern Mediterranean, particularly in Cyprus. Her work includes publications and involvement in European research projects that combine art history with digital documentation and curation. Together with art historian Dimitris Minasidis, she co-authored a remarkable study that delves into the interplay between lighting and spiritual contemplation within the dimly lit rock-cut painted Enkleistra during the Dark Ages [12]. Through her dedicated study and fieldwork, conducted in partnership with the Holy Monastery of Agios Neophytos and the Cyprus Department of Antiquities, Paschali has collected a wealth of valuable textual and visual data on the Enkleistra and its mural paintings. Her overarching goal is to enrich the content related to this site, catering to the diverse needs and interests of art history professors, curators, archaeologists, conservators, students, and the general public.

The Italian team, led by professors Giulia Bordi (Roma Tre University) and Paola Pogliani (Tuscia University), is highly experienced in studying and conducting fieldwork on palimp-sest mural paintings, with a focus on their documentation and enhancement. Since 2000, they have been researching the pictorial decoration at Santa Maria Antiqua, aiming to understand the complex succession of decorative phases spanning from the sixth to the eleventh century. Collaborating with the Archaeological Park of the Colosseum, they closely followed the stages of the recent restoration campaign (2000–2015), allowing them to investigate the techniques and materials used in each phase. The team is recognized as a leading expert in Rome’s Byzantine paintings and has also contributed to the project Corpus and Atlas of Medieval Painting in Rome (313 to 1431). The Atlas, designed by Maria Andaloro, aims to recontextualize wall paintings, mosaics, and panel paintings in Rome’s sacred buildings. Their expertise extends to the dissemination of complex pictorial cycles, exemplified by the palimpsest wall at Santa Maria Antiqua. This wall, consisting of eight layers of painted frescoes, is regarded as a significant milestone in the history of medieval painting. Their efforts culminated in the exhibition Santa Maria Antiqua. Between Rome and Byzantium (2016), featuring video mapping and virtual restoration techniques. Through this exhibition, they provided a comprehensive understanding of the development of the church’s decorative phases, captivating the audience. The team’s research on Santa Maria Antiqua has resulted in the publication of three volumes and several scientific papers [5,6,7, 13,14,15].

The art historian team from the Spanish partner consists of professors specializing in the History of Medieval Art at the Universities of Barcelona and Lleida, including Milagros Guardia, Carles Mancho, Immaculada Lorés, Juan Antonio Olañeta, and the independent researcher Begonya Cayuela. Their extensive research experience includes national and international projects on Romanesque art. Their expertise in Pedret’s church is vital for the EHEM project, where they have developed a comprehensive database for collecting textual and graphical information. Guardia and Mancho are renowned specialists in Hispanic Romanesque Art with multiple publications. Guardia and Lorés have directed the Romanesque Art department at MNAC, where a portion of Pedret’s paintings are preserved. Lorés has also led the museography of the MDCS room dedicated to Pedret’s paintings. Mancho has devoted his attention to the earlier paintings of Pedret [9], while Guardia and Cayuela published a chapter in the Enciclopedia del Románico de Santa María la Real with a renovated vision of the Romanesque frescoes [10]. Cayuela’s research focuses on Sant Quirze de Pedret, leading to several publications, including a recent monograph. Their research and dedication contribute to the preservation, understanding, and dissemination of heritage to a diverse audience [11].

2.3 Common requirements between sites

A 3D digital representation of a medieval monument should satisfy certain functional requirements in order to be useful for both scholars and the general public. The most important requirements identified by the experts of the consortium follow:

• R1. To be a tool to manage and communicate heritage to stakeholders

  • (a) Interesting for curator-restorers, art historians and the academic community, and the general public.

  • (b) Useful to explain the monument in different places: in the churches, in the museums, and at home.

  • (c) Showing the lifecycle of the monuments as a time machine.

  • (d) Allowing to test different hypotheses on the buildings related to middle-age lighting and audience experiences

• R2. To store all the important data and information related to the physical counterpart

  • (a) Store buildings information in 3D at their different architectural phases

  • (b) Store paintings information in 2D and 3D

  • (c) Store information about painting layers and their corresponding decorative phase.

  • (d) Store information on the sites of several sources such as iconography, materials and techniques used, detachments and interventions, preservation state, etc.

  • (e) Store annotations on the model which can link with stored site information

  • (f) Store information about related medieval artifacts and liturgical furniture

  • (g) Store information about original sources of artificial light

• R3. To allow for easy navigation and access to the stored data and

  • (a) Users can virtually visit the sites using applications, web browsers, and VR devices.

  • (b) Paintings need to be visible in the virtual models in their original location

  • (c) Users should differentiate the different painting layers

  • (d) Users need to access annotations and modify them (or add new ones)

  • (e) Users should be able to see preserved furniture and artifacts in their original location

  • (f) Users should modify the lighting conditions of the building including natural and artificial sources

  • (g) Users should be able to modify buildings to test different hypothesis

• R4. Accessing to data should vary depending on the different users

  • (a) Accessing devices will vary between users

  • (b) Not all the users should have access to all the 3D models

  • (c) The resolution of the model should vary between users

  • (d) Visibility of annotations has to be different between users

  • (e) Only experts users need to test hypotheses

Table 2 Overview of the specific requirements per site

Fig. 5

Santa Maria Antiqua, Rome: the apsidal wall (left) and the mapping of the painted layers (right)

2.4 Specific requirements per site

Although there are requirements that are common to the three sites, due to their own characteristics and conditions, each of them has its specific requirements. In the following text, we will define them in detail, but also, to facilitate the reader’s understanding, Table 2 summarizes them.

Enkleistra of Agios Neophytos Research hypotheses on the appearance of the Enkleistra and its Byzantine-style painted adornment have been put forward on the basis of surviving textual and material evidence. Hypothetical plans were thus created (Fig. 3). As a starting point for further investigation, at least four digital 3D models of this monument are needed to be developed, displaying phases from the lifetime of its founder Neophytos in the twelfth century to the first major restoration of the site in 1503 [3, 16]. The proposed models are showcasing the Enkleistra at most significant times.

Digital models with annotations will be developed from four hypothetical layouts of the monument which give emphasis on the known parts, based on what is still visible today and what had been recorded in the past. Furthermore, the digital models can put forward scenarios to showcase hypotheses of modifications of the rock-cut structures and their mural decoration, for instance around the sanctuary screen that change the experience of the liturgical place in different times during the Middle Ages. Digital models experimenting with the interaction of natural and artificial lighting in the interior can be used by art historians and conservators as a tool to extend their premises and arguments, offering a way forward for a more critical analysis of this complex monument [12].

Enkleistra was developed in multiple levels along the cliff during the Middle Ages, and it should be showcased digitally in its entirety to enhance the understanding of the spirituality of the monastic life, that is ascending to Heaven according to the ascetic beliefs. The inaccessibility of the upper parts of the monument requires the development of an application for a virtual visit that can function on-site or remotely through Augmented, Virtual, Mixed, and Extended Reality. It should, however, be noted that the precarious nature and small size of the monument prevent the use of VR on-site because its use may accidentally damage the mural paintings [17].

In order to meet this special requirement, visualizations can integrate 360-degree short videos and AR or Mixed Reality for partially visible spaces (e.g., site of the tomb of Neophytos on his initial cell at the site) and 360-degree short videos or VR for inaccessible places to visitors such as the other cells in the upper levels of the Enkleistra site, namely Neophytos’ Hagiasterion as well as his second cell—dedicated to the New Zion—and the chapel of Saint John the Baptist. VR should therefore be a to-the-point tool that allows the users to enter a virtual space, otherwise inaccessible to the visitors of the site. It will thereby add a level of immersion while giving the users lots of degrees of freedom and enabling nuanced layers of understanding. In addition, visualization applications aspire to integrate the late twelfth-century bilateral icons of Christ and Virgin Mary as well as the wooden cross—previously all in the Holy Cross church and today on display at the Museum of the Holy Monastery of Saint Neophytos—in the hypothetical 3D models. In this way, the applications can show research hypotheses on the placement of these religious artifacts in the Enkleistra.

Santa Maria Antiqua through its palimpsest walls [5, 6] and its more than twenty pictorial phases mapped throughout the complex (second–eleventh century) challenges scholars, conservators, and visitors to read a fragmented and multilayered pictorial decoration (Fig. 5) and to measure themselves dynamically with reuses and co-habitations of murals from different periods within the church space [7].

Due to the complexity of the pictorial decoration, and the difficulty of clearly identifying the succession of phases and reconstructing the often incomplete individual layers, it is necessary to develop a system for visualizing the paintings on the basis of a 3D model and to devise different layouts that make the information accessible to different audiences. This system should incorporate the virtual reconstruction of the paintings as one of its main characteristics.

Besides that, it is necessary to extend the reproduction of the lifecycle of the building in all chronological phases. This should show the different architectural phases of the building, distinguishing additions made during the 1900 restoration of the building; relocating the liturgical furniture and the icon, now stored elsewhere. It should showcase all phases tied to layers of fresco mapped inside the church through different layouts and sew, layer upon layer, the pictorial decoration of the church through its iconography using virtual restoration.

It will be also needed to integrate annotations on materials and techniques linked to the different pictorial phases gathered in more than 15 years of surveys and restoration campaigns [13], on archaeological, architectural, and art-historical documentation, and graphic data. Annotations linked to murals images will allow to follow, phase by phase, changes and recurrences in iconography, style and use of colors.

Finally, art historians need to be able to create virtual scenarios to test different hypotheses for covering the building—a dome or a gabled roof—and for window displacement that allows observing the impact of daylight and artificial light in the space, on the paintings, and on the perception of the sacred space in the early Middle Ages. This can help to show hypotheses on the transformation of the liturgical arrangements between the fourth and ninth centuries.

Sant Quirze de Pedret The paintings of Sant Quirze de Pedret, detached from their original walls, are preserved in two separate museums, so neither art historians nor visitors can have a comprehensive view of the decoration. Additionally, the requirements to exhibit physical vestiges of the paintings in the museum imply a constant and artificial illumination that misrepresents how could be the actual perception of the painted surfaces. These conditions are different from those of their original location, where the visibility of the pictorial surfaces fluctuated widely during the day and the period of the year.

The 3D model could improve our understanding of the materiality of mural paintings, especially the issues related to the application of colors by medieval painters: the density, the thickness of the layers, glazes, how the figures and volumes have been modeled, etc. These and some other important technical resources may help us to understand the pictorial processes and, in this way, to identify the different hands involved in their realization. Therefore, the reconstruction of the paintings in the digital model implies an essential step, that of defining a working prototype of the set of colors used to translate the material pigments into their digital representation. Achieving this has resulted to be far from an easy task. While the actual pigments used in Pedret are fairly well-known thanks to various technical analyses performed in the past decades [18], the challenge is to estimate their original chromatic values. One application is the possibility to measure the degree of fading and degradation of the current colors. In this respect, if it is necessary, the collaboration of the restorers and the collection of all the technical reports should be available for consultation.

Table 3 Description of the technologies used in the project and their relation with the corresponding requirements defined by art historian experts (see Section 2)

Fig. 6

(left) Leica RTC360 operation while scanning Sant Quirze de Pedret. (middle) Resulting point-based model. (right) Example of HDR panorama captured by a Leica RTC360 scanner (South apse at Sant Quirze de Pedret with a recreation of its original paintings)

Another challenge is to figure out how the medieval spectator used to experience the mural paintings in terms of visibility. In the specific case of Pedret, we intend to solve questions related to lighting that have important implications in this specific question. Firstly, regarding to natural lighting, we intend to draw reasonable conclusions or hypotheses about the dimensions of natural light sources (windows) and, in particular, about the manner and materials with which they were closed or protected, and, in the end, to visualize how the light was filtered. Secondly, and this is an important challenge in the project, the sources of artificial lighting, as oil lamps or chandeliers, their arrangement in the building, the intensity of lighting they provided, and, in particular, the effect of vibration they produced on the perception of the paintings will be analyzed, on the basis of archaeological and textual evidence.

3 Technologies applied and technical challenges

Having established the project’s requirements in consultation with art historians, we have identified several key technologies that are crucial for the successful implementation of our endeavor. These technologies can be categorized into two groups: transversal, which applies to all the monuments, and specific, which is unique to each of the three selected sites. Likewise, the requirements themselves can also be classified into those that are covered by multiple technologies simultaneously and those that require more specialized approaches. To provide a comprehensive overview, we have compiled a table (Table 3) that summarizes the utilized technologies and their alignment with the specified requirements as outlined by the art historians. It serves as a reference point for understanding the relationship between the technologies employed and the specific requirements they address. Furthermore, as our project is still a work in progress, this overview allows us to assess the current state of work for each site and technology, providing valuable insights into the progress made thus far. In the subsequent sections, we will delve into a detailed discussion of these technologies, highlighting important challenges and identifying any outstanding problems that require further exploration and resolution.

Fig. 7

Laser scanners apply automatic image enhancement techniques to individual panoramas (left). In scenes with varied illumination, the same surface (e.g., the wall) may appear with different colors depending on the scan (middle). Since these images are used to color the point samples, rendering multiple registered scans (right) produces visual artifacts. Using HDR images (rightmost image) does not solve the problem

Fig. 8

Our color homogenization between panoramas technique [20] considerably reduces typical visual artifacts (left) ensuring color consistency (right) when these are used to colorize multiple point clouds visualized together

3.1 Transverse technologies

We identify the following transverse technologies that are important for all monuments: (1) 3D digitization, (2) multispectral analysis, (3) rendering and exploration, (4) lighting, (5) data annotation, (6) digital color restitution, and (7) virtual restoration.

3.1.1 3D digitization

For the 3D digitization of the monuments, we combined 3D scanning and photogrammetry. For 3D scanning, a Leica RTC360 laser scanner (Fig. 6) was used. The scanner provides High Dynamic Range (HDR) 360 images (Fig. 6) by encompassing the information of multiple pictures with different exposures. It captures 432 MP (12 MP x 3 cameras x 12 positions) at 5 HDR brackets [19].

Fig. 9

Color artifacts from LiDAR scans. Left: color-geometry miss-registration (a captured person which does not appear in the geometry); right: clipped highlight. Each group shows the original image, infrared intensity captured by the scanner, and color reconstructed with neural colorization [25]

Fig. 10

Overview of the different pipelines, and their required processing steps, for the 3D reconstruction of the different sites of the EHEM project. They combine laser data, HDR panoramas, general photographs of the sites, and/or high-quality close photographs of the paintings

In state-of-the-art laser scanners, HDR images are used to colorize the point clouds. The first issue is that, unlike infrared data, which is perfectly aligned with the captured geometry, color information may exhibit color bleeding artifacts due to small misalignments between the panoramas and the point cloud, which is especially noticeable around surface edges and object silhouettes. A second major issue is that color is not consistent across scans, since image enhancements are applied per scan. When multiple registered scans are combined into a single point cloud, the same surface part appears with different colors, resulting in visual artifacts (Fig. 7).

With the goal of reducing them, we have developed a color homogenization technique [20]. It uses the geometrical scanned information to build a graph of pixel correspondences between panoramas and computes a 3-channel gain factor per panorama that reduces the color differences of the surface between panoramas. Our method keeps HDR colors during the whole process, so post-processing operators can be safely applied afterward. Applying these corrections to a set of panoramas proves to increase the color consistency among scans and prevent highly apparent visual artifacts when rendering the combined point cloud (Fig. 8). However, although the quality of the corrected panoramas is remarkable for showing the environment, they do not have enough quality (i.e., resolution, color accuracy) to display mural paintings. For this reason, we also combine laser scans with high-quality color images using photogrammetry software. Besides commercial photogrammetry solutions, for the registration of conventional photos with the Lidar model, we used the image-based registration pipeline from the works of Comino et al. and Farrás et al. [21, 22].

To get more uniform sampling rates (laser scanning sampling rate is proportional to the surface-to-scanner distance), we have developed color-aware and error-aware point subsampling techniques [23]. These subsampled models are also appropriate as an accurate reference for the architects that reconstruct the geometry of the monuments at different times in their history. Since a complete model of a monument combines surfaces with different levels of interest from an art history point of view, it is desirable to adapt the resolution to such semantic interest. Thanks to image-to-model registration algorithms [21], the user can roughly highlight semantically meaningful parts of the model on a subset of the photographs, and the resulting masks can be used to provide higher sampling rates for the regions of interest, while optimizing space for surroundings that only provide context information.

A major challenge is the acquisition of geometry and appearance in the field as compared to laboratory conditions. Lighting conditions are hard or impossible to control in most monuments, and undesirable elements (e.g., people, furniture) might appear in the scans. Postprocessing the scans to clean such objects in both the point cloud and the images takes a considerable amount of time. We have some preliminary results on automating such post-processing, through panorama-wise color correction and completion [24, 25], by encompassing the HDR color with the Lidar infrared intensity (Fig. 9).

We have divided our photogrammetric approach into two different strategies: (a) we focus on acquiring mural paintings at the highest possible resolution taking into account the calibration of the color acquisition, and (b) we perform a general photogrammetric approach to the whole building to enhance laser scan photographs and increase redundancy of data.

Fig. 11

Overview of the acquisition protocol, for laser scan, photogrammetry, and HMI, defined in the project and followed in methods described in Sections 3.1.1 and 3.1.2

Digital representations of murals are a challenging task, even more so, when dealing with paintings on medieval sacred spaces. Besides the classic problems of accuracy in the digital representation of colors, these scenarios present their specific struggles; the accessibility to the different parts of the buildings, varying lighting conditions throughout the rooms, and the preservation state of the paintings are some of the most common factors that make using a typical photogrammetric approach not feasible. In our campaigns (Fig. 12), an expert photographer member of the team, Gaetano Alfano, used DSLR cameras. These cameras have high-quality full-frame sensors that allow for maximal sharpness and resolution of images. To handle the different lighting conditions present in the buildings, we used flash units or custom diffuse lights to have uniform lighting and minimal shadows. Additional campaigns were organized to increase the redundancy and quality of the generated models. For the postprocessing of images, Adobe Lightroom and Darktable were used. Moreover, a color calibration process has been applied to the pictures of paintings.

Photogrammetry was performed using Reality Capture² and Agisoft Metashape.³ For Santa Maria Antiqua and Sant Quirze de Pedret, laser scan data were used to reconstruct the surface of each model, and images were used to reconstruct the appearance; the Enkleistra was done exclusively with Photogrammetry. For the murals, we have only used images of its specific campaign. However, for the rest of the building, we have combined the images of the scanner and the general photographs. Some color artifacts appeared in the reconstruction due to the variations in the illumination caused by moving the flashes over the scene. However, considering the changes in illumination conditions present across these buildings, this restriction works against the accuracy and calibration of the reconstructed models. We are now working on the development of new methods to face these problems. Fig. 10 overviews the different acquisition pipelines used in the project and their required processing steps.

3.1.2 Multispectral analysis

To improve image quality, the EHEM project included the development of an acquisition protocol for photogrammetry, 3D surveying, and photographic campaigns to use the same parameters for visualizing the wall paintings of the three case studies. This protocol is summarized in Fig. 11. Following this protocol, the shooting of the paintings in the case study started and provides the first 3D models of the monuments (Fig. 12).

Fig. 12

Specific photographic campaigns were carried out to obtain high-quality color images. On the left, the campaign at SMA, and on the right, at SQP

At the same time, a multispectral campaign in the MNAC Museum in Barcelona, in the Solsona Museum, in Santa Maria Antiqua in Rome, and the Enkleistra of Neophytos in Cyprus was underway to acquire images of the paintings. This was carried out with a Hypercolorimetric Multispectral Imaging (HMI) approach created by Profilocolore (https://www.profilocolore.com/). This is an innovative multispectral imaging technique that allows various types of surface investigation including accurate colorimetric measurements of the surface with $\Delta E_{00} < 2$ (CIEDE2000). Contrary to traditional colorimeters, this instrument does not require contact and can return colorimetric measurements for each pixel of the image, process thematic maps, and clustering as a function of the Lab values associated with the spectral reflectance measurements of the surface. HMI has been performed through a Nikon D810FR 36 Megapixel camera, modified in order to obtain full range spectral reflectance measurements. Nikon SB910 xenon flashes have been exploited for lighting the painting. UVF has been obtained by filtering the flashes light with a UV band-pass filter (nominal cut-off at 380 nm) and using also a UV-IR cut filter (nominal cut-off at 400–700 nm) placed onto the camera lens. The HMI image processing system consists of two main software tools: SpectraPick for the image callibration, and PickViewer for the image analysis both developed by Profilocore⁴ [26]. The experimentation conducted on the mural paintings established that this is a very valid method of investigation. The results made it possible to analyze the original colors as well as the colors added during restoration, as for example on the mural painting attributed to Pastura and conserved in the Museo Civico in Viterbo [26] For the paintings covered by the project, we will analyze sample surfaces that will allow us to produce distribution maps of pigments and chromatic backgrounds for the analyzed surface and to monitor the color before and after any restoration work.

With this technique, the University of Tuscia team coordinated by Claudia Pelosi [26] carried out color measurements on the Pedret paintings, conserved in the MNAC and Solsona museums, in Santa Maria Antiqua in Rome, and the Enkleistra of Neophytos, choosing the most representative areas that included the palette used by the painters in the Middle Ages (Fig. 13). The measurement points indicate pure colors (the palette used by the painter) and sometimes mixed colors; the respective bands produced allow the mapping of colors in the state of conservation. The measuring points are also associated with the analytical results carried out for the recognition of pigments and in the case of the Enkleistra and Santa Maria Antiqua also with the measuring points with colorimeter analysis. The integration of this information allows to obtain an objective basis for the calibration of images. The methodology that has been developed integrates different systems of investigation that enable us to understand the use of color in the Middle Ages. The results achieved make it possible to calibrate the images for the 3D model with color references responsive to the paintings examined and to propose virtual restoration solutions.

Fig. 13

With a camera, reference target, and color scale, images were acquired for HMI (left: Santa Maria Antiqua; right: Solsona Museum); with a contact colorimeter, spectral data were acquired (the right part of the Santa Maria Antiqua image)

Fig. 14

Top: Graphical User Interface to tweak the VR navigation technique to specific scenarios and user profiles [31]. The tool allows interaction designers and end-users to fully customize the mapping between user input and gestures to virtual locomotion through the virtual world. Bottom: The web page allows virtual visitors to select which parts of the monument should be inspected. By selecting a single physical space (e.g., an apse), users can inspect the 3D model more intuitively using an orbit-like navigation around the object

3.1.3 Model rendering and exploration

We have already explored different techniques for the visualization and interactive exploration of the scanned models. We wish to support the inspection of both point-based models and mesh-based models. Point-based models are more appropriate for certain uses, such as floor-plan and cross-section extractions. Point-based models are also appropriate for those tasks that require actual captured data, with no synthetic geometry or interpolated data. Mesh-based models are more appropriate for interactive applications; however, undersampled regions and missing parts of the model are often filled with synthetic data. The resulting models are thus more complete, but it brings up the need for letting users the ability to enable the visualization of local certainty in the reconstructions [27]. A major issue is the amount of detail in the digital models and the associated storage and computing requirements. Despite specific geometry simplification techniques [23], resulting models are still too large for in-core visualization. For point-based models, we have developed a web-based application using Potree [28], which supports the streaming and rendering of billions of points in web browsers through an octree that stores subsamples of the original data at different resolutions. For mesh-based models, we use Cesium, an open streaming platform for geospatial 3D data that can be easily integrated with game engines such as Unity and Unreal engine, and that also supports web-based visualization.

A second issue is 3D navigation. A recent study has identified and characterized over one hundred navigation techniques for VR [29]. The reason for such an increasing variety of navigation techniques is twofold. On the one hand, AR/VR software and hardware technologies are in constant evolution, opening new opportunities for 3D interaction. On the other hand, current interaction techniques for 3D navigation have some limitations, due to the inherent trade-off between simulation sickness and accessibility [29]. We have developed a method to facilitate the design, testing, and fine-tuning of interaction techniques following the major navigation metaphors for VR, allowing users to refine the mapping of user input onto virtual camera manipulation [30, 31] (Fig. 14). To further facilitate navigation, we also explore the use of annotated maps that give users the choice of selecting which parts of the model to show (Fig. 14). This facilitates navigation and avoids the occlusion of Regions of Interest (RoI).

Surface appearance is essential for monuments with mural paintings, and therefore, models should be colorized by projecting and merging the contribution of a collection of photos. In order to get a coherent color reproduction, the photos contributing to the colorization process must be consistent. We have developed a VR-ready prototype application to simultaneously explore the digital model together with arbitrary photo collections [22]. At any time, users can define a RoI either explicitly or implicitly, and the application retrieves, scores, groups, and shows a matching subset of the photos. Users can then select a photo to project it onto the 3D model, inspect the photo separately, or teleport to the position the photo was taken from (Fig. 15). Our approach extends PhotoCloud [32], a client–server system for the joint exploration of 3D models and thousands of photographs. Similarly to PhotoCloud, we also cluster similar images into piles that can be scrolled and selected. We however support VR headsets besides traditional browsers, and users can define the RoI implicitly or explicitly.

Fig. 15

Joint exploration of the 3D model along with registered photo collections using a standalone VR headset [22]. The application allows users to define a region of interest either explicitly (drawing a region on the 3D model) or implicitly (using the gaze direction), and the application retrieves, groups, and shows all photos that match the selected region (stacked images shown on top of the user’s view). Users can then select specific photos to project them onto the 3D model or to navigate to the location the photo was taken from. This allows art historians to quickly retrieve graphic materials (photos from other campaigns, annotated images) of specific parts of a monument

Fig. 16

Results of testing art historian hypothesis using our Blender’s add-on. On top, different materials for closing the windows: nothing (left), glass (center), and thick leather (right). On the bottom, natural lighting at different moments on August 1st: 12.00 pm (left), 1.00 pm (center), and 2.00 pm (right)

Since lighting (see below) plays an important part in this project, during visualization, we would like to have a real-time rendering system that can allow for as accurate lighting as possible. Currently, we are developing visualizations using Unreal Engine 5 (UE5 - https://www.unrealengine.com/), a modern high-performance game engine with Lumen, a real-time global illumination system. Additionally, UE5 has Nanite, a virtualized geometry system that allows for the rendering of 3D models with high polygon counts.

Fig. 17

Real-time natural lighting simulations in the 3D model of the Enkleistra of Neophytos using the Lumen Rendering system. We demonstrate lighting in the four main spaces of the monument’s first floor during three primary sun paths. (1) Church of the Holy Cross, Naos, (2) Church of the Holy Cross, Bema, (3) Cell of Neophytos the Recluse, (4) Refectory of the Enkleistra

3.1.4 Lighting

The mural paintings, whether displayed on-site or in a museum, are illuminated for their current vision with artificial lights that uniformize and even expose parts that were not intended to be shown in a particular way. In our virtual reconstructions, we try to restore the illumination and, therefore, the perception of the paintings as they were at the time. The reconstruction of the different phases of the buildings allows us to know in detail how they were illuminated from the openings at different times of the day and the seasons. Medieval documentation refers to candlesticks, oil lamps, and other light fixtures whose original arrangement is not always easy to reconstruct. This artificial illumination was particularly important at the time when the rituals were being carried out; our reconstructions should therefore help us in reproducing the vibration that light produced in the paintings, to learn about the experience of those present at those crucial moments of the liturgy. Satisfying these requirements demands incorporating the knowledge of lighting present in the monuments when they were being used. These include the openings, which are different in each case, and precise models of the artificial light sources that differ between sites.

To solve that, we are using the 3D models of the different phases of each site in combination with Radiance⁵ and Lumen⁶. Radiance is a validated lighting simulation tool. It computes the illumination of the scene considering geometry, material properties, and natural and artificial illumination sources. However, it is formed by a set of command-line tools with no GUI. This usability problem is one of the causes of its low usage in the Cultural Heritage community. To fill this gap, we have developed an easy-to-use add-on for Blender⁷. It allows users to configure the lighting simulation of the scene through a simple GUI. The user can set the rendering camera, the properties of the materials of the different models of the scene, its geo-position, the date and time of the simulation, and the conditions of the sky, besides other properties related to the quality of the obtained image. Once everything is set, our add-on automatically transforms this data into Radiance formats and executes a lighting simulation radiance process. The result of this process is an HDR render together with numerical data of the simulation. This tool has proven to be very useful for testing different art historian hypotheses related to how window openings were closed (Fig. 16). Not only to show how the closing affects the visualization of the paintings but also to estimate what is the received irradiance of the paintings, which is useful for understanding their weathering processes.

Fig. 18

Different screenshots of real-time lighting simulations of artificial light sources in the Enkleistra of Neophytos. The team is experimenting with more realistic particle-based candlelight simulations and less realistic but more efficient shading-based implementations

Table 4 Different types of annotations and their requirements in terms of accuracy of the target contour, the need for predefined labels, and how the different annotated regions should be highlighted

On the other hand, Lumen is a real-time global illumination solution developed by Epic Games. It allows for instant visualization of lighting interactions that is almost as good as Radiance. We believe that both technologies will help us to illuminate our virtual reconstructions as close as possible as they were in the past taking special attention to showing the use of light and shadow. Figure 17 demonstrates the effect of natural lighting using Lumen in the 3D model of the Enkleistra of Neophytos. We simulate real-time global illumination in different parts of the monument using a single sun light source at three different main sun paths; June 21st ($77^{o}$), September/March 21st ($50^{o}$) and December 21st ($33^{o}$). These kinds of real-time simulations allow the experts to efficiently analyze the effect of lighting through the different openings on the murals and explore the shifting perception of the mural program and the ambiance of the religious monument during different periods, feast days, and times of day. We additionally simulate artificial light sources like candlelights (Fig. 18) which could have a profound effect on the experience of the monument. We are experimenting with both particle-based and shading-based implementations of other light sources, such as oil lamps; particle-based implementations allow for more realistic dynamic lighting and shadowing effects, whereas shader-based implementations allow for relatively good lighting simulation with better performance. A specific issue to be addressed in the case of the Enkleistra is the simulation of the radiance of its wall paintings due to the documented application of chysography and varnish on selected painted details, like haloes and garments. Gilded details in particular are an important feature of the Enkleistra’s mural adornment because it made the holy personages animated through the shifts in lighting when the faithful moved and prayed in front of them, thus appearing to be living figures. Visualizations seek to reproduce this experience of the Enkleistra to their users.

From a methodological standpoint, the main contribution of our work is the development of a Blender add-on that brings the use of Radiance closer to the cultural heritage community, which previously had limited access due to its usability challenges. Additionally, to the best of our knowledge, this is the first time that the use of artificial lighting simulation techniques (e.g., candles and torches) has been proposed to gain insight into the visitor experience in medieval temples. This aspect is still a work in progress, and we hope to achieve further methodological results in the future.

3.1.5 Annotation

The ability to add annotations to a Cultural Heritage monument is essential for art historians, and software designed specifically for Cultural Heritage often allows users to annotate the models. Examples include 3DHOP [33], Cher-ob [34] and Aïoli [35]. Ponchio et al. [36] review existing methods for adding annotations over Cultural Heritage models. Croce et al. [35] conclude that it is easier to select the annotation target on 2D media than directly on the 3D model. This image-based annotation approach is widely adopted in Cultural Heritage [35, 37, 38]. In the context of the EHEM project, annotations include the attachment of text, images, or media labels to specific parts of the monuments, as well as more dense annotations involving per-pixel classification (e.g., the segmentation of a painting’s image according to pigments, colors or conservation status).

Fig. 19

A suggested workflow for annotating mural paintings. Please refer to [39] for further details

An image annotation links some resource (body, label) to a specific region (selector) of an image (target). Table 4 summarizes typical annotation needs (in the context of medieval mural paintings) regarding the concept the annotation refers to, the complexity of the image regions to be annotated, the required labels, and their visual representation. When assessing the state of conservation of a mural painting, art historians might want to create various types of annotations for documenting aspects such as the overall condition of the painting or the level of damage at specific areas (including information about cracks, flaking, discoloration, tears, loss of content, and paint delamination), as well as the identification of retouching or overpainting areas due to conservation interventions. Similarly, annotations about the painting’s history, including the different pictorial layers, are essential (see Fig. 5). These types of annotations might require simple polygonal regions or complex selectors depending on the desired accuracy on the boundary of the specific areas being documented. Since the information about conservation state is diverse, annotation bodies should support any type of text. Annotations by presumed color might be useful both for documentation and for digital color restitution. For the latter, pixel-accurate selectors might be needed so that the color restitution method can operate precisely on selected parts of the image. Since the number of distinct colors in medieval mural paintings is quite limited, annotation labels can be based on a dictionary of predefined colors (third column of the table). Concerning materials and techniques (pigments, mixtures, binders, layering techniques to get deeper or lighter shades...), annotations usually refer to large parts of the paintings and therefore free-text annotations on simple regions suffice. A last type of annotation refers to topics related to iconography. The documentation of the central theme of the painting, its overall composition, its symbols, items, and the gestures and postures of the figures can be completed with basic image selectors.

For annotations requiring simple regions, we use the Annotorious⁸ JavaScript library, since it is web-based and can be fully integrated with Claris’ FileMaker⁹, the cross-platform relational database application used by the art history teams involved in the project.

Regarding targets with complex boundaries, we have developed a tool prototype (based on TagLab [40]) intended to facilitate the annotation of mural paintings [39]. The tool assists art historians in selecting regions with complex boundaries and supports multiple annotation layers (e.g., for degradation and color), providing both hand-drawn and semi-automatic segmentation. The tool fills the gap between automatic tools for specific problems (crack detection, segmentation, inpainting) and manual (hand-drawn) techniques. Figure 19 shows the intended workflow when annotating images with the current version of the tool.

Fig. 20

Digital color restitution of St. Quirze de Pedret mural paintings: current captured color (left) and restituted one (right)

Fig. 21

The mapping of the green color with HMI at Santa Maria Antiqua highlights the characteristics of the green color obtained with celadonite-type earth and those of the green color obtained with yellow ochre and carbon black found in the stratigraphies of the samples taken from the grounds of the “Angelo Bello” (left) and from the grounds of St. Anne with the Infant Mary (right)

The methodological achievements of the project concerning annotation tasks are two fold. On one hand, the integration of a simple-to-use, web-based annotation tool with the database system used by the art history team allows them to create, edit (both body and selector), inspect, and share the annotated images, using the database interface. Most importantly, annotation bodies and selectors are fields of database entities, allowing art historians to retrieve, using the database tools they are used to, specific annotations containing some search string within their body. This fully integrates image annotations with the rest of traditional fields and entities art historians deal with regarding CH monuments. On the other hand, our adapted version of TagLab greatly facilitates the segmentation of images of paintings, according to the different needs in Table 4. Such segmentations can be imported into the database and then used as the other annotations created manually through Annotorious.

3.1.6 Digital color restitution

One of the objectives of the project is to create virtual reconstructions that represent the original appearance of the sites. In particular, restoring the appearance of the paintings plays a key role in their understanding, either for researchers, restorers, or the general public.

Digital color restitution aims at estimating how the colors of a painting looked at a precise time (e.g., creation time), by trying to reverse the physical and chemical changes that degrade the visual color appearance. Its scope is limited to those regions of the painting whose reflectance properties (at the time the painting was digitized) provide some evidence or suggest that a particular color was used; it does not attempt to recover content in severely degraded parts where color information is lost, compared to virtual reconstruction (see Section 3.1.7).

For St. Quirze de Pedret murals, we have developed a technique to semi-automatically restitute the color of the captured paintings. The method starts by selecting a set of representative colors over the set of captured images, which have been previously color-calibrated. The selection is performed by art historians, which have knowledge of the characteristic pigments and mixtures used on each particular site. For each representative color, a set of points are chosen over different images, to make the method more robust to subtle differences. In addition, a second set of colors is provided that represents the original colors according to their understanding. Using these two sets, we have developed an automatic transformation that transforms the colors from one set to another while transforming the surrounding color space. Compared to previous work, our approach does not rely on a uniform transformation of the whole color space [41], nor requires learning a transformation from a large set of examples [42]. Instead, we provide a solution that locally deforms the color space according to the provided color samples. An example of this restitution is shown in Fig. 20.

Fig. 22

Santa Maria Antiqua, palimpsest wall. On the right, the painting with Maria Regina in its current state; on the left, a portion of it showing its virtual restoration with reconstruction of the lost part (@Katatexilux)

At Santa Maria Antiqua, we aim to perform digital color restitution through the acquisition of the specific physical and chemical characteristics of the pigments [14]. The work is in progress and is focused on combining the characteristics of materials and the execution methods of the paintings by integrating the analysis of art historians with the results of scientific investigations [15]. In this direction, we proceed in analyzing how colors appear today (modified by their state of conservation and restoration works) in order to recompose, with the autoptic investigation and analytical-scientific characterization of pigments, the palette used by painters who worked at Santa Maria Antiqua. These investigations are complemented by multispectral and colorimetric surveys that implement the knowledge of the paintings’ color characteristics. The colorimetric analysis allows the color coordinates to be described by means of the CIELAB color system, while multispectral surveys with HMI allow creating distribution maps of pigments over the entire analyzed surface (Fig. 21). These techniques, applied on medieval wall paintings in a compromised state of conservation, make it possible to “recompose” the image and to enhance it where finishings, highlights, or brushstrokes, that have faded, are not perceptible in visible light; while the color tone mapping allows us to “reestablish” the chromatic reading of the images for their digital reproduction. With this data, we are working on developing an image calibration system for the 3D model that enables digital color restitution based on the current state of the murals’ pictorial layer and the composition of the pigments used.

3.1.7 Virtual restoration of paintings

Virtual restoration is more ambitious than digital color restoration, in the sense that it also aims at giving an idea of how a painting might have looked like at a precise time. It attempts to reconstruct content in damaged regions, for example, where parts of the painting are missing. The digital restorer has to rely on contextual information to recover the content, for example, by drawing manually the boundaries of regions of different colors. In Santa Maria Antiqua, an initial trial of virtual restoration was carried out on the occasion of the exhibition Santa Maria Antiqua. Between Rome and Byzantium (2016). Subsequently, Valeria Valentini (restorer) continued her experimentation by working on the Recolor project (@Katatexilux, scientific coordinator Luca Villa) by virtually restoring the early medieval paintings of the Lombard temple in Cividale (Friuli-Italy).¹⁰ In any case, these works represent only a first approximation to the problem, and more projects that deal with virtual restoration in more depth are still necessary.

In the Santa Maria Antiqua 3D model, due to the fragmentary state of conservation of the wall paintings and the difficulty of reading the iconography of different layers, we aim to propose a virtual restoration of the sanctuary murals to optimize their legibility (Fig. 22). This is an attempt to reconstruct their presumed original state using only critical and philological tools such as the study of iconography, knowledge of materials and techniques, and survey on restoration works carried out in the twentieth century. Virtual restoration acts almost exclusively on the damaged parts of the murals’ surface. It is performed directly on digital images, which are obtained by digital survey (photogrammetry) and are subsequently subject to procedures designed to optimize the image. On this photographic basis, all the interventions can then be performed using AutoCAD and Photoshop: from the rendering of the color of the pictorial layer to the filling in of lacunae with neutral, attenuated, or imitative colors [43]. The main stages of virtual restoration are (a) stitching up the abrasions of the pictorial layer; (b) mimetic integration of small lacunae; (c) virtual restoration of the most extensive lacunae. Where we have reliable data about iconography, we will apply a mimetic virtual restoration, whereas for the most hypothetical missing parts, we intend to suggest iconography using a line drawing reconstruction.

Fig. 23

The process of digitizing the Enkleistra of Neophytos. On the top, the mesh 3D mesh acquisition and reconstruction; on the bottom, the texturing of the resulting mesh

Fig. 24

Real-time renderings of the exterior (left) and interior (right) of the Enkleistra

3.2 Practical challenges per monument

In the following paragraphs, we highlight some specific challenges and how we approach them for each of the monuments.

3.2.1 Enkleistra of Agios Neophytos

Acquisition campaign and 3D model generation

For the Enkleistra of Agios Neophytos and its mural paintings, we used photogrammetric reconstructions. A total of 2500 images were obtained in three fieldwork campaigns; a drone was also used for documenting the exterior of the site. We performed a 3D reconstruction of the Enkleistra using the techniques described before, processing and simplifying a point cloud of 40 m points. Figure 23 shows the progression of the reconstruction from images, to point clouds, meshes, surfaces, and finally the colored model, whereas Fig. 24 demonstrates example real-time renders of the interior and exterior of the monument.

By exploiting the point cloud of the current site, we created four hypothetical plans of the Enkleistra that showcase the possible appearance of the site at different times during the Middle Ages. For the purposes of the creation of these layouts, the historian of Byzantine and Medieval art in our team used material and textual evidence. Figure 3 shows the four layouts of the first level of the Enkleistra; these will be used to generate four different models for the different phases of the site.

Reinstatement of artifacts

We also used photogrammetry to digitize a painted cross and two byzantine panel paintings which were initially in the Enkleistra’s interior, yet are now on display in the nearby museum of the Monastery of Saint Neophytos. The objective is to digitally display these artifacts inside the Enkleistra’s models and visualize hypotheses of their original location in different times during the Middle Ages [44]. Furthermore, representative images were captured, utilizing the HMI approach as discussed previously, which can be used in the annotation of pigments of artists who followed Byzantine painting techniques and in the virtual restoration of the acquired murals.

Fig. 25

Different visualizations of lighting in the Refectory of the Enkleistra through manually edited openings in a model of the Enkleistra in its current state

Fig. 26

The scanning process of Santa Maria Antiqua and its surroundings (left) results in 283 scans (red spheres) registered using 795 links (green lines). We obtained a point cloud of 4.6B points that represents accurately the outside (center) and the inside (right) of the site but contains dynamic objects, as scaffoldings and visitors (right), to be removed in a filtering step

Lighting hypothesis

Finally, a 3D artist from the team manually created and modified openings in the model to account for several openings that were present at different periods in the monument. Figure 3 showcases this differentiation of openings—doors and windows—at different times during the Middle Ages. Visualizations of lighting, such as those in Fig. 25, are thereby significant because they show the dim ambiance of the Enkleistra’s interior. Indeed, its setting on a cliff allowed natural light only from the east, through the fewer openings in the “Dark Ages” than today.

A particular challenge for the team will be to visualize sunlight filtering through the small windows within paintings at the Church of the Holy Cross, namely the Crucifixion of Christ in the naos and the Ascension of Christ in the bema. Such mise-en-scènes as points of brightness are of particular interest to art historians because of the agency of natural light in understanding the messages of the mural decoration within a ritual context. Circular perforations—ostensibly imitating medieval ones—are found in the window’s transenna of the bema, accentuating the spectacle of sunlight while evoking the “splendid sun” as Neophytos expressed the shine of the Ascending Christ. At the same time, the visibility of these images of Christ could have been impeded by natural light coming in through their windows, echoing the apostles’ experience of theophanies [12].

3.2.2 Santa Maria Antiqua

Acquisition campaign and 3D model generation

Due to site dimensions, we completed the surface acquisition campaign on Santa Maria Antiqua using 283 scans. We registered them by generating 795 links between the different scans. As the monument was open to the public during the capture, we processed them by removing visitors and other dynamic objects. We finally obtained a point cloud of 4.6B points. Figure 26 shows the resulting point cloud, the scan locations, and the links between scans created during the registration. We also carried out a photographic campaign to support the color acquisition of the scanner. In that case, we have tried to ensure the success of the future photogrammetric process by increasing the redundancy of the images. We obtained 7165 images that we are currently processing. The photographic campaign to capture the paintings is almost finished. Up to now, we have obtained 2419 photographs and we estimate that the final set will consist of around 2.7K images.

Fig. 27

The resulting 3D model of Santa Maria Antiqua (left, only the mesh of the central nave; middle, the textured mesh of the central nave; and right, textured mesh of the imperial ramp). It has been generated using the point cloud and the panoramas obtained during the scanning of the site. It has 11.4M faces and a surface area of 130K squared meters since it also includes the monument surroundings

Fig. 28

(Left) Locations for the laser scans in St. Quirze de Pedret: indoors (A), outdoors (B), lateral apses in MNAC (C, D), and main apse in Solsona (E). (Right) Overhead map of the scan locations in the church’s interior

We have used the point cloud and the scanned panoramas to generate a 3D mesh with 11.4M faces with a covered surface of 130K squared meters. Figure 27 shows the generated model using the open platform Cesium and Unreal engine. Although the quality of the model is considerable, it still lacks the desired details to inspect the paintings. We are now currently working on the enrichment of this model with the images of the paintings’ photographic campaign. To be able to succeed in this task, considering the amount of data and the desired resolution of the paintings’ reconstruction, we have split the paintings into different parts to reconstruct them individually to later combine them on the 3D model.

The main challenge of this part is related to the characteristics of the site; several overlapped layers of paintings make the photographic process difficult to obtain high-quality representations of the frescoes. Moreover, due to the size of the monument, ladders and scaffoldings were used; this slows down the acquisition procedure of both, scans and photographs while it increases the post-processing step to remove them. Another important challenge is the size of the acquired data; Leica software can manage this data; however, this is not the case with most processing and visualization tools. So, both, point cloud and 3D mesh, will need a semantic simplification process to be usable. Moreover, the manual filtering process of the undesired captured objects has been tedious. But, it brings out the need for guided techniques to filter the acquired data considering spatial and temporal coherence. Last but not least, the dimensions of the site and the generated amount of data make the resulting models impossible to use in most Cultural Heritage applications. Although there are multiple mesh simplification techniques and visualization approaches for large models focused on cultural heritage [45], they are mainly based on general processes that only consider the geometry of the model and not the historical-artistic importance of each specific area or the available related information. For this reason, we believe that there is room for research on new semantic-aware simplification techniques that lead to the creation of 3D models with semantic levels of detail.

3.2.3 Sant Quirze de Pedret

Acquisition campaign and 3D model generation

The church of Sant Quirze de Pedret is one of the medieval monuments preserved with more profound transformations. Most of the original paintings were detached from the church and moved to museums for their preservation. As a consequence, we had to perform three Lidar scanning campaigns: one for the church, another for the original mural paintings of the main apse (in Solsona’s Museum), and a last one for the paintings from the lateral apses (in MNAC). Figure 28 shows the scan locations for the three different sites.

Fig. 29

For Sant Quirze de Pedret, we textured the model generated from the point cloud acquired using the scanner (left) using the high-quality photographs of the paintings (middle-left) and the general photographs of the current church building. Collaborating architects built 3D models of the significant historical moments of the church (middle-right). We transferred also the paintings to them in combination with synthetic materials (right)

Besides the Lidar scans, we also acquired 1125 images with the help of a professional photographer. After processing for exposure correction and color calibration, we used Capturing Reality’s Reality Capture to generate a 3D model for each of the sites, by combining the point cloud from the Lidar scans and the high-quality color from the photographic campaigns. We generated five different models (the church’s interior and exterior, two side apses, and the central apse) that were aligned using the ICP algorithm. Two architects, Joan Font and Genís Àvila, built different 3D models for each of the significant stages of the church. They started from the point cloud of the current building, and they modified it according to existing documentation and the criteria of the art historians in our team.

Mural painting transfer

Since the church’s decorative elements were relocated to different museums, visitors have little possibility to understand the monument. The digital model of Pedret allowed us to perform a digital reintegration of the distributed mural paintings [46]. This reintegration involved digitally transferring the original paintings from the apses’ digital models, onto the appropriate walls of the church’s 3D model. The support of the paintings exhibited in the two museums roughly follows the original shape of the church apses and walls, but we had to combine ICP with a manual refinement due to minor geometric differences. The painting transfer was computed by using a ray-casting algorithm to project the color from the source photogrammetric textures to the destination model textures. The details of these digital reintegration processes can be found in Munoz-Pandiella et al. work [46]. Figure 29 shows the results of this process.

3.3 Summary of provided solutions

For each stage of the project, we now summarize the solutions we have provided to deal with the challenges of the project and the use cases.

Geometry and appearance acquisition

We have combined laser scanning and photogrammetry for digitizing the 3D models, along with in-deep photographic campaigns for the mural paintings. We have proposed several solutions for improving the HDR images coming from laser scanning, including color homogenization, correction, and colorization. We also use adapted subsampling techniques for obtaining uniform point-based models.

Color analysis and digital restoration

We have carried out multispectral (HMI) campaigns to accurately capture colorimetric measurements for color analysis and calibration. We have provided solutions for automatic color restitution of the paintings based on selected representative colors, as well as in-deep virtual reconstructions of damaged parts.

Rendering and exploration

We visualize the 3D models using both point-based and mesh representations. For point clouds, we provide a web-based application that supports the streaming and rendering of billions of points. We also provide VR exploration with specific visualization needs along with photo exploration over the 3D models.

Lighting

We account for both natural and artificial light sources. We have developed a Blender add-on for accurate Radiance lighting simulations, and we use Lumen for real-time rendering.

Data annotation

We allow the annotation of the models and we have developed a specific solution for annotating mural paintings.

4 Conclusions

In this work, we have studied the analysis and documentation of medieval heritage and its temporal transformation. First, we have analyzed the main requirements of these monuments through the study of three user cases: Enkleistra of Neophytos, Santa Maria Antiqua, and Sant Quirze de Pedret. We have shown that, although important requirements are shared between most medieval sites, each location has its specific needs that must be taken into consideration to succeed. After that, we have depicted the main transverse technologies that are required for the study and analysis of medieval heritage monuments. They consist of 3D digitization, multispectral analysis, rendering and exploration, lighting, data annotation, digital color restitution, and virtual restoration. For each of them, we have analyzed the technical challenges that we have found and the applied solutions to overtake them. However, just like we have observed establishing art historian requirements, each site also has its own technical dares. We have studied them and their solutions, and we have presented the obtained results. For comprehensive information on how to experiment with the obtained results, we encourage readers to visit our webpage,¹¹ where they can access detailed resources and guidance on harnessing the outcomes of our project.

In conclusion, in this work, we have shown that having cross-disciplinary expertise allows for the creation of Digital Twins that include both visually and metrically accurate 3D models which are informed by accurate historical data from experts in the respective fields. Moreover, we have proved that technological processes are more robust and accurate when driven by interactions between art historians and technology experts. We believe that these kinds of collaborations are what is needed in order to move to the next step; the creation of Digital Twins that are also driven by the sensor and operational data to preserve and monitor important monuments.