Keywords

Introduction

According to Europol [1], cultural goods’ crime revolves around three main phenomena:

  • Theft, when original cultural goods are robbed from their owners or caretakers;

  • Looting, which refers to the removal of ancient relics from archeological sites and old buildings;

  • Forgery, which involves the illegal imitation of cultural goods.

Looting and trafficking of cultural property can seriously undermine cultural heritage on a global scale. The revenues from such illicit activities are extremely high and are estimated to be billions of dollars annually. These phenomena are very usual in countries facing crises and conflicts, while the money from such activities is also used, in many cases, to support terrorist activities [2]. On the other hand, the demand for illicit antiquities often comes from economically and politically secure states which fund looters and sellers from less politically/economically secure states. The social political and economic structure of the antiquities market which formulates the way different actors are involved in illicit trades is quite complex [3]. Robbery and trafficking of cultural property are often characterized by traits met in organized crime [4]. Illicit online sales on the dark web using cryptocurrencies, the utilization of false documentation, and person-to-person trade are some examples of how the aforementioned activities take place [5].

Thus, the prevention of looting and illicit trafficking of cultural objects is of paramount importance for the protection and preservation of the global cultural heritage. Fighting such phenomena requires cooperation among different actors as well as a clear understanding of how illicit antiquities and trafficking networks work [6]. Any loss of time is in favor of the looters/traffickers and can seriously undermine the global cultural heritage. The highly promising Artificial Intelligence (AI) and advanced Machine Learning (ML) techniques and tools can increase the research capabilities of Law Enforcement Agencies (LEAs) and drastically reduce the time required for combatting trafficking cases [7].

The semantic engine presented herein will apply rule-based reasoning and will reveal hidden relations relevant to the activities of looters/traffickers. Such relations can be identified in the source and destination points of cultural artifacts, in the traits and activities of illicit traders, in the distribution channels of cultural property, in illicit online listings referring to artifacts, etc. Furthermore, the tool under consideration will create unified graphs which can help LEAs, archaeologists or other practitioners and end users to timely detect suspicious activities related to the illicit trading of antiquities and cultural property. Thus, the envisioned toolset is expected to play an active role in the fight of LEAs against trafficking of cultural heritage.

The remainder of this paper is organized as follows: Section “Related Works” features related research works. Section “Semantic Engine” provides a detailed description of the semantic engine together with a brief overview of the ANCHISE project. Finally, section “Conclusion-Future Work” draws conclusions and discusses future work related to the Semantic Engine.

Related Works

The usefulness of image analysis for tracking looting marks as well as for detecting illegal excavations is presented by Agapiou et al. in [8]. In a use case implemented in a village of Cyprus, the authors utilized Google Earth images, WorldView-2 images and images from the Cyprus’ Department of Land and Surveys. Extensive editing and transformation of images, then, took place for improving the extracted results (e.g., linear histogram enhancements, extraction of vegetation indices, spectral transformations, brightness and contrast corrections). Based on the results of their study, the authors highlighted the capabilities of image analysis in facilitating the detection of looting and illegal excavations. In [9], Tapete and Cigna highlighted how the use of Synthetic Aperture Radar (SAR) images can play a vital role in detecting, monitoring, and assessing the condition of cultural heritage sites due to the impact of certain activities such as looting, mining or agriculture. More specifically the authors explored the use of COSMO-Skymed data whose characteristics (e.g., high spatial resolution, frequent site revisit) make it particularly useful for archaeologists. Different use cases from archeological sites in Syria, Italy, Iraq, and Italy were showcased. The authors noted that Digital Elevation Models (DEM) making use of SAR data can be particularly important for surveying archeological features. Additionally, the spatial and temporal interpretation of SAR backscatter can also play an important role toward this direction. As mentioned before, optical remote sensing plays an important role in the detection of looting. However, it can be ineffective in areas with very dense vegetation. Danese et al. [10] showcased how LIDAR (LIght Detection And Ranging) data can be of particular usefulness in such cases and how it can importantly benefit looting detection cases. Towards this direction, they utilized spatial visualization methodologies as well as the geomorphon method for landform extraction. A relevant use case was implemented in Lazio, Italy, where the authors’ methodology yielded very satisfactory results for looting prediction, reaching a 95% detection rate in one of the four test areas.

Solutions for cultural heritage protection encompass large amounts of data which need to be collected and associated. In this light, the generation and adoption of a common representational model or data retrieval, processing, and storage system can be of benefit. Oreste Signore [11] mapped the available technologies and ontologies that can lift the restrictions imposed by the heterogeneity and scarcity of cultural heritage data. According to his work the CIDOC Common Representational Model (CRM), which is based on an ontology, can aid academics, experts as well as LEAs in storing, retrieving, and processing data effectively. Ontologies consist an effective way to deal with heterogeneous data sources. They can capture properties and metadata of a certain artifact or archeological site, as well as complex relationships through predefined classes. Based on the ontology scheme and by examining the need for better annotation and correlation processes, Bobasheva et al. [12] proposed a solution which engaged and combined semantic reasoning and machine learning. Their solution mainly aimed to improve the annotation process and enrich the metadata of artifacts contained in museums in picture format. Even though their solution referred mainly to museum curators, it has many similarities with the solution presented later in this paper. More specifically, they introduced a combination of semantic reasoning techniques alongside machine and deep learning approaches. Their results indicated that this combination could lead to enhanced metadata, automated annotation procedures, and better search results based on visual relevance and search criteria. For their experiments, they used the Joconde knowledge base which was accessed by SPARQL queries [13].

Semantic Engine

ANCHISE is a Horizon Europe project, aiming to tackle the following problems: (i) looting and trafficking of cultural goods affecting different countries which can have a devastating impact on the global cultural heritage; (ii) trafficking of cultural heritage adopting new, harder-to-trace, digital methods (e.g., social networks, dark web); and (iii) the fact that data related to cultural heritage protection is dispersed, under various formats and mostly lacking a clear ontological description that would allow large-scale interoperability. In addition to the problems that ANCHISE project is called to address, there are some pertinent end user (experts, academics, LEA officers, etc.) requirements that are envisioned to be addressed. More specifically, end users can benefit from the integration of cross-domain knowledge and expertise in order to lift traditional barriers and exploit available technologies alongside the combination of state-of-the-art technologies in order to create a common framework for fighting looting and trafficking of cultural goods. This leads to another crucial demand of the end users that of proactiveness in looting and trafficking procedures in order to decrease the time required for tackling such criminal acts.

The ANCHISE project aims at offering European societies efficient methods, knowledge, and a toolkit to enhance the protection of cultural heritage against looting and illicit trafficking, by facilitating four main aspects: (1) understand; (2) prevent; (3) act; and (4) repair. More specifically, ANCHISE consists of:

  • a hub of social science, politics and economics (for in-depth results likely to lead to structural evolutions in heritage protection),

  • a large-scale evaluation of technologies and needs,

  • a toolkit of innovative solutions,

  • pilot experimentation areas (museums, border control, archeological sites), and,

  • a unique and wide network of practitioners.

The aim and the concept of ANCHISE underline the importance of the acquisition, correlation and processing of heterogeneous data in an efficient way. In this light, semantic reasoning techniques and tools will be a major part of the ANCHISE toolkit. Thus, in the next paragraphs, two different components of the semantic technologies, which are currently being developed in the ANCHISE project, are presented, namely:

  • ART-CH: An Advanced Reasoning Tool for Fighting Trafficking of Cultural Heritage.

  • CTD-TRAC: A Complex Threat Detection Tool for Detecting Illicit Trafficking of Cultural Artefacts.

ART-CH

ART-CH which is presented herein will apply rule-based reasoning with a view to reveal hidden relations relevant to the activities of looters/traffickers. Such relations can be found in the source and destination points of cultural artifacts, in the traits and activities of illicit traders, in the distribution channels of cultural property, in illicit online listings referring to artifacts, etc. Thus, the envisioned tool is expected to play an active role in the fight of LEAs against the trafficking of cultural heritage.

ART-CH, based on the SPARQL Query language, will apply rule-based reasoning in the existing data, will be able to infer logical conclusions from stated facts, and evaluate if these conclusions are complete/consistent. In addition to this, ART-CH will use predefined rules to detect potential relations among different events or entities which may be present in a considered database. ART-CH can drastically increase the investigation and anticipation capabilities of LEAs regarding the illicit trading of cultural property, underpinning activities for identifying the traffickers’ modus operandi, the source and destination places of looted or stolen artifacts, the distribution channels of traffickers, illicit marketplaces, flows of cultural property, etc.

The integration of SPARQL queries will enable the user to explore more results in the ontology. Both the predefined rules as well as the SPARQL queries will be further assessed by practitioners, in order to insure that they are suitable for the analysis of a given data input format. Soon after that, the tool will infer useful results for the end user. It should be noted that both logical as well as probabilistic rules can be integrated in ART-CH.

Figure 17.1 presents a high-level architecture of ART-CH.

Fig. 17.1
A block diagram of the high-level architecture of the A R T C H tool, featuring S W R L rules feeding into a logical reasoner. The reasoner processes these rules to generate inferred results, which are stored and accessed in a semantic database.

High-level architecture of the ART-CH tool

Full size image

As a first step, the expert defines the SWRL [14] logical and probabilistic rules. The SWRL rules definition is a crucial part for this methodology as in this step the knowledge of experts is practically integrated into the ART-CH tool. In this way, the proposed solution is directly connected with the end users’ needs. Then, the Logical Reasoner integrates these rules and applies them on the data stored in the semantic database, which is an ontology. The Logical Reasoner is a component that integrates advanced semantic methods as well as SPARQL queries that reflect the rules proposed by the experts in the previous step. So, through automated processes taking place in the backend, the Logical Reasoner concludes new knowledge deriving from the predefined rules and the data stored in the semantic database. This newly inferred knowledge then is stored back to the semantic database enhancing in this way the previously stored data which at first sight seemed irrelevant and unconnected. This procedure then provides the capability to the end users to gain better insights, discover hidden patterns and correlate different pieces of information in a timely manner and without consuming valuable personnel and time resources.

CTD-TRAC

CTD-TRAC is a threat detection tool, which will generate diverse unified graphs related to illicit cultural heritage trading activities. Through this kind of visualization, end users will be able to easily identify correlations between different entities/activities and detect suspicious illegal trading activities as well as suspects for these activities. In addition to this, the tool will generate alerts and broadcast them via Kafka topics. Through these alerts, LEAs can timely detect traffickers/looters and stop them before they laterally move.

More specifically, the types of inputs which can be used by the tool include, but are not limited to, databases related to the illicit trading of antiquities, illicit listings of cultural property on sites or the social media, coordinates of places where illicit activities frequently take place (e.g., near the borders of countries), data from suspicious financial transactions, travel data of suspects of illegal cultural heritage trading, etc.

The tool can also be connected to other tools (e.g., make use of the inferred results based on the declared rules of a semantic reasoner or make use of the data deriving from a data fusion tool tailored to the needs of cultural trafficking detection). For creating the unified graphs, CTD-TRAC will integrate weights calculated based on the existing connections between entities/activities and the number of existing alerts.

CTD-TRAC will offer an innovative way of identifying diverse types of suspicious/illegal trading activities which undermine the global cultural heritage. The incidents will be detected in real-time or in near-real-time. The above can potentially increase the investigation capabilities of LEAs under the execution of advanced Machine Learning (ML) algorithms, responsible for identifying structured relationships among the different types of entities/activities involved.

Figure 17.2 contains an indicative screenshot from the User Interface (UI) of CTD-TRAC. In this figure, the connections among different entities are visible, together with some integrated controls which will be available to the user in order to customize the visualization results. It should be noted that the UI may be modified according to the feedback received by end users.

Fig. 17.2
A screenshot of the C T D T R A C user interface. It has control sliders on the left for center, charge, and collide. The right side has 4 and 3 nodes as a square and a line with arrows marked from Anastasios Galanis and Smyrlis to crime committed F A L S O and True.

CTD-TRAC user interface

Full size image

Conclusion-Future Work

This paper presents the main two components (i.e., ART-CH and CTD-TRAC) of a semantic engine which is currently being developed in the context of the ANCHISE project for fighting the trafficking of cultural property. ART-CH will apply rule-based reasoning and reveal previously unknown relations between the source and destination points of stolen artifacts, among diverse distribution channels, among different activities of traffickers, etc. The tool will be based on the SPARQL Query Language and future developments will encompass the inclusion of more semantic rules tailored to the end users’ needs. In addition to this, the system will also generate alerts and will publish them via Kafka topics. CTD-TRAC will create unified graphs illustrating potential illicit trading cases and connections among different entities/activities. It will also generate alerts for suspicious trafficking activities of cultural property. These alerts will be provided in real-time and/or in near-real-time to LEAs in order to stop illicit trading before it is completed. Both components are currently under development and future steps include the integration of more input sources, the generation of more types of real-time alerts, and the completion of an API of the tool.