A New Database and Protocol for Image Reuse Detection

One of the main focus of art historical research is the detection of stylistic similarities between works of art. As early as 1915, Heinrich Wöllflin, who is deemed by many as the “father” of art history as a discipline, introduced the notion of comparing artworks to define the style of a period [1]. Art historians, connoisseurs, and art critics are trained to detect whether certain features of an artwork are apparent in another one, and whether two artworks belong to the same artist or not. The experts not only use their visual understanding for such detection, but also rely heavily on historical records and archival information, which are not always sufficiently clear or available. Hence, for decades, art historical research has applied scientific methods such as infrared and x-ray photographic techniques (among others) to help in different instances where the trained eye faltered. Using computational approaches in detecting stylistic traditions of artworks is a relatively new addition to the field [2]. In this paper, we introduce a new digital image database that consists of original artworks that are re-used to create new artworks. We use this database to examine approaches for image reuse detection. In the long run, trying to detect which image is reused with computational methods will help in detecting stylistic similarities between artworks in general [3].

In Western tradition, artists learned their trade by joining ateliers of masters as apprentices. With the introduction of printing press and the wider availability of paper, and especially due to the replacement of etchings on woodblocks with engravings on metal proliferated the art education in ateliers. Metal engravings started to be widely used to teach apprentices drawing, by copying known forms and designs. Novices used these models as the basis of their new artworks, and in that sense, these designs might be the first ones that were massively re-used in visual art. Today, a similar tendency is to use the so called “stock-images” for the same purpose: to help facilitate the design of a new artwork. These images are made freely available online, and can be found in repositories and dedicated websites. With the help of multimedia technologies and digital drawing tools, as well as the availability of free stock images, it has become a common approach in digital image creation to reuse existing images. The digital re-use scenarios are on the one hand quite different than their forefathers from centuries ago: they heavily rely on photo manipulation tools to generate a desired effect or design. On the other hand, certain photo manipulation tools offer the same (basic) design changes that were commonly used centuries ago. Unlike early archives with erroneous and missing data, today, we may have access to precise information about who has reused which image for which artwork. Social networks and online communities for digital artworks, such as DeviantArt¹, and 500px², help us to follow the interaction between artists to minute detail, and build a reliable database of artworks which have re-used other images.

Image reuse detection in digital art is a high-level semantic task, which can be challenging even for humans. Despite the advances in image retrieval and image copy detection techniques, automatic detection of image reuse remains a challenge due to the lack of annotated data and specific tools designed for the analysis of reuse in digital artworks. Image reuse detection differs from general-purpose image retrieval in its scale and amount of the reused pictorial elements. A small object in an artwork can constitute a major part in another composition. An image can be featured in another image in a variety of forms. Developing a global method that addresses all types of image reuse is challenging, as the types of image reuse and modifications vary greatly among different artists and genres of digital art. Another challenge in reuse detection is that the images can have similar content without actually reusing parts from each other. For example, a famous architectural structure can be depicted by several artists. An ideal image reuse detection system should be able to detect even a small amount of reuse without retrieving false positive images. To develop a robust framework for image reuse detection, it is essential to develop tools and datasets that are designed for the task.

The prolific expansion in the reuse of pictorial elements introduces problems related to the detection and analysis of image reuse. Automatic detection of image reuse would be useful for numerous tasks, including source image location, similar image retrieval [4, 5], popularity and influence analysis [6], image manipulation and forgery detection [7, 8, 9], and copyright violation detection [8, 10, 11]. Information about the sources of elements in an image could be used in image search as a semantic variable in addition to low-level image features. Furthermore, such information would be useful for image influence analysis, for discovering the relationships between different genres of (digital) artworks, measuring the popularity of a specific piece of art, and detecting possible copyright violations.

In this paper, we first introduce a novel database called BODAIR (Bogazici-DeviantArt Image Reuse Database³). The BODAIR database is open for research use under a license agreement. To annotate BODAIR, we introduce a taxonomy in image re-use types and techniques. Next, we evaluate a set of baseline image retrieval methods on this database, discussing their strengths and weaknesses. Finally, we propose a saliency-based image retrieval approach to detect reuse on images.

The rest of the paper is organized as follows: Sect. 2 introduces the BODAIR database. Section 3 describes the methods that we employ in reuse detection and Sect. 4 presents the experimental results. Finally, Sect. 5 summarizes our contributions and conclusions.

Open image in new window

DeviantArt⁴ is a social network for artists and art enthusiasts with more than 38 million registered users. DeviantArt members post over 160,000 images every day. Images posted under the stock image category are usually published under an open license and are free to use by others. Using these images, we built an image reuse database. Being artistic creations, the images in our database pose a real challenge for reuse detection.

Open image in new window

The etiquette of DeviantArt requires members to leave each other comments if they reuse an image. This tradition helped us to track down which stock images are used in which new works, by performing link and text analysis on the stock image comments. We used regular expressions in the comments to detect any reference to another artwork and crawled more than 16,000 images in the following six subcategories of stock images: animals, food, nature, places, plants, and premade backgrounds (e.g., Fig. 1). Our image crawler used a depth limited recursive search to download the reused images (children images) and related the images to their source images (parent images). In addition to the automatically extracted parent-children relationships between images, we manually annotated a total of 1,200 images for four reuse types and nine manipulation types:

Each image in the database has an ID and a reference to the ID of the original work if the image is a reused one. The manually annotated images also include the information about the aforementioned types of reuse and manipulation. The manually annotated images include 200 original images, selected among the most popular posts, and their derivatives in each of the six subcategories. The distribution of the partial reuse, direct reuse, use as a background, and remake/inspiration among the manually annotated images are 27 %, 47 %, 44 %, and 6 %, respectively. The direct reuse and background categories have a considerable overlap, since the background images are generally used as a whole without excessive cropping. In this classification, only the direct and partial reuse categories are considered to be mutually exclusive.

Examples of the types of reuse are shown in Fig. 2. Figure 3 shows the overlaps between different categories of image reuse in the database. The matrix is symmetric, and the diagonals show the total number of images annotated with a given reuse or manipulation category. Only remake is not included in this figure. There are 75 exemplars of remake, and since they are very different than the original stock images, they are not annotated with any manipulations. Subsequently, the remake category has no overlaps with other categories. Manipulation examples are shown in Fig. 4.

Open image in new window

Open image in new window

In this section, we describe the methods we apply for image reuse detection. We first summarize several image description methods that are used in matching-based tasks in computer vision, such as content-based image retrieval, image copy detection, and object recognition. Then, we discuss how saliency maps could be combined with image descriptors to improve matching accuracy and reduce computation time in image reuse detection.

Representing an image by its most discriminant properties is an important factor to achieve higher accuracies. Different feature descriptors extract different features from the images to achieve invariance to certain conditions such as color, illumination or viewpoint changes. Traditional image recognition methods usually involve sampling keypoints, computing descriptors from the keypoints, and matching the descriptors [12]. The image descriptors can also be computed over the entire image without sampling keypoints. However, these global features usually perform poorly in detecting partial correspondences between images where a small portion of an image constitutes a major part of another image. Local descriptors, such as SIFT [13] and its color variants CSIFT [14], Opponent-SIFT [14], on the other hand, are more robust in the detection of partial matches. Although the local descriptors usually perform better than global approaches, computation of local descriptors can be computationally expensive as they usually produce a high-dimensional representation of the image which may create a bottleneck in large-scale image retrieval tasks. As suggested by Bosch et al. [12], Bag-of-visual-words (BoW) methods reduce the high-dimensional representation to a fixed-size feature vector, sacrificing some accuracy [14].

More recent approaches make use of convolutional neural networks (CNNs) to learn powerful models from the data itself [15, 16, 17, 18]. Training these models usually requires a large dataset, such as the ImageNet [19] which consists of over 15 million images in more than 22,000 categories. However, it has been shown that the models trained on a set of natural images can be generalized to other datasets [20], and the features learned by a model can be transferred to another model with another task [21].

In this work, we evaluate five image descriptors that are commonly used in image matching and content-based image retrieval problems for image reuse detection: color histograms, Histogram of Oriented Gradients (HOG) [22], Scale Invariant Feature Transform (SIFT) [13], and the SIFT-variants OpponentSIFT and C-SIFT, which are shown to have a better overall performance than the original SIFT and many other color descriptors [14]. In addition, we also use a CNN model [15] pretrained on the ImageNet [19] as a feature extractor, using the fully connected layer outputs (FC6 and FC7) as feature vectors.

Different strategies in image description can lead to a fixed or variable size description of an image. A fixed-size vector representation of images allows the use of vector distance metrics, such as the Euclidean distance, to measure image similarity. As the color histograms and HOG features produce fixed-size image descriptions, candidate matches for a query image can be ranked in order of ascending standardized Euclidean distance. On the other hand, the local descriptors, SIFT and its variants, can extract features from a different number of keypoints on each image, resulting in a variable-size representation. Variable-size representations of images usually require computation-intensive pairwise matching processes. Such a matching process can be improved using an inlier selection algorithm, such as RANSAC, which selects random feature pairs and keep the largest set of inliers to find corresponding image matches [23].

Open image in new window

Image saliency can help narrow down the areas of interest in image reuse detection. In our earlier work [24], we showed the effectiveness of using saliency maps in image description for image reuse detection. The purpose of the saliency map is to represent the conspicuity or saliency at every spatial location in the visual field by a scalar quantity and to guide the selection of attended locations [25]. Many stock images feature a foreground object that is more likely to be used in other artworks. Therefore, features can be extracted only from the salient regions, which will reduce the processing time and improve the matching accuracy. We use the saliency maps only in the stock images, assuming that each stock image provides such a region of interest to the composition images. We extract features from the query images as a whole, as the use of saliency maps could exclude some references completely.

The overall proposed framework (see Fig. 5) consists of four modules: salient region detection, salient object segmentation, feature extraction, and feature matching. For saliency map estimation, we use a recently proposed Boolean Map based Saliency (BMS) model [26], which is an efficient and simple-to-implement estimator of saliency. Despite its simplicity, BMS achieved state-of-the-art performance on five eye tracking datasets. To segment salient objects, we threshold the saliency maps at their mean intensity to create a binary segmentation mask.

In this work, we focused on how to detect image reuse in digitally created artworks. To that end, we first collected stock images from DeviantArt, a website where digital artworks are posted by users, and built the BODAIR database. Using automatic link and text analysis in the images’ comment sections, as well as manual labeling, we made available a database that has two sets of images: stock images, and images that reuse those stock images. We furthermore made the distinction between “type of reuse” and “type of manipulation”, i.e. we highlighted the difference between the contextual approach, and technical approach in reuse. We have detected four type of reuse scenarios, and nine ways of manipulations. We evaluated methods for image reuse detection that are widely used in related tasks, such as image retrieval and object recognition. Lastly, we improved the performance of these methods by using saliency maps. The methods we evaluated provide a baseline for the future research on image reuse detection.