Keywords

1 Introduction

In the recent past Multimedia and the Semantic Web have moved closer together, reflected in many efforts like W3C recommendations for Media Annotations [2] and Fragment URIs [7], which are also taken up by industry like described e.g. in [4, 5]. These efforts mainly focus on a standardized representation of multimedia metadata and the connection to knowledge using Semantic Web technologies, but currently do not consider multimedia specific query functionalities. On our poster we present SPARQL-MM, an extension for SPARQL that introduces spatio-temporal filter and aggregation functions to the de-facto standard query language in the Semantic Web. In the following we give an example, how media is currently represented in the Semantic Web and which functions are missing for current SPARQL.

2 The Hidden Knowledge of Fragment Annotations

Figure 1 outlines an example of an annotated video showing the winners ceremony of an extreme sports event. We use Media Fragment URIs to link annotations to specific spatio-temporal parts of the video. The specification provides a media-format independent, standard means of addressing media fragments on the Web using Uniform Resource Identifiers. It supports particular name-value pairs, e.g. (’t=’start’,’end) for temporal and (’xywh=’,x’,’y’,’width ’,’height) for regional fragments. In our example Connor Macfarlane appears from second 194 to 198 on the left side, while Lewis Jones is marked from second 193 to 198 on the right side. If a user wants to retrieve the (spatio-temporal) snippet, that covers exactly these both person, she may issue a query like: “Give me the spatio-temporal snippet that shows Lewis Jones right beside Connor Macfarlane”. Currently SPARQL does not support such queries, because some necessary information is not explicitly expressed in RDF, but hidden within the Media Fragment URIs. Neither does SPARQL support filter functions like rightBeside or temporalOverlap, nor aggregation functions like boundingBox.

Fig. 1.
figure 1

A sample for an annotated video

Full size image

3 SPARQL Multimedia Functions

Table 1 lists all functions that we took into account for our extension. We follow well known standards like DE-9IM [3] for topological and [1] for temporal relations. The parameters r1, r2 are of type media fragment resource, the boolean flag decides, if equal ranges are included or not (whereby false is default). A detailed description of all functions in human and machine readable format (following the sparql-service-description extension for describing SPARQL extensions and function librariesFootnote 1) can be found on the source repository of our reference implementationFootnote 2. Each function is identified by a unique URI but all together share the same base URI mm: \(\mathtt{{<}}\) http://linkedmultimedia. org/sparql-mm/functions# \(\mathtt{{>}}\). Currently the implementation uses the OpenRDF SesameFootnote 3 API and its extension interfaces, which makes it backend agnostic but requires expensive and inefficient in-memory calculations. We plan to improve this by a backend specific implementation for KiWi TriplestoreFootnote 4 in combination with SQL/MM [6].

Table 1. SPARQL-MM functions
Full size table

Using SPARQL-MM functions we can now formulate the users need from Sect. 2 (“Give me the spatio-temporal snippet that shows Lewis Jones right beside Connor Macfarlane”), as a SPARQL query like in Listing 1.1. We use mm:temporalOverlaps to get fragments that appear in the identical temporal sequence. mm:rightBeside handles the spatial relation and mm:boundingBox merges every two fragments that match the filters. You can test the query by selecting the example from the samples menu in the demo pageFootnote 5.

figure a

4 Conclusion and Further Work

In this paper we presented SPARQL-MM, a function set that adds spatio-temporal filters and aggregators to SPARQL. In further steps we will extend the function set to fragment feature extractors (e.g. getDuration) that can for example be used for sorting. We also try to include more complex spatial structures like polygons, e.g. by extending the MediaFragments URIs with a rel attribute that points to SVG shapes.

SPARQL-MM is implemented as Open Source project hosted on GitHub.