1 Introduction

Understanding the interplay between different factors influencing movement is a critical challenge, e.g. to support planning of new roads, public areas or buildings. Does the park need a special landmark for better orientation? Is a building easy to navigate in? Is there something distracting at a street that leads people to a different kind of movement and may result in more accidents? These and similar questions could be answered with a better understanding of contextual information in trajectory analysis. A trajectory is a sequence of geometric points, referring to the current position of the observed object. With the availability of more and better GPS devices, trajectory data becomes more relevant for explaining movements of objects. New devices are able not only to track the current location and time, but also to enrich the data with annotated context information. Context information includes intrinsic and extrinsic factors, such as the current temperature of the surroundings, the mode of transportation or the age of an observed person.

The movement of dynamic objects is a complex interplay of different external and internal factors [6]. If we assume that we know every influencing factor of an observed movement, the idea of our work is that we can derive an underlying function that defines the individual movement. While research on spatio-temporal trajectories has been intense over the past years, there has been less attention to the usage of additionally available features in the analysis of semantic trajectories [23]. The goal of our work therefore is to introduce a novel approach for modeling and working with what we call semantically enriched trajectories, where we assume each point of a (discrete) trajectory to be annotated with a set of attribute-value pairs that describe known annotated features. In the following we therefore assume for each annotated point to be in the form of \(p = (\text {lat},\text {long},\{(\text {att}_1,v_1),(\text {att}_2,v_2), \dots , (\text {att}_m,v_m)\})\), where lat and long describe the spatial position of the point and v holds the value for each of the m attributes. Note that \(m=0\) is possible and results in no extra annotated attributes besides the spatial position. In such a case we call them geometric trajectories. Attributes can hold different types of values. Specifically numerical attributes, like temperature or velocity, but also categorical attributes, like mode of transportation or current season.

We recognize the similarity of semantically enriched trajectory data to the data used during the so-called reverse engineering process. Here the goal consists of modeling an unknown software system by regarding only the created software traces of the system. These traces consist of multiple sets of attribute-value pairs and therefore match the appearance of semantically enriched trajectories as defined. We choose Extended Finite State Machines (EFSM) [21] as a modeling option for the semantically enriched trajectories. The EFSM extends the well-known finite state machine formalism by adding transition guards that must be matched by the traversing points. An EFSM is created by a set of given input traces and, due to its determinism, each of these traces follows a unique path in the received EFSM. One example for an EFSM can be found in Fig. 1.

Therefore the basic idea of this work is as follows: We model a set of semantically enriched trajectories with an EFSM. The EFSM itself already holds information about the creating set in its general appearance and the annotated guards of the transitions. We enrich the EFSM further by tracing each of the trajectories of the creating set through the EFSM. This means that we follow the unique path of every trajectory through the EFSM and enrich the transitions with the information of the annotated points. As a result we have a complete model of the entry set of trajectories and use it for different tasks in the trajectory analysis. It is important that with the developed approach we are able to work even on a small set of trajectories. Another advantage is that with the calculated model we always have an easy to read representation of the input trajectories and that every result of our further analyses is explainable by regarding the EFSM.

2 Inferring Semantically Enriched Representative Trajectories

In trajectory analysis a common first step is to calculate a representative trajectory for a given cluster of trajectories. In the context of our work a representative trajectory is defined as a trajectory that holds all “relevant” informationFootnote 1 of the given cluster and represents the group of trajectories [19]. Representative trajectories can be used to reduce calculation costs in further analysis steps and are useful in the context of visualization [22]. Currently they are often calculated based only on spatial characteristics (e.g. [2, 11, 20]). A given cluster might be grouped together according to the trajectories’ spatial characteristics, but also according to their semantic properties. Therefore, the inferred representative trajectory of our approach should be calculated based on all properties and should still hold an acceptable geometric representation for the set of trajectories.

In our work [17] we first introduce an algorithm for deriving an EFSM from a given cluster of trajectories. The EFSM models the cluster and we assign each trajectory to one specific path in the EFSM. We annotate each transition of the path with the attribute-value pairs of the points of the aligned trajectories and receive an annotated EFSM. We then define an algorithm for calculating a representative path in the EFSM with a greedy approach. The approach determines in every state, beginning with the initial state of the EFSM, which transition has been taken by the most trajectories at the current index of the representative path. Because the EFSM might include loops and therefore transitions might have been traversed by the same trajectory multiple times with different points, it is important to consider the current index of the representative path and the indices of the belonging points to a transition.

The algorithm receives a path of the EFSM and further enriches it with the data of transitions. The annotated attributes are not only the semantic attribute-value pairs, but also the spatial latitude and longitude of the points. Therefore we can reinterpret the path and receive a semantically enriched representative trajectory. An example of the whole process of inferring an enriched path and a geometric representation of the trajectory can be found in Fig. 1.

Fig. 1
figure 1

Left: Hurricane data set (gray, from [9], with permission) and representative trajectory inferred using our approach (blue). Right: EFSM inferred from the input data and derived representative path (blue). Letters represent semantic (shape) annotations (see [9]), latitude and longitude refer to the spatial position of the points. To reduce clutter, only guards at branching points are shown

Full size image

To evaluate our approach we compare it against current state-of-the-art approaches regarding the geometric quality according to the Fréchet distance [1]. The results can be found in the table in Fig. 2.

Fig. 2
figure 2

Number of points, mean and median fréchet distance of all given clusters in [9] for different approaches

Full size image

They show that our approach is able to calculate a representative trajectory that is similar to the input trajectories in the context of the number of points and also is close to the geometric quality provided by solely geometric approaches. We also assess the semantic quality of our results by manually inspecting the annotations of the generated trajectories. As we are not aware of any other algorithm using semantic annotations for calculating a representative trajectory, we were not able to use any other benchmark.

3 K-Means for Semantically Enriched Trajectories

In order to calculate a representative trajectory for a given group of trajectories there is a need for a clustering algorithm. The classification process usually relies only on spatio-temporal properties of the trajectories (see references in [23]). Motivated by our previous work we aim to include the semantic factors in a clustering algorithm, because solely spatio-temporal clustering might not reflect the characteristics of the additional semantic data.

We introduce EFSMClust [18] which is based on the k-means algorithm [13]. We modify the centroid-based approach to work with semantically enriched trajectories. To do so we use EFSMs and infer representative paths as the needed centroids. Additionally, we define a new similarity measure EFSMSim for scoring the similarity between semantically enriched trajectories and representative paths by aligning a trajectory with a representative path. EFSMSim returns a score reflecting how well the attributes of each point of the trajectory fit the range of attribute values of the aligned representative segment. EFSMClust works as follows: We regard an input consisting of a set of semantic trajectories and a parameter k indicating the wished number of clusters for these trajectories. The algorithm first calculates an EFSM and a representative path for the whole set of trajectories. The received representative path is used as the first initial center. Afterwards \(k-1\) initial centers are iteratively calculated, using an initialization strategy following [4]. When all initial centers are calculated, the trajectories are assigned to the centers according to the score calculated by EFSMSim. The algorithm then calculates an EFSM for each of the received clusters and the representative path of the new EFSM as the new centroid. Some trajectories might change the assigned cluster based on the new calculations. The algorithm terminates when the calculated average radius of all clusters does not improve after one iteration. It returns k clusters of the input trajectories, the EFSMs, and the semantically enriched representative central trajectories for each cluster.

We evaluate EFSMClust by comparing it with state-of-the-art spatial clustering algorithms. Some results for a set of trajectories of the GeoLife data set [24] can be found in Fig. 3. The GeoLife data set describes the movement of persons in a big city. The data set is annotated with the spatial position, time, weekday, season and mode of transport. We observe that EFSMClust identifies a single outlier in a cluster, which is semantically reasonable, because it is the only trajectory which was created by using the bus only, while all other trajectories were created using other modes of transportation or a mixture of them. Spatial algorithms fail to recognize this property. Our experiments show that our algorithm achieves a similar geometric quality and at the same time returns a representation of the centroids that can be used by domain experts to interpret both spatio-temporal and semantic information as well as to explore their possible relationships.

Fig. 3
figure 3

(Data extracted from [24]) Results for a person of the GeoLife data set

Full size image

4 Application in Wayfinding Tasks

During wayfinding, a person has to find a certain goal in an unfamiliar environment by first planning a route to get there and second moving along the route and making adaptations to it based on experienced environmental factors [7]. In order to understand different wayfinding strategies and routing decisions it is important to include all internal and external factors into the analysis to come up with a suitable model of the process [8]. In current research such a model is missing [7] and therefore domain experts have to classify wayfinding strategies manually (as in [12]). To support the process of classifying wayfinding strategies, we use the introduced tools for the analysis of semantically enriched trajectories.

In a first experiment we use 96 trajectories which were created during an experiment for measuring navigational map reading competencies of children using the location-based game GeoGami [15]. GeoGami provides a map of the environment (real or VR) and includes different wayfinding tasks. Using EFSMClust we calculate a set of clusters for the given semantically enriched trajectories for one of the tasks (task 11). To receive a comparable ground truth we ask four domain experts familiar with the regarded experiment to classify the trajectories manually according to the applied strategies. Afterwards we ask the experts in a two iteration Delphi process [5] to rate all other results, including the one by EFSMClust. Furthermore we calculate the Rand index [14] to give a calculated impression on how well EFSMClust matches the results of the experts. In the expert rating EFSMClust is placed fourth out of five. However, comments of the experts reveal that they struggle with understanding certain clusters and miss an explanation of the made decisions during the clustering process. This can be solved by providing the cluster EFSMs and central paths. Moreover the process shows strongly differing ratings of the experts along the two iterations. We assume that the classification of the trajectories is not unique and a challenging task. The Rand index reveals that almost all results by the domain experts have the largest overlap with the EFSMClust result. This leads us to the result that the application of EFSMClust can work as a common start point for a discussion by experts.

We aim to confirm these results by providing a second experiment with a clearer ground truth. Therefore we apply EFSMClust on a set of 128 wayfinding trajectories received in a wayfinding experiment in a virtual mall setting with several floors [12]. Domain experts classified the received trajectories into three different categories of applied strategies: A central, a directional and a floor strategy [10]. Roughly one quarter of the trajectories remained unclassified by the experts. In our experiment we show that we can reproduce the given ground truth with an accuracy of over \(93\%\), where the difficulties appear solely in the classification of trajectories of the central strategy. We justify the misclassifications of trajectories of the central strategy with EFSMSim being a more local measure, which fails to recognize a more global strategy, such as the central. Additionally, each trajectory of the central strategy could partly be interpreted as directional and floor strategy, which is an insight we gain as the result of applying EFSMClust. In order to gain insights into the unclassified trajectories, we compare them using EFSMSim with the clusters defined by the domain experts. This results in three values for each trajectory, indicating the similarity to the given wayfinding strategies. Trajectories with high similarity to one or more strategies can be shown to actually include parts of these strategies. Trajectories with low similarity to all of the clusters can be shown to include participants without any known strategy. We conclude that we can use EFSMClust to produce insights into the unclassified trajectories and create suggestions for domain experts.

5 Conclusion

In our work we introduce an approach to use so called Extended Finite State Machines (EFSM) to model a set of semantically enriched trajectories. By this we open the possibility to learn about the impact that different intrinsic and extrinsic factors might have on the behavior of the individual. We use the novel modeling of trajectories to infer a semantically enriched representative trajectory of a given cluster of trajectories. Evaluations show that we are able to retain a certain geometric quality and also meet our aim of gaining insights into the semantic relations of the trajectories. Based on the novel modeling approach we introduce the k-means based clustering algorithm EFSMClust with its similarity measure EFSMSim. EFSMClust enables us to calculate a set of k clusters not solely based on spatial characteristics, but also on semantic attributes. We evaluate our work in the context of wayfinding strategies. In two different experiments we show that EFSMClust is usable to support domain experts in their task of classifying trajectories regarding applied wayfinding strategies.