## Abstract

Surveillance camera usage has increased significantly for visual surveillance. Manual analysis of large video data recorded by cameras may not be feasible on a larger scale. In various applications, deep learning-guided supervised systems are used to track and identify unusual patterns. However, such systems depend on learning which may not be possible. Unsupervised methods relay on suitable features and demand cluster analysis by experts. In this paper, we propose an unsupervised trajectory clustering method referred to as t-Cluster. Our proposed method prepares indexes of object trajectories by fusing high-level interpretable features such as origin, destination, path, and deviation. Next, the clusters are fused using multi-criteria decision making and trajectories are ranked accordingly. The method is able to place abnormal patterns on the top of the list. We have evaluated our algorithm and compared it against competent baseline trajectory clustering methods applied to videos taken from publicly available benchmark datasets. We have obtained higher clustering accuracies on public datasets with significantly lesser computation overhead.

## Introduction and related works

Object motion pattern identification and trajectory analysis are two important steps in various computer vision applications (Ahmed et al. 2018b). Trajectory analysis is used in many video analysis tasks such as video summarization (Dogra et al. 2016; Ajmal et al. 2017), event detection (Reddy and Veena 2018), and visual surveillance (Vishwakarma and Agrawal 2013; Huang et al. 2018). Analysis of large volume trajectory can be effective in traffic analysis (Santhosh et al. 2018) and crowd monitoring (Bera et al. 2016). The primary application of such analysis is abnormality detection (Roshtkhari and Levine 2013; Mabrouk and Zagrouba 2018). However, unsupervised clustering of trajectories is a difficult task. Clustering using simple features extracted from object trajectories, e.g. object location (\(x_i,y_i,t_i\)), produces poor results (Xu et al. 2015). They cannot be used for complex and long-term analysis. High-level features like source, destination, path, and activity can be used to represent moving objects. These high-level features can help to find patterns and group them together. The objective can be to classify the trajectories into frequent patterns; abnormal patterns belong to infrequent movements or outliers.

We note three different approaches of trajectory analysis. The first one is supervised learning approaches. It uses a set of known or unknown patterns to train neural networks. The second one is semi-supervised methods that use minimal labelled data for learning. The third one is unsupervised methods that primarily depend on feature selection, clustering, and analysis of clusters. Next, we will discuss these methods and how our method bridges the gap.

*Supervised trajectory analysis* Various form of supervised methods utilizes different trajectory features for learning. Hidden Markov model (HMM)-based learning method (Kwon et al. 2017) is used to extract semantic region for trajectory analysis. Artificial neural network-based method such as convolutional neural network (CNN) (Mehrasa et al. 2018) is used to analyse player trajectory and team activity and also in detecting, tracking, and traffic behaviour analysis (Ren et al. 2018). Recently, Zhao et al. (2018) used trajectory convolution for human action classification and a variation of CNN model for trajectory-based video action recognition (Dai and Srivastava 2019). Many surveillance applications such as pedestrian trajectory and crowd interaction analysis (Xu et al. 2018b) also utilize the power of supervised machine intelligence. Recurrent neural network (RNN) (Ma et al. 2018) is also used in abnormal trajectory detection and sequence learning. Xu et al. (2018a) proposed to use a dual mode (static and dynamic) for supervised traffic analysis. All these methods demand a manual annotated training data and majority of the algorithms are scene specific and do not support transferable learning mechanism.

*Semi-supervised trajectory analysis* Semi-supervised methods overcome some of the problems of supervised learning such as a demand of large volume training samples. In this area, a trajectory histogram-based semi-supervised method (Chen et al. 2017) is proposed for dangerous event detection. The method uses minimal training samples only for the dangerous events. Maximum likelihood-based method (Chakraborty et al. 2018) is also used to detect freeway traffic. Topic models are popular in many semi-supervised tasks and also used in trajectory analysis (Wang et al. 2019) to explore human activity analysis. Modelling approach (Feizi 2019) is also proposed for abnormal behaviour detection. Graph-based structural learning by combining structure representation (Michelioudakis et al. 2019) is also proposed for trajectory learning and composite event detection. Although these methods utilized minimal training samples still all the benefits of unsupervised learning are not achieved.

*Unsupervised trajectory analysis* Unsupervised methods are free from large volume training samples and usually not design for specific applications. Majority of these methods depends on feature selection, distance measurement policies, and clustering algorithms. Incremental trajectory clustering based on Dirichlet process mixture model (DPMM) (Hu et al. 2013) and dense point-based trajectory clustering framework (Ochs et al. 2014) is used to represent long-term videos. Lin et al. (2016b) have proposed droplet-based features to find the abnormalities. Clustering trajectories using low-level information such as position often produces poor results (Xu et al. 2015). To overcome this problem, state-of-the-art mean shift algorithm (Comaniciu and Meer 2002) and shrinkage-based frameworks (Xu et al. 2015) for unsupervised trajectory clustering have already been proposed. Xu et al. (2015) have proposed adaptive multi-kernel-based shrinkage (AMKS), and Wang and Carreira-Perpinán (2010) have proposed manifold blurring mean shift (MBMS) algorithms as improvements. However, the majority of these existing techniques rely on a single feature of the trajectory. Fuzzy theory and multiple independent features-based method (Anjum and Cavallaro 2008) is applied to identify distinct patterns. Ahmed et al. (2018a) present a fuzzy aggregation scheme for abnormality detection. Recently, Saini et al. (2019) proposed a graph-based trajectory classification method that can be used in traffic analysis. Particle swarm-based trajectory clustering (Izakian et al. 2016) is applied on a synthetic dataset. In Feng et al. (2017) and Xu et al. (2017), authors have used deep appearance and motion features together to detect abnormality. Choong et al. (2016) have proposed a similarity function to achieve clustering of spatio-temporal data. Short duration trajectory (Lin et al. 2016a; Sharma and Guha 2016) extracted from feature tracker also used for clustering and understanding crowd behaviour. Density-based approach is popular among unsupervised algorithms and also applied in vessel trajectory analysis (Li et al. 2018). Zhao et al. (2019) used an unsupervised decision module to identify traffic abnormality. Unsupervised trajectory modelling using location, velocity, and time appearance (Campo et al. 2018) is used to cluster trajectories. Das and Mishra (2018) proposed a mean shift-based method for crowd trajectory analysis and abnormality detection. Recent approaches such as adversarial framework (Spampinato et al. 2020) are also used in abnormal event detection. Yue et al. (2019) utilized deep trajectory representation and proposed a deep trajectory clustering (DETECT) for behaviour analysis. Neural network-based trajectory analysis for traffic analysis proposed in Bandaragoda et al. (2019). Reviews on trajectory analysis (Ahmed et al. 2018b) and clustering (Yuan et al. 2017) show the methods and applications of trajectory-based analysis in detail.

*Challenges and gaps bridged by our work* Majority of trajectory analysis methods used for abnormality and event detection. The main challenges of such supervised and semi-supervised trajectory analysis are (i) demand of manual annotated trajectory dataset and in most of the case the dataset is scene specific, and (ii) a concrete definition of normal and abnormal patterns for detection and classification. It is noted that the concept of “normality” and “abnormality” is not fixed always. For example, a high-speed car is abnormal where the speed is restricted by upper limit, whereas a low moving car is treated as abnormal when a highway lane is restricted by minimum speed. In unsupervised methods, selection of features played most vital role. State-of-the-art low-level features such as speed, velocity, and movement patterns are well explored. The challenges of unsupervised methods are solved here using a suitable selection of high-level features such as origin, destination, and path deviation of trajectory and choice of suitable clustering framework for extracting logical meaning of movement clusters. Finally, multi-criteria decision making is used to rank the patterns to identify abnormalities. The motivation for such work is manifold. The primary application is to identify unusual patterns by analysing unsupervised clusters and rank them accordingly. At many public places like subway stations, railway junctions, highway junctions, or airports, the method can be used for detecting unusual movement patterns from a large volume of camera footage. The method also significantly reduces the volume of data by representing trajectories using high-level interpretable features such as path and deviation which makes the method suitable for large volume trajectory analysis.

Rest of the paper is organized as follows. Section 2 explains the proposed framework. Experimental results are provided in Sect. 3. Finally, Sect. 4 concludes the paper.

## Proposed framework

In this section, we present the approach of unsupervised pattern searching and ranking to understand abnormal movements. Let a spatio-temporal scene of finite duration be represented using a set of trajectories \(\tau =\{T_1,T_2,\ldots ,T_n\}\). Each trajectory can be represented by a pair of points, namely entry and exit points (Ahmed et al. 2018a). We perform a partitional trajectory clustering and the ranking scheme that combines multiple features to generate crisp partitions for indexing. Independent features are aggregated to obtain a higher degree of descriptiveness of the trajectories as opposed to using a single feature. Each feature produces an abnormality score of a moving object. Object trajectories are then represented by this abnormality score. Initially, the trajectories are extracted using multi-object tracker. Then, entry/exit regions are identified from the cumulative patterns. Next, entry-to-exit paths are extracted; hence, any deviations from the path (PD) can be easily obtained. In the next stage, each trajectory is assigned independent abnormality scores based on these high-level features and the scores are aggregated using multi-criteria decision making (MCDM) and the score is used to rank each object. We have experimented with two state-of-the-art MCDM methods, namely entropy-based simple additive weighting (SAW) (Abdullah and Adawiyah 2014) and weighted Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) (Hwang et al. 1993). Figure 1 depicts the different modules of the proposed approach. Next, we will discuss each module in detail.

### Unsupervised trajectory clustering

We introduce an unsupervised trajectory clustering method that is referred to as t-Cluster. The method takes a set of trajectories extracted using MOT and returns three sets of clusters. A trajectory belongs to exactly three clusters taken from the different sets. The method is presented in Algorithm 1.

### Trajectory representation, fusion and ranking

In visual surveillance context, abnormality can be defined in various ways. For example, infrequent motion patterns or abnormality in average velocity or time spent can be very important to understand abnormal situations. A method to calculate abnormality scores by fusing individual feature-based scores is presented here. All movement patterns (clusters) that are based on entry \((C_{entry})\), exit \((C_{exit})\), and entry-to-exit \((C_{path})\) are extracted at the beginning using t-Cluster. Assume the patterns based on any one of the above mentioned criteria is represented as given in (1), where frequency of each pattern is given in (2)

A pattern of movement can be considered as discrete random variable. Hence, the weight of a pattern \(\omega (p)\) is defined by the probability density function given in (3)

such that the condition given in (4) is satisfied.

According to our assumption, lower frequency represents higher abnormality. The abnormality score (\(\sigma \)) of a pattern is scaled between 0 and 1. \(\sigma \) is defined by (5), where higher score represents more abnormality.

A trajectory may belong to three different patterns, namely entry based, exit based, and entry-to-exit path based. \(\alpha \), \(\beta \), and \(\gamma \) represent abnormality scores based on the above three criteria. A DTW barycentre averaging (DBA) (Petitjean et al. 2014) method is used to estimate the average path between entry-to-exit. The dynamic time wrapping (DTW) algorithm is heuristic in nature and used to calculate a global average of various time series applications. DTW distance between two trajectory \(t_a\) and \(t_b\) is calculated recursively between \(t^a(1\ldots i)\) and \(t^b(1\ldots j)\) such that the Euclidean distance (ED) is aligned and mapped as:

The method iteratively refines the initial average sequence in order to minimize the squared distance (DTW). The path is defined by \(\kappa =DBA(\tau )\), where \(\tau ={\tau _1,\tau _2,\ldots ,\tau _m}\) is the set of all trajectories that belong to similar path. For example, Fig. 2 depicts the construction of average path using DBA.

Deviation-based abnormality score of a trajectory is then defined by the maximum deviation of the trajectory from the average path. The Hausdorff distance between the path and the trajectory is estimated using (7)

The displacement from the average path is a high-level feature, and it represents the trajectories. Higher the value of the deviation, more the abnormality. Path deviation (PD) of a trajectory is formally defined in (8)

PD can be useful to measure spatial displacement as well as temporal displacement. It is observed that normal moving targets have lower PD compared to abnormal cases such as moving in the wrong direction, moving with high speed, loitering, unusual stop, and moving slowly. For example, Fig. 3 shows deviation of a normal target, Some abnormal cases like fast moving target (Fig. 4), loitering (Fig. 5), side walker (Fig. 6), and moving in opposite direction (Fig. 7) are shown. It is noted that the deviation of an abnormal target may have a higher deviation score.

Next, path normality score (\(\gamma \)) and path deviation abnormality score (\(\zeta \)) are calculated in a similar fashion as done in case of \(\alpha \) and \(\beta \). Higher the value of \(\gamma \) or \(\zeta \), more unusual the trajectory. Finally, a set of local ranks (\(L_{\kappa }\)) based on \(\alpha , \beta , \gamma \), and \(\zeta \) are estimated. The highest abnormality scores represent the most unusual patterns of a given scene.

#### Fusion techniques

Multi-modal feature fusion has gained considerable attention of the researchers for data-mining related tasks. Fusion can be done using lower-level features that are often referred to as early fusion. On the other hand, decision-level or late-fusion and a hybrid approach can also be taken. In visual surveillance, fusion at low level is common. However, decision-level fusion is still evolving. Here, we present two state-of-the-art fusion methodologies, namely entropy-based SAW (Abdullah and Adawiyah 2014) and TOPSIS (Hwang et al. 1993) have been used to fuse multiple criteria and produce a meaningful abnormality score. Entropy has been used to estimate the weight of each criterion.

*Simple Additive Weighting (SAW) fusion* Shannon entropy (Shannon 2001) is refereed as an important measure of disorder or uncertainty. Using entropy, we can measure the uncertainty present in the information. Probabilistic distribution of the patterns can represent uncertainty. In case of multi-criteria decision making (MCDM), entropy decides the weight of the criterion. Higher entropy represents higher diversity in the information. Our assumption is that most uncertain event has a higher abnormality score for randomly moving objects. The entropy of the random variable *X* is measured using (9), where \(x_i=\{x_1,x_2,\ldots ,x_N\}\) is the set observations of *X* and \(p(x_i)\) is the probability of taking the value \(x_i\).

Hence, entropy of different criteria is calculated using (10).

The weight of a pattern or trajectory follows the pattern (\(\chi \)) that is defined in (11), where \(\omega \) is the abnormality score of the pattern and \(\psi \) is the weight of the criterion.

Simple additive weighting (SAW) is a widely known method of MCDM. In this work, it has been used to fuse the trajectory and parameter weights into a single parameter. \(\Omega \) represents the combined weight of trajectory, and it is calculated using (12).

Finally, the trajectories are ranked according to their aggregated abnormality scores.

*TOPSIS-guided fusion* The Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) method is mainly used to get an optimal solution, which is farthest from the negative ideal solution and closest to the ideal solution. Similar to SAW method, TOPSIS is also used to aggregate individual abnormality scores (\(\omega \)). Positive ideal score and negative ideal score of a set of *N* trajectories can be defined using (13) and (14).

Aggregated abnormality scores using TOPSIS (\(\varphi \)) can then be estimated using (15). Our assumption is that all criteria are evenly probable.

A global ranking of the trajectories based on \(\alpha , \beta , \gamma \), and \(\zeta \) can be obtained using fusion. We denote \(G_{\kappa }\) to be the global rank of a trajectory obtained using SAW(\(\Omega \)) or TOPSIS (\(\varphi \)), where highest rank (e.g. rank 1) represents most abnormal/unusual pattern according to our assumption. Global ranks help to resolve ambiguity in local ranking when they more than one trajectories share same local rank.

## Experimental results

In this section, we present the results and comparisons with state-of-the-art methods using publicly available datasets.

### Dataset

We have used four public datasets, namely MIT trajectory dataset (MIT) (Wang et al. 2011), QMUL junction dataset (QMUL) (Long et al. 2016), and crowd dataset (UCF) (Ali and Shah 2007). Details of about these datasets are summarized in Table 1. These datasets have been selected because they provide challenging as well as the simplistic scenario. Some of the videos in these datasets contain a large number of moving objects and the presence of occlusions.

### Unsupervised trajectories clustering

In this section, we present the results of the trajectory clustering (t-Custer). We also present comparative results using other distance measures and clustering techniques. We have compared the proposed method (only using \(C_{path}\)) and popularly known state-of-the-art path clustering techniques such as mean shift, MBMS (Wang and Carreira-Perpinán 2010), AKMS (Xu et al. 2015), Fast AKMS (Xu et al. 2015), DETECT Yue et al. (2019), and deep representation-based feature (Bandaragoda et al. 2019). Results reveal the superiority of our proposed method over the existing techniques.

We have also experimented with distance-based trajectory clustering methods aided by Euclidean distance and dynamic time warping (DTW) combined with Kmeans to extract patterns of movements form cumulative trajectories. Table 2 summarizes the accuracy of path-based pattern clustering using various methods. We have calculated the adjusted rand index (ARI) to measure clustering similarity. The experiment is carried out by taking random 80% trajectories in each run and repeated 20 times. The mean ARI and the distribution of accuracy are shown in Fig. 8 (QMUL), Fig. 9 (MIT), and Fig. 10 (UCF) dataset. The results support the superiority of our method in terms of higher ARI and lower deviation of accuracy in randomly selected trajectories. Table 2 summarizes the average ARI of the baselines and proposed method.

Figures 11, 12, and 13 present source, destination, and path-based cluster analysis obtain using t-Cluster when applied on QMUL dataset. The box plot in source and destination is obtained by considering the distance of each trajectory from the cluster centre, and in path-based clustering, the Hausdorff distance of each trajectory from the path is considered. The outlier points of the boxes are more likely to be unusual.

Our proposed method is closely related with shrinkage-based methods such as mean shift, AMKS, MBMS. The quantitative results of shrinking are presented in Fig. 14. It is observed that the proposed clustering method produces much distinguishable cluster compared to others.

The computational overhead of the proposed method is also lower as compared to other methods since our method does not compare pairwise points. Table 3 shows the average execution time of various clustering methods (20 runs) in Intel core i7, 3.6 GHz processor with 16 GB of RAM.

### Ranking of trajectories

Entry/exit and entry-to-exit paths are good features to summarize a scene. However, clustering using these features may result in loss of information, e.g. speed of the objects, path deviation. To overcome such problems, path deviation has been included and an aggregate of all features using MCDM has been performed. In this section, we present patterns (i.e. clusters) obtained using various features. Table 4 summarizes the results. It has been found that inclusion of path deviation actually splits the clusters and produces larger movement patterns. Figure 15 presents examples of some trajectories and corresponding ranks using different parameters.

### Comparison between SAW and TOPSIS

Spearman’s rank correlation coefficient has been used to measure the relationship between two ranking mechanisms. It is calculated using (16), where \(d_i\) is the difference between two ranks and *n* represents the number of observations.

Table 5 presents the Spearman’s rank correlation coefficients between the ranks obtained using TOPSIS and SAW methods over all datasets. Results reveal that ranks obtained by SAW and TOPSIS are similar in nature.

We also perform the nonparametric Wilcoxon signed rank test to determine the *p* value for four different datasets using SAW and TOPSIS which are listed in Table 5. For each of these datasets, we set the null hypothesis \((H_0)\) since there is no significant difference between the ranking mechanism SAW and TOPSIS. However, for all these datasets, \(H_0\) is set to 0 at \(5\%\) level of significance. This reveals that there is not much evidence to reject \(H_0\). Therefore, we conclude that the notable difference does not exist between SAW and TOPSIS ranking mechanisms at \(5\%\) significance level.

## Conclusion

In this paper, we present an unsupervised method for unsupervised trajectory clustering and indexing. Our technique is based on a clustering and ranking method using entry/exit regions and entry-to-exit paths. Trajectory abnormality scores obtained with respect to entry/exit patterns, entry-to-exit paths, path deviation, and a local ranks (\(L_\kappa \)) are generated for each moving object. MCDM fusion has been applied to aggregate individual abnormality scores and a global abnormality score is obtained. In the next step, a global rank (\(G_\kappa \)) is assigned to each object. All moving objects are then represented using \(L_\kappa \) and \(G_\kappa \) pair, where lower value represents higher abnormality. The proposed algorithm can be thought as a generalized framework for unsupervised trajectory clustering and ranking and the method can be applied for intelligent browsing of large volume of surveillance videos.

## References

Abdullah L, Adawiyah CR (2014) Simple additive weighting methods of multi criteria decision making and applications: a decade review. Int J Inf Process Manag 5(1):39

Ahmed SA, Dogra DP, Kar S, Roy PP (2018a) Surveillance scene representation and trajectory abnormality detection using aggregation of multiple concepts. Expert Syst Appl 101:43–55

Ahmed SA, Dogra DP, Kar S, Roy PP (2018b) Trajectory-based surveillance analysis: a survey. IEEE Trans Circuits Syst Video Technol 29:1985–1997

Ajmal M, Naseer M, Ahmad F, Saleem A (2017) Human motion trajectory analysis based video summarization. In: 2017 16th IEEE international conference on machine learning and applications (ICMLA). IEEE, pp 550–555

Ali S, Shah M (2007) A Lagrangian particle dynamics approach for crowd flow segmentation and stability analysis. In: Proceedings of the IEEE Computer Society conference on computer vision and pattern recognition, pp 1–6

Anjum N, Cavallaro A (2008) Multifeature object trajectory clustering for video analysis. IEEE Trans Circuits Syst Video Technol 18(11):1555–1564

Bandaragoda T, De Silva D, Kleyko D, Osipov E, Wiklund U, Alahakoon D (2019) Trajectory clustering of road traffic in urban environments using incremental machine learning in combination with hyperdimensional computing. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC). IEEE, pp 1664–1670

Bera A, Kim S, Manocha D (2016) Realtime anomaly detection using trajectory-level crowd behavior learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 50–57

Campo D, Baydoun M, Marcenaro L, Cavallaro A, Regazzoni CS (2018) Unsupervised trajectory modeling based on discrete descriptors for classifying moving objects in video sequences. In: 2018 25th IEEE international conference on image processing (ICIP). IEEE, pp 833–837

Chakraborty P, Sharma A, Hegde C (2018) Freeway traffic incident detection from cameras: a semi-supervised learning approach. In: 2018 21st international conference on intelligent transportation systems (ITSC). IEEE, pp 1840–1845

Chen Z, Wu C, Huang Z, Lyu N, Hu Z, Zhong M, Cheng Y, Ran B (2017) Dangerous driving behavior detection using video-extracted vehicle trajectory histograms. J Intell Transp Syst 21(5):409–421

Choong MY, Chin RKY, Yeo KB, Teo KTK (2016) Trajectory pattern mining via clustering based on similarity function for transportation surveillance. Int J Simul Syst Sci Technol 17(34):19-1

Comaniciu D, Meer P (2002) Mean shift: a robust approach toward feature space analysis. IEEE Trans Pattern Anal Mach Intell 24(5):603–619

Dai M, Srivastava A (2019) Video-based action recognition using dimension reduction of deep covariance trajectories. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops

Das D, Mishra D (2018) Unsupervised anomalous trajectory detection for crowded scenes. In: 2018 IEEE 13th international conference on industrial and information systems (ICIIS). IEEE, pp 27–31

Dogra D, Ahmed A, Bhaskar H (2016) Smart video summarization using mealy machine-based trajectory modelling for surveillance applications. Multimed Tools Appl 75(11):6373–6401

Feizi A (2019) Hierarchical detection of abnormal behaviors in video surveillance through modeling normal behaviors based on AUC maximization. Soft Comput. https://doi.org/10.1007/s00500-019-04544-9

Feng Y, Yuan Y, Lu X (2017) Learning deep event models for crowd anomaly detection. Neurocomputing 219:548–556

Hu W, Li X, Tian G, Maybank S, Zhang Z (2013) An incremental dpmm-based method for trajectory clustering, modeling, and retrieval. IEEE Trans Pattern Anal Mach Intell 35(5):1051–1065

Huang H, Fu S, Cai ZQ, Li B (2018) Video abstract system based on spatial-temporal neighborhood trajectory analysis algorithm. Multimed Tools Appl 77(9):11321–11338

Hwang CL, Lai YJ, Liu TY (1993) A new approach for multiple objective decision making. Comput Oper Res 20(8):889–899

Izakian Z, Mesgari MS, Abraham A (2016) Automated clustering of trajectory data using a particle swarm optimization. Comput Environ Urban Syst 55:55–65

Kwon Y, Kang K, Jin J, Moon J, Park J (2017) Hierarchically linked infinite hidden markov model based trajectory analysis and semantic region retrieval in a trajectory dataset. Expert Syst Appl 78:386–395

Li H, Liu J, Wu K, Yang Z, Liu RW, Xiong N (2018) Spatio-temporal vessel trajectory clustering based on data mapping and density. IEEE Access 6:58939–58954

Lin W, Mi Y, Wang W, Wu J, Wang J, Mei T (2016a) A diffusion and clustering-based approach for finding coherent motions and understanding crowd scenes. IEEE Trans Image Process 25(4):1674–1687

Lin W, Zhou Y, Xu H, Yan J, Xu M, Wu J, Liu Z (2016b) A tube-and-droplet-based approach for representing and analyzing motion trajectories. IEEE Trans Pattern Anal Mach Intell 39(8):1489–1503

Long C, Hua G, Kapoor A (2016) A joint gaussian process model for active visual recognition with expertise estimation in crowdsourcing. Int J Comput Vis 116(2):136–160

Mabrouk AB, Zagrouba E (2018) Abnormal behavior recognition for intelligent video surveillance systems: a review. Expert Syst Appl 91:480–491

Ma C, Miao Z, Li M, Song S, Yang MH (2018) Detecting anomalous trajectories via recurrent neural networks. In: Asian Conference on Computer Vision. Springer, pp 370–382

Mehrasa N, Zhong Y, Tung F, Bornn L, Mori G (2018) Deep learning of player trajectory representations for team activity analysis. In: 11th MIT Sloan Sports Analytics Conference

Michelioudakis E, Artikis A, Paliouras G (2019) Semi-supervised online structure learning for composite event recognition. Mach Learn 108(7):1085–1110

Ochs P, Malik J, Brox T (2014) Segmentation of moving objects by long term video analysis. IEEE Trans Pattern Anal Mach Intell 36(6):1187–1200

Petitjean F, Forestier G, Webb G, Nicholson AE, Chen Y, Keogh E et al (2014) Dynamic time warping averaging of time series allows faster and more accurate classification. In: IEEE international conference on data mining (ICDM), pp 470–479

Reddy GT, Veena M (2018) Video surveillance of abnormal trajectory analysis and event detection. Int J Appl Eng Res 13(16):12928–12931

Ren X, Wang D, Laskey M, Goldberg K (2018) Learning traffic behaviors by extracting vehicle trajectories from online video streams. In: 2018 IEEE 14th international conference on automation science and engineering (CASE). IEEE, pp 1276–1283

Roshtkhari MJ, Levine MD (2013) An on-line, real-time learning method for detecting anomalies in videos using spatio-temporal compositions. Comput Vis Image Underst 117(10):1436–1452

Saini R, Kumar P, Roy PP, Pal U (2019) Modeling local and global behavior for trajectory classification using graph based algorithm. Pattern Recognit Lett. https://doi.org/10.1016/j.patrec.2019.05.014

Santhosh KK, Dogra DP, Roy PP (2018) Temporal unknown incremental clustering model for analysis of traffic surveillance videos. IEEE Trans Intell Transp Syst 20(5):1762–1773

Shannon CE (2001) A mathematical theory of communication. ACM SIGMOBILE Mobile Comput Commun Rev 5(1):3–55

Sharma R, Guha T (2016) A trajectory clustering approach to crowd flow segmentation in videos. In: 2016 IEEE international conference on image processing (ICIP). IEEE, pp 1200–1204

Spampinato C, Palazzo S, D’Oro P, Giordano D, Shah M (2020) Adversarial framework for unsupervised learning of motion dynamics in videos. Int J Comput Vis 128(1):1378–1397

Vishwakarma S, Agrawal A (2013) A survey on activity recognition and behavior understanding in video surveillance. Visual Comput 29(10):983–1009

Wang W, Carreira-Perpinán MA (2010) Manifold blurring mean shift algorithms for manifold denoising. In: IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 1759–1766

Wang X, Ma K, Ng G, Grimson W (2011) Trajectory analysis and semantic region modeling using nonparametric hierarchical bayesian models. Int J Comput Vis 95(3):287–312

Wang J, Xia L, Hu X, Xiao Y (2019) Abnormal event detection with semi-supervised sparse topic model. Neural Comput Appl 31(5):1607–1617

Xu H, Zhou Y, Lin W, Zha H (2015) Unsupervised trajectory clustering via adaptive multi-kernel-based shrinkage. In: Proceedings of the IEEE international conference on computer vision, pp 4328–4336

Xu D, Yan Y, Ricci E, Sebe N (2017) Detecting anomalous events in videos by learning deep representations of appearance and motion. Comput Vis Image Underst 156:117–127

Xu Y, Ouyang X, Cheng Y, Yu S, Xiong L, Ng CC, Pranata S, Shen S, Xing J (2018a) Dual-mode vehicle motion pattern learning for high performance road traffic anomaly detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 145–152

Xu Y, Piao Z, Gao S (2018b) Encoding crowd interaction with deep neural network for pedestrian trajectory prediction. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5275–5284

Yuan G, Sun P, Zhao J, Li D, Wang C (2017) A review of moving object trajectory clustering algorithms. Artif Intell Rev 47(1):123–144

Yue M, Li Y, Yang H, Ahuja R, Chiang YY, Shahabi C (2019) Detect: Deep trajectory clustering for mobility-behavior analysis. In: 2019 IEEE international conference on Big Data (Big Data). IEEE, pp 988–997

Zhao Y, Xiong Y, Lin D (2018) Trajectory convolution for action recognition. In: Advances in neural information processing systems, pp 2204–2215

Zhao J, Yi Z, Pan S, Zhao Y, Zhuang B (2019) Unsupervised traffic anomaly detection using trajectories. In: Proceedings of CVPR workshops

## Acknowledgements

Open Access funding provided by UiT The Arctic University of Norway. This study is not funded from anywhere.

## Author information

### Authors and Affiliations

### Corresponding author

## Ethics declarations

### Conflict of interest

The authors declare that there is no conflict of interest regarding the publication of this paper (declaration attached).

### Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

### Informed consent

Informed consent was obtained from all individual participants included in the study.

## Additional information

Communicated by V. Loia.

### Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

## About this article

### Cite this article

Sekh, A.A., Dogra, D.P., Kar, S. *et al.* Video trajectory analysis using unsupervised clustering and multi-criteria ranking.
*Soft Comput* **24**, 16643–16654 (2020). https://doi.org/10.1007/s00500-020-04967-9

Published:

Issue Date:

DOI: https://doi.org/10.1007/s00500-020-04967-9

### Keywords

- Unsupervised clustering
- Object trajectory
- Motion analysis