Anomaly detection in spatiotemporal data via regularized non-negative tensor analysis

Abstract

Anomaly detection in multidimensional data is a challenging task. Detecting anomalous mobility patterns in a city needs to take spatial, temporal, and traffic information into consideration. Although existing techniques are able to extract spatiotemporal features for anomaly analysis, few systematic analysis about how different factors contribute to or affect the anomalous patterns has been proposed. In this paper, we propose a novel technique to localize spatiotemporal anomalous events based on tensor decomposition. The proposed method employs a spatial-feature-temporal tensor model and analyzes latent mobility patterns through unsupervised learning. We first train the model based on historical data and then use the model to capture the anomalies, i.e., the mobility patterns that are significantly different from the normal patterns. The proposed technique is evaluated based on the yellow-cab dataset collected from New York City. The results show several interesting latent mobility patterns and traffic anomalies that can be deemed as anomalous events in the city, suggesting the effectiveness of the proposed anomaly detection method.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Notes

  1. 1.

    https://data.cityofnewyork.us/view/gn7m-em8n.

References

  1. Bahadori MT, Yu QR, Liu Y (2014) Fast multivariate spatio-temporal analysis via low rank tensor learning. In: Advances in neural information processing systems, pp 3491–3499

  2. Bai Y, Tezcan J, Cheng Q, Cheng J (2013) A multiway model for predicting earthquake ground motion. In: ACIS international conference on software engineering, artificial intelligence, networking and parallel/distributed computing (SNPD), pp 219–224

  3. Breunig MM, Kriegel HP, Ng RT, Sander J (2000) Lof: identifying density-based local outliers. ACM Sigmod Rec 29:93–104

    Article  Google Scholar 

  4. Chandola V, Banerjee A, Kumar V (2009) Anomaly detection: a survey. ACM Comput Surv (CSUR) 41(3):15

    Article  Google Scholar 

  5. Chen Y, Zhou XS, Huang TS (2001) One-class svm for learning in image retrieval. IEEE Image Process 1:34–37

    Google Scholar 

  6. Cichocki A, Zdunek R, Phan AH, Amari SI (2009) Nonnegative matrix and tensor factorizations: applications to exploratory multi-way data analysis and blind source separation. Wiley, New York

    Google Scholar 

  7. Fanaee-T H, Gama J (2015) Eigenevent: an algorithm for event detection from complex data streams in syndromic surveillance. Intell Data Anal 19(3):597–616

    Article  Google Scholar 

  8. Fanaee-T H, Gama J (2016a) Tensor-based anomaly detection: an interdisciplinary survey. Knowl Based Syst 98:130–147

    Article  Google Scholar 

  9. Fanaee-T H, Gama J (2016b) Event detection from traffic tensors: a hybrid model. Neurocomputing 203:22–33

    Article  Google Scholar 

  10. Fanaee-T H, Gama J (2014) An eigenvector-based hotspot detection. arXiv preprint arXiv:1406.3191

  11. Gauvin L, Panisson A, Cattuto C (2014) Detecting the community structure and activity patterns of temporal networks: a non-negative tensor factorization approach. PloS ONE 9(1):e86028

    Article  Google Scholar 

  12. Jiang M, Cui P, Faloutsos C (2016) Suspicious behavior detection: current trends and future directions. IEEE Intell Syst 31(1):31–39

    Article  Google Scholar 

  13. Kim J, He Y, Park H (2014) Algorithms for nonnegative matrix and tensor factorizations: a unified view based on block coordinate descent framework. J Global Optim 58(2):285–319

    MathSciNet  Article  MATH  Google Scholar 

  14. Kolda TG, Bader BW (2009) Tensor decompositions and applications. SIAM Rev 51(3):455–500

    MathSciNet  Article  MATH  Google Scholar 

  15. Kotsia I, Guo W, Patras I (2012) Higher rank support tensor machines for visual recognition. Pattern Recogn 45(12):4192–4203

    Article  MATH  Google Scholar 

  16. Liu S, Cui W, Wu Y, Liu M (2014) A survey on information visualization: recent advances and challenges. Visual Comput 30(12):1373–1393

    Article  Google Scholar 

  17. Liu D, Weng D, Li Y, Bao J, Zheng Y, Qu H, Wu Y (2017) SmartAdP: Visual analytics of large-scale taxi trajectories for selecting billboard locations. IEEE Trans. Vis. Comput. Graphics 23(1):1–10

    Article  Google Scholar 

  18. Liu Y, Zhou B, Chen F, Cheung DW (2016) Graph topic scan statistic for spatial event detection. In: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. ACM, pp 489–498

  19. Mao HH, Wu CJ, Papalexakis EE, Faloutsos C, Lee KC, Kao TC (2014) Malspot: Multi2 malicious network behavior patterns analysis. In: Pacific-Asia conference on knowledge discovery and data mining. Springer, pp 1–14

  20. Matsubara Y, Sakurai Y, Faloutsos C, Iwata T, Yoshikawa M (2012) Fast mining and forecasting of complex time-stamped events. In: Proceedings of the ACM SIGKDD international conference on Knowledge discovery and data mining, pp 271–279

  21. Nomikos P, MacGregor JF (1994) Monitoring batch processes using multiway principal component analysis. AIChE J 40(8):1361–1375

    Article  Google Scholar 

  22. Papalexakis EE, Faloutsos C, Sidiropoulos ND (2012) Parcube: sparse parallelizable tensor decompositions. In: Joint European conference on machine learning and knowledge discovery in databases. Springer, pp 521–536

  23. Papalexakis E, Pelechrinis K, Faloutsos C (2014) Spotting misbehaviors in location-based social networks using tensors. In: Proceedings of the international conference on world wide web. ACM, pp 551–552

  24. Paschalidis IC, Smaragdakis G (2009) Spatio-temporal network anomaly detection by assessing deviations of empirical measures. IEEE/ACM Trans Netw (TON) 17(3):685–697

    Article  Google Scholar 

  25. Prada MA, Dominguez M, Barrientos P, Garcia S (2012a) Dimensionality reduction for damage detection in engineering structures. Int J Mod Phys B 26(25):1246004

    Article  Google Scholar 

  26. Prada MA, Toivola J, Kullaa J, HollméN J (2012b) Three-way analysis of structural health monitoring data. Neurocomputing 80:119–128

    Article  Google Scholar 

  27. Rendle S (2012) Factorization machines with libfm. ACM Trans Intell Syst Technol (TIST) 3(3):57

    Google Scholar 

  28. Rozenshtein P, Anagnostopoulos A, Gionis A, Tatti N (2014) Event detection in activity networks. In: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pp 1176–1185

  29. Shi L, Gangopadhyay A, Janeja VP (2015) Stensr: spatio-temporal tensor streams for anomaly detection and pattern discovery. Knowl Inf Syst 43(2):333

    Article  Google Scholar 

  30. Sun GD, Liang R, Qu H, Wu Y (2017a) Embedding spatiotemporal information into maps by route-zooming. IEEE Trans. Vis. Comput. Graphics 23(5):1506–1519

    Article  Google Scholar 

  31. Sun G, Tang T, Peng TQ, Liang R, Wu Y (2017b) Socialwave: visual analysis of spatio-temporal diffusion of information on social media. ACM Trans Intell Syst Technol 9(2):15

    Article  Google Scholar 

  32. Sun J, Tao D, Papadimitriou S, Yu PS, Faloutsos C (2008) Incremental tensor analysis: Theory and applications. ACM Trans Knowl Discov Data (TKDD) 2(3):11

    Google Scholar 

  33. Sun J, Tao D, Faloutsos C (2006) Beyond streams and graphs: dynamic tensor analysis. In: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pp 374–383

  34. Sun G, Wu YC, Liang RH, Liu SX (2013) A survey of visual analytics techniques and applications: state-of-the-art research and future challenges. J Comput Sci Tech 28(5):852–867

    Article  Google Scholar 

  35. Tao D, Li X, Hu W, Maybank S, Wu X (2005) Supervised tensor learning. In: IEEE international conference on data mining

  36. Thai-Nghe N, Horváth T, Schmidt-Thieme L (2010) Factorization models for forecasting student performance. In: Educational Data Mining 2011

  37. Tian X, Zhang X, Deng X, Chen S (2009) Multiway kernel independent component analysis based on feature samples for batch process monitoring. Neurocomputing 72(7):1584–1596

    Article  Google Scholar 

  38. Tork HF, Oliveira M, Gama J, Malinowski S, Morla R (2012) Event and anomaly detection using tucker3 decomposition. In: Workshop on ubiquitous data mining, p 8

  39. Wang XR, Lizier JT, Obst O, Prokopenko M, Wang P (2008) Spatiotemporal anomaly detection in gas monitoring sensor networks. In: Wireless sensor networks: 5th European conference, EWSN 2008. Springer, pp 90–105

  40. Wang J, Gao F, Cui P, Li C, Xiong Z (2014) Discovering urban spatio-temporal structure from time-evolving traffic networks. In: Asia-Pacific web conference. Springer, pp 93–104

  41. Weng D, Zhu H, Bao J, Zheng Y, Wu Y (2018) Homefinder revisited: finding ideal homes with reachability centric multi-criteria decision making. In Proceedings of ACM CHI

  42. Wu Y, Lan J, Shu X, Ji C, Zhao K, Wang J, Zhang H (2018) ITTVIS: Interactive visualization of table tennis data. IEEE Trans Visualization and Comp Graphics 24(1):709–718

    Article  Google Scholar 

  43. Xia J, Chen W, Hou Y, Hu W, Huang X, Ebertk DS (2016) DimScanner: A relation-based visual exploration approach towards data dimension inspection. In: IEEE conference on visual analytics science and technology (VAST). pp 81–90

  44. Xie C, Chen W, Huang X, Hu Y, Barlowe S, Yang J (2014) VAET: A visual analytics approach for e-transactions time-series. IEEE Trans. Vis. Comput. Graphics 20(12):1743–1752

    Article  Google Scholar 

  45. Xu P, Mei H, Ren L, Chen W (2017) ViDX: Visual diagnostics of assembly line performance in smart factories. IEEE Trans. Vis. Comput. Graphics 23(1):291–300

    Article  Google Scholar 

  46. Young WC, Blumenstock JE, Fox EB, McCormick TH (2014) Detecting and classifying anomalous behavior in spatiotemporal network data. In: Proceedings of KDD workshop on learning about emergencies from social information (KDD-LESI 2014), pp 29–33

  47. Yuan J, Zheng Y, Xie X (2012) Discovering regions of different functions in a city using human mobility and pois. In: Proceedings of the ACM SIGKDD international conference on Knowledge discovery and data mining, pp 186–194

  48. Zhang T, Wang X, Li Z, Guo F, Ma Y, Chen W (2017) A survey of network anomaly visualization. Sc China Infor Sci 60(12):121101

    Article  Google Scholar 

  49. Zhao Z, Cheng Z, Hong L, Chi EH (2015) Improving user topic interest profiles by behavior factorization. In: Proceedings of the international conference on world wide web. ACM, pp 1406–1416

  50. Zheng Y, Liu T, Wang Y, Zhu Y, Liu Y, Chang E (2014) Diagnosing New York city’s noises with ubiquitous data. In: Proceedings of the ACM international joint conference on pervasive and ubiquitous computing, pp 715–725

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Nan Cao.

Additional information

Responsible editor: Jieping Ye.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lin, C., Zhu, Q., Guo, S. et al. Anomaly detection in spatiotemporal data via regularized non-negative tensor analysis. Data Min Knowl Disc 32, 1056–1073 (2018). https://doi.org/10.1007/s10618-018-0560-3

Download citation

Keywords

  • Tensor analysis
  • Anomaly detection
  • Outlier detection
  • Urban computing
  • Traffic analysis