Applied Intelligence

, Volume 44, Issue 2, pp 340–361 | Cite as

An anomaly detection approach for multiple monitoring data series based on latent correlation probabilistic model

  • Jianwei DingEmail author
  • Yingbo Liu
  • Li Zhang
  • Jianmin Wang
  • Yonghong Liu


Condition monitoring systems are widely used to monitor the working condition of equipment, generating a vast amount and variety of monitoring data in the process. The main task of surveillance focuses on detecting anomalies in these routinely collected monitoring data, intended to help detect possible faults in the equipment. However, with the rapid increase in the volume of monitoring data, it is a nontrivial task to scan all the monitoring data to detect anomalies. In this paper, we propose an approach called latent correlation-based anomaly detection (LCAD) that efficiently and effectively detects potential anomalies from a large number of correlative isomerous monitoring data series. Instead of focusing on one or more isomorphic monitoring data series, LCAD identifies anomalies by modeling the latent correlation among multiple correlative isomerous monitoring data series, using a probabilistic distribution model called the latent correlation probabilistic model, which helps to detect anomalies according to their relations with the model. Experimental results on real-world data sets show that when dealing with a large number of correlative isomerous monitoring data series, LCAD yields better performances than existing anomaly detection approaches.


Anomaly detection Abnormal pattern Multiple monitoring data series Latent correlation 


  1. 1.
    Aggarwal CC, Philip SY (2008) Outlier detection with uncertain data. In: SDM. SIAM, pp 483–493Google Scholar
  2. 2.
    Atzori L, Iera A, Morabito G (2010) The internet of things: A survey. Comput Netw 54(15):2787–2805CrossRefzbMATHGoogle Scholar
  3. 3.
    Bilmes JA, et al. (1998) A gentle tutorial of the em algorithm and its application to parameter estimation for gaussian mixture and hidden markov models. Int Comput Sci Inst 4(510):126Google Scholar
  4. 4.
    Chan PK, Mahoney MV (2005) Modeling multiple time series for anomaly detection. In: Data Mining, Fifth IEEE International Conference on, pages 8–pp. IEEEGoogle Scholar
  5. 5.
    Das K, Schneider J (2007) Detecting anomalous records in categorical datasets. In: Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pp 220–229. ACMGoogle Scholar
  6. 6.
    Dutton WH (2013) The internet of things. Available at SSRNGoogle Scholar
  7. 7.
    Fawcett T (2006) An introduction to roc analysis. Pattern Recogn Lett 27(8):861–874CrossRefMathSciNetGoogle Scholar
  8. 8.
    Fujimaki R, Nakata T, Tsukahara H, Sato A, Yamanishi K (2009) Mining abnormal patterns from heterogeneous time-series with irrelevant features for fault event detection. Stat Anal Data Min 2(1):1–17CrossRefMathSciNetzbMATHGoogle Scholar
  9. 9.
    Ghoting A, Parthasarathy S, Otey ME (2008) Fast mining of distance-based outliers in high-dimensional datasets. Data Min Knowl Disc 16(3):349–364CrossRefMathSciNetGoogle Scholar
  10. 10.
    Heard NA, Weston DJ, Platanioti K, Hand DJ, et al. (2010) Bayesian anomaly detection methods for social networks. Ann Appl Stat 4(2):645–662CrossRefMathSciNetzbMATHGoogle Scholar
  11. 11.
    Hecht-Nielsen R (1989) Theory of the backpropagation neural network. In: Neural Networks, 1989. IJCNN., International Joint Conference on, pp 593–605. IEEEGoogle Scholar
  12. 12.
    Hu X, Hu S, Zhang X, Zhang H, Luo L (2014) Anomaly detection based on local nearest neighbor distance descriptor in crowded scenes. Sci World J:2014Google Scholar
  13. 13.
    Izakian H, Pedrycz W (2013) Anomaly detection in time series data using a fuzzy c-means clustering. In: IFSA World Congress and NAFIPS Annual Meeting (IFSA/NAFIPS), 2013 Joint, pp 1513–1518. IEEEGoogle Scholar
  14. 14.
    Janakiram D, Adi Mallikarjuna Reddy V, Phani Kumar AVU (2006) Outlier detection in wireless sensor networks using bayesian belief networks. In: Communication System Software and Middleware, 2006. Comsware 2006. First International Conference on, pp 1–6. IEEEGoogle Scholar
  15. 15.
    Keogh E, Chakrabarti K, Pazzani M, Mehrotra S (2001) Locally adaptive dimensionality reduction for indexing large time series databases. ACM SIGMOD Rec 30(2):151–162CrossRefGoogle Scholar
  16. 16.
    Keogh E, Kasetty S (2003) On the need for time series data mining benchmarks: a survey and empirical demonstration. Data Min Knowl Disc 7(4):349–371CrossRefMathSciNetGoogle Scholar
  17. 17.
    Kou Y, Lu C-T, Chen D (2006) Spatial weighted outlier detection. In: SDM, pp 614–618. SIAMGoogle Scholar
  18. 18.
    Liu D, Lung C-H, Lambadaris I, Seddigh N (2013) Network traffic anomaly detection using clustering techniques and performance comparison. In: Electrical and Computer Engineering (CCECE), 2013 26th Annual IEEE Canadian Conference on, pp 1–4. IEEEGoogle Scholar
  19. 19.
    Ma J, Perkins S (2003) Online novelty detection on temporal sequences. In: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, pp 613–618. ACMGoogle Scholar
  20. 20.
    Mabu S, Chen C, Lu N, Shimada K, Hirasawa K (2011) An intrusion-detection model based on fuzzy class-association-rule mining using genetic network programming. IEEE Trans Syst Man Cybern Part C Appl Rev 41(1):130–139CrossRefGoogle Scholar
  21. 21.
    Mahoney MV, Chan PK (2002) Learning nonstationary models of normal network traffic for detecting novel attacks. In: Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pp 376–385. ACMGoogle Scholar
  22. 22.
    Marx ML, Larsen RJ (2006) Introduction to mathematical statistics and its applications. Pearson/Prentice HallGoogle Scholar
  23. 23.
    Mascaro S, Nicholso AE, Korb KB (2014) Anomaly detection in vessel tracks using bayesian networks. Int J Approx Reason 55(1):84–98CrossRefGoogle Scholar
  24. 24.
    Moshtaghi M, Havens TC, Bezdek JC, Park L, Leckie C, Rajasegarar S, Keller JM, Palaniswami M (2011) Clustering ellipses for anomaly detection. Pattern Recogn 44(1):55–69CrossRefzbMATHGoogle Scholar
  25. 25.
    Otey ME, Ghoting A, Parthasarathy S (2006) Fast distributed outlier detection in mixed-attribute data sets. Data Min Knowl Disc 12(2-3):203–228CrossRefMathSciNetGoogle Scholar
  26. 26.
    Papadimitriou S, Sun J, Faloutsos C (2005) Streaming pattern discovery in multiple time-series. In: Proceedings of the 31st international conference on Very large data bases, pp 697–708. VLDB EndowmentGoogle Scholar
  27. 27.
    Peterson LE (2009) K-nearest neighbor. Scholarpedia 4(2):1883CrossRefGoogle Scholar
  28. 28.
    Pokrajac D, Lazarevic A, Latecki LJ (2007) Incremental local outlier detection for data streams. In: Computational Intelligence and Data Mining, 2007. CIDM 2007. IEEE Symposium on, pp 504–515. IEEEGoogle Scholar
  29. 29.
    Povinelli RJ, Johnson MT, Lindgren AC, Ye J (2004) Time series classification using gaussian mixture models of reconstructed phase spaces. IEEE Trans Knowl Data Eng 16(6):779–783CrossRefGoogle Scholar
  30. 30.
    Qin M, Hwang K (2004) Frequent episode rules for internet anomaly detection. In: Network Computing and Applications, 2004.(NCA 2004). Proceedings. Third IEEE International Symposium on, pp 161–168. IEEEGoogle Scholar
  31. 31.
    Schlechtingen M, Santos IF (2011) Comparative analysis of neural network and regression based condition monitoring approaches for wind turbine fault detection. Mech Syst Signal Process 25(5):1849–1875CrossRefGoogle Scholar
  32. 32.
    Sheikhan M, Jadidi Z (2014) Flow-based anomaly detection in high-speed links using modified gsa-optimized neural network. Neural Comput Appl 24(3-4):599–611CrossRefGoogle Scholar
  33. 33.
    Sotiris VA, Tse PW, Pecht MG (2010) Anomaly detection through a bayesian support vector machine. IEEE Trans Reliab 59(2):277–286CrossRefGoogle Scholar
  34. 34.
    Su M-Y (2011) Real-time anomaly detection systems for denial-of-service attacks by weighted k-nearest-neighbor classifiers. Expert Syst Appl 38(4):3492–3498CrossRefGoogle Scholar
  35. 35.
    Tandon G, Chan PK (2007) Weighting versus pruning in rule validation for detecting network and host anomalies. In: Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pp 697–706. ACMGoogle Scholar
  36. 36.
    Thatte G, Mitra U, Heidemann J (2011) Parametric methods for anomaly detection in aggregate traffic. IEEE/ACM Trans Networking (TON) 19(2):512–525CrossRefGoogle Scholar
  37. 37.
    Tsai C-F, Lin C-Y (2010) A triangle area based nearest neighbors approach to intrusion detection. Pattern Recogn 43(1):222–229CrossRefMathSciNetzbMATHGoogle Scholar
  38. 38.
    Turner R, Fernandes E (2013) Optimizing back propagation parameters for anomaly detection. Eur J Eng Innov 10(1):10–19Google Scholar
  39. 39.
    Wang X, Mueen A, Ding H, Trajcevski G, Scheuermann P, Keogh E (2013) Experimental comparison of representation methods and distance measures for time series data. Data Min Knowl Disc 26(2):275–309CrossRefMathSciNetGoogle Scholar
  40. 40.
    Wu M, Jermaine C (2006) Outlier detection by sampling with accuracy guarantees. In: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp 767–772. ACMGoogle Scholar
  41. 41.
    Yamanishi K, Takeuchi J-I, Williams G, Milne P (2004) On-line unsupervised outlier detection using finite mixtures with discounting learning algorithms. Data Min Knowl Disc 8(3):275–300CrossRefMathSciNetGoogle Scholar
  42. 42.
    Yi B-K, Faloutsos C (2000) Fast time sequence indexing for arbitrary lp norms. In: proceedings of the 26st Intl Conference on Very Large Databases. VLDBGoogle Scholar
  43. 43.
    Yu JX, Qian W, Lu H, Zhou A (2006) Finding centric local outliers in categorical/numerical spaces. Knowl Inf Syst 9(3):309–338CrossRefGoogle Scholar
  44. 44.
    Yu R, He X, Liu Y (2014) Glad: group anomaly detection in social media analysis. In: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp 372–381. ACMGoogle Scholar
  45. 45.
    Zhang C, Weng N, Chang J, Zhou A (2009) Detecting abnormal trend evolution over multiple data streams. In: Advances in Data and Web Management, pp 285–296. SpringerGoogle Scholar
  46. 46.
    Zhang J, Wang H (2006) Detecting outlying subspaces for high-dimensional data: the new task, algorithms, and performance. Knowl Inf Syst 10(3):333–355CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Jianwei Ding
    • 1
    • 2
    • 4
    Email author
  • Yingbo Liu
    • 2
    • 4
  • Li Zhang
    • 2
    • 4
  • Jianmin Wang
    • 2
    • 4
  • Yonghong Liu
    • 3
  1. 1.Department of Computer Science and TechnologyTsinghua UniversityBeijingChina
  2. 2.Institute of Information System & Engineering, School of SoftwareTsinghua UniversityBeijingChina
  3. 3.General Research Institute of SANY GroupChangshaChina
  4. 4.East Main Building, School of SoftwareTsinghua UniversityBeijingChina

Personalised recommendations