Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

A comparative evaluation of novelty detection algorithms for discrete sequences

Abstract

The identification of anomalies in temporal data is a core component of numerous research areas such as intrusion detection, fault prevention, genomics and fraud detection. This article provides an experimental comparison of candidate methods for the novelty detection problem applied to discrete sequences. The objective of this study is to identify which state-of-the-art methods are efficient and appropriate candidates for a given use case. These recommendations rely on extensive novelty detection experiments based on a variety of public datasets in addition to novel industrial datasets. We also perform thorough scalability and memory usage tests resulting in new supplementary insights of the methods’ performance, key selection criteria to solve problems relying on large volumes of data and to meet the expectations of applications subject to strict response time constraints.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

References

  1. Aggarwal CC (2015) Outlier analysis. Springer, Cham, pp 237–263

  2. Bergroth L, Hakonen H, Raita T (2000) A survey of longest common subsequence algorithms. In: Proceedings seventh international symposium on string processing and information retrieval. SPIRE 2000, pp 39–48

  3. Breiman L, Friedman J, Olsen R, Stone C (1984) Classification and regression trees. Wadsworth and Brooks

  4. Breunig MM, Kriegel H-P, Ng RT, Sander J (2000) LOF: identifying density-based local outliers. SIGMOD Rec 29(2):93–104

  5. Budalakoti S, Srivastava AN, Akella R,Turkov E (2006) Anomaly detection in large sets of high-dimensional symbol sequences. Technical Report NASA TM-2006-214553

  6. Budalakoti S, Srivastava AN, Otey ME (2009) Anomaly detection and diagnosis algorithms for discrete symbol sequences with applications to airline safety. IEEE Trans Syst Cybern C (Appl Rev) 39(1):101–113

  7. Chandola V, Banerjee A, Kumar V (2012) Anomaly detection for discrete sequences: a survey. IEEE Trans Knowl Data Eng 24(5):823–839

  8. Chandola V, Mithal V, Kumar V (2008) Comparative evaluation of anomaly detection techniques for sequence data. In: 2008 Eighth IEEE international conference on data mining, pp 743–748

  9. Chang D, Jones NA, Li D, Ouyang M, Ragade RK (2008) Compute pairwise Euclidean distances of data points with gpus. In: Proceedings of the iASTED international symposium on computational biology and bioinformatics, pp 278–283

  10. Christ M, Kempa-Liehr AW, Feindt M (2016) Distributed and parallel time series feature extraction for industrial big data applications. arXiv:1610.07717

  11. Cohen WW (1995) Fast effective rule induction. In: Prieditis A, Russell S (eds) Machine learning proceedings 1995. Morgan Kaufmann, San Francisco, pp 115–123

  12. Crochemore M, Iliopoulous CS, Pinzon YJ (2003) Speeding-up hirschberg and hunt-szymanski lcs algorithms. Fundamenta Informaticae 56(1–2):89–103

  13. Davis J, Goadrich M (2006) The relationship between precision-recall and roc curves. In: Proceedings of the 23rd international conference on Machine learning. ACM, pp 233–240

  14. de Fortuny EJ, Martens D (2015) Active learning-based pedagogical rule extraction. IEEE Trans Neural Netw Learn Syst 26(11):2664–2677

  15. Domingues R, Michiardi P, Zouaoui J, Filippone M (2018) Deep gaussian process autoencoders for novelty detection. Mach Learn

  16. Emmott A, Das S, Dietterich T, Fern A, Wong W-K (2016) A meta-analysis of the anomaly detection problem. arXiv:1503.01158v2

  17. Etchells TA, Lisboa PJG (2006) Orthogonal search-based rule extraction (osre) for trained neural networks: a practical and efficient approach. IEEE Trans Neural Netw 17(2):374–384

  18. Fowkes J, Sutton C (2016) A subsequence interleaving model for sequential pattern mining. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, KDD ’16, New York, NY, USA, 2016. ACM, pp 835–844

  19. Gan W, Lin JC-W, Fournier-Viger P, Chao H-C, Yu PS (2018) A survey of parallel sequential pattern mining. arXiv:1805.10515

  20. García S, Fernández A, Luengo J, Herrera F (2010) Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: experimental analysis of power. Inf Sci 180(10):2044–2064 (Special Issue on Intelligent Distributed Information Systems)

  21. Gupta M, Gao J, Aggarwal CC, Han J (2014) Outlier detection for temporal data: a survey. IEEE Trans Knowl Data Eng 26(9):2250–2267

  22. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

  23. Hodge V, Austin J (2004) A survey of outlier detection methodologies. Artif Intell Rev 22(2):85–126

  24. Hofmeyr SA, Forrest S, Somayaji A (1998) Intrusion detection using sequences of system calls. J Comput Secur 6(3):151–180

  25. Hunt JW, Szymanski TG (1977) A fast algorithm for computing longest common subsequences. Commun ACM 20(5):350–353

  26. Huysmans J, Baesens B, Vanthienen J (2006) Iter: an algorithm for predictive regression rule extraction. In: Tjoa AM, Trujillo J (eds) Data warehousing and knowledge discovery. Springer, Berlin, pp 270–279

  27. Karpathy A, Johnson J, Li F (2016) Visualizing and understanding recurrent networks. In: Proceedings of the fourth international conference on learning representations (ICLR 2016)

  28. Kundzewicz ZW, Robson AJ (2004) Change detection in hydrological records—a review of the methodology. Hydrol Sci J 49(1):7–19

  29. Lazarevic A, Ertoz L, Kumar V, Ozgur A, Srivastava J (2003) A comparative study of anomaly detection schemes in network intrusion detection. In: Proceedings of the 2003 SIAM international conference on data mining, pp 25–36

  30. Lee W, Stolfo SJ, Chan PK (1997) Learning patterns from unix process execution traces for intrusion detection. In: AAAI workshop on AI approaches to fraud detection and risk management, pp 50–56

  31. Levenshtein VI (1966) Binary codes capable of correcting deletions, insertions and reversals. Sov. Phys. Dokl. 10:707

  32. Lipton ZC, Berkowitz J, Elkan C (2015) A critical review of recurrent neural networks for sequence learning

  33. Liu FT, Ting KM, Zhou Z-H (2008) Isolation forest. In: Proceedings of the 2008 eighth IEEE international conference on data mining, ICDM ’08. IEEE Computer Society, pp 413–422

  34. Luong M-T, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. arXiv:1508.04025

  35. Marchi E, Vesperini F, Eyben F, Squartini S, Schuller B (2015) A novel approach for automatic acoustic novelty detection using a denoising autoencoder with bidirectional lstm neural networks. In: 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 1996–2000

  36. Maxion RA, Townsend TN (2002) Masquerade detection using truncated command lines. In: Proceedings international conference on dependable systems and networks, pp 219–228

  37. Onderwater M (2015) Outlier preservation by dimensionality reduction techniques. Int J Data Anal Tech Strateg 7(3):231–252

  38. Park H-S, Jun C-H (2009) A simple and fast algorithm for k-medoids clustering. Expert Syst Appl 36(2, Part 2):3336–3341

  39. Petrović S, Osborne M, Lavrenko V (2010) Streaming first story detection with application to twitter. In: Human Language technologies: the 2010 annual conference of the North American Chapter of the Association for Computational Linguistics, HLT ’10. Association for Computational Linguistics, Stroudsburg, pp 181–189

  40. Pihur V, Datta S, Datta S (2009) RankAggreg, an r package for weighted rank aggregation. BMC Bioinform 10(1):62

  41. Rabiner LR (1989) A tutorial on hidden Markov models and selected applications in speech recognition. Proc IEEE 77(2):257–286

  42. Ramaswamy S, Rastogi R, Shim K (2000) Efficient algorithms for mining outliers from large data sets. In: Proceedings of the 2000 ACM SIGMOD international conference on management of data, SIGMOD ’00, New York, NY, USA. ACM, pp 427–438

  43. Saad EW, Wunsch DC (2007) Neural network explanation using inversion. Neural Netw 20(1):78–93

  44. Saidi R, Maddouri M, MephuNguifo E (2010) Protein sequences classification by means of feature extraction with substitution matrices. BMC Bioinform 11(1):175

  45. Sakurada M, Yairi T (2014) Anomaly detection using autoencoders with nonlinear dimensionality reduction. In: Proceedings of the MLSDA 2014 2nd workshop on machine learning for sensory data analysis. ACM, pp 4–11

  46. Schubert E, Rousseeuw PJ (2018) Faster k-Medoids clustering: improving the PAM, CLARA, and CLARANS algorithms. arXiv:1810.05691

  47. Sculley D, Brodley CE (2006) Compression and machine learning: a new perspective on feature space vectors. In: Data compression conference (DCC’06), pp 332–341

  48. Setiono R, Leow WK, Zurada JM (2002) Extraction of rules from artificial neural networks for nonlinear regression. IEEE Trans Neural Netw 13(3):564–577

  49. Sun P, Chawla S, Arunasalam B (2006) Mining for outliers in sequential databases. In: Proceedings of the 2006 SIAM international conference on data mining, pp 94–105

  50. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: Ghahramani Z, Welling M, Cortes C, Lawrence ND, Weinberger KQ (eds) Advances in neural information processing systems, vol 27. Curran Associates Inc., New York, pp 3104–3112

  51. Taylor SJ, Letham B (2018) Forecasting at scale. Am Stat 72(1):37–45

  52. Wang JTL, Ma Q, Shasha D, Wu CH (2001) New techniques for extracting features from protein sequences. IBM Syst J 40(2):426–441

  53. Warrender C, Forrest S, Pearlmutter B (1999) Detecting intrusions using system calls: alternative data models. In: Proceedings of the 1999 IEEE symposium on security and privacy (Cat. No. 99CB36344), pp 133–145

  54. Zhang Q-S, Zhu S-C (2018) Visual interpretability for deep learning: a survey. Front Inf Technol Electron Eng 19(1):27–39

Download references

Acknowledgements

The authors wish to thank the Amadeus Middleware Fraud Detection team directed by Virginie Amar and Jérémie Barlet, led by the product owner Christophe Allexandre and composed of Jean-Blas Imbert, Jiang Wu, Yang Pu and Damien Fontanes for building the rights, transactions-fr and transactions-mo datasets. MF gratefully acknowledges support from the AXA Research Fund.

Author information

Correspondence to Rémi Domingues.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Domingues, R., Michiardi, P., Barlet, J. et al. A comparative evaluation of novelty detection algorithms for discrete sequences. Artif Intell Rev (2019). https://doi.org/10.1007/s10462-019-09779-4

Download citation

Keywords

  • Novelty detection
  • Discrete sequences
  • Temporal data
  • Fraud detection
  • Outlier detection
  • Anomaly detection