A comparative evaluation of novelty detection algorithms for discrete sequences

  • Rémi DominguesEmail author
  • Pietro Michiardi
  • Jérémie Barlet
  • Maurizio Filippone


The identification of anomalies in temporal data is a core component of numerous research areas such as intrusion detection, fault prevention, genomics and fraud detection. This article provides an experimental comparison of candidate methods for the novelty detection problem applied to discrete sequences. The objective of this study is to identify which state-of-the-art methods are efficient and appropriate candidates for a given use case. These recommendations rely on extensive novelty detection experiments based on a variety of public datasets in addition to novel industrial datasets. We also perform thorough scalability and memory usage tests resulting in new supplementary insights of the methods’ performance, key selection criteria to solve problems relying on large volumes of data and to meet the expectations of applications subject to strict response time constraints.


Novelty detection Discrete sequences Temporal data Fraud detection Outlier detection Anomaly detection 



The authors wish to thank the Amadeus Middleware Fraud Detection team directed by Virginie Amar and Jérémie Barlet, led by the product owner Christophe Allexandre and composed of Jean-Blas Imbert, Jiang Wu, Yang Pu and Damien Fontanes for building the rights, transactions-fr and transactions-mo datasets. MF gratefully acknowledges support from the AXA Research Fund.


  1. Aggarwal CC (2015) Outlier analysis. Springer, Cham, pp 237–263Google Scholar
  2. Bergroth L, Hakonen H, Raita T (2000) A survey of longest common subsequence algorithms. In: Proceedings seventh international symposium on string processing and information retrieval. SPIRE 2000, pp 39–48Google Scholar
  3. Breiman L, Friedman J, Olsen R, Stone C (1984) Classification and regression trees. Wadsworth and BrooksGoogle Scholar
  4. Breunig MM, Kriegel H-P, Ng RT, Sander J (2000) LOF: identifying density-based local outliers. SIGMOD Rec 29(2):93–104CrossRefGoogle Scholar
  5. Budalakoti S, Srivastava AN, Akella R,Turkov E (2006) Anomaly detection in large sets of high-dimensional symbol sequences. Technical Report NASA TM-2006-214553Google Scholar
  6. Budalakoti S, Srivastava AN, Otey ME (2009) Anomaly detection and diagnosis algorithms for discrete symbol sequences with applications to airline safety. IEEE Trans Syst Cybern C (Appl Rev) 39(1):101–113CrossRefGoogle Scholar
  7. Chandola V, Banerjee A, Kumar V (2012) Anomaly detection for discrete sequences: a survey. IEEE Trans Knowl Data Eng 24(5):823–839CrossRefGoogle Scholar
  8. Chandola V, Mithal V, Kumar V (2008) Comparative evaluation of anomaly detection techniques for sequence data. In: 2008 Eighth IEEE international conference on data mining, pp 743–748Google Scholar
  9. Chang D, Jones NA, Li D, Ouyang M, Ragade RK (2008) Compute pairwise Euclidean distances of data points with gpus. In: Proceedings of the iASTED international symposium on computational biology and bioinformatics, pp 278–283Google Scholar
  10. Christ M, Kempa-Liehr AW, Feindt M (2016) Distributed and parallel time series feature extraction for industrial big data applications. arXiv:1610.07717
  11. Cohen WW (1995) Fast effective rule induction. In: Prieditis A, Russell S (eds) Machine learning proceedings 1995. Morgan Kaufmann, San Francisco, pp 115–123CrossRefGoogle Scholar
  12. Crochemore M, Iliopoulous CS, Pinzon YJ (2003) Speeding-up hirschberg and hunt-szymanski lcs algorithms. Fundamenta Informaticae 56(1–2):89–103MathSciNetzbMATHGoogle Scholar
  13. Davis J, Goadrich M (2006) The relationship between precision-recall and roc curves. In: Proceedings of the 23rd international conference on Machine learning. ACM, pp 233–240Google Scholar
  14. de Fortuny EJ, Martens D (2015) Active learning-based pedagogical rule extraction. IEEE Trans Neural Netw Learn Syst 26(11):2664–2677MathSciNetCrossRefGoogle Scholar
  15. Domingues R, Michiardi P, Zouaoui J, Filippone M (2018) Deep gaussian process autoencoders for novelty detection. Mach LearnGoogle Scholar
  16. Emmott A, Das S, Dietterich T, Fern A, Wong W-K (2016) A meta-analysis of the anomaly detection problem. arXiv:1503.01158v2
  17. Etchells TA, Lisboa PJG (2006) Orthogonal search-based rule extraction (osre) for trained neural networks: a practical and efficient approach. IEEE Trans Neural Netw 17(2):374–384CrossRefGoogle Scholar
  18. Fowkes J, Sutton C (2016) A subsequence interleaving model for sequential pattern mining. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, KDD ’16, New York, NY, USA, 2016. ACM, pp 835–844Google Scholar
  19. Gan W, Lin JC-W, Fournier-Viger P, Chao H-C, Yu PS (2018) A survey of parallel sequential pattern mining. arXiv:1805.10515
  20. García S, Fernández A, Luengo J, Herrera F (2010) Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: experimental analysis of power. Inf Sci 180(10):2044–2064 (Special Issue on Intelligent Distributed Information Systems)CrossRefGoogle Scholar
  21. Gupta M, Gao J, Aggarwal CC, Han J (2014) Outlier detection for temporal data: a survey. IEEE Trans Knowl Data Eng 26(9):2250–2267CrossRefGoogle Scholar
  22. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780CrossRefGoogle Scholar
  23. Hodge V, Austin J (2004) A survey of outlier detection methodologies. Artif Intell Rev 22(2):85–126CrossRefGoogle Scholar
  24. Hofmeyr SA, Forrest S, Somayaji A (1998) Intrusion detection using sequences of system calls. J Comput Secur 6(3):151–180CrossRefGoogle Scholar
  25. Hunt JW, Szymanski TG (1977) A fast algorithm for computing longest common subsequences. Commun ACM 20(5):350–353MathSciNetCrossRefGoogle Scholar
  26. Huysmans J, Baesens B, Vanthienen J (2006) Iter: an algorithm for predictive regression rule extraction. In: Tjoa AM, Trujillo J (eds) Data warehousing and knowledge discovery. Springer, Berlin, pp 270–279CrossRefGoogle Scholar
  27. Karpathy A, Johnson J, Li F (2016) Visualizing and understanding recurrent networks. In: Proceedings of the fourth international conference on learning representations (ICLR 2016)Google Scholar
  28. Kundzewicz ZW, Robson AJ (2004) Change detection in hydrological records—a review of the methodology. Hydrol Sci J 49(1):7–19CrossRefGoogle Scholar
  29. Lazarevic A, Ertoz L, Kumar V, Ozgur A, Srivastava J (2003) A comparative study of anomaly detection schemes in network intrusion detection. In: Proceedings of the 2003 SIAM international conference on data mining, pp 25–36Google Scholar
  30. Lee W, Stolfo SJ, Chan PK (1997) Learning patterns from unix process execution traces for intrusion detection. In: AAAI workshop on AI approaches to fraud detection and risk management, pp 50–56Google Scholar
  31. Levenshtein VI (1966) Binary codes capable of correcting deletions, insertions and reversals. Sov. Phys. Dokl. 10:707MathSciNetGoogle Scholar
  32. Lipton ZC, Berkowitz J, Elkan C (2015) A critical review of recurrent neural networks for sequence learningGoogle Scholar
  33. Liu FT, Ting KM, Zhou Z-H (2008) Isolation forest. In: Proceedings of the 2008 eighth IEEE international conference on data mining, ICDM ’08. IEEE Computer Society, pp 413–422Google Scholar
  34. Luong M-T, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. arXiv:1508.04025
  35. Marchi E, Vesperini F, Eyben F, Squartini S, Schuller B (2015) A novel approach for automatic acoustic novelty detection using a denoising autoencoder with bidirectional lstm neural networks. In: 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 1996–2000Google Scholar
  36. Maxion RA, Townsend TN (2002) Masquerade detection using truncated command lines. In: Proceedings international conference on dependable systems and networks, pp 219–228Google Scholar
  37. Onderwater M (2015) Outlier preservation by dimensionality reduction techniques. Int J Data Anal Tech Strateg 7(3):231–252CrossRefGoogle Scholar
  38. Park H-S, Jun C-H (2009) A simple and fast algorithm for k-medoids clustering. Expert Syst Appl 36(2, Part 2):3336–3341CrossRefGoogle Scholar
  39. Petrović S, Osborne M, Lavrenko V (2010) Streaming first story detection with application to twitter. In: Human Language technologies: the 2010 annual conference of the North American Chapter of the Association for Computational Linguistics, HLT ’10. Association for Computational Linguistics, Stroudsburg, pp 181–189Google Scholar
  40. Pihur V, Datta S, Datta S (2009) RankAggreg, an r package for weighted rank aggregation. BMC Bioinform 10(1):62CrossRefGoogle Scholar
  41. Rabiner LR (1989) A tutorial on hidden Markov models and selected applications in speech recognition. Proc IEEE 77(2):257–286CrossRefGoogle Scholar
  42. Ramaswamy S, Rastogi R, Shim K (2000) Efficient algorithms for mining outliers from large data sets. In: Proceedings of the 2000 ACM SIGMOD international conference on management of data, SIGMOD ’00, New York, NY, USA. ACM, pp 427–438Google Scholar
  43. Saad EW, Wunsch DC (2007) Neural network explanation using inversion. Neural Netw 20(1):78–93CrossRefGoogle Scholar
  44. Saidi R, Maddouri M, MephuNguifo E (2010) Protein sequences classification by means of feature extraction with substitution matrices. BMC Bioinform 11(1):175CrossRefGoogle Scholar
  45. Sakurada M, Yairi T (2014) Anomaly detection using autoencoders with nonlinear dimensionality reduction. In: Proceedings of the MLSDA 2014 2nd workshop on machine learning for sensory data analysis. ACM, pp 4–11Google Scholar
  46. Schubert E, Rousseeuw PJ (2018) Faster k-Medoids clustering: improving the PAM, CLARA, and CLARANS algorithms. arXiv:1810.05691
  47. Sculley D, Brodley CE (2006) Compression and machine learning: a new perspective on feature space vectors. In: Data compression conference (DCC’06), pp 332–341Google Scholar
  48. Setiono R, Leow WK, Zurada JM (2002) Extraction of rules from artificial neural networks for nonlinear regression. IEEE Trans Neural Netw 13(3):564–577CrossRefGoogle Scholar
  49. Sun P, Chawla S, Arunasalam B (2006) Mining for outliers in sequential databases. In: Proceedings of the 2006 SIAM international conference on data mining, pp 94–105Google Scholar
  50. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: Ghahramani Z, Welling M, Cortes C, Lawrence ND, Weinberger KQ (eds) Advances in neural information processing systems, vol 27. Curran Associates Inc., New York, pp 3104–3112Google Scholar
  51. Taylor SJ, Letham B (2018) Forecasting at scale. Am Stat 72(1):37–45MathSciNetCrossRefGoogle Scholar
  52. Wang JTL, Ma Q, Shasha D, Wu CH (2001) New techniques for extracting features from protein sequences. IBM Syst J 40(2):426–441CrossRefGoogle Scholar
  53. Warrender C, Forrest S, Pearlmutter B (1999) Detecting intrusions using system calls: alternative data models. In: Proceedings of the 1999 IEEE symposium on security and privacy (Cat. No. 99CB36344), pp 133–145Google Scholar
  54. Zhang Q-S, Zhu S-C (2018) Visual interpretability for deep learning: a survey. Front Inf Technol Electron Eng 19(1):27–39CrossRefGoogle Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.Department of Data ScienceEURECOMSophia AntipolisFrance
  2. 2.AmadeusSophia AntipolisFrance

Personalised recommendations