Artificial Intelligence Review

, Volume 22, Issue 2, pp 85–126 | Cite as

A Survey of Outlier Detection Methodologies

  • Victoria Hodge
  • Jim Austin
Article

Abstract

Outlier detection has been used for centuries to detect and, where appropriate, remove anomalous observations from data. Outliers arise due to mechanical faults, changes in system behaviour, fraudulent behaviour, human error, instrument error or simply through natural deviations in populations. Their detection can identify system faults and fraud before they escalate with potentially catastrophic consequences. It can identify errors and remove their contaminating effect on the data set and as such to purify the data for processing. The original outlier detection methods were arbitrary but now, principled and systematic techniques are used, drawn from the full gamut of Computer Science and Statistics. In this paper, we introduce a survey of contemporary techniques for outlier detection. We identify their respective motivations and distinguish their advantages and disadvantages in a comparative review.

anomaly detection deviation noise novelty outlier recognition 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Aggarwal, C.C. & Yu, P.S.(2001). Outlier Detection for High Dimensional Data. Proceedings of the ACM SIGMOD Conference 2001.Google Scholar
  2. Aha, D.W. & Bankert, R.B.(1994). Feature Selection for Case-Based Classification of Cloud Types:An Empirical Comparison. Proceedings of the AAAI-94 Workshop on Case-Based Reasoning. Google Scholar
  3. Allan, J., Carbonell, J., Doddington, G., Yamron, J. & Yang, Y.(1998). Topic Detection and Tracking Pilot Study:Final Report. Proceedings of the DARPA Broadcast News Transcription and Understanding Workshop. Google Scholar
  4. Arning, A., Agrawal, R. & Raghavan, P.(1996). A Linear Method for Deviation Detection in Large Databases. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,164–169.Google Scholar
  5. Baker, L.D., Hofmann, T., McCallum, A.K. & Yang, Y.(1999). A Hierarchical Probabilistic Model for Novelty Detection in Text. NIPS '99,Unpublished manuscript. Google Scholar
  6. Barnett, V. & Lewis, T.(1994). Outliers in Statistical Data, 3rd edn. John Wiley & Sons.Google Scholar
  7. Beale, R. & Jackson, T.(1990). Neural Computing: An Introduction. Bristol, UK and Philadelphia,PA: Institute of Physics Publishing.Google Scholar
  8. Bishop, C.M.(1994). Novelty detection & Neural Network validation. Proceedings of the IEE Conference on Vision,Image and Signal Processing, 217–222.Google Scholar
  9. Bishop, C.M.(1995). Neural Networks for Pattern Recognition. Oxford University Press.Google Scholar
  10. Blake, C.L. & Merz, C.J.(1998). UCI Repository of Machine Learning Databases, http://www.ics.uci.edu/mlearn/MLRepository.html, University of California,Irvine,Department of Information and Computer Sciences.Google Scholar
  11. Bolton, R.J. & Hand, D.J.(2001). Unsupervised Profiling Methods for Fraud Detection. Credit Scoring and Credit Control VII,Edinburgh,UK, 5-7 September. Google Scholar
  12. Bradley, P.S., Fayyad, U.M. & Mangasarian, O.L.(1999). Mathematical Programming for Data Mining:Formulations and Challenges.INFORMS Journal on Computing 11(3):217–238.Google Scholar
  13. Breiman, L., Friedman, J., Olshen,R. & Stone,C.(1984). Classification and Regression Trees Belmont,CA: Wadsworth International Group.Google Scholar
  14. Brodley, C.E. & Friedl, M.A.(1996). Identifying and Eliminating Mislabeled Training Instances. Proceedings of the 13th National Conference on Artificial Intelligence, 799–805, AAAI Press.Google Scholar
  15. Brotherton, T., Johnson, T. & Chadderdon,G.(1998). Classification and Novelty Detection using Linear Models and a Class Dependent-Elliptical Bassi Function Neural Network. Proceedings of the International conference on neural networks. Anchorage,Alaska.Google Scholar
  16. Byers, S. & Raftery, A.E.(1998). Nearest Neighbor Clutter Removal for Estimating Features in Spatial Point Processes. Journal of the American Statistical Association 93(442): 577–584.Google Scholar
  17. Carpenter, G. & Grossberg, S.(1987). 'A Massively Parallel Architecture for a Self-Organizing Neural Pattern Recognition Machine'. Computer Vision,Graphics,and Image Processing 37: 54–115.Google Scholar
  18. Caudell, T.P. & Newman, D.S. (1993). An Adaptive Resonance Architecture to Define Normality and Detect Novelties in Time Series and Databases. IEEE World Congress on Neural Networks,Portland,Oregon. 166–176.Google Scholar
  19. Cohen, W.W.(1995). Fast Effective Rule Induction. International Conference on Machine Learning, 115–123.Google Scholar
  20. Crook,P. & Hayes, G. (1995). A Robot Implementation of a Biologically Inspired Method for Novelty Detection. Proceedings of TIMR-2001,Towards Intelligent Mobile Robots Manchester.Google Scholar
  21. Dasgupta, D. & Forrest, S. (1996). Novelty Detection in Time Series Data Using Ideas from Immunology. Proceedings of the Fifth International Conference on Intelligent Systems.Google Scholar
  22. Datta, P. & Kibler, D. (1995). Learning prototypical concept descriptions. Proceedings of the 12th International Conference on Machine Learning,158–166, Morgan Kaufmann.Google Scholar
  23. DeCoste, D. & Levine, M.B. (2000). Automated Event Detection in Space Instruments: A Case Study Using IPEX-2 Data and Support Vector Machines. Proceedings of the SPIE Conference on Astronomical Telescopes and Space Instrumentation. Google Scholar
  24. Dietterich, T.G. & Michalski, R.S. (1986). Learning to Predict Sequences. In Michalski, Carbonell & Mitchell (eds.) Machine Learning:An Artificial Intelligence Approach San Mateo,CA: Morgan Kaufmann.Google Scholar
  25. Ester, M., Kriegel, H.-P. & Xu, X. (1996). A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. Proceedings of the Second International Conference on Knowledge Discovery and Data Mining,Portland,Oregon, 226–231. AAAI Press.Google Scholar
  26. Faloutsos, C., Korn, F., Labrinidis, A., Kotidis, Y., Kaplunovich, A. & Perkovic, D. (1997). Quantitable Data Mining Using Principal Component Analysis. Technical Report CS-TR-3754, Institute for Systems Research,University of Maryland, College Park, MD.Google Scholar
  27. Fawcett, T. & Provost, F.J. (1999). Activity Monitoring:Noticing Interesting Changes in Behavior. Proceedings of the 5th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 53–62.Google Scholar
  28. Francis, J., Addison, D., Wermter, S. & MacIntyre, J. (1999). Effectiveness of Feature Extraction in Neural Network Architectures for Novelty Detection. Proceedings of the ICANN Conference.Google Scholar
  29. Grubbs, F.E.(1969). Procedures for Detecting Outlying Observations in Samples. Technometrics 11: 1–21.Google Scholar
  30. Hickinbotham, S. & Austin, J.(2000). Novelty Detection in Airframe Strain Data. Proceedings of 15th International Conference on Pattern Recognition, Barcelona, 536–539.Google Scholar
  31. Himberg, J., Jussi, A., Alhoniemi, E., Vesanto, J. & Simula, O. (2001). The Self-Organizing Map as a Tool in Knowledge Engineering,In Pattern Recognition in Soft Computing Paradigm, 38–65.Soft Computing.World Scientific Publishing.Google Scholar
  32. Hollier, G. & Austin, J. (2002). Novelty Detection for Strain-Gauge Degradation Using Maximally Correlated Components.Proceedings of the European Symposium on Artificial Neural Networks,ESANN '2002,Bruges,257–262.Google Scholar
  33. Hollmen,J. & Tresp,V.(1999).Call-based Fraud Detection in Mobile Communication Networks using a Hierarchical Regime-Switching Model.{tiAdvances in Neural Information Processing Systems-Proceedings of the 1998 Conference (NIPS '11)}, 889–895,MIT Press.Google Scholar
  34. Japkowicz,N., Myers,C. & Gluck M.A.(1995).A Novelty Detection Approach to Classi cation.Proceedings of the 14th International Conference on Artificial Intelligence (IJCAI-95) ,518–523.Google Scholar
  35. John,G.H.(1995). Robust Decision Trees:Removing Outliers from Databases. Proceedings of the First International Conference on Knowledge Discovery and Data Mining, 174–179. Menlo Park,CA: AAAI Press.Google Scholar
  36. Knorr,E.M. & Ng,R.T.(1998). Algorithms for Mining Distance-Based Outliers in Large Datasets. Proceedings of the VLDB Conference, 392–403, New York,USA.Google Scholar
  37. Kohonen,T.(1997). Self-Organizing Maps, Vol.2. Springer-Verlag, Heidelberg.Google Scholar
  38. Lane,T. & Brodley,C.E.(1997a). Applications of Machine Learning to Anomaly Detection.In Adey,R.A., Rzevski,G,and Teti,T.(eds.) Applications of Artificial Intelligence in Engineering X11, 113–14, Southampton,UK: Comput.Mech.Publications.Google Scholar
  39. Lane,T. & Brodley,C.E.(1997b). Sequence matching and learning in anomaly detection for computer security. AAAI Workshop:AI Approaches to Fraud Detection and Risk Management, 43–49.AAAI Press.Google Scholar
  40. Laurikkala,J., Juhola,M. & Kentala,E.(2000). Informal Identification of Outliers in Medical Data. Fifth International Workshop on Intelligent Data Analysis in Medicine and Pharmacology IDAMAP-2000 Berlin,22 August.Organized as a workshop of the 14th European Conference on Artificial Intelligence ECAI-2000.Google Scholar
  41. Marsland,S.(2001). On-Line Novelty Detection Through Self-Organisation,with Application to Inspection Robotics. Ph.D.Thesis, Faculty of Science and Engineering,University of Manchester, UK.Google Scholar
  42. Nairac,A., Townsend,N., Carr,R., King,S., Cowley,P. & Tarassenko,L.(1999). A System for the Analysis of Jet System Vibration Data.Integrated ComputerAided Engineering 6(1): 53–65.Google Scholar
  43. Ng,R.T. & Han,J.(1994).Efficient and Effective Clustering Methods for Spatial Data Mining. Proceedings of the 20th International Conference on Very Large Data Bases, September 12-15,1994,Santiago,Chile, 144–155. Morgan Kaufmann Publishers.Google Scholar
  44. Parra,L., Deco,G. & Miesbach S.(1996). Statistical Independence and Novelty Detection with Information Preserving Nonlinear Maps.Neural Computation 8(2): 260–269.Google Scholar
  45. Prodromidis,A.L. & Stolfo,S.J.(1998). Mining Databases with Different Schemas: Integrating Incompatible Classifiers. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 314–318.Google Scholar
  46. Quinlan,J.R.(1986). Induction of Decision Trees. Machine Learning 1 (1): 81–106.Google Scholar
  47. Quinlan,J.R.(1993). C4.5:Programs for Machine Learning. Morgan Kaufmann.Google Scholar
  48. Ramaswamy,S., Rastogi,R. & Shim,K.(2000). Efficient Algorithms for Mining Outliers from Large Data Sets. Proceedings of the ACM SIGMOD Conference on Management of Data,Dallas,TX, 427–438.Google Scholar
  49. Roberts,S.J.(1998). Novelty Detection Using Extreme Value Statistics. IEE Proceedings on Vision,Image and Signal Processing 146 (3): 124–129.Google Scholar
  50. Roberts,S. & Tarassenko,L.(1995). A Probabilistic Resource Allocating Network for Novelty Detection.Neural Computation 6: 270–284.Google Scholar
  51. Rousseeuw,P. & Leroy,A.(1996). Robust Regression and Outlier Detection, 3rd edn. John Wiley & Sons.Google Scholar
  52. Saunders,R. & Gero,J.S.(2001a). A Curious Design Agent:A Computational Model of Novelty-Seeking Behaviour in Design. Proceedings of the Sixth Conference on Computer Aided Architectural Design Research in Asia (CAADRIA 2001 ),Sydney.Google Scholar
  53. Saunders,R. & Gero,J.S.(2001b). Designing for Interest and Novelty:Motivating Design Agents. Proceedings of CAAD Futures 2001,Eindhoven.Google Scholar
  54. Shekhar,S., Lu,C. & Zhang,P.(2001). Detecting Graph-Based Spatial Outliers: Algorithms and Applications. Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Google Scholar
  55. Skalak,D.B.(1994). Prototype and feature selection by sampling and random mutation hill climbing algorithms.Machine Learning: Proceedings of the Eleventh International Conference, 293–301.Google Scholar
  56. Skalak,D.B. & Rissland,E.L.(1990). Inductive Learning in a Mixed Paradigm Setting. Proceedings of the Eighth National Conference on Artificial Intelligence,Boston,MA, 840–847.Google Scholar
  57. Smyth,P.(1994). Markov Monitoring with Unknown States. IEEE Journal on Selected Areas in Communications,Special Issue on Intelligent Signal Processing for Communications 12 (9): 1600–1612.Google Scholar
  58. Stolfo,S.J., Prodromidis,A.L., Tselepis,S., Lee,W., Fan,D.W. & Chan,P.K.(1997). JAM:Java Agents for Meta-Learning over Distributed Databases. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 74–81.Google Scholar
  59. Tang,J., Chen,Z., Fu,A. & Cheung,D.(2002). A Robust Outlier Detection Scheme in Large Data Sets,6th Pacific-Asia Conference on Knowledge Discovery and Data Mining,Taipei,Taiwan,May,2002.Google Scholar
  60. Tax,D.M.J., Ypma,A. & Duin,R.P.W.(1999). Support Vector Data Description Applied to Machine Vibration Analysis. Proceedings of ASCI '99,Heijen,Netherlands. Google Scholar
  61. Taylor,O. & Addison,D.(2000). Novelty Detection Using Neural Network Technology. Proceedings of the COMADEN Conference.Google Scholar
  62. Torr,P.H.S. & Murray,D.W.(1993). Outlier Detection and Motion Segmentation. Proceedings of SPIE.Google Scholar
  63. Vesanto,J., Himberg,J., Siponen,M. & Simula,O.(1998). Enhancing SOM Based Data Visualization. Proceedings of the 5th International Conference on Soft Computing and Information/Intelligent Systems.Methodologies for the Conception,Design and Application of Soft Computing,Vol.1,64–67,Singapore:World ScientificGoogle Scholar
  64. Wettschereck,D.(1994). A Study of Distance-based Machine Learning Algorithms. Ph.D.Thesis,Department of Computer Science,Oregon State University, Corvallis.Google Scholar
  65. Ypma,A. & Duin,R.P.W.(1997). Novelty Detection Using Self-Organizing Maps.In Kasabov, N., Kozma, R. Ko, K., O'Shea, R., Coghill, G. & Gedeon, T.(eds.) Progress in Connectionist-Based Information Systems,Vol.2,London: Springer. 1322–1325.Google Scholar
  66. Zhang,T., Ramakrishnan,R. & Livny,M.(1996). BIRCH:An Efficient Data Clustering Method for Very Large Databases'.In Jagadish,H.V. & Mumick,I.S.(eds.) Proceedings of the 1996 ACM SIGMOD International Conference on Management of Data Montreal,Quebec,Canada,June 4-6,1996,103–114.ACM Press.Google Scholar

Copyright information

© Kluwer Academic Publishers 2004

Authors and Affiliations

  • Victoria Hodge
    • 1
  • Jim Austin
    • 1
  1. 1.Department of Computer ScienceUniversity of YorkUK

Personalised recommendations