Abstract
Outlier detection has been used for centuries to detect and, where appropriate, remove anomalous observations from data. Outliers arise due to mechanical faults, changes in system behaviour, fraudulent behaviour, human error, instrument error or simply through natural deviations in populations. Their detection can identify system faults and fraud before they escalate with potentially catastrophic consequences. It can identify errors and remove their contaminating effect on the data set and as such to purify the data for processing. The original outlier detection methods were arbitrary but now, principled and systematic techniques are used, drawn from the full gamut of Computer Science and Statistics. In this paper, we introduce a survey of contemporary techniques for outlier detection. We identify their respective motivations and distinguish their advantages and disadvantages in a comparative review.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aggarwal, C. C. Yu, P. S. (2001). Outlier Detection for High Dimensional Data. Proceedings of the ACM SIGMOD Conference 2001.
Aha, D. W. Bankert, R. B. (1994). Feature Selection for Case-Based Classification of Cloud Types: An Empirical Comparison. Proceedings of the AAAI-94 Workshop on Case-Based Reasoning.
Allan J, Carbonell J, Doddington G, Yamron J, Yang Y (1998) Topic Detection and Tracking Pilot Study: Final Report. Proceedings of the DARPA Broadcast News Transcription and Understanding Workshop.
A. Arning R. Agrawal P. Raghavan (1996) ArticleTitleA Linear Method for Deviation Detection in Large Databases Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining ␣ 164–169
Baker, L. D., Hofmann, T., McCallum, A. K. Yang, Y. (1999). A Hierarchical Probabilistic Model for Novelty Detection in Text. NIPS’99, Unpublished manuscript.
V. Barnett T. Lewis (1994) Outliers in Statistical Data 3rd edn John Wiley Sons NY
R. Beale T. Jackson (1990) Neural Computing: An Introduction Institute of Physics Publishing Bristol, UK and Philadelphia, PA
C.M. Bishop (1994) ArticleTitleNovelty detection Neural Network validation Proceedings of the IEE Conference on Vision, Image and Signal Processing ␣ 217–222
C.M. Bishop (1995) Neural Networks for Pattern Recognition Oxford University Press Oxford
Blake, C. L. Merz, C. J. (1998). UCI Repository of Machine Learning Databases, http://www.ics.uci.edu/mlearn/MLRepository.html, University of California, Irvine, Department of Information and Computer Sciences.
R. J. Bolton D. J. Hand (2001) Unsupervised Profiling Methods for Fraud Detection. Credit Scoring and Credit Control VII ␣ Edinburgh 5–7
P.S. Bradley O.L. FayyadUM Mangasarian U.M. Mangasarian (1999) ArticleTitleMathematical Programming for Data Mining: Formulations and Challenges. INFORMS Journal on Computing 11 IssueID3 217–238
L. Breiman J. Friedman R. Olshen C. Stone (1984) Classification and Regression Trees Wadsworth International Group. Belmont CA
C. E. Brodley M. A. Friedl (1996) Identifying and Eliminating Mislabeled Training Instances. Proceedings of the 13th National Conference on Artificial Intelligence, AAAI Press ␣ 799–805
T. Brotherton T. Johnson G. Chadderdon (1998) Classification and Novelty Detection using Linear Models and a Class Dependent – Elliptical Bassi Function Neural Network Proceedings of the International conference on neural networks Anchorage Alaska
S. Byers A.E. Raftery (1998) ArticleTitleNearest Neighbor Clutter Removal for Estimating Features in Spatial Point Processes Journal of the American Statistical Association 93 IssueID442 577–584
G. Carpenter S. Grossberg (1987) ArticleTitle‘A Massively Parallel Architecture for a Self-Organizing Neural Pattern Recognition Machine’ Computer Vision, Graphics, and Image Processing 37 54–115
T P Caudell DS Newman (1993) An Adaptive Resonance Architecture to Define Normality and Detect Novelties in Time Series and Databases. IEEE World Congress on Neural Networks Oregon Portland 166–176
Cohen, W. W. (1995). Fast Effective Rule Induction. International Conference on Machine Learning, 115–123.
Crook P, Hayes G (1995) A Robot Implementation of a Biologically Inspired Method for Novelty Detection. Proceedings of TIMR-2001, Towards Intelligent Mobile Robots Manchester.
D. Dasgupta S. Forrest (1996) Novelty Detection in Time Series Data Using Ideas from Immunology. Proceedings of the Fifth International Conference on Intelligent Systems. ␣ ␣
P. Datta D. Kibler (1995) Learning prototypical concept descriptions Proceedings of the 12th International Conference on Machine Learning Morgan Kaufmann ␣ 158–166
DeCoste, D. Levine, M. B. (2000). Automated Event Detection in Space Instruments: A Case Study Using IPEX-2 Data and Support Vector Machines. Proceedings of the SPIE Conference on Astronomical Telescopes and Space Instrumentation.
T. G. Dietterich R. S. Michalski (1986) Learning to Predict Sequences. Michalski Carbonell Mitchell (Eds) Machine Learning: An Artificial Intelligence Approach San Mateo, CA Morgan Kaufmann
M. Ester H. -P. Kriegel X. Xu (1996) A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise Proceedings of the Second International Conference on Knowledge Discovery and Data Mining AAAI Press Portland, Oregon 226–231
Faloutsos, C., Korn, F., Labrinidis, A., Kotidis, Y., Kaplunovich, A. Perkovic, D. (1997). Quantifiable Data Mining Using Principal Component Analysis. Technical Report CS-TR-3754, Institute for Systems Research, University of Maryland, College Park, MD.
T. Fawcett F.J. Provost (1999) ArticleTitleActivity Monitoring: Noticing Interesting Changes in Behavior Proceedings of the h ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 12 ␣ 53–62
Francis, J., Addison, D., Wermter, S. MacIntyre, J. (1999). Effectiveness of Feature Extraction in Neural Network Architectures for Novelty Detection. Proceedings of the ICANN Conference.
F.E. Grubbs (1969) ArticleTitleProcedures for Detecting Outlying Observations in Samples Technometrics 11 1–21
S. Hickinbotham J. Austin (2000) ArticleTitleNovelty Detection in Airframe Strain Data Proceedings of the International Conference on Pattern Recognition Barcelona 12 15 536–539
Himberg J, Jussi, A., Alhoniemi, E., Vesanto, J. Simula, O. (2001). The Self-Organizing Map as a Tool in Knowledge Engineering, In Pattern Recognition in Soft Computing Paradigm, 38–65. Soft Computing. World Scientific Publishing.
G. Hollier J. Austin (2002) Novelty Detection for Strain-Gauge Degradation Using Maximally Correlated Components. Proceedings of the European Symposium on Artificial Neural Networks ESANN’2002 Bruges 257–262
J. Hollmen V. Tresp (1999) Call-based Fraud Detection in Mobile Communication Networks using a Hierarchical Regime-Switching Model Advances in Neural Information Processing Systems – Proceedings of the 1998 Conference (NIPS’11) MIT Press ␣ 889–895
N. Japkowicz C. Myers M. A. Gluck (1995) A Novelty Detection Approach to Classification. Proceedings of the 14th International Conference on Artificial Intelligence (IJCAI-95), ␣ ␣ 518–523
G. H. John (1995) Robust Decision Trees: Removing Outliers from Databases. Proceedings of the First International Conference on Knowledge Discovery and Data Mining AAAI Press Menlo Park, CA 174–179
E. M. Knorr R. T. Ng (1998) Algorithms for Mining Distance-Based Outliers in Large Datasets. Proceedings of the VLDB Conference ␣ New York, USA. 392–403
T. Kohonen (1997) Self-Organizing Maps Springer-Verlag Heidelberg
T. Lane C. E. Brodley (1997a) Applications of Machine Learning to Anomaly Detection. R.A. Adey G. Rzevski T. Teti (Eds) Applications of Artificial Intelligence in Engineering X11, Southampton, UK Comput. Mech. Publications. 113–14
T. Lane C. E. Brodley (1997b) Sequence matching and learning in anomaly detection for computer security AAAI Workshop: AI Approaches to Fraud Detection and Risk Management, AAAI Press ␣ 43–49
Laurikkala, J., Juhola, M. Kentala, E. (2000). Informal Identification of Outliers in Medical Data. Fifth International Workshop on Intelligent Data Analysis in Medicine and Pharmacology IDAMAP-2000 Berlin, 22 August. Organized as a workshop of the 14th European Conference on Artificial Intelligence ECAI-2000.
Marsland, S. (2001). On-Line Novelty Detection Through Self-Organisation, with Application to Inspection Robotics. Ph.D. Thesis, Faculty of Science and Engineering, University of Manchester, UK.
A. Nairac N. Townsend R. Carr S. King L. CowleyP Tarassenko (1999) ArticleTitleA System for the Analysis of Jet System Vibration Data Integrated ComputerAided Engineering 6 IssueID1 53–65
Ng, R. T. Han, J. (1994). Efficient and Effective Clustering Methods for Spatial Data Mining. Proceedings of the 20th International Conference on Very Large Data Bases, September 12–15, 1994, Santiago, Chile, 144–155. Morgan Kaufmann Publishers.
L. Parra G. Deco S. Miesbach (1996) ArticleTitleStatistical Independence and Novelty Detection with Information Preserving Nonlinear Maps Neural Computation 8 IssueID2 260–269
A. L. Prodromidis S. J. Stolfo (1998) Mining Databases with Different Schemas: Integrating Incompatible Classifiers. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining ␣ ␣ 314–318
J.R. Quinlan (1986) ArticleTitleInduction of Decision Trees Machine Learning 1 IssueID1 81–106
Quinlan, J. R. (1993). C4.5: Programs for Machine Learning. Morgan Kaufmann.
S. Ramaswamy R. Rastogi K. Shim (2000) Efficient Algorithms for Mining Outliers from Large Data Sets. Proceedings of the ACM SIGMOD Conference on Management of Data ␣ Dallas, TX 427–438
S.J. Roberts (1998) ArticleTitleNovelty Detection Using Extreme Value Statistics IEE Proceedings on Vision, Image and Signal Processing 146 IssueID3 124–129
S. Roberts L. Tarassenko (1995) ArticleTitleA Probabilistic Resource Allocating Network for Novelty Detection Neural. Computation 6 270–284
P. Rousseeuw A. Leroy (1996) Robust Regression and Outlier Detection 3rd edn John Wiley & Sons NY
R. Saunders J.S. Gero (2001a) A Curious Design Agent: A Computational Model of Novelty-Seeking Behaviour in Design Proceedings of the Sixth Conference on Computer Aided Architectural Design Research in Asia (CAADRIA 2001) ␣ Sydney
R. Saunders J.S. Gero (2001b) Designing for Interest and Novelty: Motivating Design Agents Proceedings of CAAD Futures 2001 ␣ Eindhoven
S. Shekhar C. Lu P. Zhang (2001) Detecting Graph-Based Spatial Outliers: Algorithms and Applications Proceedings of the Seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ␣ ␣
D. B. Skalak (1994) Prototype and feature selection by sampling and random mutation hill climbing algorithms Machine Learning: Proceedings of the Eleventh International Conference, ␣ ␣ 293–301
D. B. Skalak E. L. Rissland (1990) Inductive Learning in a Mixed Paradigm Setting Proceedings of the Eighth National Conference on Artificial Intelligence ␣ Boston, MA 840–847
P. Smyth (1994) ArticleTitleMarkov Monitoring with Unknown States IEEE Journal on Selected Areas in Communications, Special Issue on Intelligent Signal Processing for Communications 12 IssueID9 1600–1612
S. J. Stolfo A. L. Prodromidis S. Tselepis W. Lee D. W. Fan P. K. Chan (1997) JAM: Java Agents for Meta-Learning over Distributed Databases Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining ␣ ␣ 74–81
Tang, J., Chen, Z., Fu, A. Cheung, D. (2002). A Robust Outlier Detection Scheme in Large Data Sets, 6th Pacific-Asia Conference on Knowledge Discovery and Data Mining, Taipei, Taiwan, May, 2002.
D. M. J. Tax A. Ypma R. P. W. Duin (1999) Support Vector Data Description Applied to Machine Vibration Analysis. Proceedings of ASCI’99 Heijen Netherlands
O. Taylor D. Addison (2000) Novelty Detection Using Neural Network Technology. Proceedings of the COMADEN Conference ␣ ␣
P. H. S. Torr D. W. Murray (1993) Outlier Detection and Motion Segmentation Proceedings of SPIE ␣ ␣
Vesanto, J., Himberg, J., Siponen, M. Simula, O. (1998). Enhancing SOM Based Data Visualization. Proceedings of the 5th International Conference on Soft Computing and Information/Intelligent Systems. Methodologies for the Conception, Design and Application of Soft Computing, Vol. 1, 64–67, Singapore: World Scientific.
Wettschereck, D. (1994). A Study of Distance-based Machine Learning Algorithms. Ph.D. Thesis, Department of Computer Science, Oregon State University, Corvallis.
A. Ypma R. P. Duin (1997) Novelty Detection Using Self-Organizing Maps. N. Kasabov R. Kozma K. Ko R. O’Shea G. Coghill T. Gedeon (Eds) Progress in Connectionist-Based Information Systems, Springer London 1322–1325
Zhang, T., Ramakrishnan, R. Livny, M. (1996). BIRCH: An Efficient Data Clustering Method for Very Large Databases’. In Jagadish, H. V. Mumick, I. S. (eds.) Proceedings of the 1996 ACM SIGMOD International Conference on Management of Data, Montreal, Quebec, Canada, June 4–6, 1996, 103–114. ACM Press.