Skip to main content
Book cover

Ibero-American Conference on Artificial Intelligence

IBERAMIA 2008: Advances in Artificial Intelligence – IBERAMIA 2008 pp 163–172Cite as

Detection of Anomalies in Large Datasets Using an Active Learning Scheme Based on Dirichlet Distributions

  • Conference paper

Part of the Lecture Notes in Computer Science book series (LNAI,volume 5290)

Abstract

Today, the detection of anomalous records is a highly valuable application in the analysis of current huge datasets. In this paper we propose a new algorithm that, with the help of a human expert, efficiently explores a dataset with the goal of detecting relevant anomalous records. Under this scheme the computer selectively asks the expert for data labeling, looking for relevant semantic feedback in order to improve its knowledge about what characterizes a relevant anomaly. Our rationale is that while computers can process huge amounts of low level data, an expert has high level semantic knowledge to efficiently lead the search. We build upon our previous work based on Bayesian networks that provides an initial set of potential anomalies. In this paper, we augment this approach with an active learning scheme based on the clustering properties of Dirichlet distributions. We test the performance of our algorithm using synthetic and real datasets. Our results indicate that, under noisy data and anomalies presenting regular patterns, our approach significantly reduces the rate of false positives, while decreasing the time to reach the relevant anomalies.

Keywords

  • Active Learning
  • Bayesian Network
  • Anomaly Detection
  • Synthetic Dataset
  • Dirichlet Distribution

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-540-88309-8_17
  • Chapter length: 10 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   84.99
Price excludes VAT (USA)
  • ISBN: 978-3-540-88309-8
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   109.00
Price excludes VAT (USA)
Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Aamodt, A., Plaza, E.: Case-based reasoning: Foundational issues, methodological variations, and system approaches. Artificial Intelligence Communications 7(1), 39–59 (1994)

    Google Scholar 

  2. Asuncion, A., Newman, D.J.: UCI machine learning repository (2007), http://www.ics.uci.edu/~mlearn/MLRepository.html

  3. Blackwell, D., MacQueen, J.: Ferguson distribution via polya urn schemes. The Annals of Statistics 1(2), 353–355 (1973)

    MATH  CrossRef  MathSciNet  Google Scholar 

  4. Cansado, A., Soto, A.: Unsupervised anomaly detection in large databases using bayesian networks. Applied Artificial Intelligence 22(4), 309–330 (2008)

    CrossRef  Google Scholar 

  5. Ferguson, T.: A bayesian analysis of some nonparametric problems. The Annals of Statistics 1(2), 209–230 (1973)

    MATH  CrossRef  MathSciNet  Google Scholar 

  6. Hodge, V., Austin, J.: A survey of outlier detection methodologies. Artificial Intelligence Review 22(2), 85–126 (2004)

    MATH  CrossRef  Google Scholar 

  7. Jackson, P.: Introduction to Expert Systems. Addison-Wesley, Reading (1998)

    Google Scholar 

  8. Kou, Y., Lu, C., Sirwongwattana, S., Huang, Y.: Survey of fraud detection techniques. In: Proc. of the IEEE Int. Conf. on Networking, Sensing and Control, pp. 749–754 (2004)

    Google Scholar 

  9. Lewis, D., Gale, W.: A sequential algorithm for training text classifiers. In: Proc. of 17th Int. Conf. ACM SIGIR, pp. 3–12 (1994)

    Google Scholar 

  10. Neapolitan, R.: Learning Bayesian Networks. Prentice-Hall, Englewood Cliffs (2004)

    Google Scholar 

  11. Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, San Francisco (1988)

    Google Scholar 

  12. Pelleg, D., Moore, A.: Active learning for anomaly and rare-category detection. In: Proc. of the 18th Conf. on Advances in Neural Information Processing Systems, NIPS (2004)

    Google Scholar 

  13. Roy, N., McCallum, A.: Toward optimal active learning through sampling estimation of error reduction. In: Proc. of 18th Int. Conf. on Machine Learning, ICML, pp. 441–448 (2001)

    Google Scholar 

  14. Seung, S., Opper, M., Sompolinski, H.: Query by committee. In: Proc. of 5th Annual ACM Workshop on Computational Learning Theory, pp. 287–294 (1992)

    Google Scholar 

  15. Soto, A., Zavala, F., Araneda, A.: An accelerated algorithm for density estimation in large databases using Gaussian mixtures. Cybernetics and Systems 38(2), 123–139 (2007)

    CrossRef  MATH  Google Scholar 

  16. Tong, S., Koller, D.: Active learning for parameter estimation in bayesian networks. In: Proc. of the 13th Conf. on Advances in Neural Information Processing Systems, NIPS, pp. 647–653 (2001)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Pontificia Universidad Católica de, Chile

    Karim Pichara, Alvaro Soto & Anita Araneda

Authors
  1. Karim Pichara
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alvaro Soto
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anita Araneda
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. ICREA & Universitat Pompeu Fabra, Paseo de Circumvalacion 8, 08003, Barcelona, Spain

    Hector Geffner

  2. IST-UTL and INESC-ID, Av. Prof. Cavaco Silva - Taguspark, 2744-016, Porto Salvo, Portugal

    Rui Prada

  3. ADETTI/ISCTE and ISCTE, Lisbon University Institute, Av. das Forças Armadas, 1649-026, Lisbon, Portugal

    Isabel Machado Alexandre

  4. ADETTI/ISCTE and ISCTE, Lisbon University Institute, , Av. das Forças Armadas, 1649-026, Lisbon, Portugal

    Nuno David

Rights and permissions

Reprints and Permissions

Copyright information

© 2008 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Pichara, K., Soto, A., Araneda, A. (2008). Detection of Anomalies in Large Datasets Using an Active Learning Scheme Based on Dirichlet Distributions. In: Geffner, H., Prada, R., Machado Alexandre, I., David, N. (eds) Advances in Artificial Intelligence – IBERAMIA 2008. IBERAMIA 2008. Lecture Notes in Computer Science(), vol 5290. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-88309-8_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-88309-8_17

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-88308-1

  • Online ISBN: 978-3-540-88309-8

  • eBook Packages: Computer ScienceComputer Science (R0)