Rare Category Detection on O(dN) Time Complexity

  • Zhenguang Liu
  • Hao Huang
  • Qinming He
  • Kevin Chiew
  • Lianhang Ma
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8444)


Rare category detection (RCD) aims at finding out at least one data example of each rare category in an unlabeled data set with the help of a labeling oracle to prove the existence of such a rare category. Various approaches have been proposed for RCD with quadratic or even cubic time complexity. In this paper, by using histogram density estimation and wavelet analysis, we propose FRED algorithm and its prior-free version iFRED algorithm for RCD, both of which achieve linear time complexity w.r.t. either the data set size N or the data dimension d. Theoretical analysis guarantees its effectiveness, and comprehensive experiments on both synthetic and real data sets verify the effectiveness and efficiency of our algorithms.


Rare category detection wavelet analysis linear time complexity 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Asuncion, A., Newman, D.: UCI Machine Learning Repository (2007)Google Scholar
  2. 2.
    Bay, S., Kumaraswamy, K., Anderle, M., Kumar, R., Steier, D.: Large scale detection of irregularities in accounting data. In: ICDM, Hong Kong, China, pp. 75–86 (2006)Google Scholar
  3. 3.
    Garcia-Pedrajas, N., Garcia-Osorio, C.: Boosting for class-imbalanced datasets using genetically evolved supervised non-linear projections. Progress in AI 2(1), 29–44 (2013)Google Scholar
  4. 4.
    He, J., Carbonell, J.: Nearest-neighbor-based active learning for rare category detection. In: NIPS 2007, Vancouver, British Columbia, Canada, December 3-6, pp. 633–640 (2007)Google Scholar
  5. 5.
    He, J., Carbonell, J.: Prior-free rare category detection. In: SDM 2009, Sparks, Nevada, USA, April 30-May 2, pp. 155–163 (2009)Google Scholar
  6. 6.
    He, J., Carbonell, J.: Coselection of features and instances for unsupervised rare category analysis. Statistical Analysis and Data Mining 3(6), 417–430 (2010)CrossRefMathSciNetGoogle Scholar
  7. 7.
    He, J., Liu, Y., Lawrence, R.: Graph-based rare category detection. In: ICDM 2008, Pisa, Italy, December 15-19, pp. 833–838 (2008)Google Scholar
  8. 8.
    He, J., Tong, H., Carbonell, J.: An effective framework for characterizing rare categories. Frontiers of Computer Science 6(2), 154–165 (2012)zbMATHMathSciNetGoogle Scholar
  9. 9.
    Hospedales, T.M., Gong, S., Xiang, T.: Finding rare classes: Active learning with generative and discriminative models. TKDE 25(2), 374–386 (2013)Google Scholar
  10. 10.
    Huang, H., Chiew, K., Gao, Y., He, Q., Li, Q.: Rare category exploration. Expert Systems with Applications (2014)Google Scholar
  11. 11.
    Huang, H., He, Q., Chiew, K., Qian, F., Ma, L.: Clover: A faster prior-free approach to rare category detection. Knowledge and Information Systems 35(3), 713–736 (2013)CrossRefGoogle Scholar
  12. 12.
    Huang, H., He, Q., He, J., Ma, L.: Radar: Rare category detection via computation of boundary degree. In: Huang, J.Z., Cao, L., Srivastava, J. (eds.) PAKDD 2011, Part II. LNCS (LNAI), vol. 6635, pp. 258–269. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  13. 13.
    Khor, K., Ting, C., Phon-Amnuaisuk, S.: A cascaded classifier approach for improving detection rates on rare attack categories in network intrusion detection. Applied Intelligence 36(2), 320–329 (2012)CrossRefGoogle Scholar
  14. 14.
    Nenadic, Z., Burdick, J.: Spike detection using the continuous wavelet transform. IEEE Transactions on Biomedical Engineering 52(1), 74–87 (2005)CrossRefGoogle Scholar
  15. 15.
    Pelleg, D., Moore, A.: Active learning for anomaly and rare-category detection. In: NIPS 2004, Vancouver, British Columbia, Canada, December 13-18, pp. 1073–1080 (2004)Google Scholar
  16. 16.
    Scott, D.W.: Multivariate Density Estimation: Theory, Practice, and Visualization. Wiley, New York (1992)CrossRefzbMATHGoogle Scholar
  17. 17.
    Vatturi, P., Wong, W.: Category detection using hierarchical mean shift. In: KDD 2009, Paris, France, June 28-July 1, pp. 847–856 (2009)Google Scholar
  18. 18.
    Wand, M.P.: Data-based choice of histogram bin width. The American Statistician 51(1), 59–64 (1997)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Zhenguang Liu
    • 1
  • Hao Huang
    • 2
  • Qinming He
    • 1
  • Kevin Chiew
    • 3
  • Lianhang Ma
    • 1
  1. 1.College of Computer Science and TechnologyZhejiang UniversityHangzhouChina
  2. 2.School of ComputingNational University of SingaporeSingapore
  3. 3.Provident Technology Pte. Ltd.Singapore

Personalised recommendations