A Density-Based Re-ranking Technique for Active Learning for Data Annotations

  • Jingbo Zhu
  • Huizhen Wang
  • Benjamin K. Tsou
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5459)

Abstract

One of the popular techniques of active learning for data annotations is uncertainty sampling, however, which often presents problems when outliers are selected. To solve this problem, this paper proposes a density-based re-ranking technique, in which a density measure is adopted to determine whether an unlabeled example is an outlier. The motivation of this study is to prefer not only the most informative example in terms of uncertainty measure, but also the most representative example in terms of density measure. Experimental results of active learning for word sense disambiguation and text classification tasks using six real-world evaluation data sets show that our proposed density-based re-ranking technique can improve uncertainty sampling.

Keywords

active learning uncertainty sampling density-based re-ranking data annotation text classification word sense disambiguation 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Eduard, H., Marcus, M., Palmer, M., Ramshaw, L., Weischedel, R.: Ontonotes: The 90% Solution. In: Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, poster session (2006)Google Scholar
  2. 2.
    David, C., Atlas, L., Ladner, R.: Improving generalization with active learning. Machine Learning 15(2), 201–221 (1994)Google Scholar
  3. 3.
    Seung, H.S., Opper, M., Sompolinsky, H.: Query by committee. In: Proceedings of the Fifth Annual ACM Conference on Computational Learning Theory, pp. 287–294. ACM Press, New York (1992)Google Scholar
  4. 4.
    Lewis David, D., Gale, W.A.: A sequential algorithm for training text classifiers. In: Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 3–12 (1994)Google Scholar
  5. 5.
    Jinying, C., Schein, A., Ungar, L., Palmer, M.: An empirical study of the behavior of active learning for word sense disambiguation. In: Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, pp. 120–127 (2006)Google Scholar
  6. 6.
    Seng, C.Y., Ng, H.T.: Domain adaptation with active learning for word sense disambiguation. In: Proceedings of the 45th annual meeting on Association for Computational Linguistics, pp. 49–56 (2007)Google Scholar
  7. 7.
    Jingbo, Z., Wang, H., Yao, T., Tsou, B.: Active Learning with Sampling by Uncertainty and Density for Word Sense Disambiguation and Text Classification. In: Proceedings of the 22nd International Conference on Computational Linguistics, pp. 1137–1144 (2008)Google Scholar
  8. 8.
    Min, T., Luo, X., Roukos, S.: Active learning for statistical natural language parsing. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 120–127 (2002)Google Scholar
  9. 9.
    Dan, S., Zhang, J., Su, J., Zhou, G., Tan, C.-L.: Multi-criteria-based active learning for named entity recognition. In: Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics (2004)Google Scholar
  10. 10.
    Nicholas, R., McCallum, A.: Toward optimal active learning through sampling estimation of error reduction. In: Proceedings of the Eighteenth International Conference on Machine Learning, pp. 441–448 (2001)Google Scholar
  11. 11.
    Cohn David, A., Ghahramani, Z., Jordan, M.I.: Active learning with statistical models. Journal of Artificial Intelligence Research 4, 129–145 (1996)Google Scholar
  12. 12.
    Jingbo, Z., Hovy, E.: Active learning for word sense disambiguation with methods for addressing the class imbalance problem. In: Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 783–790 (2007)Google Scholar
  13. 13.
    Ying, Z., Hildebrand, A.S., Vogel, S.: Distributed language modeling for N-best list reranking. In: Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pp. 216–223 (2006)Google Scholar
  14. 14.
    Michael, C., Koo, T.: Discriminative reranking for natural language parsing. In: Proceedings of 17th International Conference on Machine Learning, pp. 175–182 (2000)Google Scholar
  15. 15.
    Eduard, H., Lin, C.-Y.: Automated Text Summarization in SUMMARIST. In: Maybury, M., Mani, I. (eds.) Advances in Automatic Text Summarization. MIT Press, Cambridge (1998)Google Scholar
  16. 16.
    Schein Andrew, I., Ungar, L.H.: Active learning for logistic regression: an evaluation. Machine Learning 68(3), 235–265 (2007)CrossRefGoogle Scholar
  17. 17.
    Tou, N.H., Lee, H.B.: Integrating multiple knowledge sources to disambiguate word sense: an exemplar-based approach. In: Proceedings of the Thirty-Fourth Annual Meeting of the Association for Computational Linguistics, pp. 40–47 (1996)Google Scholar
  18. 18.
    Claudia, L., Towell, G., Voorhees, E.: Corpus-Based Statistical Sense Resolution. In: Proceedings of the ARPA Workshop on Human Language Technology, pp. 260–265 (1993)Google Scholar
  19. 19.
    Andrew, P., Hovy, E., Pantel, P.: The Omega Ontology. In: Proceedings of OntoLex 2005 - Ontologies and Lexical Resources, pp. 59–66 (2005)Google Scholar
  20. 20.
    Andrew, M., Nigam, K.: A comparison of event models for naïve bayes text classification. In: AAAI 1998 workshop on learning for text categorization (1998)Google Scholar
  21. 21.
    Berger Adam, L., Vincent, J., Della Pietra, S.A.D.: A maximum entropy approach to natural language processing. Computational Linguistics 22(1), 39–71 (1996)Google Scholar
  22. 22.
    Keok, L.Y., Ng, H.T.: An empirical evaluation of knowledge sources and learning algorithm for word sense disambiguation. In: Proceedings of the ACL 2002 conference on Empirical methods in natural language processing, pp. 41–48 (2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Jingbo Zhu
    • 1
  • Huizhen Wang
    • 1
  • Benjamin K. Tsou
    • 2
  1. 1.Natural Language Processing LaboratoryNortheastern UniversityShenyangP.R. China
  2. 2.Language Information Sciences Research CentreCity University of Hong KongHong Kong

Personalised recommendations