Improving SVM Text Classification Performance through Threshold Adjustment

  • James G. Shanahan
  • Norbert Roma
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2837)

Abstract

In general, support vector machines (SVM), when applied to text classification provide excellent precision, but poor recall. One means of customizing SVMs to improve recall, is to adjust the threshold associated with an SVM. We describe an automatic process for adjusting the thresholds of generic SVM which incorporates a user utility model, an integral part of an information management system. By using thresholds based on utility models and the ranking properties of classifiers, it is possible to overcome the precision bias of SVMs and insure robust performance in recall across a wide variety of topics, even when training data are sparse. Evaluations on TREC data show that our proposed threshold adjusting algorithm boosts the performance of baseline SVMs by at least 20% for standard information retrieval measures.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Arampatzis, A., Unbiased, S.-D.: Threshold Optimization, Initial Query Degradation, Decay, and Incrementality, for Adaptive Document Filtering. In: Tenth Text Retrieval Conference (TREC 2001), pp. 596–605 (2002)Google Scholar
  2. 2.
    Ault, T., Yang, Y.: kNN, Rocchio and Metrics for Information Filtering at TREC-10. In: Tenth Text Retrieval Conference (TREC 2001), pp. 84–93 (2002)Google Scholar
  3. 3.
    Cancedda, N., et al.: Kernel Methods for Document Filtering. In: Eleventh Text Retrieval Conference, TREC 2002 (2003)Google Scholar
  4. 4.
    Cortes, C., Vapnik, V.: Support vector networks. Machine Learning 20, 273–297 (1995)MATHGoogle Scholar
  5. 5.
    Evans, D.A., Shanahan, J., Tong, X., Roma, N., Stoica, E., Sheftel, V., Montgomery, J., Bennett, J., Fujita, S., Grefenstette, G.: Topic Specific Optimization and Structuring. In: Tenth Text Retrieval Conference (TREC 2001), pp. 132–141 (2002)Google Scholar
  6. 6.
    Joachims, T.: Text categorization with support vector machines: Learning with many relevant features. In: Nédellec, C., Rouveirol, C. (eds.) ECML 1998. LNCS, vol. 1398, Springer, Heidelberg (1998)Google Scholar
  7. 7.
    Keerthi, S.S., Shevade, S.K., Bhattacharyya, C., Murthy, K.R.K.: Improvements to Platt’s SMO algorithm for SVM classifier design. Technical report, Dept of CSA, IISc, Bangalore, India (1999)Google Scholar
  8. 8.
    LeCun, Y., Jackel, L.D., Bottou, L., Cortes, C., Denker, J.S., Drucker, H., Guyon, I., Muller, U.A., Sackinger, E., Simard, P., Vapnik, V.: Learning algorithms for classification: A comparison on handwritten digit recognition. Neural Networks: The Statistical Mechanics Perspective, 261–276 (1995)Google Scholar
  9. 9.
    Lewis, D.D., Schapire, R.E., Callan, J.P., Papka, R.: Training algorithms for linear text classifiers. In: Int’l ACM Conf. on Research and Development in Information Retrieval (SIGIR 1996), pp. 298-306 (1996)Google Scholar
  10. 10.
    Lewis, D.D.: The Reuters-21578 text categorization test collection, Checked on 11 May 1998, http://www.research.att.com/~lewis/reuters21578.html; Timestamp Tue Jan 20 21:07:21 EST (1998)
  11. 11.
    Lewis, D.D.: Applying Support Vector Machines to the TREC-2001 Batch Filtering and Routing Tasks. In: Tenth Text Retrieval Conference (TREC 2001), pp. 286–294 (2002)Google Scholar
  12. 12.
    Li, Y., Zaragoza, H., Herbrich, R., Shawe-Taylor, J., Kandola, J.S.: The Perceptron Algorithm with Uneven Margins. In: ICML 2002, pp. 379–386 (2002)Google Scholar
  13. 13.
    Mayfield, J., McNamee, P., Costello, C., Piatko, C., Banerjee, A.: JHU/APL at TREC 2001: Experiments in Filtering and in Arabic, Video, and Web Retrieval, at TREC-10. In: Tenth Text Retrieval Conference (TREC 2001), pp. 322–332 (2002)Google Scholar
  14. 14.
    Morik, K., Brockhausen, P., Joachims, T.: Combining statistical learning with a knowledge-based approach - A case study in intensive care monitoring. In: Proc. 16th Int’l Conf. on Machine Learning, ICML 1999 (1999)Google Scholar
  15. 15.
    Osuna, E., Freund, R., Girosi, F.: Training support vector machines: An application to face detection. In: Proceedings of Computer Vision and Pattern Recognition 1997, pp. 130–136 (1997)Google Scholar
  16. 16.
    Platt, J.: Fast training of SVMs using sequential minimal optimization. In: Scholkopf, B., Burges, C., Smola, A. (eds.) Advances in Kernel Methods – Support Vector Learning, MIT Press, Cambridge (1998)Google Scholar
  17. 17.
    Robertson, S., Soboroff, I.: The TREC, Filtering Track Report. In: Tenth Text Retrieval Conference (TREC 2001), pp. 26–37 (2001)Google Scholar
  18. 18.
    Salton, G.: Introduction to Modern Information Retrieval. McGraw Hill, New York (1983)MATHGoogle Scholar
  19. 19.
    Vapnik, V.: The Nature of Statistical Learning Theory. Springer, Heidelberg (1995)MATHGoogle Scholar
  20. 20.
    Vapnik, V.: Statistical Learning Theory. Wiley, Chichester (1998)MATHGoogle Scholar
  21. 21.
    Voorhees, E.M.: Overview of TREC. In: Eleventh Text Retrieval Conference, TREC 2002, pp. 1–16 (2002)Google Scholar
  22. 22.
    Yang, Y.: A study on thresholding strategies for text categorization. In: Proceedings of ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2001, pp. 137-145 (2001)Google Scholar
  23. 23.
    Zhai, C., Jansen, P., Stoica, E., Grot, N., Evans, D.A.: Threshold Calibration in CLARIT Adaptive Filtering. In: Seventh Text Retrieval Conference (TREC-7), pp. 149–156 (1999)Google Scholar
  24. 24.
    Zhang, Y., Callan, J.: YFilter at TREC-9. In: Proceedings of the Ninth Text Retrieval Conference (TREC-9), pp. 135–140, 500-249. National Institute of Standards and Technology (2001)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • James G. Shanahan
    • 1
  • Norbert Roma
    • 1
  1. 1.Clairvoyance CorporationPittsburghUSA

Personalised recommendations