Information Retrieval

, Volume 2, Issue 4, pp 303–336 | Cite as

Learning Algorithms for Keyphrase Extraction

  • Peter D. Turney

Abstract

Many academic journals ask their authors to provide a list of about five to fifteen keywords, to appear on the first page of each article. Since these key words are often phrases of two or more words, we prefer to call them keyphrases. There is a wide variety of tasks for which keyphrases are useful, as we discuss in this paper. We approach the problem of automatically extracting keyphrases from text as a supervised learning task. We treat a document as a set of phrases, which the learning algorithm must learn to classify as positive or negative examples of keyphrases. Our first set of experiments applies the C4.5 decision tree induction algorithm to this learning task. We evaluate the performance of nine different configurations of C4.5. The second set of experiments applies the GenEx algorithm to the task. We developed the GenEx algorithm specifically for automatically extracting keyphrases from text. The experimental results support the claim that a custom-designed algorithm (GenEx), incorporating specialized procedural domain knowledge, can generate better keyphrases than a general-purpose algorithm (C4.5). Subjective human evaluation of the keyphrases generated by GenEx suggests that about 80% of the keyphrases are acceptable to human readers. This level of performance should be satisfactory for a wide variety of applications.

machine learning summarization indexing keywords keyphrase extraction 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Brandow R, Mitze K and Rau LR (1995) The automatic condensation of electronic publications by sentence selection. Information Processing and Management, 31(5):675–685.Google Scholar
  2. Breiman L (1996a) Arcing Classifiers. Technical Report 460, University of California at Berkeley, Statistics Department.Google Scholar
  3. Breiman L (1996b) Bagging predictors. Machine Learning, 24(2):123–140.Google Scholar
  4. Buntine W (1989) Stratifying samples to improve learning. In: Proceedings of the IJCAI-89Workshop on Knowledge Discovery in Databases, Detroit, MichiganGoogle Scholar
  5. Carter C and Catlett J (1987) Assessing credit card applications using machine learning. IEEE Expert, Fall issue, 71–79.Google Scholar
  6. Catlett J (1991) Megainduction: Machine learning on very large databases. PhD Dissertation, University of Sydney, Basser Department of Computer Science.Google Scholar
  7. Croft WB, Turtle H and Lewis D (1991) The use of phrases and structured queries in information retrieval. In: SIGIR-91: Proceedings of the 14th Annual InternationalACMSIGIR Conference on Research and Development in Information Retrieval, New York, ACM, pp. 32–45.Google Scholar
  8. Deming WE (1978) Sample surveys: The field. In: William H, Kruskal and Judith M Tanur, Eds., International Encyclopedia of Statistics. Free Press, New York.Google Scholar
  9. Edmundson HP (1969) Newmethods in automatic extracting. Journal of the Association for Computing Machinery, 16(2):264–285.Google Scholar
  10. Fagan JL (1987) Experiments in automatic phrase indexing for document retrieval: A comparison of syntactic and non-syntactic methods. PhD Dissertation, Cornell University, Department of Computer Science, Report #87–868, Ithaca, New York.Google Scholar
  11. Feelders A and Verkooijen W (1995) Which method learns the most from data? Methodological issues in the analysis of comparative studies. In: Fifth International Workshop on Artificial Intelligence and Statistics, Ft. Lauderdale, Florida, pp. 219–225.Google Scholar
  12. Field BJ (1975) Towards automatic indexing: Automatic assignment of controlled-language indexing and classification from free indexing. Journal of Documentation, 31(4):246–265.Google Scholar
  13. Frank E, Paynter GW, Witten IH, Gutwin C and Nevill-Manning CG (1999) Domain-specific keyphrase extraction. In: Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence (IJCAI-99), Morgan Kaufmann, California, pp. 668–673.Google Scholar
  14. Fraser DAS (1976) Probability and Statistics: Theory and Applications. Duxbury Press, Massachusetts.Google Scholar
  15. Freund Y and Schapire RE (1996) Experiments with a newboosting algorithm. In: Machine Learning: Proceedings of the Thirteenth International Conference (ICML-96), Morgan Kaufmann, California, pp. 148–156.Google Scholar
  16. Ginsberg A (1993) A unified approach to automatic indexing and information retrieval. IEEE Expert,8:46–56.Google Scholar
  17. Grefenstette JJ (1983) A user's guide to GENESIS. Technical Report CS-83–11, Vanderbilt University, Computer Science Department.Google Scholar
  18. Grefenstette JJ (1986) Optimization of control parameters for genetic algorithms. IEEE Transactions on Systems, Man, and Cybernetics, 16:122–128.Google Scholar
  19. Gutwin C, Paynter GW, Witten IH, Nevill-Manning CG and Frank E (1999) Improving browsing in digital libraries with keyphrase indexes. Decision Support Systems, 27:81–104.Google Scholar
  20. Jang D-H and Myaeng SH (1997) Development of a document summarization system for effective information services. In: RIAO 97 Conference Proceedings: Computer-Assisted Information Searching on Internet, Montreal, Canada, pp. 101–111.Google Scholar
  21. Johnson FC, Paice CD, Black WJ and Neal AP (1993) The application of linguistic processing to automatic abstract generation. Journal of Document and Text Management, 1:215–241.Google Scholar
  22. Krovetz R (1993) Viewing morphology as an inference process. In: Proceedings of the Sixteenth Annual InternationalACMSIGIR Conference on Research and Development in Information Retrieval, SIGIR'93, pp. 191–203.Google Scholar
  23. Krulwich B and Burkey C (1996) Learning user information interests through the extraction of semantically significant phrases. In: Hearst M and Hirsh H, Eds., AAAI 1996 Spring Symposium on Machine Learning in Information Access. AAAI Press, California.Google Scholar
  24. Krupka G (1995) SRA: Description of the SRA system as used for MUC-6. In: Proceedings of the Sixth Message Understanding Conference, Morgan Kaufmann, California.Google Scholar
  25. Kubat M, Holte R and Matwin S (1998) Machine learning for the detection of oil spills in satellite radar images. Machine Learning, 30(2/3):195–215.Google Scholar
  26. Kupiec J, Pedersen J and Chen F (1995) A trainable document summarizer. In: Fox EA, Ingwersen P and Fidel R, Eds., In: SIGIR-95: Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, New York, pp. 68–73.Google Scholar
  27. Leung C-H and Kan W-K (1997) A statistical learning approach to automatic indexing of controlled index terms. Journal of the American Society for Information Science, 48(1):55–66.Google Scholar
  28. Lovins JB (1968) Development of a stemming algorithm. Mechanical Translation and Computational Linguistics, 11:22–31.Google Scholar
  29. Luhn HP (1958) The automatic creation of literature abstracts. I.B.M. Journal of Research and Development, 2(2):159–165.Google Scholar
  30. Maclin R and Opitz D (1997) An empirical evaluation of bagging and boosting. In: Proceedings of the Fourteenth National Conference on Artificial Intelligence (AAAI-97), AAAI Press, pp. 546–551.Google Scholar
  31. Marsh E, Hamburger H and Grishman R (1984) A production rule system for message summarization. In: AAAI-84, Proceedings of the American Association for Artificial Intelligence, AAAI Press/MIT Press, Cambridge, MA, pp. 243–246.Google Scholar
  32. Mathieu J (1999) Adaptation of a keyphrase extractor for Japanese text. In: Proceedings of the 27th Annual Conference of the Canadian Association for Information Science (CAIS-99), Sherbrooke, Quebec, pp. 182–189.Google Scholar
  33. MUC-3 (1991) In: Proceedings of the Third Message Understanding Conference, Morgan Kaufmann, California.Google Scholar
  34. MUC-4 (1992) In: Proceedings of the Fourth Message Understanding Conference, Morgan Kaufmann, California.Google Scholar
  35. MUC-5 (1993) In: Proceedings of the Fifth Message Understanding Conference, Morgan Kaufmann, California.Google Scholar
  36. MUC-6 (1995) In: Proceedings of the Sixth Message Understanding Conference, Morgan Kaufmann, California.Google Scholar
  37. MuQnoz A (1996) Compound key word generation from document databases using a hierarchical clustering ART model. Intelligent Data Analysis, 1(1): Elsevier, Amsterdam.Google Scholar
  38. Nakagawa H (1997) Extraction of index words from manuals. In: RIAO 97 Conference Proceedings: Computer-Assisted Information Searching on Internet, Montreal, Canada, pp. 598–611.Google Scholar
  39. Paice CD (1990) Constructing literature abstracts by computer: Techniques and prospects. Information Processing and Management, 26(1):171–186.Google Scholar
  40. Paice CD and Jones PA (1993) The identification of important concepts in highly structured technical papers. In: SIGIR-93: Proceedings of the 16th Annual International ACMSIGIR Conference on Research and Development in Information Retrieval, ACM, New York, pp. 69–78.Google Scholar
  41. Porter MF (1980) An algorithm for suffix stripping. Program; Automated Library and Information Systems, 14(3):130–137.Google Scholar
  42. Quinlan JR (1987) Decision trees as probabilistic classifiers. In: P Langley, Ed., Proceedings of the Fourth International Workshop on Machine Learning, Morgan Kaufmann, California, pp. 31–37.Google Scholar
  43. Quinlan JR (1990) Probabilistic decision trees. In: Kodratoff Y and Michalski RS, Eds., Machine Learning: An Artificial Intelligence Approach, Volume III, Morgan Kaufmann, California, pp. 140–152.Google Scholar
  44. Quinlan JR (1993) C4.5: Programs for Machine Learning, Morgan Kaufmann, California.Google Scholar
  45. Quinlan JR (1996) Bagging, boosting, and C4.5. In: Proceedings of the Thirteenth National Conference on Artificial Intelligence (AAAI-96), AAAI Press, pp. 725–730.Google Scholar
  46. Salton G (1988) Syntactic approaches to automatic book indexing. In: Proceedings of the 26th Annual Meeting of the Association for Computational Linguistics, ACM, New York, pp. 120–138.Google Scholar
  47. Salton G, Allan J, Buckley C and Singhal A (1994) Automatic analysis, theme generation, and summarization of machine-readable texts. Science, 264:1421–1426.Google Scholar
  48. Soderland S and Lehnert W(1994) Wrap-Up: A trainable discourse module for information extraction. Journal of Artificial Intelligence Research, 2:131–158.Google Scholar
  49. Sparck Jones K (1973) Does indexing exhaustivity matter? Journal of the American Society for Information Science, September-October, 313–316.Google Scholar
  50. Steier AM and Belew RK (1993) Exporting phrases: A statistical analysis of topical language. In: R Casey and B Croft, Eds., Second Symposium on Document Analysis and Information Retrieval, pp. 179–190.Google Scholar
  51. Turney PD (1997) Extraction of keyphrases from text: Evaluation of four algorithms. National Research Council, Institute for Information Technology, Technical Report ERB-1051.Google Scholar
  52. Turney PD (1999) Learning to extract keyphrases from text. National Research Council, Institute for Information Technology, Technical Report ERB-1057.Google Scholar
  53. Whitley D (1989) The GENITOR algorithm and selective pressure. In: Proceedings of the Third International Conference on Genetic Algorithms (ICGA-89), Morgan Kaufmann, California, pp. 116–121.Google Scholar

Copyright information

© Kluwer Academic Publishers 2000

Authors and Affiliations

  • Peter D. Turney
    • 1
  1. 1.National Research Council of CanadaInstitute for Information TechnologyOttawaCanada

Personalised recommendations