Abstract
In this paper, the concept of finding an appropriate classifier ensemble for named entity recognition is posed as a multiobjective optimization (MOO) problem. Our underlying assumption is that instead of searching for the best-fitting feature set for a particular classifier, ensembling of several classifiers those are trained using different feature representations could be a more fruitful approach, but it is crucial to determine the appropriate subset of classifiers that are most suitable for the ensemble. We use three heterogenous classifiers namely maximum entropy, conditional random field, and support vector machine in order to build a number of models depending upon the various representations of the available features. The proposed MOO-based ensemble technique is evaluated for three resource-constrained languages, namely Bengali, Hindi, and Telugu. Evaluation results yield the recall, precision, and F-measure values of 92.21, 92.72, and 92.46%, respectively, for Bengali; 97.07, 89.63, and 93.20%, respectively, for Hindi; and 80.79, 93.18, and 86.54%, respectively, for Telugu. We also evaluate our proposed technique with the CoNLL-2003 shared task English data sets that yield the recall, precision, and F-measure values of 89.72, 89.84, and 89.78%, respectively. Experimental results show that the classifier ensemble identified by our proposed MOO-based approach outperforms all the individual classifiers, two different conventional baseline ensembles, and the classifier ensemble identified by a single objective–based approach. In a part of the paper, we formulate the problem of feature selection in any classifier under the MOO framework and show that our proposed classifier ensemble attains superior performance to it.
Similar content being viewed by others
References
Cunningham H.: GATE, a general architecture for text engineering. Comput. Human. 36, 223–254 (2002)
Babych, B., Hartley, A.: Improving machine translation quality with automatic named entity recognition. In: Proceedings of EAMT/EACL 2003 Workshop on MT and other Language Technology Tools, pp. 1–8 (2003)
Moldovan, D., Harabagiu, S., Girju, R., Morarescu, P., Lacatusu, F., Novischi, A., Badulescu, A., Bolohan, O.: LCC tools for question answering. In: Text REtrieval Conference (TREC) (2002)
Nobata, C., Sekine, S., Isahara, H., Grishman, R.: Summarization system integrated with named entity tagging and IE pattern discovery. In: Proceedings of Third International Conference on Language Resources and Evaluation (LREC 2002), Spain (2002)
Humphreys, K., Gaizauskas, R., Azzam, S., Huyck, C., Mitchell, B., Cunnigham, H., Wilks, Y.: University of sheffield: description of the LaSIE-II system as used for MUC-7. In: MUC-7, Fairfax, Virginia (1998)
Aone, C., Halverson, L., Hampton, T., Ramos-Santacruz, M.: SRA: description of the IE2 system used for MUC-7. In: MUC-7, Fairfax, Virginia (1998)
Mikheev, A., Grover, C., Moens, M.: Description of the LTG system used for MUC-7. In: MUC-7, Fairfax, Virginia (1998)
Mikheev, A., Grover, C., Moens, M.: Named entity recognition without gazeteers. In: Proceedings of EACL, Bergen, Norway, pp. 1–8 (1999)
Miller, S., Crystal, M., Fox, H., Ramshaw, L., Schawartz, R., Stone, R., Weischedel, R., The Annotation~Group: BBN: Description of the SIFT system as used for MUC-7. In: MUC-7, Fairfax, Virginia (1998)
Bikel D.M., Schwartz R.L., Weischedel R.M.: An algorithm that learns what’s in a name. Mach. Learn. 34(1–3), 211–231 (1999)
Borthwick, A.: Maximum entropy approach to named entity recognition. PhD thesis, New York University (1999)
Borthwick, A., Sterling, J., Agichtein, E., Grishman, R.: NYU: description of the MENE named entity system as used in MUC-7. In: MUC-7, Fairfax (1998)
Sekine, S.: Description of the Japanese NE system used for MET-2. In: MUC-7, Fairfax, Virginia (1998)
McCallum, A., Li, W.: Early results for named entity recognition with conditional random fields, Feature Induction and Web-enhanced Lexicons. In: Proceedings of CoNLL, Canada, pp. 188–191 (2003)
Lafferty, J.D., McCallum, A., Pereira, F.C.N.: Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: ICML, pp. 282–289 (2001)
Collins, M., Singer, Y.: Unsupervised models for named entity classification. In: Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora (1999)
Yangarber, R., Lin, W., Grishman, R.: Unsupervised learning of generalized names. In: Proceedings of the 19th International Conference on Computational Linguistics (COLING-2002), pp. 1–7 (2002)
Miller, S., Guinness, J., Zamanian, A.: Name tagging with word clusters and discriminative training. In: Proceedings of HLT-NAACL, USA (2004)
Ji, H., Grishman, R.: Data selection in semi-supervised learning for name tagging. In: Proceedings of the Workshop on Information Extraction Beyond The Document, COLING/ACL, Sydney (2006)
Alfonseca, E., Manandhar, S.: An unsupervised method for general named entity recognition and automated concept discovery. In: Proceedings AAAI ’99/IAAI ’99: Proceedings of the Sixteenth National Conference on Artificial Intelligence and the Eleventh Conference on Innovative Applications of Artificial Intelligence, pp. 474–479 (1999)
Shinyama, Y., Sekine, S.: Named entity discovery using comparable news articles. In: Proceedings of the International Conference on Computational Linguistics (COLING), Switzerland, pp. 848–855 (2004)
Etzioni, O., Cafarrella, M., Downey, D., Popescu, A.M., Shaked, T., Soderland, S., Weld, D.S., Yates, A.: Unsupervised named entity extraction from the web: an experimental study. Artif. Intell. 165:91–134 (2005)
Yu, X.: Chinese named entity recognition with cascaded hybrid model. In: Proceedings of NAACL HLT 2007, Prague, pp. 197–200 (2007)
Srihari, R., Niu, C., Li, W.: A hybrid approach for named entity and sub-type tagging. In: Proceedings of Sixth Conference on Applied Natural Language Processing (ANLP), pp. 247–254 (2002)
Ekbal, A., Bandyopadhyay, S.: Lexical pattern learning from corpus data for named entity recognition. In: Proceedings of the 5th International Conference on Natural Language Processing (ICON), India, pp. 123–128 (2002)
Ekbal A., Naskar S., Bandyopadhyay S.: Named entity recognition and transliteration in Bengali (Special Issue of Lingvisticae Investigationes Journal). Named Entities Recogn. Classif. Use 30(1), 95–114 (2007)
Ekbal, A., Haque, R., Bandyopadhyay, S.: Named entity recognition in Bengali: a conditional random field approach. In: Proceedings of the 3rd International Joint Conference on Natural Language Processing (IJCNLP 2008), pp. 589–594 (2008)
Ekbal, A., Bandyopadhyay, S.: Bengali named entity recognition using support vector machine. In: Proceedings of Workshop on NER for South and South East Asian Languages, 3rd International Joint Conference on Natural Language Processing (IJCNLP), India, pp. 51–58 (2008)
Ekbal, A., Bandyopadhyay, S.: Voted NER system using appropriate unlabeled data. In: Proceedings of the 2009 Named Entities Workshop: Shared Task on Transliteration (NEWS 2009), ACL-IJCNLP 2009, pp. 202–210 (2009)
Li W., McCallum A.: Rapid development of Hindi named entity recognition using conditional random fields and feature induction. ACM Trans. Asian Lang. Inf. Process. 2(3), 290–294 (2004)
Patel, A., Ramakrishnan, G., Bhattacharya, P.: Relational Learning Assisted Construction of Rule Base for Indian Language NER. In: Proceedings of ICON 2009: 7th International Conference on Natural Language Processing, India (2009)
Saha, S., Sarkar, S., Mitra, P.: A hybrid feature set based maximum entropy Hindi named entity recognition. In: Proceedings of the 3rd International Joint Conference in Natural Langauge Processing (IJCNLP 2008), pp. 343–350 (2008)
Gali, K., Sharma, H., Vaidya, A., Shisthla, P., Sharma, D.M.: Aggregrating machine learning and rule-based heuristics for named entity recognition. In: Proceedings of the IJCNLP-08 Workshop on NER for South and South East Asian Languages, pp. 25–32 (2008)
Srikanth, P., Murthy, K.N.: Named entity recognition for Telugu. In: Proceedings of the IJCNLP-08 Workshop on NER for South and South East Asian Languages, pp. 41–50 (2008)
Shishtla, P.M., Pingali, P., Varma, V.: A character n-gram based approach for improved recall in Indian language NER. In: Proceedings of the IJCNLP-08 Workshop on NER for South and South East Asian Languages, pp. 101–108 (2008)
Goldberg D.E.: Genetic Algorithms in Search, Optimization and Machine Learning. Addison-Wesley, New York (1989)
Deb K.: Multi-objective Optimization Using Evolutionary Algorithms. Wiley, England (2001)
Coello~Coello C.A.: A comprehensive survey of evolutionary-based multiobjective optimization techniques. Knowl. Inf. Syst. 1(3), 269–308 (1999)
Veldhuizen D.V., Lamont G.: Multiobjective evolutionary algorithms: analyzing the state-of-the-artn. Evol. Comput. 2, 125–1473 (2000)
Florian, R., Ittycheriah, A., Jing, H., Zhang, T.: Named entity recognition through classifier combination. In: Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003 (2003)
Tjong Kim~Sang, E.F., De~Meulder, F.: Introduction to the Conll-2003 shared task: language independent named entity recognition. In: Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pp. 142–147 (2003)
Deb K., Pratap A., Agarwal S., Meyarivan T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 181–197 (2002)
Ekbal A., Bandyopadhyay S.: A web-based Bengali news corpus for named entity recognition. Lang. Resour. Evaluat. J. 42(2), 173–182 (2008)
Asif Ekbal., Sriparna Saha.: Classifier Ensemble Selection Using genetic algorithm for named entity Recognition. Res Lang and Comput. 8(1), 73–99 (2010)
Darroch J., Ratcliff D.: Generalized iterative scaling for log-linear models. Ann. Math. Stat. 43, 1470–1480 (1972)
Sha, F., Pereira, F.: Shallow parsing with conditional random fields. In: Proceedings of NAACL ’03, Canada, pp. 134–141 (2003)
Pinto, D., McCallum, A., Wei, X., Croft, W.B.: Table extraction using conditional random fields. In: Proceedings of SIGIR’03 Conference (2003)
Vapnik V.N.: The Nature of Statistical Learning Theory. Springer, New York (1995)
Joachims, T.: Making Large Scale SVM Learning Practical. MIT Press, Cambridge, pp. 169–184 (1999)
Taira, H., Haruno, M.: Feature selection in SVM text categorization. In: Proceedings of AAAI-99 (1999)
Bandyopadhyay S., Pal S.K., Aruna B.: Multi-objective GAs, quantitative indices and pattern classification. IEEE Trans. Syst. Man Cybern. B 34(5), 2088–2099 (2004)
Anderson T.W., Scolve S.: Introduction to the Statistical Analysis of Data. Houghton Mifflin, Boston (1978)
Lin, D., Wu, X.: Phrase clustering for discriminative learning. In: Proceedings of 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pp. 1030–1038 (2009)
Suzuki, J., Isozaki, H.: Semi-supervised sequential labeling and segmentation using Gigaword scale unlabeled data. In: Proceedings of ACL/HLT-08, pp. 665–673 (2008)
Chieu, H.L., Ng, H.T.: Named entity recognition with a maximum entropy approach. In: Proceedings of CoNLL-2003, HLT-NAACL 2003, pp. 160–163 (2003)
Dekai~Wu, G.N., Carput, M.: A stacked, voted, stacked model for named entity recognition. In: Proceedings of the CoNLL-2003, HLT-NAACL (2003)
Klein, D., Smarr, J., Nguyen, H.N., Manning, C.D.: Named entity recognition with character-level models. In: Proceedings of CoNLL-2003, HLT-NAACL 2003, pp. 188–191 (2003)
Singh, A.K.: Named entity recognition for South and South East Asian languages: taking stock. In: Proceedings of the IJCNLP-08 Workshop on NER for South and South East Asian Languages, IJCNLP-08, India (2008)
Author information
Authors and Affiliations
Corresponding author
Additional information
Both authors equally contributed for this work.
Rights and permissions
About this article
Cite this article
Ekbal, A., Saha, S. Multiobjective optimization for classifier ensemble and feature selection: an application to named entity recognition. IJDAR 15, 143–166 (2012). https://doi.org/10.1007/s10032-011-0155-7
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10032-011-0155-7