Hybrid Algorithms with Instance-Based Classification

  • Iris Hendrickx
  • Antal van den Bosch
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3720)


In this paper we aim to show that instance-based classification can replace the classifier component of a rule learner and of maximum-entropy modeling, thereby improving the generalization accuracy of both algorithms. We describe hybrid algorithms that combine rule learning models and maximum-entropy modeling with instance-based classification. Experimental results show that both hybrids are able to outperform the parent algorithm. We analyze and compare the overlap in errors and the statistical bias and variance of the hybrids, their parent algorithms, and a plain instance-based learner. We observe that the successful hybrid algorithms have a lower statistical bias component in the error than their parent algorithms; the fewer errors they make are also less systematic.


Hybrid Algorithm Rule Learning Parent Algorithm Machine Learning Database Complementary Rate 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Cover, T.M., Hart, P.E.: Nearest neighbor pattern classification. Institute of Electrical and Electronics Engineers Transactions on Information Theory 13, 21–27 (1967)zbMATHGoogle Scholar
  2. 2.
    Aha, D.W., Kibler, D., Albert, M.: Instance-based learning algorithms. Machine Learning 6, 37–66 (1991)Google Scholar
  3. 3.
    Domingos, P.: Unifying instance-based and rule-based induction. Machine Learning 24, 141–168 (1996)Google Scholar
  4. 4.
    Ting, K., Cameron-Jones, R.: Exploring a framework for instance based learning and naive bayesian classifiers. In: Proceedings of the Seventh Australian Joint Conference on Artificial Intelligence, pp. 100–107 (1994)Google Scholar
  5. 5.
    Sebag, M., Schoenauer, M.: A rule-based similarity measure. In: Wess, S., Richter, M., Althoff, K.-D. (eds.) EWCBR 1993. LNCS, vol. 837, pp. 119–130. Springer, Heidelberg (1994)Google Scholar
  6. 6.
    Cost, S., Salzberg, S.: A weighted nearest neighbour algorithm for learning with symbolic features. Machine Learning 10, 57–78 (1993)Google Scholar
  7. 7.
    Wettschereck, D., Aha, D.W., Mohri, T.: A review and comparative evaluation of feature-weighting methods for a class of lazy learning algorithms. Artificial Intelligence Review, special issue on Lazy Learning 11, 273–314 (1997)Google Scholar
  8. 8.
    Daelemans, W., Zavrel, J., Van der Sloot, K., Van den Bosch, A.: TiMBL: Tilburg Memory Based Learner, version 5.1, reference manual. Technical Report ILK-0402, ILK, Tilburg University (2004)Google Scholar
  9. 9.
    Guiasu, S., Shenitzer, A.: The principle of maximum entropy. The Mathematical Intelligencer 7 (1985)Google Scholar
  10. 10.
    Nocedal, J.: Updating quasi-Newton matrices with limited storage. Mathematics of Computation 35, 773–782 (1980)zbMATHMathSciNetCrossRefGoogle Scholar
  11. 11.
    Cohen, W.: Fast effective rule induction. In: Proceedings 12th International Conference on Machine Learning, pp. 115–123 (1995)Google Scholar
  12. 12.
    Hendrickx, I., Van den Bosch, A.: Maximum-entropy parameter estimation for the k-nn modified value-difference kernel. In: Proceedings of the 16th Belgian-Dutch Conference on Artificial Intelligence, pp. 19–26 (2004)Google Scholar
  13. 13.
    Van den Bosch, A.: Feature transformation through rule induction, a case study with the k-nn classifier. In: Proceedings on the workshop on advances in Inductive rule learning at the ECML/PKDD 2004, pp. 1–15 (2004)Google Scholar
  14. 14.
    Blake, C., Merz, C.: UCI repository of machine learning databases (1998),
  15. 15.
    Kohavi, R., John, G.: Wrappers for feature subset selection. Artificial Intelligence Journal 97, 273–324 (1997)zbMATHCrossRefGoogle Scholar
  16. 16.
    Van den Bosch, A.: Wrapped progressive sampling search for optimizing learning algorithm parameters. In: Proceedings of the 16th Belgian-Dutch Conference on Artificial Intelligence, pp. 219–226 (2004)Google Scholar
  17. 17.
    Fayyad, U., Irani, K.: Multi-interval discretization of continuous-valued attributes for classification learning. In: Proceedings of the 13th International Joint Conference on Artificial Intelligence, pp. 1022–1027 (1993)Google Scholar
  18. 18.
    Brill, E., Wu, J.: Classifier combination for improved lexical disambiguation. In: Proceedings of the COLING-ACL 1998, pp. 191–195 (1998)Google Scholar
  19. 19.
    Kohavi, R., Wolpert, D.: Bias plus variance decomposition for zero-one loss functions. In: Proceedings of the thirteenth International Conference on Machine Learning, pp. 275–283 (1996)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Iris Hendrickx
    • 1
  • Antal van den Bosch
    • 1
  1. 1.ILK / Computational Linguistics and AITilburg UniversityTilburgThe Netherlands

Personalised recommendations