Advertisement

Analogy-Based Reasoning in Classifier Construction

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3700)

Abstract

Analogy-based reasoning methods in machine learning make it possible to reason about properties of objects on the basis of similarities between objects. A specific similarity based method is the k nearest neighbors (k-nn) classification algorithm. In the k-nn algorithm, a decision about a new object x is inferred on the basis of a fixed number k of the objects most similar to x in a given set of examples. The primary contribution of the dissertation is the introduction of two new classification models based on the k-nn algorithm.

The first model is a hybrid combination of the k-nn algorithm with rule induction. The proposed combination uses minimal consistent rules defined by local reducts of a set of examples. To make this combination possible the model of minimal consistent rules is generalized to a metric-dependent form. An effective polynomial algorithm implementing the classification model based on minimal consistent rules has been proposed by Bazan. We modify this algorithm in such a way that after addition of the modified algorithm to the k-nn algorithm the increase of the computation time is inconsiderable. For some tested classification problems the combined model was significantly more accurate than the classical k-nn classification algorithm.

For many real-life problems it is impossible to induce relevant global mathematical models from available sets of examples. The second model proposed in the dissertation is a method for dealing with such sets based on locally induced metrics. This method adapts the notion of similarity to the properties of a given test object. It makes it possible to select the correct decision in specific fragments of the space of objects. The method with local metrics improved significantly the classification accuracy of methods with global models in the hardest tested problems.

The important issues of quality and efficiency of the k-nn based methods are a similarity measure and the performance time in searching for the most similar objects in a given set of examples, respectively. In this dissertation both issues are studied in detail and some significant improvements are proposed for the similarity measures and for the search methods found in the literature.

Keywords

analogy-based reasoning case-based reasoning k nearest neighbors similarity measure distance based indexing hybrid decision system local metric induction 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Aggarwal, C.C., Hinneburg, A., Keim, D.A.: On the surprising behaviour of distance metrics in high dimensional space. In: Proceedings of the Eighth Internatinal Conference on Database Theory, London, UK, pp. 420–434 (2001)Google Scholar
  2. 2.
    Aha, D.W.: Tolerating noisy, irrelevant and novel attributes in instance-based learning algorithms. International Journal of Man-Machine Studies 36, 267–287 (1992)CrossRefGoogle Scholar
  3. 3.
    Aha, D.W.: The omnipresence of case-based reasoning in science and applications. Knowledge-Based Systems 11(5-6), 261–273 (1998)CrossRefGoogle Scholar
  4. 4.
    Aha, D.W., Kibler, D., Albert, M.K.: Instance-based learning algorithms. Machine Learning 6, 37–66 (1991)Google Scholar
  5. 5.
    Ajdukiewicz, K.: Logika Pragmatyczna. PWN, Warszawa (1974)Google Scholar
  6. 6.
    Bazan, J.G.: Discovery of decision rules by matching new objects against data tables. In: Polkowski, L., Skowron, A. (eds.) RSCTC 1998. LNCS (LNAI), vol. 1424, pp. 521–528. Springer, Heidelberg (1998)CrossRefGoogle Scholar
  7. 7.
    Bazan, J.G., Szczuka, M.: RSES and RSESlib - a collection of tools for rough set computations. In: Ziarko, W.P., Yao, Y. (eds.) RSCTC 2000. LNCS (LNAI), vol. 2005, pp. 106–113. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  8. 8.
    Bazan, J.G., Szczuka, M., Wojna, A.G., Wojnarski, M.: On the evolution of Rough Set Exploration System. In: Tsumoto, S., Słowiński, R., Komorowski, J., Grzymała-Busse, J.W. (eds.) RSCTC 2004. LNCS (LNAI), vol. 3066, pp. 592–601. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  9. 9.
    Beckmann, N., Kriegel, H.P., Schneider, R., Seeger, B.: The R  ⋆ -tree: an efficient and robust access method for points and rectangles. In: Proceedings of the 1990 ACM SIGMOD International Conference on Management of Data, Atlantic City, NJ, pp. 322–331 (1990)Google Scholar
  10. 10.
    Bellman, R.E.: Dynamic Programming. Princeton University Press, Princeton (1957)zbMATHGoogle Scholar
  11. 11.
    Bentley, J.L.: Multidimensional binary search trees used for associative searching. Communications of the ACM 18(9), 509–517 (1975)zbMATHCrossRefMathSciNetGoogle Scholar
  12. 12.
    Berchtold, S., Keim, D., Kriegel, H.P.: The X-tree: an index structure for high dimensional data. In: Proceedings of the Twenty Second International Conference on Very Large Databases, pp. 28–39 (1996)Google Scholar
  13. 13.
    Beyer, K.S., Goldstein, J., Ramakrishnan, R., Shaft, U.: When is “nearest neighbor” meaningful? In: Proceedings of the Seventh International Conference on Database Theory, Jerusalem, Israel, pp. 217–235 (1999)Google Scholar
  14. 14.
    Biberman, Y.: A context similarity measure. In: Proceedings of the Ninth European Conference on Machine Learning, Catania, Italy, pp. 49–63 (1994)Google Scholar
  15. 15.
    Bishop, C.M.: Neural Networks for Pattern Recognition. Oxford University Press, Oxford (1996)zbMATHGoogle Scholar
  16. 16.
    Blake, C.L., Merz, C.J.: UCI repository of machine learning databases. Department of Information and Computer Science. University of California, Irvine (1998), http://www.ics.uci.edu/~mlearn/MLRepository.html
  17. 17.
    Breiman, L.: Statistical modeling - the two cultures. Statistical Science 16(3), 199–231 (2001)zbMATHCrossRefMathSciNetGoogle Scholar
  18. 18.
    Brin, S.: Near neighbor search in large metric spaces. In: Proceedings of the Twenty First International Conference on Very Large Databases, pp. 574–584 (1995)Google Scholar
  19. 19.
    Chavez, E., Navarro, G., Baeza-Yates, R., Marroquin, J.L.: Searching in metric spaces. Technical Report TR/DCC-99-3, Department of Computer Science. University of Chile (1999)Google Scholar
  20. 20.
    Ciaccia, P., Patella, M., Zezula, P.: M-tree: an efficient access method for similarity search in metric spaces. In: Proceedings of the Twenty Third International Conference on Very Large Databases, pp. 426–435 (1997)Google Scholar
  21. 21.
    Clark, P., Niblett, T.: The CN2 induction algorithm. Machine Learning 3, 261–284 (1989)Google Scholar
  22. 22.
    Cost, S., Salzberg, S.: A weighted nearest neighbor algorithm for learning with symbolic features. Machine Learning 10, 57–78 (1993)Google Scholar
  23. 23.
    Cover, T.M., Hart, P.E.: Nearest neighbor pattern classification. IEEE Transactions on Information Theory 13, 21–27 (1967)zbMATHCrossRefGoogle Scholar
  24. 24.
    Domeniconi, C., Gunopulos, D.: Efficient local flexible nearest neighbor classification. In: Proceedings of the Second SIAM International Conference on Data Mining (2002)Google Scholar
  25. 25.
    Domingos, P.: Unifying instance-based and rule-based induction. Machine Learning 24(2), 141–168 (1996)MathSciNetGoogle Scholar
  26. 26.
    Duda, R.O., Hart, P.E.: Pattern Classification and Scene Analysis. Wiley, New York (1973)zbMATHGoogle Scholar
  27. 27.
    Dudani, S.: The distance-weighted k-nearest-neighbor rule. IEEE Transactions on Systems, Man and Cybernetics 6, 325–327 (1976)Google Scholar
  28. 28.
    Fikes, R.E., Nilsson, N.J.: STRIPS: A new approach to the application of theorem proving to problem solving. Artificial Intelligence 2(3-4), 189–208 (1971)zbMATHCrossRefGoogle Scholar
  29. 29.
    Finkel, R., Bentley, J.: Quad-trees: a data structure for retrieval and composite keys. ACTA Informatica 4(1), 1–9 (1974)zbMATHCrossRefGoogle Scholar
  30. 30.
    Fisher, R.A.: Applications of “student”s’ distribution. Metron 5, 3–17 (1925)Google Scholar
  31. 31.
    Fix, E., Hodges, J.L.: Discriminatory analysis, non-parametric discrimination: Consistency properties. Technical Report 4, USAF School of Aviation and Medicine, Randolph Air Field (1951)Google Scholar
  32. 32.
    Friedman, J.: Flexible metric nearest neighbor classification. Technical Report 113. Department of Statistics, Stanford University, CA (1994)Google Scholar
  33. 33.
    Friedman, J., Hastie, T., Tibshirani, R.: The Elements of Statistical Learning. Springer, New York (2001)zbMATHGoogle Scholar
  34. 34.
    Friedman, J.H., Kohavi, R., Yun, Y.: Lazy decision trees. In: Proceedings of the Thirteenth National Conference on Artificial Intelligence, Cambridge, pp. 717–724 (1996)Google Scholar
  35. 35.
    Fukunaga, K., Narendra, P.M.: A branch and bound algorithm for computing k-nearest neighbors. IEEE Transactions on Computers 24(7), 750–753 (1975)zbMATHCrossRefMathSciNetGoogle Scholar
  36. 36.
    Gaede, V., Gunther, O.: Multidimensional access methods. ACM Computing Surveys 30(2), 170–231 (1998)CrossRefGoogle Scholar
  37. 37.
    Golding, A.R., Rosenbloom, P.S.: Improving accuracy by combining rule-based and case-based reasoning. Artificial Intelligence 87(1-2), 215–254 (1996)CrossRefGoogle Scholar
  38. 38.
    Góra, G., Wojna, A.G.: Local attribute value grouping for lazy rule induction. In: Alpigini, J.J., Peters, J.F., Skowron, A., Zhong, N. (eds.) RSCTC 2002. LNCS (LNAI), vol. 2475, pp. 405–412. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  39. 39.
    Góra, G., Wojna, A.G.: RIONA: a classifier combining rule induction and k-nn method with automated selection of optimal neighbourhood. In: Elomaa, T., Mannila, H., Toivonen, H. (eds.) ECML 2002. LNCS (LNAI), vol. 2430, pp. 111–123. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  40. 40.
    Góra, G., Wojna, A.G.: RIONA: a new classification system combining rule induction and instance-based learning. Fundamenta Informaticae 51(4), 369–390 (2002)zbMATHMathSciNetGoogle Scholar
  41. 41.
    Gosset, W.S.(Student): The probable error of a mean. Biometrika 6, 1–25 (1908)Google Scholar
  42. 42.
    Grzymala-Busse, J.W.: LERS - a system for learning from examples based on rough sets. In: Slowinski, R. (ed.) Intelligent Decision Support, Handbook of Applications and Advances of the Rough Sets Theory, pp. 3–18. Kluwer Academic Publishers, Dordrecht (1992)Google Scholar
  43. 43.
    Guttman, A.: R-trees: a dynamic index structure for spatial searching. In: Proceedings of the 1984 ACM SIGMOD International Conference on Management of Data, Boston, MA, pp. 47–57 (1984)Google Scholar
  44. 44.
    Hastie, T., Tibshirani, R.: Discriminant adaptive nearest neighbor classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 18(6), 607–616 (1996)CrossRefGoogle Scholar
  45. 45.
    Jensen, F.V.: An Introduction to Bayesian Networks. Springer, New York (1996)Google Scholar
  46. 46.
    Kalantari, I., McDonald, G.: A data structure and an algorithm for the nearest point problem. IEEE Transactions on Software Engineering 9(5), 631–634 (1983)CrossRefGoogle Scholar
  47. 47.
    Katayama, N., Satoh, S.: The SR-tree: an index structure for high dimensional nearest neighbor queries. In: Proceedings of the 1997 ACM SIGMOD International Conference on Management of Data, Tucson, Arizona, pp. 369–380 (1997)Google Scholar
  48. 48.
    Kira, K., Rendell, L.A.: A practical approach to feature selection. In: Proceedings of the Ninth International Conference on Machine Learning, Aberdeen, Scotland, pp. 249–256. Morgan Kaufmann, San Francisco (1992)Google Scholar
  49. 49.
    Kleinberg, J., Papadimitriou, C., Raghavan, P.: Segmentation problems. Journal of the ACM 51(2), 263–280 (2004)CrossRefMathSciNetGoogle Scholar
  50. 50.
    Klösgen, W., Żytkow, J.M. (eds.): Handbook of Data Mining and Knowledge Discovery. Oxford University Press, Inc., New York (2002)zbMATHGoogle Scholar
  51. 51.
    Kononenko, I.: Estimating attributes: Analysis and extensions of RELIEF. In: Bergadano, F., De Raedt, L. (eds.) ECML 1994. LNCS, vol. 784, pp. 171–182. Springer, Heidelberg (1994)Google Scholar
  52. 52.
    Leake, D.B. (ed.): Case-Based Reasoning: Experiences, Lessons and Future Directions. AAAI Press/MIT Press (1996)Google Scholar
  53. 53.
    Li, J., Dong, G., Ramamohanarao, K., Wong, L.: DeEPs: a new instance-based discovery and classification system. Machine Learning (2003) (to appear)Google Scholar
  54. 54.
    Li, J., Ramamohanarao, K., Dong, G.: Combining the strength of pattern frequency and distance for classification. In: Proceedings of the Fifth Pacific-Asia Conference on Knowledge Discovery and Data Mining, Hong Kong, pp. 455–466 (2001)Google Scholar
  55. 55.
    Lin, K.I., Jagadish, H.V., Faloustos, C.: The TV-tree: an index structure for high dimensional data. VLDB Journal 3(4), 517–542 (1994)CrossRefGoogle Scholar
  56. 56.
    Lowe, D.: Similarity metric learning for a variable kernel classifier. Neural Computation 7, 72–85 (1995)CrossRefGoogle Scholar
  57. 57.
    Luce, D.R., Raiffa, H.: Games and Decisions. Wiley, New York (1957)zbMATHGoogle Scholar
  58. 58.
    Macleod, J.E.S., Luk, A., Titterington, D.M.: A re-examination of the distance-weighted k-nearest-neighbor classification rule. IEEE Transactions on Systems, Man and Cybernetics 17(4), 689–696 (1987)CrossRefGoogle Scholar
  59. 59.
    Michalski, R.S.: A theory and methodology of inductive learning. Artificial Intelligence 20, 111–161 (1983)CrossRefMathSciNetGoogle Scholar
  60. 60.
    Michalski, R.S., Mozetic, I., Hong, J., Lavrac, H.: The multi-purpose incremental learning system AQ15 and its testing application to three medical domains. In: Proceedings of the Fifth National Conference on Artificial Intelligence, pp. 1041–1045 (1986)Google Scholar
  61. 61.
    Mitchell, T.M.: Machine Learning. McGraw-Hill, Portland (1997)zbMATHGoogle Scholar
  62. 62.
    Nievergelt, J., Hinterberger, H., Sevcik, K.: The grid file: an adaptable symmetric multikey file structure. ACM Transactions on Database Systems 9(1), 38–71 (1984)CrossRefGoogle Scholar
  63. 63.
    Pawlak, Z.: Rough Sets - Theoretical Aspects of Reasoning about Data. Kluwer Academic Publishers, Dordrecht (1991)zbMATHGoogle Scholar
  64. 64.
    Polkowski, L., Skowron, A.: Synthesis of decision systems from data tables. In: Lin, T.Y., Cercone, N. (eds.) Rough Sets and Data Mining: Analysis of Imprecise Data, pp. 259–299. Kluwer Academic Publishers, Dordrecht (1997)Google Scholar
  65. 65.
    Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo (1993)Google Scholar
  66. 66.
    Robinson, J.: The K-D-B-tree: a search structure for large multi-dimensional dynamic indexes. In: Proceedings of the 1981 ACM SIGMOD International Conference on Management of Data, New York, pp. 10–18 (1981)Google Scholar
  67. 67.
    Rosenblueth, A., Wiener, N., Bigelow, J.: Behavior, purpose, and teleology. Philosophy of Science 10, 18–24 (1943)CrossRefGoogle Scholar
  68. 68.
    Russell, S.J.: Use of Knowledge in Analogy and Induction. Morgan Kaufmann, San Francisco (1989)zbMATHGoogle Scholar
  69. 69.
    Salzberg, S.: A nearest hyperrectangle learning method. Machine Learning 2, 229–246 (1991)Google Scholar
  70. 70.
    Savaresi, S.M., Boley, D.L.: On the performance of bisecting K-means and PDDP. In: Proceedings of the First SIAM International Conference on Data Mining, Chicago, USA, pp. 1–14 (2001)Google Scholar
  71. 71.
    Sellis, T., Roussopoulos, N., Faloustos, C.: The R+-tree: a dynamic index for multi-dimensional objects. In: Proceedings of the Thirteenth International Conference on Very Large Databases, pp. 574–584 (1987)Google Scholar
  72. 72.
    Shepard, R.N.: Toward a universal law of generalization for psychological science. science 237, 1317–1323 (1987)CrossRefMathSciNetGoogle Scholar
  73. 73.
    Skowron, A., et al.: Rough set exploration system. Institute of Mathematics, Warsaw University, Poland, http://logic.mimuw.edu.pl/~rses
  74. 74.
    Skowron, A., Rauszer, C.: The discernibility matrices and functions in information systems. In: Slowinski, R. (ed.) Intelligent Decision Support, Handbook of Applications and Advances of the Rough Sets Theory, pp. 331–362. Kluwer Academic Publishers, Dordrecht (1992)Google Scholar
  75. 75.
    Skowron, A., Stepaniuk, J.: Information granules and rough-neural computing. In: Rough-Neural Computing: Techniques for Computing with Words. Cognitive Technologies, pp. 43–84. Springer-Verlag, Heidelberg (2003)Google Scholar
  76. 76.
    Skowron, A., Wojna, A.G.: K nearest neighbors classification with local induction of the simple value difference metric. In: Tsumoto, S., Słowiński, R., Komorowski, J., Grzymała-Busse, J.W. (eds.) RSCTC 2004. LNCS (LNAI), vol. 3066, pp. 229–234. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  77. 77.
    Stanfill, C., Waltz, D.: Toward memory-based reasoning. Communications of the ACM 29(12), 1213–1228 (1986)CrossRefGoogle Scholar
  78. 78.
    Uhlmann, J.: Satisfying general proximity/similarity queries with metric trees. Information Processing Letters 40(4), 175–179 (1991)zbMATHCrossRefGoogle Scholar
  79. 79.
    Vapnik, V.: Statistical Learning Theory. Wiley, Chichester (1998)zbMATHGoogle Scholar
  80. 80.
    Veloso, M.: Planning and Learning by Analogical Reasoning. Springer, Heidelberg (1994)zbMATHGoogle Scholar
  81. 81.
    van Neumann, J., Morgenstern, O.: Theory of Games and Economic Behavior. Princeton University Press, Princeton (1944)zbMATHGoogle Scholar
  82. 82.
    Ward Jr, J.: Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association 58, 236–244 (1963)CrossRefMathSciNetGoogle Scholar
  83. 83.
    Weber, R., Schek, H.J., Blott, S.: A quantitative analysis and performance study for similarity-search methods in high-dimensional spaces. In: Proceedings of the Twenty Fourth International Conference on Very Large Databases, pp. 194–205 (1998)Google Scholar
  84. 84.
    Wettschereck, D.: A Study of Distance-Based Machine Learning Algorithms. PhD thesis, Oregon State University (1994)Google Scholar
  85. 85.
    Wettschereck, D., Aha, D.W., Mohri, T.: A review and empirical evaluation of feature weighting methods for a class of lazy learning algorithms. Artificial Intelligence Review 11, 273–314 (1997)CrossRefGoogle Scholar
  86. 86.
    White, D.A., Jain, R.: Similarity indexing with the SS-tree. In: Proceedings of the Twelve International Conference on Data Engineering, New Orleans, USA, pp. 516–523 (1996)Google Scholar
  87. 87.
    Wiener, N.: Cybernetics. Wiley, New York (1948)Google Scholar
  88. 88.
    Wilson, D.R., Martinez, T.R.: Improved heterogeneous distance functions. Journal of Artificial Intelligence Research 6, 1–34 (1997)zbMATHMathSciNetGoogle Scholar
  89. 89.
    Wilson, D.R., Martinez, T.R.: An integrated instance-based learning algorithm. Computational Intelligence 16(1), 1–28 (2000)CrossRefMathSciNetGoogle Scholar
  90. 90.
    Wojna, A.G.: Adaptacyjne definiowanie funkcji boolowskich z przykladow. Master’s thesis, Warsaw University (2000)Google Scholar
  91. 91.
    Wojna, A.G.: Center-based indexing for nearest neighbors search. In: Proceedings of the Third IEEE International Conference on Data Mining, Melbourne, Florida, USA, pp. 681–684. IEEE Computer Society Press, Los Alamitos (2003)CrossRefGoogle Scholar
  92. 92.
    Wojna, A.G.: Center-based indexing in vector and metric spaces. Fundamenta Informaticae 56(3), 285–310 (2003)zbMATHMathSciNetGoogle Scholar
  93. 93.
    Wolpert, D.: Constructing a generalizer superior to NETtalk via meithematical theory of generalization. Neural Networks 3, 445–452 (1989)CrossRefGoogle Scholar
  94. 94.
    Wróblewski, J.: Covering with reducts - a fast algorithm for rule generation. In: Polkowski, L., Skowron, A. (eds.) RSCTC 1998. LNCS (LNAI), vol. 1424, pp. 402–407. Springer, Heidelberg (1998)CrossRefGoogle Scholar
  95. 95.
    Yianilos, P.N.: Data structures and algorithms for nearest neighbor search in general metric spaces. In: Proceedings of the Fourth Annual ACM/SIGACT-SIAM Symposium on Discrete Algorithms, Austin, Texas, pp. 311–321 (1993)Google Scholar
  96. 96.
    Zavrel, J.: An empirical re-examination of weighted voting for k-nn. In: Proceedings of the Seventh Belgian-Dutch Conference on Machine Learning, Tilburg, The Netherlands, pp. 139–148 (1997)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  1. 1.Institute of InformaticsWarsaw UniversityWarsawPoland

Personalised recommendations