A guided tour across the boundaries of learning recursive languages

  • Thomas Zeugmann
  • Steffen Lange
1 Inductive Inference Theory 1.2 Inductive Inference of Formal Languages

Abstract

The present paper deals with the learnability of indexed families of uniformly recursive languages from positive data as well as from both, positive and negative data. We consider the influence of various monotonicity constraints to the learning process, and provide a thorough study concerning the influence of several parameters. In particular, we present examples pointing to typical problems and solutions in the field. Then we provide a unifying framework for learning. Furthermore, we survey results concerning learnability in dependence on the hypothesis space, and concerning order independence. Moreover, new results dealing with the efficiency of learning are provided. First, we investigate the power of iterative learning algorithms. The second measure of efficiency studied is the number of mind changes a learning algorithm is allowed to perform. In this setting we consider the problem whether or not the monotonicity constraints introduced do influence the efficiency of learning algorithms.

The paper mainly emphasis to provide a comprehensive summary of results recently obtained, and of proof techniques developed. Finally, throughout our guided tour we discuss the question of what a natural language learning algorithm might look like.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Adleman, L.M., and Blum, M. (1991), Inductive inference and unsolvability, Journal of Symbolic Logic56, 891–900.Google Scholar
  2. Angluin, D. (1980a), Finding patterns common to a set of strings, Journal of Computer and System Sciences, 21, 46–62.Google Scholar
  3. Angluin, D. (1980b), Inductive inference of formal languages from positive data, Information and Control, 45 (1980), 117–135.Google Scholar
  4. Angluin, D. (1992), Computational learning theory: Survey and selected bibliography, in “Proceedings 24th Annual ACM Workshop on Theory of Computing,” pp. 351–369, ACM Press.Google Scholar
  5. Angluin, D., and Smith, C.H. (1983), Inductive inference: theory and methods, Computing Surveys15, 237–269.Google Scholar
  6. Angluin, D., and Smith, C.H. (1987), Formal inductive inference, in “Encyclopedia of Artificial Intelligence” (St.C. Shapiro, Ed.), Vol. 1, pp. 409–418, Wiley-Interscience Publication, New York.Google Scholar
  7. Anthony, M. and Biggs, N. (1992), “Computational Learning Theory,” Cambridge University Press, Cambridge.Google Scholar
  8. Arikawa, S., Goto, S., Ohsuga, S., and Yokomori, T. (Eds.) (1990) “Proceedings 1st International Workshop on Algorithmic Learning Theory,” October 1990, Tokyo, Japanese Society for Artificial Intelligence.Google Scholar
  9. Arikawa, S., Maruoka, A., and Sato, T. (Eds.) (1991) “Proceedings 2nd International Workshop on Algorithmic Learning Theory,” October 1991, Tokyo, Japanese Society for Artificial Intelligence.Google Scholar
  10. Arikawa, S., Kuhara, S., Miyano, S., Mukouchi, Y., Shinohara, A. and Shinohara, T. (1992), A machine discovery from amino acid sequences by decision trees over regular patterns, in Proceedings International Conference on Fifth Generation Computer Systems, Vol. 2, pp. 618–625, Institute for New Generation Computer Technology (ICOT), Tokyo, Japan.Google Scholar
  11. Barzdin, Ya. M. (1974), Inductive inference of automata, functions and programs, in “Proceedings International Congress of Math.,” Vancouver, pp. 455–460.Google Scholar
  12. Barzdin, Ya.M., and Freivalds, R.V. (1972), On the prediction of general recursive functions, Sov. Math. Dokl.13, 1224–1228.Google Scholar
  13. Barzdin, Ya.M., and Freivalds, R.V. (1974), Прогнозированиye и преqdелQwный синтез еффективно перечислимыqh классов фунций, in “Теория АлгоритмОв и Программ,” Vol. 1 (Ya. M. Barzdin, ed.) Latvian State University, Riga, pp. 101–111.Google Scholar
  14. Barzdin, Ya.M., Kinber, E.B., and Podnieks, K.M. (1974), ОБ укорении синтеза и прогрозирования функций, in “Теориц Алгоритмов и Программ,” Vol. 1 (Ya.M. Barzdin, Ed.) Latvian State University, Riga, pp. 117–128.Google Scholar
  15. Berwick, R. (1985), “The Acquisition of Syntatic Knowledge,” MIT Press, Cambridge, Massachusetts.Google Scholar
  16. Blum, A., and Singh, M. (1990), Learning functions of k terms, in “Proceedings 3rd Workshop on Computational Learning Theory, July 1990, Rochester,” (M. Fulk and J. Case, Eds.), pp. 144–153, Morgan Kaufmann Publishers Inc., San Mateo.Google Scholar
  17. Brewka, G. (1991), “Nonmonotonic Reasoning: Logical Foundations of Commonsense,” Cambridge University Press, Cambridge.Google Scholar
  18. Case, J. (1988), The power of vacillation, in “Proceedings 1st Workshop on Computational Learning Theory, August 1988, Boston,” (D. Haussler and L. Pitt, Eds.), pp. 196–205, Morgan Kaufmann Publishers Inc., San Mateo.Google Scholar
  19. Case, J., and Lynes, C. (1982), Machine inductive inference and language identification, in “Proceedings Automata, Languages and Programming, Ninth Colloquium, Aarhus, Denmark,” (M. Nielsen and E.M. Schmidt, Eds.), Lecture Notes in Computer Science Vol. 140, pp. 107–115, Springer-Verlag, Berlin.Google Scholar
  20. Case, J., and Smith, C.H. (1983), Comparison of identification criteria for machine inductive inference, Theoretical Computer Science25, 193–220.Google Scholar
  21. Freivalds, R., Kinber, E.B. and Wiehagen, R. (1988), Probabilistic versus deterministic inductive inference in nonstandard numberings, Zeitschrift für Mathematische Logik und Grundlagen der Mathematik, 34 (1988), 531–539.Google Scholar
  22. Freivalds, R., Kinber, E.B. and Wiehagen, R. (1992), Convergently versus divergently incorrect hypotheses in inductive inference, GOSLER-Report 02/92, January 1992, FB Mathematik und Informatik, TH Leipzig.Google Scholar
  23. FulK, M. (1990), Prudence and other restrictions in formal language learning, Information and Computation, 85 1–11.Google Scholar
  24. Fulk, M., and Case, J. (Eds.) (1990), Proceedings of the 3rd Annual Workshop on Computational Learning Theory, July 1990, Rochester, Morgan Kaufmann Publishers Inc., San Mateo.Google Scholar
  25. Gasarch, W.I., and Velauthapillai, M. (1992), Asking questions versus verifiability, in “Proceedings 3rd International Workshop on Analogical and Inductive Inference,” October 1992, Dagstuhl, (K.P. Jantke, ed.) Lecture Notes in Artificial Intelligence Vol. 642, pp. 197–213, Springer-Verlag, Berlin.Google Scholar
  26. Gold, M.E. (1965), Limiting recursion, Journal of Symbolic Logic, 30 28–48.Google Scholar
  27. Gold, M.E. (1967), Language identification in the limit, Information and Control10, 447–474Google Scholar
  28. Haussler, D. (Ed.) (1992), Proceedings of the 5th Annual Workshop on Computational Learning Theory, July 1992, Pittsburgh, ACM Press, New York.Google Scholar
  29. Hopcroft, J.E., and Ullman, J.D. (1969), “Formal Languages and their Relation to Automata,” Addison-Wesley, Reading, Massachusetts.Google Scholar
  30. Ishizaka, H. (1989), Learning simple deterministic languages. in “Proceedings of the 2nd Annual Workshop on Computational Learning Theory, Santa Cruz, August 1989, (R. Rivest, D. Haussler and M.K. Warmuth, Eds.), pp. 162–174, Morgan Kaufmann Publishers Inc., San Mateo.Google Scholar
  31. Jain, S., and Sharma, A. (1989), Recursion theoretic characterizations of language learning, The University of Rochester, Dept. of Computer Science, TR 281.Google Scholar
  32. Jantke, K.P. (Ed.) (1989), “Proceedings 2nd International Workshop on Analogical and Inductive Inference, October 1989, Reinhardsbrunn Castle,” Lecture Notes in Artificial Intelligence Vol. 397.Google Scholar
  33. Jantke, K.P. (1991a), Monotonic and non-monotonic inductive inference, New Generation Computing8, 349–360.Google Scholar
  34. Jantke, K.P. (1991b), Monotonic and non-monotonic inductive inference of functions and patterns, in “Proceedings 1st International Workshop on Nonmonotonic and Inductive Logics, December 1990, Karlsruhe,” (J. Dix, K.P. Jantke and P.H. Schmitt, Eds.), Lecture Notes in Artificial Intelligence Vol. 543, pp. 161–177, Springer-Verlag, Berlin.Google Scholar
  35. Jantke, K.P. (Ed.) (1992), “Proceedings 3rd International Workshop on Analogical and Inductive Inference, October 1992, Dagstuhl Castle,” Lecture Notes in Artificial Intelligence Vol. 642.Google Scholar
  36. Kapur, S. (1992), Monotonic language learning, in “Proceedings 3rd Workshop on Algorithmic Learning Theory,” October 1992, Tokyo, (S. Doshita, K. Furukawa, K.P. Jantke and T. Nishida, Eds.), Lecture Notes in Artificial Intelligence Vol. 743, pp. 147–158, Springer-Verlag, Berlin.Google Scholar
  37. Kapur, S., and Bilardi, G. (1992), Language learning without overgeneralization, in “Proceedings 9th Annual Symposium on Theoretical Aspects of Computer Science, Cachan, France, February 13–15,” (A. Finkel and M. Jantzen, Eds.), Lecture Notes in Computer Science Vol. 577, pp. 245–256, Springer-Verlag, Berlin.Google Scholar
  38. Kearns, M., and Pitt, L. (1989), A polynomial-time algorithm for learning k-variable pattern languages from examples, in “Proceedings 1st Annual Workshop on Computational Learning Theory, August 1988, Boston,” (D. Haussler and L. Pitt, Eds.), pp. 196–205, Morgan Kaufmann Publishers Inc., San Mateo.Google Scholar
  39. Kinber, E.B. (1992), personal communication.Google Scholar
  40. Kodratoff, Y., and Michalski, R.S. (1990), “Machine Learning, An Artificial Intelligence Approach,” Vol. 3, Morgan Kaufmann Publishers Inc., San Mateo.Google Scholar
  41. Lange, S. (1994), The representation of recursive languages and its impact on the efficiency of learning, in “Proceedings 7th Annual ACM Conference on Computational Learning Theory, New Brunswick, July 1994,” (M. Warmuth, Ed.), pp. 256–267, ACM Press, New York.Google Scholar
  42. Lange, S., and Wiehagen, R. (1991), Polynomial-time inference of arbitrary pattern languages, New Generation Computing8, 361–370.Google Scholar
  43. Lange, S., and Zeugmann, T. (1992), Types of monotonic language learning and their characterization, in “Proceedings 5th Annual ACM Workshop on Computational Learning Theory, Pittsburgh, July 1992,” (D. Haussler, Ed.), pp. 377–390, ACM Press, New York.Google Scholar
  44. Lange, S., and Zeugmann, T. (1993a), Monotonic versus non-monotonic language learning, in “Proceedings 2nd International Workshop on Nonmonotonic and Inductive Logic, December 1991, Reinhardsbrunn,” (G. Brewka, K.P. Jantke and P.H. Schmitt, Eds.), Lecture Notes in Artificial Intelligence Vol. 659, pp. 254–269, Springer-Verlag, Berlin.Google Scholar
  45. Lange, S., and Zeugmann, T. (1993b), Learning recursive languages with bounded mind changes, International Journal of Foundations of Computer Science4, 157–178.Google Scholar
  46. Lange, S., and Zeugmann, T. (1993c), Language learning in dependence on the space of hypotheses, in “Proceedings 6th Annual ACM Conference on Computational Learning Theory,” Santa Cruz, July 1993, pp. 127–136, ACM Press, New York.Google Scholar
  47. Lange, S., and Zeugmann, T. (1993d), The learnability of recursive languages in dependence on the space of hypotheses, GOSLER-Report 20/93, July 1993, Fachbereich Mathematik und Informatik, TH Leipzig.Google Scholar
  48. Lange, S., and Zeugmann, T. (1993e), On the impact of order independence to the learnability of recursive languages, Research Report ISIS-RR-93-17E, Institute for Social Information Science, FUJITSU Laboratories Ltd, Numazu.Google Scholar
  49. Lange, S., and Zeugmann, T. (1994), Characterization of language learning on informant under various monotonicity constraints, Journal of Experimental and Theoretical Artificial Intelligence6, 73–94.Google Scholar
  50. Lange, S., Zeugmann, T., and Kapur, S (1992), Class preserving monotonic language learning, submitted to Theoretical Computer Science, and GOSLER-Report 14/92, FB Mathematik und Informatik, TH Leipzig.Google Scholar
  51. Machtey, M., and Young, P. (1978), “An Introduction to the General Theory of Algorithms,” North-Holland, New York.Google Scholar
  52. Michalski, R.S., Carbonell, J.G., and Mitchell, T.M. (1984), “Machine Learning, An Artificial Intelligence Approach,” Vol. 1, Springer-Verlag, Berlin.Google Scholar
  53. Michalski, R.S., Carbonell, J.G., and Mitchell, T.M. (1986), “Machine Learning, An Artificial Intelligence Approach,” Vol. 2, Morgan Kaufmann Publishers Inc., San Mateo.Google Scholar
  54. Mukouchi, Y. (1992), Inductive inference with bounded mind changes, in “Proceedings 3rd Workshop on Algorithmic Learning Theory,” October 1992, Tokyo, (S. Doshita, K. Furukawa, K.P. Jantke and T. Nishida, Eds.), Lecture Notes in Artificial Intelligence Vol. 743, pp. 125–134, Springer-Verlag, Berlin.Google Scholar
  55. Mukouchi, Y. (1994), Inductive inference of recursive concepts, Ph.D. Thesis, RI-FIS, Kyushu University 33, RIFIS-TR-CS-82, March 25th.Google Scholar
  56. Natarajan, B.K. (1991), “Machine Learning, A Theoretical Approach,” Morgan Kaufmann Publishers, Inc., San Mateo.Google Scholar
  57. Nix, R.P. (1983), Editing by examples, Yale University, Dept. Computer Science, Technical Report 280.Google Scholar
  58. Osherson, D., Stob, M., and Weinstein, S. (1986), “Systems that Learn, An Introduction to Learning Theory for Cognitive and Computer Scientists,” MIT-Press, Cambridge, Massachusetts.Google Scholar
  59. Pitt, L., and Valiant, L.G. (1988), Computational limitations on learning from examples, Journal of the ACM35, 965–984.Google Scholar
  60. Popper, K. (1968), “The Logic of Scientific Discovery,” Harper Torch Books.Google Scholar
  61. Rivest, R., Haussler, D., and Warmuth, M.K. (Eds.) (1989), Proceedings of the 2nd Annual Workshop on Computational Learning Theory, August 1989, Santa Cruz, Morgan Kaufmann Publishers Inc., San Mateo.Google Scholar
  62. Schäfer-Richter, G. (1984), Über Eingabeabhängigkeit und Komplexität von Inferenzstrategien, Rheinisch Westfälische Technische Hochschule Aachen, Dissertation.Google Scholar
  63. Shinohara, T. (1982), Polynomial time inference of extended regular pattern languages, in “Proceedings RIMS Symposia on Software Science and Engineering,” Kyoto, Lecture Notes in Computer Science 147, pp. 115–127, Springer-Verlag, Berlin.Google Scholar
  64. Solomonoff, R. (1964), A formal theory of inductive inference, Information and Control7, 1–22, 234–254.Google Scholar
  65. Trakhtenbrot, B.A., and Barzdin, Ya.M. (1970) “Конеч ые втоматы (Поведение и Синтез),” Наука, Москва, English translation: “Finite Automata-Behavior and Synthesis, Fundamental Studies in Computer Science 1,” North-Holland, Amsterdam, 1973.Google Scholar
  66. Wexler, K. (1992), The subset principle is an intensional principle, in “Knowledge and Language: Issues in Representation and Acquisition,” ((E. Reuland and W. Abraham, Eds.), Kluwer Academic Publishers.Google Scholar
  67. Wexler, K., and Culicover, P. (1980), “Formal Principles of Language Acquisition,” MIT Press, Cambridge, Massachusetts.Google Scholar
  68. Wiehagen, R. (1976), Limes-Erkennung rekursiver Funktionen durch spezielle Strategien, Journal of Information Processing and Cybernetics (EIK), 12, 93–99.Google Scholar
  69. Wiehagen, R. (1977), Identification of formal languages, in “Proceedings Mathematical Foundations of Computer Science, Tatranska Lomnica,” (J. Gruska, Ed.), Lecture Notes in Computer Science 53, pp. 571–579, Springer-Verlag, Berlin.Google Scholar
  70. Wiehagen, R. (1978), Characterization problems in the theory of inductive inference, in “Proceedings 5th Colloquium on Automata, Languages and Programming,” (G. Ausiello and C. Böhm, Eds.), Lecture Notes in Computer Science 62, pp. 494–508, Springer-Verlag, Berlin.Google Scholar
  71. Wiehagen, R. (1991), A thesis in inductive inference, in “Proceedings First International Workshop on Nonmonotonic and Inductive Logic,” (J. Dix, K.P. Jantke and P.H. Schmitt, Eds.), Lecture Notes in Artificial Intelligence 543, pp. 184–207, Springer-Verlag, Berlin.Google Scholar
  72. Wiehagen, R., Freivalds, R., and Kinber, E.B. (1984), On the power of probabilistic strategies in inductive inference, Theoretical Computer Science28, 111–133.Google Scholar
  73. Wiehagen, R., and Zeugmann, T. (1994), Learning and Consistency, this volume.Google Scholar
  74. Zeugmann, T., Lange, S., and Kapur, S. (199x), Characterizations of monotonic and dual monotonic language learning, Information and Computation, to appear.Google Scholar

Copyright information

© Springer-Verlag 1995

Authors and Affiliations

  • Thomas Zeugmann
    • 1
  • Steffen Lange
    • 2
  1. 1.Research Institute of Fundamental Information ScienceKyushu University 33FukuokaJapan
  2. 2.FB Mathematik und InformatikHTWK LeipzigLeipzig

Personalised recommendations