CrySSMEx, a Novel Rule Extractor for Recurrent Neural Networks: Overview and Case Study

  • Henrik Jacobsson
  • Tom Ziemke
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3697)


In this paper, it will be shown that it is feasible to extract finite state machines in a domain of, for rule extraction, previously unencountered complexity. The algorithm used is called the Crystallizing Substochastic Sequential Machine Extractor, or CrySSMEx. It extracts the machine from sequence data generated from the RNN in interaction with its domain. CrySSMEx is parameter free, deterministic and generates a sequence of increasingly deterministic extracted stochastic models until a fully deterministic machine is found.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Jacobsson, H.: Rule extraction from recurrent neural networks: A taxonomy and review. Neural Computation 17, 1223–1263 (2005)CrossRefMATHMathSciNetGoogle Scholar
  2. 2.
    Giles, C.L., Miller, C.B., Chen, D., Chen, H.H., Sun, G.Z.: Learning and extracting finite state automata with second-order recurrent neural networks. Neural Computation 4, 393–405 (1992)CrossRefGoogle Scholar
  3. 3.
    Tiňo, P., Vojtek, V.: Extracting stochastic machines from recurrent neural networks trained on complex symbolic sequences. Neural Network World 8, 517–530 (1998)Google Scholar
  4. 4.
    Casey, M.: The dynamics of discrete-time computation, with application to recurrent neural networks and finite state machine extraction. Neural Computation 8, 1135–1178 (1996)CrossRefGoogle Scholar
  5. 5.
    Wiles, J., Elman, J.L.: Learning to count without a counter: A case study of dynamics and activation landscapes in recurrent neural networks. In: Proceedings of the Seventeenth Annual Conference of the Cognitive Science Society, pp. 482–487. MIT Press, Cambridge (1995)Google Scholar
  6. 6.
    Tonkes, B., Blair, A., Wiles, J.: Inductive bias in context-free language learning. In: Proceedings of the Ninth Australian Conference on Neural Networks (1998)Google Scholar
  7. 7.
    Rodriguez, P., Wiles, J., Elman, J.L.: A recurrent network that learns to count. Connection Science 11, 5–40 (1999)CrossRefGoogle Scholar
  8. 8.
    Tonkes, B., Wiles, J.: Learning a context-free task with a recurrent neural network: An analysis of stability. In: Heath, R., Hayes, B., Heathcote, A., Hooker, C. (eds.): Dynamical Cognitive Science: Proceedings of the Fourth Biennial Conference of the Australasian Cognitive Science Society (1999)Google Scholar
  9. 9.
    Bodén, M., Wiles, J., Tonkes, B., Blair, A.: Learning to predict a context-free language: Analysis of dynamics in recurrent hidden units. In: Proceedings of ICANN 1999, pp. 359–364. IEEE, Edinburgh (1999)Google Scholar
  10. 10.
    Bodén, M., Wiles, J.: Context-free and context-sensitive dynamics in recurrent neural networks. Connection Science 12, 196–210 (2000)Google Scholar
  11. 11.
    Gers, F.A., Schmidhuber, J.: LSTM recurrent networks learn simple context free and context sensitive languages. IEEE Transactions on Neural Networks 12, 1333–1340 (2001)CrossRefGoogle Scholar
  12. 12.
    Jacobsson, H., Ziemke, T.: Improving procedures for evaluation of connectionist context-free language predictors. IEEE Transactions on Neural Networks 14, 963–966 (2003)CrossRefGoogle Scholar
  13. 13.
    Zeng, Z., Goodman, R.M., Smyth, P.: Learning finite state machines with self-clustering recurrent networks. Neural Computation 5, 976–990 (1993)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Henrik Jacobsson
    • 1
  • Tom Ziemke
    • 1
  1. 1.School of Humanities and InformaticsUniversity of SkövdeSkövdeSweden

Personalised recommendations