Heterogeneous recurrent neural networks for natural language model

  • Masayuki Tsuji
  • Teijiro IsokawaEmail author
  • Takayuki Yumoto
  • Nobuyuki Matsui
  • Naotake Kamiura
Original Article


Neural networks for language model are proposed and their performances are explored. The proposed network consists of two recurrent networks of which structures are different to each other. Both networks accept words as their inputs, translate their distributed representation, and produce the probabilities of words to occur from their sequence of input words. Performances for the proposed network are investigated through constructions for language models, as compared with a single recurrent neural and a long short-term memory network.


Recurrent neural network Neural probabilistic language model Heterogeneous network 


  1. 1.
    Mikolov T, Karafiát M, Burget L, Cernocký J, Khudanpur S (2010) Recurrent neural network based language model. In: Proceedings of interspeech, pp 1045–1048Google Scholar
  2. 2.
    Gers FA, Schmidhuber JA, Cummins FA (2000) Learning to Forget: Continual Prediction with LSTM. Neural Comput 12(10):2451–2471CrossRefGoogle Scholar
  3. 3.
    Sundermeyer M, Schlüter R, Ney H (2012) LSTM neural networks for language modeling. In: Proceedings of interspeechGoogle Scholar
  4. 4.
    Chung J, Gulcehre C, Cho K, Bengio Y (2015) Gated feedback recurrent neural networks. In: Proceedings of the 32nd international conference on machine learning, pp 2067–2075Google Scholar
  5. 5.
    Marcus MP, Marcinkiewicz MA, Santorini B (1993) Building a large annotated corpus of English: the penn treebank. Comput Linguist 19(2):313–330Google Scholar
  6. 6.
    Francis WN, Kucera H (1979) Brown University Standard Corpus of Present-Day American English, revised version. Accessed 15 Sept 2018

Copyright information

© ISAROB 2018

Authors and Affiliations

  • Masayuki Tsuji
    • 1
    • 2
  • Teijiro Isokawa
    • 1
    Email author
  • Takayuki Yumoto
    • 1
  • Nobuyuki Matsui
    • 1
  • Naotake Kamiura
    • 1
  1. 1.Graduate School of EngineeringUniversity of HyogoHimejiJapan
  2. 2.Cadence Design SystemsYokohamaJapan

Personalised recommendations