Abstract
Energy-based models are a specific class of neural networks. The simplest energy model is the Hopfield Network dating back from the 1980s (Hopfield Proc Nat Acad Sci USA 79(8):2554–2558, 1982, [1]). Hopfield networks are often thought to be very simple, but they are quite different from what we have seen before.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
For a fully detailed view, see the blog entry of one of the creators of the NTM, https://medium.com/aidangomez/the-neural-turing-machine-79f6e806c0a1.
- 2.
By default, memory networks make one hop, but it has been shown that multiple hops are beneficial, especially in natural language processing.
- 3.
Winograd sentences are sentences of a particular form, whare the computer should resolve the coreference of a pronoun. They were proposed as an alternative to the Turing test, since the turing test has some deep flaws (deceptive behaviour is encouraged), and it is hard to quantify its results and evaluate it on a large scale. Winograd sentences are sentances of the form ‘I tried to put the book in the drwer but it was too [big/small]’, and they are named after Terry Winograd who first considered them in the 1970s [13].
References
J.J. Hopfield, Neural networks and physical systems with emergent collective computational abilities. Proc. Nat. Acad. Sci. U.S.A 79(8), 2554–2558 (1982)
D.H. Ackley, G.E. Hinton, T. Sejnowski, A learning algorithm for boltzmann machines. Cogn. Sci. 9(1), 147–169 (1985)
P. Smolensky, Information processing in dynamical systems: foundations of harmony theory, in Parallel Distributed Processing: Explorations in the Microstructure of Cognition, ed. by D.E. Rumelhart, J.L. McClelland, the PDP Research Group, (MIT Press, Cambridge)
G.E. Hinton, S. Osindero, Y.-W. Teh, A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–1554 (2006)
Y. Bengio, P. Lamblin, D. Popovici, H. Larochelle, Greedy layer-wise training of deep networks, in Proceedings of the 19th International Conference on Neural Information Processing Systems (MIT Press, Cambridge, 2006), pp. 153–160
Y. Bengio, Learning deep architectures for AI. Found. Trends Mach. Learn. 2(1), 1–127 (2009)
I. Goodfellow, Y. Bengio, A. Courville, Deep Learning (MIT Press, Cambridge, 2016)
W. Bechtel, A. Abrahamsen, Connectionism and the Mind: Parallel Processing, Dynamics and Evolution in Networks (Blackwell, Oxford, 2002)
A. Graves, G. Wayne, I. Danihelka, Neural turing machines (2014), arXiv:1410.5401
J. Weston, S. Chopra, A. Bordes, Memory networks, in ICLR (2015), arXiv:1410.3916
S. Sukhbaatar, A. Szlam, J. Weston, End-to-end memory networks (2015), arXiv:1503.08895
J. Weston, A. Bordes, S. Chopra, A.M. Rush, B. van Merriënboer, A. Joulin, T. Mikolov, Towards ai-complete question answering: A set of prerequisite toy tasks, in ICLR (2016), arXiv:1502.05698
T. Winograd, Understanding Natural Language (Academic Press, New York, 1972)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this chapter
Cite this chapter
Skansi, S. (2018). An Overview of Different Neural Network Architectures. In: Introduction to Deep Learning. Undergraduate Topics in Computer Science. Springer, Cham. https://doi.org/10.1007/978-3-319-73004-2_10
Download citation
DOI: https://doi.org/10.1007/978-3-319-73004-2_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-73003-5
Online ISBN: 978-3-319-73004-2
eBook Packages: Computer ScienceComputer Science (R0)