© 2012

Neural Networks: Tricks of the Trade

Second Edition

  • Grégoire Montavon
  • Geneviève B. Orr
  • Klaus-Robert Müller

Part of the Lecture Notes in Computer Science book series (LNCS, volume 7700)

Table of contents

  1. Front Matter
  2. Introduction

    1. Klaus-Robert Müller
      Pages 1-5
  3. Speeding Learning

    1. Klaus-Robert Müller
      Pages 7-8
    2. Yann A. LeCun, Léon Bottou, Genevieve B. Orr, Klaus-Robert Müller
      Pages 9-48
  4. Regularization Techniques to Improve Generalization

    1. Klaus-Robert Müller
      Pages 49-51
    2. Lutz Prechelt
      Pages 53-67
    3. Thorsteinn S. Rögnvaldsson
      Pages 69-89
    4. Jan Larsen, Claus Svarer, Lars Nonboe Andersen, Lars Kai Hansen
      Pages 111-130
    5. David Horn, Ury Naftaly, Nathan Intrator
      Pages 131-137
  5. Improving Network Models and Algorithmic Tricks

    1. Klaus-Robert Müller
      Pages 139-141
    2. Rich Caruana
      Pages 163-189
    3. Patrick van der Smagt, Gerd Hirzinger
      Pages 191-203
    4. Nicol N. Schraudolph
      Pages 205-223
  6. Representing and Incorporating Prior Knowledge in Neural Network Training

    1. Patrice Y. Simard, Yann A. LeCun, John S. Denker, Bernard Victorri
      Pages 235-269
    2. Steve Lawrence, Ian Burns, Andrew Back, Ah Chung Tsoi, C. Lee Giles
      Pages 295-309

About this book


The twenty last years have been marked by an increase in available data and computing power. In parallel to this trend, the focus of neural network research and the practice of training neural networks has undergone a number of important changes, for example, use of deep learning machines.

The second edition of the book augments the first edition with more tricks, which have resulted from 14 years of theory and experimentation by some of the world's most prominent neural network researchers. These tricks can make a substantial difference (in terms of speed, ease of implementation, and accuracy) when it comes to putting algorithms to work on real problems.


back-propagation graphics processing unit multilayer perceptron neural reinforcement learning optimization

Editors and affiliations

  • Grégoire Montavon
    • 1
  • Geneviève B. Orr
    • 2
  • Klaus-Robert Müller
    • 1
  1. 1.Dept. of Computer ScienceTechnische Universität BerlinBerlinGermany
  2. 2.Dept. of computer ScienceWillamette UniversitySalemUSA

Bibliographic information