Neural Networks: Tricks of the Trade

  • Genevieve B. Orr
  • Klaus-Robert Müller

Part of the Lecture Notes in Computer Science book series (LNCS, volume 1524)

Table of contents

  1. Front Matter
    Pages I-VI
  2. Introduction

    1. Pages 1-5
  3. Speeding Learning

    1. Front Matter
      Pages 7-8
    2. Yann LeCun, Leon Bottou, Genevieve B. Orr, Klaus -Robert Müller
      Pages 9-50
  4. Regularization Techniques to Improve Generalization

    1. Front Matter
      Pages 51-53
    2. Lutz Prechelt
      Pages 55-69
    3. Thorsteinn S. Rognvaldsson
      Pages 71-92
    4. Jan Larsen, Claus Svarer, Lars Nonboe Andersen, Lars Kai Hansen
      Pages 113-132
    5. David Horn, Ury Naftaly, Nathan Intrator
      Pages 133-139
  5. Improving Network Models and Algorithmic Tricks

    1. Front Matter
      Pages 141-143
    2. Rich Caruana
      Pages 165-191
    3. Patrick van der Smagt, Gerd Hirzinger
      Pages 193-206
    4. Nicol N. Schraudolph
      Pages 207-226
  6. Representing and Incorporating Prior Knowledge in Neural Network Training

    1. Front Matter
      Pages 235-237
    2. Patrice Y. Simard, Yann A. LeCun, John S. Denker, Bernard Victorri
      Pages 239-274
    3. Steve Lawrence, Ian Burns, Andrew Back, Ah Chung Tsoi, C. Lee Giles
      Pages 299-313
    4. Jürgen Fritsch, Michael Finke
      Pages 315-342
  7. Tricks for Time Series

    1. Front Matter
      Pages 343-345
    2. Ralph Neuneier, Hans Georg Zimmermann
      Pages 373-423
  8. Back Matter
    Pages 425-432

About this book


It is our belief that researchers and practitioners acquire, through experience and word-of-mouth, techniques and heuristics that help them successfully apply neural networks to di cult real world problems. Often these \tricks" are theo- tically well motivated. Sometimes they are the result of trial and error. However, their most common link is that they are usually hidden in people’s heads or in the back pages of space-constrained conference papers. As a result newcomers to the eld waste much time wondering why their networks train so slowly and perform so poorly. This book is an outgrowth of a 1996 NIPS workshop called Tricks of the Trade whose goal was to begin the process of gathering and documenting these tricks. The interest that the workshop generated motivated us to expand our collection and compile it into this book. Although we have no doubt that there are many tricks we have missed, we hope that what we have included will prove to be useful, particularly to those who are relatively new to the eld. Each chapter contains one or more tricks presented by a given author (or authors). We have attempted to group related chapters into sections, though we recognize that the di erent sections are far from disjoint. Some of the chapters (e.g., 1, 13, 17) contain entire systems of tricks that are far more general than the category they have been placed in.


Artificial Neural Networks Learning Neural Algorithms Pattern Recognition Time Series Analysis algorithms neural networks

Editors and affiliations

  • Genevieve B. Orr
    • 1
  • Klaus-Robert Müller
    • 2
  1. 1.Department of Computer ScienceWillamette UniversitySalemUSA
  2. 2.GMD First (Forschungszentrum Informationstechnik)BerlinGermany

Bibliographic information

  • DOI
  • Copyright Information Springer-Verlag Berlin Heidelberg 1998
  • Publisher Name Springer, Berlin, Heidelberg
  • eBook Packages Springer Book Archive
  • Print ISBN 978-3-540-65311-0
  • Online ISBN 978-3-540-49430-0
  • Series Print ISSN 0302-9743
  • Buy this book on publisher's site