Joint European Conference on Machine Learning and Knowledge Discovery in Databases

ECML PKDD 2015: Machine Learning and Knowledge Discovery in Databases pp 498-515

Difference Target Propagation

  • Dong-Hyun Lee
  • Saizheng Zhang
  • Asja Fischer
  • Yoshua Bengio
Conference paper

DOI: 10.1007/978-3-319-23528-8_31

Part of the Lecture Notes in Computer Science book series (LNCS, volume 9284)
Cite this paper as:
Lee DH., Zhang S., Fischer A., Bengio Y. (2015) Difference Target Propagation. In: Appice A., Rodrigues P., Santos Costa V., Soares C., Gama J., Jorge A. (eds) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2015. Lecture Notes in Computer Science, vol 9284. Springer, Cham

Abstract

Back-propagation has been the workhorse of recent successes of deep learning but it relies on infinitesimal effects (partial derivatives) in order to perform credit assignment. This could become a serious issue as one considers deeper and more non-linear functions, e.g., consider the extreme case of non-linearity where the relation between parameters and cost is actually discrete. Inspired by the biological implausibility of back-propagation, a few approaches have been proposed in the past that could play a similar credit assignment role. In this spirit, we explore a novel approach to credit assignment in deep networks that we call target propagation. The main idea is to compute targets rather than gradients, at each layer. Like gradients, they are propagated backwards. In a way that is related but different from previously proposed proxies for back-propagation which rely on a backwards network with symmetric weights, target propagation relies on auto-encoders at each layer. Unlike back-propagation, it can be applied even when units exchange stochastic bits rather than real numbers. We show that a linear correction for the imperfectness of the auto-encoders, called difference target propagation, is very effective to make target propagation actually work, leading to results comparable to back-propagation for deep networks with discrete and continuous units and denoising auto-encoders and achieving state of the art for stochastic networks.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Dong-Hyun Lee
    • 1
  • Saizheng Zhang
    • 1
  • Asja Fischer
    • 1
  • Yoshua Bengio
    • 1
    • 2
  1. 1.Université de MontréalMontrealCanada
  2. 2.CIFAR Senior FellowMontrealCanada

Personalised recommendations