Negative Log Likelihood Ratio Loss for Deep Neural Network Classification

  • Hengshuai YaoEmail author
  • Dong-lai Zhu
  • Bei Jiang
  • Peng Yu
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1069)


In deep neural network, the cross-entropy loss function is commonly used for classification. Minimizing cross-entropy is equivalent to maximizing likelihood under assumptions of uniform feature and class distributions. It belongs to generative training criteria which does not directly discriminate correct class from competing classes. We propose a discriminative loss function with negative log likelihood ratio between correct and competing classes. It significantly outperforms the cross-entropy loss on the CIFAR-10 image classification task.


Loss function Cross entropy Likelihood ratio Deep neural network 


  1. 1.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)Google Scholar
  2. 2.
    Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)zbMATHGoogle Scholar
  3. 3.
    Boer, D., et al.: A tutorial on the cross-entropy method. Ann. Oper. Res. 134, 19–67 (2005)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Vapnik, V.: The Nature of Statistical Learning Theory. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  5. 5.
    Rumelhart, E., Hinton, G., Williams, J.: Learning representations by back-propagating errors. Nature 323, 533–536 (1986)CrossRefGoogle Scholar
  6. 6.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  7. 7.
    Casella, G., Berger, R.L.: Statistical Inference, Duxbury Thomson Learning (2002)Google Scholar
  8. 8.
    Krizhevsky, A.: Learning Multiple Layers of Features from Tiny Images (2009)Google Scholar
  9. 9.
    Murphy, K.: Machine Learning: A Probabilistic Perspective. MIT Press, Cambridge (2012)zbMATHGoogle Scholar
  10. 10.

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Hengshuai Yao
    • 1
    Email author
  • Dong-lai Zhu
    • 2
  • Bei Jiang
    • 3
  • Peng Yu
    • 3
  1. 1.Huawei Hi-SiliconEdmontonCanada
  2. 2.Huawei Noah’s Ark LabEdmontonCanada
  3. 3.Department of Mathematical and Statistical SciencesUniversity of AlbertaEdmontonCanada

Personalised recommendations