Skip to main content

Overfitting effect of artificial neural network based nonlinear equalizer: from mathematical origin to transmission evolution


Overfitting effect of artificial neural network (ANN) based nonlinear equalizer (NLE) leads to a trap of bit error ratio (BER) overestimation in optical fiber communication system, especially when the performance is evaluated by the commonly-used pseudo-random binary sequence (PRBS). First, we mathematically investigate the PRBS generation and Gray code mapping rules, in comparison with the use of Mersenne Twister random sequence (MTRS). Under the condition of a symbol erasure channel, we identify that ANN can recognize both the PRBS generation and symbol mapping rules, by increasing the weights of NLE at specific positions, whereas the MTRS is currently safe owing to the limited input length of current ANN based NLE. Then, we design four channel models of fiber optical transmission to experimentally examine various impairments on the evolution of overfitting effect. When both the additive white Gaussian noise (AWGN) channel and the bandwidth limited channel are considered, the mitigation of overfitting becomes possible by the use of pruned PRBS (P-PRBS) training set with removing the generation and mapping rules determined input symbols. However, as for both the chromatic dispersion (CD) uncompensated channel and the CD managed channel, the overfitting effect becomes serious, because both CD and fiber nonlinearity induced inter-symbol interference (ISI) is beneficial for ANN to identify the PRBS symbol rules. Finally, possible solutions to mitigate the overfitting effect are summarized.

This is a preview of subscription content, access via your institution.


  1. Krizhevsky A, Sutskever I, Hinton G E. Imagenet classification with deep convolutional neural networks. In: Proceedings of Advances in neural information processing systems (NIPS), 2012. 1097–1105

  2. Hinton G, Deng L, Yu D, et al. Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process Magaz, 2012, 28: 82–97

    Article  Google Scholar 

  3. Sagiroglu S, Yavanoglu U, Guven E N. Web based machine learning for language identification and translation. In: Proceedings of the 6th International Conference on Machine Learning and Applications, Cincinnati, 2007. 280–285

  4. Jarajreh M A, Giacoumidis E, Aldaya I, et al. Artificial neural network nonlinear equalizer for coherent optical OFDM. IEEE Photon Technol Lett, 2015, 27: 387–390

    Article  Google Scholar 

  5. Giacoumidis E, Le S T, Ghanbarisabagh M, et al. Fiber nonlinearity-induced penalty reduction in CO-OFDM by ANN-based nonlinear equalization. Opt Lett, 2015, 40: 5113–5116

    Article  Google Scholar 

  6. Luo M, Gao F, Li X, et al. Transmission of 4×50-Gb/s PAM-4 signal over 80-km single mode fiber using neural network. In: Proceedings of Optical Fiber Communication Conference, 2018. M2F.2

  7. Yang Z, Gao F, Fu S, et al. Radial basis function neural network enabled C-band 4×50-Gb/s PAM-4 transmission over 80 km SSMF. Opt Lett, 2018, 43: 3542–3545

    Article  Google Scholar 

  8. Chuang C, Liu L, Wei C, et al. Convolutional neural network based nonlinear classifier for 112-Gbps high speed optical link. In: Proceedings of Optical Fiber Communication Conference, 2018. W2A.43

  9. Ye C, Zhang D, Hu X, et al. Recurrent neural network (RNN) based end-to-end nonlinear management for symmetrical 50 Gbps NRZ PON with 29 dB+ loss budget. In: Proceedings of European Conference on Optical Communication, 2018. 1–3

  10. Karanov B, Chagnon M, Thouin F, et al. End-to-end deep learning of optical fiber communications. J Lightw Technol, 2018, 36: 4843–4855

    Article  Google Scholar 

  11. Karanov B, Lavery B, Bayvel P, et al. End-to-end optimized transmission over dispersive intensity-modulated channels using bidirectional recurrent neural networks. Opt Express, 2019, 27: 19650–19663

    Article  Google Scholar 

  12. Wang D, Zhang M, Li Z, et al. Modulation format recognition and OSNR estimation using CNN-based deep learning. IEEE Photon Technol Lett, 2017, 29: 1667–1670

    Article  Google Scholar 

  13. Dong Z, Khan F N, Sui Q, et al. Optical performance monitoring: a review of current and future technologies. J Lightw Technol, 2016, 34: 525–543

    Article  Google Scholar 

  14. Chen X, Li B, Shamsabardeh M, et al. On real-time and self-taught anomaly detection in optical networks using hybrid unsupervised/supervised learning. In: Proceedings of European Conference on Optical Communication, 2018. 1–3

  15. Charalabopoulos G, Stavroulakis P, Aghvami A H. A frequency-domain neural network equalizer for OFDM. In: Proceedings of IEEE Global Telecommunications Conference, 2003. 571–575

  16. Rajbhandari S, Ghassemlooy Z, Angelova M. Effective denoising and adaptive equalization of indoor optical wireless channel with artificial light using the discrete wavelet transform and artificial neural network. J Lightw Technol, 2009, 27: 4493–4500

    Article  Google Scholar 

  17. ITU-T. Digital test patterns for performance measurements on digital transmission equipment. CCITT Recommendation O.150.

  18. IEEE Standards Association. IEEE Standard for Ethernet Amendment 10: Media Access Control Parameters, Physical Layers, and Management Parameters for 200 Gb/s and 400 Gb/s Operation. IEEE Std 802.3bs.

  19. Eriksson T A, Bülow H, Leven A. Applying neural networks in optical communication systems: possible pitfalls. IEEE Photon Technol Lett, 2017, 29: 2091–2094

    Article  Google Scholar 

  20. Shu L, Li J, Wan Z, et al. Overestimation trap of artificial neural network: learning the rule of PRBS. In: Proceedings of European Conference on Optical Communication, 2018. 1–3

  21. Chuang C, Liu L, Wei C, et al. Study of training patterns for employing deep neural networks in optical communication systems. In: Proceedings of European Conference on Optical Communication, 2018. 1–3

  22. Yi L, Liao T, Huang L, et al. Machine learning for 100 Gb/s/A passive optical network. J Lightw Technol, 2019, 37: 1621–1630

    Article  Google Scholar 

  23. Matsumoto M, Nishimura T. Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator. ACM Trans Model Comput Simul, 1998, 8: 330

    MATH  Google Scholar 

  24. Doran R W. The Gray code. J Univ Comput Sci, 2007, 13: 1573–1597

    MathSciNet  Google Scholar 

  25. Agrawal G P. Nonlinear Fiber Optics. 4th ed. San Diego: Academic Press, 2001

    MATH  Google Scholar 

Download references


This work was supported by National Key R&D Program of China (Grant No. 2018YFB1801301) National Natural Science Foundation of China (Grant No. 61875061), and Key Project of R&D Program of Hubei Province (Grant No. 2018AAA041).

Author information

Authors and Affiliations


Corresponding author

Correspondence to Songnian Fu.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Yang, Z., Gao, F., Fu, S. et al. Overfitting effect of artificial neural network based nonlinear equalizer: from mathematical origin to transmission evolution. Sci. China Inf. Sci. 63, 160305 (2020).

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI:


  • artificial neural network
  • nonlinear equalizer
  • pseudo-random binary sequence
  • overfitting