On the possibilities of the limited precision weights neural networks in classification problems

  • Sorin Draghici
  • Ishwar K. Sethi
Neural Nets Simulation, Emulation and Implementation
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1240)


Limited precision neural networks are better suited for hardware implementations. Several researchers have proposed various algorithms which are able to train neural networks with limited precision weights. Also it has been suggested that the limits introduced by the limited precision weights can be compensated by an increased number of layers. This paper shows that, from a theoretical point of view, neural networks with integer weights in the range [-p,p] can solve classification problems for which the minimum euclidian distance in-between two patterns from opposite classes is 1/p. This result can be used in an information theory context to calculate a bound on the number of bits necessary for solving a problem. It is shown that the number of bits is limited by m*n*log(2pD) where m is the number of patterns, n is the dimensionality of the space, p is the weight range and D is the radius of a sphere including all patterns.


neural networks entropy classification problems integer weights number of bits 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [Abu-Mostafa, 1988]-
    Abu-Mostafa Y.S., Connectivity versus Entropy, NIPS'87, D.Z. Anderson (Ed.), Amer. Inst. Of Phys., New York, 1988, 1–8Google Scholar
  2. [Beiu, 1994]-
    Beiu V., Peperstraete J.A., Vandewalle J., Lauwereins R., Area-time performances of some neural computations, Int. Symp. On Signal Proc., Robotics and NN's, P. Borne, T. Fukuda and S.G. Tzafestas (Eds.), GERF EC, Lille, 1994, pp. 664–668Google Scholar
  3. [Beiu, 1997a]-
    Beiu V.-Neural Networks Using Threshold Gates: A Complexity Analysis of Their Area and Time Efficient VLSI Representations, Ph.D. thesis, Katholieke Universiteit Leuven, 1994. Extended version to appear as “VLSI Complexity of Discrete Neural Networks”, Gordon & Breach, 1997 (in press).Google Scholar
  4. [Beiu, 1995]-
    Beiu, V., Optimal VLSI Implementations of Neural Networks, Chap. 18 in J.G. Taylor (ed.): Neural Networks and Their Applications, John Wiley & Sons, Chichester, UK, 255–276, 1996.Google Scholar
  5. [Beiu, 1995a]-
    Beiu, V., Taylor J.G., VLSI optimal neural network learning algorithm, Artif. NN's and Genetic Algs., D.W. Pearson, N.C. Steele and R.F. Albrecht (Eds.), Springer-Verlag, Vienna, 1995, pp. 61–64Google Scholar
  6. [Beiu, 1995b]-
    Beiu, V., Taylor J.G., Area efficient constructive learning algorithms, Proc. 10th Intl.Conf. on Control Sys. and Comp. Sci., PU Bucharest, Bucharest, 1995, pp. 293–310Google Scholar
  7. [Beiu, 1996]-
    Beiu, V., Entropy bounds for classification algorithms, Neural Network World, Vol. 6, No. 4, pp. 497–505, IDG Press, 1996Google Scholar
  8. [Beiu, 1997]-
    Beiu, V., T. De Pauw, Tight bounds on the size of neural networks for classification problems, submitted for IWANN'97Google Scholar
  9. [Bruck, 1990]-
    Bruck J., Goodman J.W., On the power of neural networks for solving hard problems, NIPS'87, D.Z. Anderson (Ed.), Amer. Inst. Of Phys., NY, 1988, 137–143 (also in J. of Complexity, 6, 1990, 129–135)Google Scholar
  10. [Coggins, 1994]-
    Coggins R., M. Jabri, Wattle: A Trainable Gain Analogue VLSI Neural Network, Advances in NIPS 6 (NIPS*93, Denver, CO), Morgan Kaufman, San Mateo, CA, 874–881, 1994Google Scholar
  11. [Denker, 1988]-
    Denker J.S., Wittner B.S., Network Generality, Training Required and Precision Required, NIPS'88, D.Z. Anderson (Ed.), Amer. Inst of Phys., New York, 1988, 219–222Google Scholar
  12. [Dundar, 1995]-
    Dundar G., K. Rose, The Effect of Quantization on Multilayer Neural Networks, IEEE Transactions on Neural Networks 6(6), pp. 1446–1451, 1995Google Scholar
  13. [Hammerstrom, 1988]-
    Hammerstrom D., The connectivity analysis of simple associations — or — How many connections do you need?, NIPS'87, D.Z. Anderson (Ed.), Amer. Inst. Of Phys., New York, 1988, 338–347Google Scholar
  14. [Hohfeld, 1991a]-
    Hohfeld M., S.E. Fahlman, Learning with limited numerical precision using the Cascade-Correlation Algorithm, Tech.Rep. CMU-CS-91-130, School of Comp. Sci. Carnegie Mellon, May 1991. Also in IEEE Transactions on Neural Networks, NN-3(4) 602–611, 1992Google Scholar
  15. [Hohfeld, 1991b]-
    Hohfeld M., S.E. Fahlman, Probabilistic rounding in neural networks with limited precision. In U. Ruckert and J.A. Nossek (eds.): Microelectronics for Neural Networks (Proc. MicroNeuro'91-Munich, Germany), Kyrill & Method Verlag, 1–8, October 1991. Also in Neurocomputing, 4, 291–299, 1992Google Scholar
  16. [Khan, 1994]-
    Khan A.H., E.L. Hines, Integer weight neural networks, Electronics Letters 30(15), pp. 1237–1238, 1994Google Scholar
  17. [Khan, 1996]-
    Khan A.H., R.G. Wilson, Integer weight approximation of continuous-weight multilayer feedforward nets, Proc. IEEE Int. Conf. on Neural Networks, vol. 1, pp. 392–397, Washington DC, June 1996, IEEE Press, New York, NY.Google Scholar
  18. [Klaggers, 1993]-
    Klaggers H., Soegtrop M., Limited fan-in random wired cascade-correlation learning architecture, MicroNeuro'93, D.J. Myers and A.F.Murray (Eds.), Univ.Ed Tech. Ltd. Edinburgh, 1993, pp. 79–82Google Scholar
  19. [Kwan, 1992]-
    Kwan H.K., Tang C.Z., Designing Multilayer Feedforward Neural Networks Using Simplified Activation Functions and One-Power-of-Two Weights. Electronic Letters, 28(25), pp. 2343–2344, 1992Google Scholar
  20. [Kwan, 1993]-
    Kwan H.K., Tang C.Z., Multiplierless Multilayer Feedforward Neural Networks Design Suitable for Continuous Input-Output Mapping, Electronic Letters, 29(14), pp. 1259–1260, 1993Google Scholar
  21. [Mason, 1995]-
    Mason R.D., Robertson W., Mapping Hierarchical Neural Networks to VLSI Hardware, Neural Networks, vol. 8, 6, 1995, 905–913Google Scholar
  22. [Marchesi, 1990]-
    Marchesi M., G. Orlandi, F. Piazza, L. Pollonara, A. Uncini, Multilayer Perceptrons with Discrete Weights, Proc. Int. Joint Conf. on Neural Networks IJCNN'90, San Diego, Vol. II, pp. 623–630, June 1990Google Scholar
  23. [Marchesi, 1990]-
    Marchesi M., G. Orlandi, F. Piazza, A. Uncini, Fast Neural Networks without Multipliers, IEEE Transactions on Neural Networks, NN-4(1), pp. 53–62, 1993Google Scholar
  24. [Phatak, 1994]-
    Phatak D.S., Koren I., Connectivity and performance tradeoffs in the cascade-correlation learning architecture, IEEE Trans. NN's, 5, 6, 1994, 930–935Google Scholar
  25. [Tang, 1993]-
    Tang C.Z., H.K. Kwan, Multilayer Feedforward Neural Networks with Single Power-of-Two Weights. IEEE Trans. On Signal Processing, SP-41(8), 2724–2727, 1993Google Scholar
  26. [Vincent, 1992]-
    Vincent J.M., D.J.Myers, Weight Dithering and Wordlength Selection for Digital Backpropagation Networks, BT Technology J., 10(3), pp. 124–133, 1992Google Scholar
  27. [Williamson, 1991]-
    Williamson R.C., Entropy and the complexity of feedforward neural networks, NIPS'90, R.P. Lippmann, J.E.Moody and D.S. Touretzky (Eds.), Morgan Kaufmann, San Mateo, 1991, pp. 946–952Google Scholar
  28. [Xie, 1991]-
    Xie Y., M.A. Jabri, Training Algorithms for Limited Precision Feedforward Neural Networks, SEDAL TR 1991-8-3, School of EE, University of Sydney, Australia, 1991. Also in Proc. of the Australian Conference on Neural Networks, Canberra, Australia, 67–71, 1992Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1997

Authors and Affiliations

  • Sorin Draghici
    • 1
  • Ishwar K. Sethi
    • 1
  1. 1.Vision and Neural Networks Laboratory Department of Computer ScienceWayne State UniversityDetroitUSA

Personalised recommendations