# On the possibilities of the limited precision weights neural networks in classification problems

## Abstract

Limited precision neural networks are better suited for hardware implementations. Several researchers have proposed various algorithms which are able to train neural networks with limited precision weights. Also it has been suggested that the limits introduced by the limited precision weights can be compensated by an increased number of layers. This paper shows that, from a theoretical point of view, neural networks with integer weights in the range [-p,p] can solve classification problems for which the minimum euclidian distance in-between two patterns from opposite classes is 1/p. This result can be used in an information theory context to calculate a bound on the number of bits necessary for solving a problem. It is shown that the number of bits is limited by m^{*}n^{*}log(2pD) where m is the number of patterns, n is the dimensionality of the space, p is the weight range and D is the radius of a sphere including all patterns.

## Keywords

neural networks entropy classification problems integer weights number of bits## Preview

Unable to display preview. Download preview PDF.

## Bibliography

- [Abu-Mostafa, 1988]-Abu-Mostafa Y.S., Connectivity versus Entropy, NIPS'87, D.Z. Anderson (Ed.), Amer. Inst. Of Phys., New York, 1988, 1–8Google Scholar
- [Beiu, 1994]-Beiu V., Peperstraete J.A., Vandewalle J., Lauwereins R., Area-time performances of some neural computations, Int. Symp. On Signal Proc., Robotics and NN's, P. Borne, T. Fukuda and S.G. Tzafestas (Eds.), GERF EC, Lille, 1994, pp. 664–668Google Scholar
- [Beiu, 1997a]-Beiu V.-Neural Networks Using Threshold Gates: A Complexity Analysis of Their Area and Time Efficient VLSI Representations, Ph.D. thesis, Katholieke Universiteit Leuven, 1994. Extended version to appear as “VLSI Complexity of Discrete Neural Networks”, Gordon & Breach, 1997 (in press).Google Scholar
- [Beiu, 1995]-Beiu, V., Optimal VLSI Implementations of Neural Networks, Chap. 18 in J.G. Taylor (ed.): Neural Networks and Their Applications, John Wiley & Sons, Chichester, UK, 255–276, 1996.Google Scholar
- [Beiu, 1995a]-Beiu, V., Taylor J.G., VLSI optimal neural network learning algorithm, Artif. NN's and Genetic Algs., D.W. Pearson, N.C. Steele and R.F. Albrecht (Eds.), Springer-Verlag, Vienna, 1995, pp. 61–64Google Scholar
- [Beiu, 1995b]-Beiu, V., Taylor J.G., Area efficient constructive learning algorithms, Proc. 10
^{th}Intl.Conf. on Control Sys. and Comp. Sci., PU Bucharest, Bucharest, 1995, pp. 293–310Google Scholar - [Beiu, 1996]-Beiu, V., Entropy bounds for classification algorithms, Neural Network World, Vol. 6, No. 4, pp. 497–505, IDG Press, 1996Google Scholar
- [Beiu, 1997]-Beiu, V., T. De Pauw, Tight bounds on the size of neural networks for classification problems, submitted for IWANN'97Google Scholar
- [Bruck, 1990]-Bruck J., Goodman J.W., On the power of neural networks for solving hard problems, NIPS'87, D.Z. Anderson (Ed.), Amer. Inst. Of Phys., NY, 1988, 137–143 (also in J. of Complexity, 6, 1990, 129–135)Google Scholar
- [Coggins, 1994]-Coggins R., M. Jabri, Wattle: A Trainable Gain Analogue VLSI Neural Network, Advances in NIPS 6 (NIPS
^{*}93, Denver, CO), Morgan Kaufman, San Mateo, CA, 874–881, 1994Google Scholar - [Denker, 1988]-Denker J.S., Wittner B.S., Network Generality, Training Required and Precision Required, NIPS'88, D.Z. Anderson (Ed.), Amer. Inst of Phys., New York, 1988, 219–222Google Scholar
- [Dundar, 1995]-Dundar G., K. Rose, The Effect of Quantization on Multilayer Neural Networks, IEEE Transactions on Neural Networks 6(6), pp. 1446–1451, 1995Google Scholar
- [Hammerstrom, 1988]-Hammerstrom D., The connectivity analysis of simple associations — or — How many connections do you need?, NIPS'87, D.Z. Anderson (Ed.), Amer. Inst. Of Phys., New York, 1988, 338–347Google Scholar
- [Hohfeld, 1991a]-Hohfeld M., S.E. Fahlman, Learning with limited numerical precision using the Cascade-Correlation Algorithm, Tech.Rep. CMU-CS-91-130, School of Comp. Sci. Carnegie Mellon, May 1991. Also in IEEE Transactions on Neural Networks, NN-3(4) 602–611, 1992Google Scholar
- [Hohfeld, 1991b]-Hohfeld M., S.E. Fahlman, Probabilistic rounding in neural networks with limited precision. In U. Ruckert and J.A. Nossek (eds.): Microelectronics for Neural Networks (Proc. MicroNeuro'91-Munich, Germany), Kyrill & Method Verlag, 1–8, October 1991. Also in Neurocomputing, 4, 291–299, 1992Google Scholar
- [Khan, 1994]-Khan A.H., E.L. Hines, Integer weight neural networks, Electronics Letters 30(15), pp. 1237–1238, 1994Google Scholar
- [Khan, 1996]-Khan A.H., R.G. Wilson, Integer weight approximation of continuous-weight multilayer feedforward nets, Proc. IEEE Int. Conf. on Neural Networks, vol. 1, pp. 392–397, Washington DC, June 1996, IEEE Press, New York, NY.Google Scholar
- [Klaggers, 1993]-Klaggers H., Soegtrop M., Limited fan-in random wired cascade-correlation learning architecture, MicroNeuro'93, D.J. Myers and A.F.Murray (Eds.), Univ.Ed Tech. Ltd. Edinburgh, 1993, pp. 79–82Google Scholar
- [Kwan, 1992]-Kwan H.K., Tang C.Z., Designing Multilayer Feedforward Neural Networks Using Simplified Activation Functions and One-Power-of-Two Weights. Electronic Letters, 28(25), pp. 2343–2344, 1992Google Scholar
- [Kwan, 1993]-Kwan H.K., Tang C.Z., Multiplierless Multilayer Feedforward Neural Networks Design Suitable for Continuous Input-Output Mapping, Electronic Letters, 29(14), pp. 1259–1260, 1993Google Scholar
- [Mason, 1995]-Mason R.D., Robertson W., Mapping Hierarchical Neural Networks to VLSI Hardware, Neural Networks, vol. 8, 6, 1995, 905–913Google Scholar
- [Marchesi, 1990]-Marchesi M., G. Orlandi, F. Piazza, L. Pollonara, A. Uncini, Multilayer Perceptrons with Discrete Weights, Proc. Int. Joint Conf. on Neural Networks IJCNN'90, San Diego, Vol. II, pp. 623–630, June 1990Google Scholar
- [Marchesi, 1990]-Marchesi M., G. Orlandi, F. Piazza, A. Uncini, Fast Neural Networks without Multipliers, IEEE Transactions on Neural Networks, NN-4(1), pp. 53–62, 1993Google Scholar
- [Phatak, 1994]-Phatak D.S., Koren I., Connectivity and performance tradeoffs in the cascade-correlation learning architecture, IEEE Trans. NN's, 5, 6, 1994, 930–935Google Scholar
- [Tang, 1993]-Tang C.Z., H.K. Kwan, Multilayer Feedforward Neural Networks with Single Power-of-Two Weights. IEEE Trans. On Signal Processing, SP-41(8), 2724–2727, 1993Google Scholar
- [Vincent, 1992]-Vincent J.M., D.J.Myers, Weight Dithering and Wordlength Selection for Digital Backpropagation Networks, BT Technology J., 10(3), pp. 124–133, 1992Google Scholar
- [Williamson, 1991]-Williamson R.C., Entropy and the complexity of feedforward neural networks, NIPS'90, R.P. Lippmann, J.E.Moody and D.S. Touretzky (Eds.), Morgan Kaufmann, San Mateo, 1991, pp. 946–952Google Scholar
- [Xie, 1991]-Xie Y., M.A. Jabri, Training Algorithms for Limited Precision Feedforward Neural Networks, SEDAL TR 1991-8-3, School of EE, University of Sydney, Australia, 1991. Also in Proc. of the Australian Conference on Neural Networks, Canberra, Australia, 67–71, 1992Google Scholar