Neural Processing Letters

, Volume 23, Issue 1, pp 55–70 | Cite as

Theoretical Properties of Projection Based Multilayer Perceptrons with Functional Inputs

Abstract

Many real world data are sampled functions. As shown by Functional Data Analysis (FDA) methods, spectra, time series, images, gesture recognition data, etc. can be processed more efficiently if their functional nature is taken into account during the data analysis process. This is done by extending standard data analysis methods so that they can apply to functional inputs. A general way to achieve this goal is to compute projections of the functional data onto a finite dimensional sub-space of the functional space. The coordinates of the data on a basis of this sub-space provide standard vector representations of the functions. The obtained vectors can be processed by any standard method.

In [43], this general approach has been used to define projection based Multilayer Perceptrons (MLPs) with functional inputs. We study in this paper important theoretical properties of the proposed model. We show in particular that MLPs with functional inputs are universal approximators: they can approximate to arbitrary accuracy any continuous mapping from a compact sub-space of a functional space to \(\mathbb{R}\). Moreover, we provide a consistency result that shows that any mapping from a functional space to \(\mathbb{R}\) can be learned thanks to examples by a projection based MLP: the generalization mean square error of the MLP decreases to the smallest possible mean square error on the data when the number of examples goes to infinity.

Keywords

consistency functional data analysis multilayer perceptron projection universal approximation 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Abraham, C., Cornillon, P.-A., Matzner-Lober, E., Molinari, N. 2003Unsupervised curve clustering using B-splinesScandinavian Journal of Statistics30581595CrossRefMathSciNetGoogle Scholar
  2. 2.
    Barron, A. R. 1994Approximation and estimation bounds for artificial neural networksMachine Learning14115133MATHGoogle Scholar
  3. 3.
    Besse, P., Cardot, H., Ferraty, F. 1997Simultaneous non-parametric regressions of unbalanced longitudinal dataComputational Statistics and Data Analysis24255270CrossRefMathSciNetGoogle Scholar
  4. 4.
    Besse, P., Ramsay, J. 1986Principal component analysis of sampled curvesPsychometrica51285311MathSciNetGoogle Scholar
  5. 5.
    Biau, G., Bunea, F., Wegkamp, M. 2005Functional classification in Hilbert spacesIEEE Transactions on Information Theory5121632172Google Scholar
  6. 6.
    Brumback, B. A., Rice, J. A. 1998Smoothing spline models for the analysis of nested and crossed samples of curvesJ. Amer. Statist. Assoc.93961994MathSciNetGoogle Scholar
  7. 7.
    Cardot, H., Ferraty, F., Sarda, P. 1999Functional linear modelStatistical and Probability Letters451122MathSciNetGoogle Scholar
  8. 8.
    Cardot, H., Ferraty, F., Sarda, P. 2003Spline estimators for the functional linear modelStatistica Sinica13571591MathSciNetGoogle Scholar
  9. 9.
    Chen, T. 1998A unified approach for neural network-like approximation of non-linear functionalNeural Networks11981983Google Scholar
  10. 10.
    Chen, T., Chen, H. 1993Approximation of continuous functionals by neural networks with application to dynamic systemsIEEE Transactions on Neural Networks4910918CrossRefGoogle Scholar
  11. 11.
    Chen, T., Chen, H. 1995aApproximation capability to functions of several variables, nonlinear functionals, and operators by radial basis function Neural NetworksIEEE Transactions on Neural Networks6904910Google Scholar
  12. 12.
    Chen, T., Chen, H. 1995bUniversal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systemsIEEE Transactions on Neural Networks6911917Google Scholar
  13. 13.
    Conan-Guez, B. and Rossi, F.: Multilayer perceptrons for functional data analysis: A projection based approach’. In: J. R. Dorronsoro (ed.), Artificial Neural Networks – ICANN, pp. 667–672, Madrid, 2002.Google Scholar
  14. 14.
    Dauxois, J., Pousse, A. 1976Les analyses factorielles en calcul des probabilités et en statistiques : essai d’étude synthétiqueUniversité Paul SabatierToulouseThèse d’étatGoogle Scholar
  15. 15.
    Dauxois, J., Pousse, A., Romain, Y. 1982Asymptotic theory for the principal component analysis of a vector of random function: some applications to statistical inferenceJournal of Multivariate Analysis12136154CrossRefMathSciNetGoogle Scholar
  16. 16.
    Delannay, N., Rossi, F. Conan-Guez, B. and Verleysen, M.: Functional radial basis function network, In: Proceedings of ESANN 2004, pp. 313–318, Bruges, Belgium, 2004.Google Scholar
  17. 17.
    Deville, J. 1974Méthodes statistiques et numériques de l’analyse harmoniqueAnnales de l’INSEE15397MathSciNetGoogle Scholar
  18. 18.
    Ferraty, F. and Vieu, P.: The functional nonparametric model and application to spectrometric data, Computational Statistics 17(4) (2002).Google Scholar
  19. 19.
    Ferraty, F., Vieu, P. 2003Curves Discriminations: a Nonparametric Functional ApproachComputational Statistics and Data Analysis44161173MathSciNetGoogle Scholar
  20. 20.
    Ferré, L. and Villa, N.: Multi-layer neural network with functional inputs: an inverse regression approach, Submitted to Scandinavian Journal of Statistics, (2005).Google Scholar
  21. 21.
    Ferré, L., Yao, A.-F. 2003Functional sliced inverse regression analysisStatistics37475488MathSciNetGoogle Scholar
  22. 22.
    Hastie, T., Buja, A., Tibshirani, R. 1995Penalized discriminant analysisAnnals of Statistics2373102MathSciNetGoogle Scholar
  23. 23.
    Hastie, T., Mallows, C. 1993A discussion of A statistical view of some chemometrics regression tools by I.E. Frank and JH. Friedman. Technometrics35140143Google Scholar
  24. 24.
    Hoover, D. R., Rice, J. A., Wu, C. O., Yang, L.-P. 1998Nonparametric smoothing estimates of time-varying coefficient models with longitudinal dataBiometrika85809822CrossRefMathSciNetGoogle Scholar
  25. 25.
    James, G. M. 2002Generalized linear models with functional predictor variablesJournal of the Royal Statistical Society Series B64411432MATHGoogle Scholar
  26. 26.
    James, G. M., Hastie, T. J. 2001Functional linear discriminant analysis for irregularly sampled curvesJournal of the Royal Statistical Society Series B63533550MathSciNetGoogle Scholar
  27. 27.
    James, G. M., Hastie, T. J., Sugar, C. A. 2000Principal component models for sparse functional dataBiometrika87587602CrossRefMathSciNetGoogle Scholar
  28. 28.
    James, G. M., Sugar, C. A. 2003Clustering for sparsely sampled functional dataJournal of American Statistical Association98397408MathSciNetGoogle Scholar
  29. 29.
    Kallenberg, O.: Foundations of Modern Probability, Probability and its Applications. Springer, 1997.Google Scholar
  30. 30.
    Leurgans, S., Moyeed, R., Silverman, B. 1993Canonical correlation analysis when the data are curvesJournal of the Royal Statistical Society B55725740MathSciNetGoogle Scholar
  31. 31.
    Lugosi, G., Zeger, K. 1995Nonparametric estimation via empirical risk minimizationIEEE Transactions on Information Theory41677687CrossRefMathSciNetGoogle Scholar
  32. 32.
    Luschgy, H., Pages, G. 2002Functional quantization of Gaussian processesJournal of Functional Analysis196486531CrossRefMathSciNetGoogle Scholar
  33. 33.
    Marx, B. D. and Eilers, P. H.: Generalized linear regression on sampled signals with penalized likelihood, In: R. H. A. Forcina, G. M. Marchetti and G. Galmacci (eds): Statistical Modelling. Proceedings of the 11th international workshop on statistical modelling. Orvietto, 1996.Google Scholar
  34. 34.
    Pinkus, A.: Approximation theory of the MLP model in neural networks, Acta Numerica (1999) 143–195.Google Scholar
  35. 35.
    Pollard, D.: A User’s Guide to Measure Theoretic Probability, Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2002.Google Scholar
  36. 36.
    Ramsay, J. and Silverman, B.: Functional Data Analysis, Springer Series in Statistics. Springer Verlag, 1997.Google Scholar
  37. 37.
    Ramsay, J. O., Dalzell, C. J. 1991Some tools for functional data analysis (with Discussion)Journal of the Royal Statistical Society Series B53539572MathSciNetGoogle Scholar
  38. 38.
    Rice, J. A., Wu, C. O. 2001Nonparametric mixed effects models for unequally sampled noisy curvesBiometrics57253259CrossRefMathSciNetGoogle Scholar
  39. 39.
    Rossi, F. and Conan-Guez, B.: Functional preprocessing for multilayer perceptrons. In: Proceedings of ESANN 2004. Bruges, Belgium (2004) 319–324.Google Scholar
  40. 40.
    Rossi, F., Conan-Guez, B. 2005aEstimation consistante des paramètres d’un modèle non linéaire pour des données fonctionnelles discrétisées aléatoirementComptes rendus de l’Académie des Sciences – Série I340167170MathSciNetGoogle Scholar
  41. 41.
    Rossi, F., Conan-Guez, B. 2005bFunctional multi-layer perceptron: a nonlinear tool for functional data analysisNeural Networks184560CrossRefGoogle Scholar
  42. 42.
    Rossi, F., Conan-Guez, B. and El Golli, A.: Clustering functional data with the SOM algorithm. In: Proceedings of ESANN, pp. 305–312, Bruges, Belgium, 2004.Google Scholar
  43. 43.
    Rossi, F., Delannay, N., Conan-Guez, B., Verleysen, M. 2005Representation of functional data in neural networksNeurocomputing64183210Google Scholar
  44. 44.
    Sandberg, I. W. 1994General structures for classificationIEEE Transactions on Circuits and Systems–I: Fundamental Theory and Applications41372376MATHMathSciNetGoogle Scholar
  45. 45.
    Sandberg, I. W. 1996Notes on weighted norms and network approximation of functionalsIEEE Transactions on Circuits and Systems–I: Fundamental Theory and Applications43600601MathSciNetGoogle Scholar
  46. 46.
    Sandberg, I. W., Xu, L. 1996Network approximation of input-output maps and functionalsCircuits Systems Signal Processing15711725MathSciNetGoogle Scholar
  47. 47.
    Stinchcombe, M. B. 1999Neural network approximation of continuous functionals and continuous functions on compactificationsNeural Networks12467477CrossRefGoogle Scholar
  48. 48.
    White, H. 1990Connectionist nonparametric regression: mutilayer feedforward networks can learn arbitrary mappingsNeural Networks3535549CrossRefGoogle Scholar

Copyright information

© Springer 2006

Authors and Affiliations

  1. 1.Projet AxIS, INRIADomaine de Voluceau, RocquencourtLe Chesnay CedexFrance
  2. 2.LITA EA3097Université de MetzMetzFrance

Personalised recommendations