Abstract
This paper applies conformal prediction techniques to compute simultaneous prediction bands and clustering trees for functional data. These tools can be used to detect outliers and clusters. Both our prediction bands and clustering trees provide prediction sets for the underlying stochastic process with a guaranteed finite sample behavior, under no distributional assumptions. The prediction sets are also informative in that they correspond to the high density region of the underlying process. While ordinary conformal prediction has high computational cost for functional data, we use the inductive conformal predictor, together with several novel choices of conformity scores, to simplify the computation. Our methods are illustrated on some real data examples.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Antoniadis, A., Xavier Brossat, J.C., Poggi, J.: Clustering functional data using wavelets. In: e-book of COMPSTAT, pp. 697–704. Springer (2010)
Cheng, Y.: Mean shift, mode seeking, and clustering. IEEE Trans. Pattern Anal. Mach. Intell. 17(8), 790–799 (1995)
Cuevas, A., Febrero, M., Fraiman, R.: Robust estimation and classification for functional data via projection-based depth notions. Comput. Stat. 22, 481–496 (2007)
Delaigle, A., Hall, P., Bathia, N.: Componentwise classification and clustering of functional data. Biometrika 99 (2012)
Efromovich, S.: Nonparametric Curve Estimation: Methods, Theory and Applications. Springer Verlag (1999)
Febrero, M., Galeano, P., González-Manteiga, W.: Outlier detection in functional data by depth measures, with application to identify abnormal nox levels. Environmetrics 19, 331–345 (2008)
Ferraty, F., Vieu, P.: Nonparametric Functional Data Analysis: Theory and Practice. Springer Verlag (2006)
Gervini, D.: Detecting and handling outlying trajectories in irregularly sampled functional datasets. Ann. Appl. Stat. 3, 1758–1775 (2009)
Hartigan, J.: Clustering Algorithms. John Wiley, New York (1975)
Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning, 2nd edn. Springer (2009)
Hyndman, R.J., Shang, H.L.: Rainbow plots, bagplots, and boxplots for functional data. J. Comput. Graph. Stat. 19, 29–45 (2010)
James, G.M., Sugar, C.A.: Clustering for sparsely sampled functional data. J. Am. Stat. Assoc. 98, 397–408 (2003)
Lei, J., Robins, J., Wasserman, L.: Efficient nonparametric conformal prediction regions. J. Am. Stat. Assoc. 108, 278–287 (2013)
Lei, J., Wasserman, L.: Distribution free prediction bands for nonparametric regression. J. R. Stat. Soc. Ser. B Stat. Methodol. (2013, to appear)
López-Pintado, S., Romo, J.: On the concept of depth for functional data. J. Am. Stat. Assoc. 104, 718–734 (2009)
Papadopoulos, H.: Inductive conformal prediction: theory and application to neural networks. In: Fritzsche, P. (ed.) Tools in Artificial Intelligence, pp. 315–330. InTech, Chapters (2008)
Ramsay, J., Silverman, B.: Functional Data Analysis. New York: Springer (1997)
Rinaldo, A., Singh, A., Nugent, R., Wasserman, L.: Stability of density-based clustering. J. Mach. Learn. Res. 13, 905–948 (2012)
Rinaldo, A., Wasserman, L.: Generalized density clustering. Ann. Stat. 38, 2678–2722 (2010)
Rousseeuw, P.J., Ruts, I., Tukey, J.W.: The bagplot: a bivariate boxplot. Am. Stat. 53, 382–387 (1999)
Shafer, G., Vovk, V.: A tutorial on conformal prediction. J. Mach. Learn. Res. 9, 371–421 (2008)
Shi, J., Wang, B.: Curve prediction and clustering with mixtures of Gaussian process functional regression models. Stat. Comput. 18, 267–283 (2008)
Sun, Y., Genton, M.G.: Functional boxplots. J. Comput. Graph. Stat. 20, 216–334 (2011)
Tarpey, T., Kinateder, K.K.J.: Clustering functional data. J. Classif. 20, 93–114 (2003)
Vovk, V.: Conditional validity of inductive conformal predictors. In: JMLR: Workshop and Conference Proceedings, vol. 25, pp. 475–490 (2012)
Vovk, V., Gammerman, A., Shafer, G.: Algorithmic Learning in a Random World. Springer (2005)
Yao, F., Müller, H.G., Wang, J.L.: Functional data analysis for sparse longitudinal data. J. Am. Stat. Assoc. 100(470), 577–590 (2005)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Lei, J., Rinaldo, A. & Wasserman, L. A conformal prediction approach to explore functional data. Ann Math Artif Intell 74, 29–43 (2015). https://doi.org/10.1007/s10472-013-9366-6
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10472-013-9366-6