Advertisement

Statistics and Computing

, Volume 29, Issue 3, pp 429–447 | Cite as

Gauss–Christoffel quadrature for inverse regression: applications to computer experiments

  • Andrew GlawsEmail author
  • Paul G. Constantine
Article

Abstract

Sufficient dimension reduction (SDR) provides a framework for reducing the predictor space dimension in statistical regression problems. We consider SDR in the context of dimension reduction for deterministic functions of several variables such as those arising in computer experiments. In this context, SDR can reveal low-dimensional ridge structure in functions. Two algorithms for SDR—sliced inverse regression (SIR) and sliced average variance estimation (SAVE)—approximate matrices of integrals using a sliced mapping of the response. We interpret this sliced approach as a Riemann sum approximation of the particular integrals arising in each algorithm. We employ the well-known tools from numerical analysis—namely, multivariate numerical integration and orthogonal polynomials—to produce new algorithms that improve upon the Riemann sum-based numerical integration in SIR and SAVE. We call the new algorithms Lanczos–Stieltjes inverse regression (LSIR) and Lanczos–Stieltjes average variance estimation (LSAVE) due to their connection with Stieltjes’ method—and Lanczos’ related discretization—for generating a sequence of polynomials that are orthogonal with respect to a given measure. We show that this approach approximates the desired integrals, and we study the behavior of LSIR and LSAVE with two numerical examples. The quadrature-based LSIR and LSAVE eliminate the first-order algebraic convergence rate bottleneck resulting from the Riemann sum approximation, thus enabling high-order numerical approximations of the integrals when appropriate. Moreover, LSIR and LSAVE perform as well as the best-case SIR and SAVE implementations (e.g., adaptive partitioning of the response space) when low-order numerical integration methods (e.g., simple Monte Carlo) are used.

Keywords

Sufficient dimension reduction Sliced inverse regression Sliced average variance estimation Orthogonal polynomials 

References

  1. Adragni, K.P., Cook, R.D.: Sufficient dimension reduction and prediction in regression. Philos. Trans. R. Soc. London A Math. Phys. Eng. Sci. 367(1906), 4385–4405 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  2. Ajtai, M., Janos, K., and Szemeredi, E.: An \({\cal{O}} (n \log n)\) sorting network. In: Fifteenth Annual ACM Symposium on Theory of Computing, pp. 1–9 (1983)Google Scholar
  3. Ben-Ari, E.N., Steinberg, D.M.: Modeling data from computer experiments: an empirical comparison of kriging with mars and projection pursuit regression. Qual. Eng. 19(4), 327–338 (2007)CrossRefGoogle Scholar
  4. Chang, J.T., Pollard, D.: Conditioning as disintegration. Stat. Neerlandica 51(3), 287–317 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  5. Clenshaw, C.W., Curtis, A.R.: A method for numerical integration on an automatic computer. Numer. Math. 2(1), 197–205 (1960)MathSciNetCrossRefzbMATHGoogle Scholar
  6. Constantine, P.G.: Active Subspaces: Emerging Ideas for Dimension Reduction in Parameter Studies. SIAM, Philadelphia (2015)CrossRefzbMATHGoogle Scholar
  7. Constantine, P.G., Phipps, E.T.: A Lanczos method for approximating composite functions. Appl. Math. Comput. 218(24), 11751–11762 (2012)MathSciNetzbMATHGoogle Scholar
  8. Constantine, P.G., Eldred, M.S., Phipps, E.T.: Sparse pseudospectral approximation method. Comput. Methods Appl. Mech. Eng. 229, 1–12 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  9. Cook, R.D.: Using dimension-reduction subspaces to identify important inputs in models of physical systems. In: Proceedings of the Section on Physical and Engineering Sciences, pp. 18–25. American Statistical Association, Alexandria, VA (1994)Google Scholar
  10. Cook, R.D.: Regression Graphics: Ideas for Studying Regression through Graphics. Wiley, New York (1998)CrossRefzbMATHGoogle Scholar
  11. Cook, R.D.: SAVE: a method for dimension reduction and graphics in regression. Commun. Stat. Theory Methods 29(9–10), 2109–2121 (2000)CrossRefzbMATHGoogle Scholar
  12. Cook, R.D., Forzani, L.: Likelihood-based sufficient dimension reduction. J. Am. Stat. Assoc. 104(485), 197–208 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  13. Cook, R.D., Ni, L.: Sufficient dimension reduction via inverse regression: a minimum discrepancy approach. J. Am. Stat. Assoc. 100(470), 410–428 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  14. Cook, R.D., Weisberg, S.: Sliced inverse regression for dimension reduction: comment. J. Am. Stat. Assoc. 86(414), 328–332 (1991)zbMATHGoogle Scholar
  15. Davis, P.J., Rabinowitz, P.: Methods Numer. Integr. Academic Press, San Diego (1984)Google Scholar
  16. de Boor, C., Golub, G.H.: The numerically stable reconstruction of a Jacobi matrix from spectral data. Linear Algebra Appl. 21(3), 245–260 (1978)MathSciNetCrossRefzbMATHGoogle Scholar
  17. Donoho, D.L.: High-dimensional data analysis: the curses and blessings of dimensionality. In: AMS Conference on Math Challenges of the 21st Century (2000)Google Scholar
  18. Forsythe, G.E.: Generation and use of orthogonal polynomials for data-fitting with a digital computer. J. Soc. Ind. Appl. Math. 5(2), 74–88 (1957)MathSciNetCrossRefzbMATHGoogle Scholar
  19. Gautschi, W.: The interplay between classical analysis and (numerical) linear algebra–a tribute to Gene H. Golub. Electron. Trans. Numer. Anal. 13, 119–147 (2002)MathSciNetzbMATHGoogle Scholar
  20. Gautschi, W.: Orthogonal Polynomials. Oxford Press, Oxford (2004)zbMATHGoogle Scholar
  21. Glaws, A., Constantine, P.G., and Cook, R.D.: Inverse regression for ridge recovery I: Theory. (2018) arXiv:1702.02227v2
  22. Golub, G.H., Meurant, G.: Matrices, Moments, and Quadrature with Applications. Princeton University, Princeton (2010)zbMATHGoogle Scholar
  23. Golub, G.H., Van Loan, C.F.: Matrix Computations. JHU Press, Baltimore (1996)zbMATHGoogle Scholar
  24. Golub, G.H., Welsch, J.H.: Calculation of Gauss quadrature rules. Math. Comput. 23(106), 221–230 (1969)MathSciNetCrossRefzbMATHGoogle Scholar
  25. Jolliffe, I.T.: Principal Component Analysis. Springer, New York (2002)zbMATHGoogle Scholar
  26. Jones, D.R.: A taxonomy of global optimization methods based on response surfaces. J. Glob. Optim. 21(4), 345–383 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  27. Koehler, J.R., Owen, A.B.: Computer experiments. Handb. Stat. 13(9), 261–308 (1996)MathSciNetCrossRefzbMATHGoogle Scholar
  28. Lanczos, C.: An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. J. Res. Natl. Bureau Stand. 45(4), 255–282 (1950)MathSciNetCrossRefGoogle Scholar
  29. Li, B., Wang, S.: On directional regression for dimension reduction. J. Am. Stat. Assoc. 102(479), 997–1008 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  30. Li, K.C.: Sliced inverse regression for dimension reduction. J. Am. Stat. Assoc. 86(414), 316–327 (1991)MathSciNetCrossRefzbMATHGoogle Scholar
  31. Li, K.C.: On principal Hessian directions for data visualization and dimension reduction: another application of Stein’s lemma. J. Am. Stat. Assoc. 87(420), 1025–1039 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  32. Li, K.C., Duan, N.: Regression analysis under link violation. Ann. Stat. 17(3), 1009–1052 (1989)MathSciNetCrossRefzbMATHGoogle Scholar
  33. Li, W., Lin, G., Li, B.: Inverse regression-based uncertainty quantification algorithms for high-dimensional models: theory and practice. J. Comput. Phys. 321, 259–278 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  34. Liesen, J., Strakoš, Z.: Krylov Subspace Methods: Principles and Analysis. Oxford Press, Oxford (2013)zbMATHGoogle Scholar
  35. Ma, Y., Zhu, L.: A review on dimension reduction. Int. Stat. Rev. 81(1), 134–150 (2013)MathSciNetCrossRefGoogle Scholar
  36. Meurant, G.: The Lanczos and Conjugate Gradient Algorithms: From Theory to Finite Precision Computations. SIAM, Philadelphia (2006)CrossRefzbMATHGoogle Scholar
  37. Myers, R.H., Montgomery, D.C.: Response Surface Methodology: Process and Product Optimization Using Designed Experiments. Wiley, New York (1995)zbMATHGoogle Scholar
  38. Owen, A.B.: Monte Carlo Theory, Methods and Examples (2013). http://statweb.stanford.edu/~owen/mc/
  39. Pinkus, A.: Ridge Functions. Cambridge University Press, Cambridge (2015)CrossRefzbMATHGoogle Scholar
  40. Sacks, J., Welch, W.J., Mitchell, T.J., Wynn, H.P.: Design and analysis of computer experiments. Stat. Sci. 4(4), 409–423 (1989)MathSciNetCrossRefzbMATHGoogle Scholar
  41. Saltelli, A., Ratto, M., Andres, T., Francesca Campolongo, J.C., Galtelli, D., Saisana, M., Tarantola, S.: Global Sensitivity Analysis. The Primer. Wiley, England (2008)zbMATHGoogle Scholar
  42. Santner, T.J., Williams, B.J., Notz, W.I.: The Design and Analysis of Computer Experiments. Springer, New York (2003)CrossRefzbMATHGoogle Scholar
  43. Stieltjes, T.J.: Quelques recherches sur la théorie des quadratures dites mécaniques. Annales scientifiques de l’ École Normale Supérieure 1, 409–426 (1884)MathSciNetCrossRefzbMATHGoogle Scholar
  44. Surjanovic, S., Bingham, D.: Virtual library of simulation experiments: test functions and datasets (2015). http://www.sfu.ca/~ssurjano
  45. Traub, J.F., Werschulz, A.G.: Complexity and Information. Cambridge University Press, Cambridge (1998)zbMATHGoogle Scholar
  46. Trefethen, L.N.: Approximation Theory and Approximation Practice. SIAM, Philadelphia (2013)zbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity of Colorado BoulderBoulderUSA

Personalised recommendations