A Bayesian Method of Online Automatic Tuning

  • Reiji Suda


This chapter discusses mathematical methods for online automatic tuning. After formulating the abstract model of automatic tuning, we review the proposed method, which comprises several novel concepts of automatic tuning, such as online automatic tuning, Bayesian data analysis for quantitative treatments of uncertainties, Bayesian suboptimal sequential experimental design, asymptotic optimality, finite startup, and infinite dilution. Experimental results reveal that the Bayesian sequential experimental design has advantages over random sampling, although random sampling combined with an accurate cost function model can be as good as the Bayesian sequential experimental design.


Objective Function Cost Function Posterior Distribution Prior Distribution Tuning Parameter 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The author sincerely appreciates the members of the Automatic Tuning Research Group for engaging in valuable discussions and collaborations and providing valuable suggestions. The author is also grateful to Prof. Akimichi Takemura, Prof. Tatsuya Kubokawa, Dr. Kazuki Yoshizoe, and Mr. Junya Honda for their invaluable and essential suggestions.

This study is supported in part by a Grant-in-Aid for Scientific Research “Research on Mathematical Core for Robust Auto-Tuning System in Information Explosion Era” from MEXT Japan and the Core Research of Evolutional Science and Technology (CREST) project “ULP-HPC: Ultra Low-Power, High-Performance Computing via Modeling and Optimization of Next Generation HPC Technologies” of the Japan Science and Technology Agency (JST).


  1. 1.
    Whaley RC, Dongarra JJ (1998) Automatically tuned linear algebra software. In: Proceedings of SC98 (CD-ROM)Google Scholar
  2. 2.
    Frigo M, Johnson SG (1998) FFTW: an adaptive software architecture for the FFT. In: Proceedings of ICASSP ’98, vol 3, pp 1381–1384Google Scholar
  3. 3.
    Püschel M et al (2005) SPIRAL: code generation for DSP transforms. Proc. IEEE 93(2):1–42CrossRefGoogle Scholar
  4. 4.
    Imamura T (2007) Recursive multi-factoring algorithm for MPI allreduce. In: Proc. IASTED int’l conf. parallel and distributed computing and networks (PDCN 2007), pp (551)135–145Google Scholar
  5. 5.
    Katagiri T, Voemel C, Demmel J (2007) Automatic performance tuning for the multi-section with multiple eigenvalues method for the symmetric eigenproblem. In: Selected papers of PARA’06, Lecture Notes in Computer Science, vol 4699. Springer, Berlin, pp 938–948Google Scholar
  6. 6.
    Naono K, Sakurai T, Egi M (2008) Research trends on automatic tuning methods for matrix computations and proposal of a new run-time automatic tuning method. In: Int’l workshop on par. mat. alg. appl. (PMAA08)Google Scholar
  7. 7.
    Fukaya T, Yamamoto Y, Zhang S-L (2008) A dynamic programming approach to optimizing the blocking strategy for the householder QR decomposition. In: Proceedings of IEEE international conference on cluster computing 2008 (Proc. int’l workshop on automatic performance tuning (iWAPT2008)), pp 402–410Google Scholar
  8. 8.
    Katagiri T, Kise K, Honda H, Yuba T (2006) ABCLibScript: a directive to support specification of an auto-tuning facility for numerical software. Parallel Comput. 32(1):92–112CrossRefGoogle Scholar
  9. 9.
    Vuduc R, Demmel JW, Bilmes JA (2004) Statistical models for empirical search-based performance tuning. Int. J. High Perform. Comput. Appl. 18(1):65–94CrossRefGoogle Scholar
  10. 10.
    Eijkhout V (2006) A self-adapting system for linear solver selection. In: Proc. 1st int’l workshop on automatic performance tuning (iWAPT2006), pp 44–53Google Scholar
  11. 11.
    Suda R (2007) A Bayesian method for online code selection: toward efficient and robust methods of automatic tuning. In: Proc. 2nd int’l workshop on automatic performance tuning (iWAPT2007), pp 23–32Google Scholar
  12. 12.
    Suda R (2008) A Bayesian approach to automatic performance tuning. In: 13th SIAM conf. para. proc. sci. comp. (PP08) (oral presentation)Google Scholar
  13. 13.
    Carlin BP, Louis TA (2000) Bayes and empirical Bayes methods for data analysis, 2nd edn. Chapman and Hall, Boco RatonMATHCrossRefGoogle Scholar
  14. 14.
    Govindarajulu Z (2004) Sequential statistics. World Scientific, SingaporeMATHGoogle Scholar
  15. 15.
    Auer P, Cesa-Bianchi N (2002) Fischer P Finite-time analysis of the multi-armed bandit problem. Mach. Learn. 47:235–256MATHCrossRefGoogle Scholar
  16. 16.
    Berry DA, Fristedt B (1985) Bandit problem. Chapman and Hall, Boco RatonCrossRefGoogle Scholar
  17. 17.
    Kubokawa T (2000) Estimation of variance and covariance components in elliptically contoured distributions. J. Japan Stat. Soc. 30:143–176MathSciNetMATHGoogle Scholar
  18. 18.
    Vermorel J (2005) Mohri M Multi-armed bandit algorithms and empirical evaluation. In: Proc. Euro. conf. machine learning (ECML 2005), Lecture Notes in Computer Science, vol 3720. Springer, Berlin, pp 437–448Google Scholar
  19. 19.
    Lai T, Robbins H (1985) Asymptotically efficient adaptive allocation rules. Adv. Appl. Math. 6:4–22MathSciNetMATHCrossRefGoogle Scholar

Copyright information

© Springer New York 2011

Authors and Affiliations

  1. 1.Department of Computer Science, Graduate School of Information Science and TechnologyThe University of TokyoTokyoJapan
  2. 2.CREST, JSTTokyoJapan

Personalised recommendations