Skip to main content

Synthesizing Efficient Low-Precision Kernels

Part of the Lecture Notes in Computer Science book series (LNPSE,volume 11781)

Abstract

In this paper, we present a fully automated approach for synthesizing fast numerical kernels with guaranteed error bounds. The kernels we target contain elementary functions such as sine and logarithm, which are widely used in scientific computing, embedded as well as machine-learning programs. However, standard library implementations of these functions are often overly accurate and therefore unnecessarily expensive. Our approach trades superfluous accuracy against performance by approximating elementary function calls by polynomials and by implementing arithmetic operations in low-precision fixed-point arithmetic. Our algorithm soundly distributes and guarantees an overall error budget specified by the user. The evaluation on benchmarks from different domains shows significant performance improvements of 2.23\(\times \) on average compared to state-of-the-art implementations of such kernel functions.

Keywords

  • Synthesis
  • Approximation
  • Finite precision
  • Elementary functions

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-31784-3_17
  • Chapter length: 20 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   79.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-31784-3
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   99.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.

Notes

  1. 1.

    These techniques are applied to underlying roundoff error analysis, and our approach can be combined with any sound roundoff error analysis. Therefore, the application domain for our technique is only limited by what a roundoff error analysis can handle. For programs with discrete decisions – like machine-learning classifiers – the effect of the approximations on decision errors can be obtained experimentally, or to a limited extent via static analysis [31].

  2. 2.

    We optimize for running time, but our algorithm is also applicable to other objectives such as energy, with an appropriate cost function. We note that running time often correlates with energy.

  3. 3.

    All cost functions are available in the source code in repo/opt/CostFunctions.scala.

References

  1. Project CORPIN. https://www-sop.inria.fr/corpin/logiciels/ALIAS/Benches/

  2. Python sklearn - multi-layer perceptron regressor (2019). https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html

  3. Alur, R., et al.: Syntax-guided synthesis. In: FMCAD, pp. 1–8. IEEE (2013)

    Google Scholar 

  4. Alur, R., Radhakrishna, A., Udupa, A.: Scaling enumerative program synthesis via divide and conquer. In: Legay, A., Margaria, T. (eds.) TACAS 2017. LNCS, vol. 10205, pp. 319–336. Springer, Heidelberg (2017). https://doi.org/10.1007/978-3-662-54577-5_18

    CrossRef  Google Scholar 

  5. Ansel, J., Wong, Y.L., Chan, C., Olszewski, M., Edelman, A., Amarasinghe, S.: Language and compiler support for auto-tuning variable-accuracy algorithms. In: CGO (2011)

    Google Scholar 

  6. Bodik, R., Jobstmann, B.: Algorithmic program synthesis: introduction. STTT 15(5), 397–411 (2013)

    CrossRef  Google Scholar 

  7. Bornholt, J., Torlak, E., Grossman, D., Ceze, L.: Optimizing Synthesis with Metasketches. In: POPL (2016)

    CrossRef  Google Scholar 

  8. Brunie, N., De Dinechin, F., Kupriianova, O., Lauter, C.: Code generators for mathematical functions. In: ARITH (2015)

    Google Scholar 

  9. Chiang, W.F., Baranowski, M., Briggs, I., Solovyev, A., Gopalakrishnan, G., Rakamarić, Z.: Rigorous floating-point mixed-precision tuning. In: POPL (2017)

    Google Scholar 

  10. Damouche, N., Martel, M.: Mixed precision tuning with salsa. In: PECCS, pp. 185–194. SciTePress (2018)

    Google Scholar 

  11. Damouche, N., Martel, M., Chapoutot, A.: Improving the numerical accuracy of programs by automatic transformation. STTT 19(4), 427–448 (2017)

    CrossRef  Google Scholar 

  12. D’Antoni, L., Samanta, R., Singh, R.: Qlose: program repair with quantitative objectives. In: Chaudhuri, S., Farzan, A. (eds.) CAV 2016. LNCS, vol. 9780, pp. 383–401. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-41540-6_21

    CrossRef  Google Scholar 

  13. Darulova, E., Volkova, A.: Sound approximation of programs with elementary functions. In: Dillig, I., Tasiran, S. (eds.) CAV 2019. LNCS, vol. 11562, pp. 174–183. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25543-5_11

    CrossRef  Google Scholar 

  14. Darulova, E., Izycheva, A., Nasir, F., Ritter, F., Becker, H., Bastian, R.: Daisy - framework for analysis and optimization of numerical programs (tool paper). In: Beyer, D., Huisman, M. (eds.) TACAS 2018. LNCS, vol. 10805, pp. 270–287. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-89960-2_15

    CrossRef  Google Scholar 

  15. Darulova, E., Kuncak, V.: Towards a compiler for reals. TOPLAS 39(2), 8 (2017)

    CrossRef  Google Scholar 

  16. Darulova, E., Sharma, S., Horn, E.: Sound mixed-precision optimization with rewriting. In: ICCPS (2018)

    Google Scholar 

  17. De Dinechin, F., Lauter, C.Q., Melquiond, G.: Assisted verification of elementary functions using Gappa. In: ACM Symposium on Applied Computing (2006)

    Google Scholar 

  18. Esmaeilzadeh, H., Sampson, A., Ceze, L., Burger, D.: Neural acceleration for general-purpose approximate programs. In: MICRO (2012)

    Google Scholar 

  19. Feng, Y., Martins, R., Wang, Y., Dillig, I., Reps, T.W.: Component-based synthesis for complex APIs. In: POPL (2017)

    Google Scholar 

  20. de Figueiredo, L.H., Stolfi, J.: Affine arithmetic: concepts and applications. Numer. Algorithms 37(1–4), 147–158 (2004)

    MathSciNet  CrossRef  Google Scholar 

  21. Goubault, E., Putot, S.: Static analysis of finite precision computations. In: Jhala, R., Schmidt, D. (eds.) VMCAI 2011. LNCS, vol. 6538, pp. 232–247. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-18275-4_17

    CrossRef  Google Scholar 

  22. Gulwani, S.: Automating string processing in spreadsheets using input-output examples. In: POPL (2011)

    CrossRef  Google Scholar 

  23. IEEE: IEEE Standard for Floating-Point Arithmetic. IEEE Std 754–2008 (2008)

    Google Scholar 

  24. Kneuss, E., Kuraj, I., Kuncak, V., Suter, P.: Synthesis modulo recursive functions. In: OOPSLA (2013)

    Google Scholar 

  25. Kuncak, V., Mayer, M., Piskac, R., Suter, P.: Complete functional synthesis. In: PLDI (2010)

    Google Scholar 

  26. Kupriianova, O., Lauter, C.: A domain splitting algorithm for the mathematical functions code generator. In: Asilomar (2014)

    Google Scholar 

  27. Kupriianova, O., Lauter, C.: Metalibm: a mathematical functions code generator. In: Hong, H., Yap, C. (eds.) ICMS 2014. LNCS, vol. 8592, pp. 713–717. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-44199-2_106

    CrossRef  MATH  Google Scholar 

  28. Lee, D.U., Gaffar, A.A., Cheung, R.C., Mencer, O., Luk, W., Constantinides, G.A.: Accuracy-guaranteed bit-width optimization. TCAD 25(10), 1990–2000 (2006)

    Google Scholar 

  29. Lee, S., John, L.K., Gerstlauer, A.: High-level synthesis of approximate hardware under joint precision and voltage scaling. In: DATE, pp. 187–192 (2017)

    Google Scholar 

  30. Lee, W., Sharma, R., Aiken, A.: On automatically proving the correctness of math.h implementations. Proc. ACM Program. Lang. 2(POPL), 47 (2018)

    Google Scholar 

  31. Lohar, D., Darulova, E., Putot, S., Goubault, E.: Discrete choice in the presence of numerical uncertainties. IEEE TCAD 37, 2381–2392 (2018)

    Google Scholar 

  32. Loncaric, C., Torlak, E., Ernst, M.D.: Fast synthesis of fast collections. ACM SIGPLAN Not. 51(6), 355–368 (2016)

    CrossRef  Google Scholar 

  33. Magron, V., Constantinides, G., Donaldson, A.: Certified roundoff error bounds using semidefinite programming. ACM Trans. Math. Softw. 43(4), 34 (2017)

    MathSciNet  CrossRef  Google Scholar 

  34. Misailovic, S., Carbin, M., Achour, S., Qi, Z., Rinard, M.C.: Chisel: reliability- and accuracy-aware optimization of approximate computational kernels. In: OOPSLA (2014)

    Google Scholar 

  35. Moore, R.: Interval Analysis. Prentice-Hall, Upper Saddle River (1966)

    MATH  Google Scholar 

  36. Moscato, M., Titolo, L., Dutle, A., Muñoz, C.A.: Automatic estimation of verified floating-point round-off errors via static analysis. In: Tonetta, S., Schoitsch, E., Bitsch, F. (eds.) SAFECOMP 2017. LNCS, vol. 10488, pp. 213–229. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66266-4_14

    CrossRef  Google Scholar 

  37. Briesbarre, N., Chevillard, S.: Efficient polynomial l-approximations. In: ARITH (2007)

    Google Scholar 

  38. Neider, D., Saha, S., Madhusudan, P.: Compositional synthesis of piece-wise functions by learning classifiers. ACM Trans. Comput. Logic 19(2), 10:1–10:23 (2018)

    MathSciNet  CrossRef  Google Scholar 

  39. Nori, A.V., Ozair, S., Rajamani, S.K., Vijaykeerthy, D.: Efficient synthesis of probabilistic programs. In: PLDI. ACM (2015)

    CrossRef  Google Scholar 

  40. Braojos, R., Ansaloni, G., Atienza, D.: A methodology for embedded classification of heartbeats using random projections. In: DATE. EPFL (2013)

    Google Scholar 

  41. Renganarayana, L., Srinivasan, V., Nair, R., Prener, D.: Programming with relaxed synchronization. In: RACES, pp. 41–50 (2012)

    Google Scholar 

  42. Schkufza, E., Sharma, R., Aiken, A.: Stochastic optimization of floating-point programs with tunable precision. In: PLDI (2014)

    Google Scholar 

  43. Sidiroglou-Douskos, S., Misailovic, S., Hoffmann, H., Rinard, M.: Managing performance vs. accuracy trade-offs with loop perforation. In: ESEC/FSE (2011)

    Google Scholar 

  44. Solovyev, A., Jacobsen, C., Rakamaric, Z., Gopalakrishnan, G.: Rigorous estimation of floating-point round-off errors with symbolic taylor expansions. In: FM (2015)

    Google Scholar 

  45. Xilinx: Vivado design suite (2018). https://www.xilinx.com/products/design-tools/vivado.html

  46. Xu, Q., Mytkowicz, T., Kim, N.S.: Approximate computing: a survey. IEEE Des. Test 33(1), 8–22 (2016)

    CrossRef  Google Scholar 

  47. Yazdanbakhsh, A., Mahajan, D., Esmaeilzadeh, H., Lotfi-Kamran, P.: AxBench: a multiplatform benchmark suite for approximate computing. IEEE Des. Test 34(2), 60–68 (2017)

    CrossRef  Google Scholar 

  48. Yi, X., Chen, L., Mao, X., Ji, T.: Efficient automated repair of high floating-point errors in numerical libraries. In: POPL (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Anastasiia Izycheva .

Editor information

Editors and Affiliations

A Benchmarks

A Benchmarks

All benchmarks are provided below. Error specification given in the ensuring clause corresponds to the small error, the larger errors are given in comments.

figure b
figure c
figure d

Rights and permissions

Reprints and Permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Izycheva, A., Darulova, E., Seidl, H. (2019). Synthesizing Efficient Low-Precision Kernels. In: Chen, YF., Cheng, CH., Esparza, J. (eds) Automated Technology for Verification and Analysis. ATVA 2019. Lecture Notes in Computer Science(), vol 11781. Springer, Cham. https://doi.org/10.1007/978-3-030-31784-3_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-31784-3_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-31783-6

  • Online ISBN: 978-3-030-31784-3

  • eBook Packages: Computer ScienceComputer Science (R0)