Skip to main content
SpringerLink
Log in

Data Analytics and Machine Learning for Coverage Closure

  • Chapter
  • First Online:
Frontiers of Quality Electronic Design (QED)

  • 787 Accesses

Abstract

Functional coverage is a de facto standard for monitoring the state of the verification process and its progress. Therefore, coverage closure, which is the process of advancing coverage goals, is an important aspect of functional verification that requires both time and expertise from the verification team. This chapter describes a collection of methods and technologies that use data analytics and machine learning to improve the quality and increase the automation in the coverage closure process. The chapter concentrates on advanced techniques and tools for coverage analysis and feedback. It starts by describing techniques for extracting concise and useful information out of coverage data and presenting them to the users. It then moves on to techniques that automate the closing of the feedback loop from coverage data to stimuli generation. All the techniques described in the chapter are accompanied by real-life examples of their usage and benefits.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 129.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Time can be measured by clock time, number of tests, number of cycles, etc.

References

  1. Aharoni, M., Asaf, S., Fournier, L., Koyfman, A., Nagel, R.: FPgen - a deep-knowledge test generator for floating point verification. In: Proceedings of the 8th High-Level Design Validation and Test Workshop, pp. 17–22 (2003)

    Google Scholar 

  2. Arar, M., et al.: The verification cockpit - creating the dream playground for data analytics over the verification process. In: Proceedings of the 11th Haifa Verification Conference, pp. 104–119 (2015)

    Google Scholar 

  3. Asaf, S., Marcus, E., Ziv, A.: Defining coverage views to improve functional coverage analysis. In: Proceedings of the 41st Design Automation Conference, pp. 41–44 (2004)

    Google Scholar 

  4. Azatchi, H., Fournier, L., Marcus, E., Ur, S., Ziv, A., Zohar, K.: Advanced analysis techniques for cross-product coverage. IEEE Trans. Comput. 55(11), 1367–1379 (2006)

    Article  Google Scholar 

  5. Birnbaum, A., Fournier, L., Mittermaier, S., Ziv, A.: Reverse coverage analysis. In: K. Eder, J. Lourenço, O. Shehory (eds.) Hardware and Software: Verification and Testing, pp. 190–202. Springer, Berlin, Heidelberg (2012)

    Chapter  Google Scholar 

  6. Blei, D., Ng, A., Jordan, M.: Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)

    MATH  Google Scholar 

  7. Carter, H.B., Hemmady, S.G.: Metric Driven Design Verification: An Engineer’s and Executive’s Guide to First Pass Success. Springer (2007)

    Google Scholar 

  8. Conn, A., Scheinberg, K., Vicente, L.: Introduction to Derivative-Free Optimization. SIAM, Philadelphia (2009)

    Book  MATH  Google Scholar 

  9. Copty, S., Fine, S., Ur, S., Yom-Tov, E., Ziv, A.: A probabilistic alternative to regression suites. Theor. Comput. Sci. 404(3), 219–234 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  10. Duff, I.S., Erisman, A.M., Reid, J.K.: Direct Methods for Sparse Matrices. Oxford University Press, USA (1986)

    MATH  Google Scholar 

  11. Elakkiya, C., Murty, N., Babu, C., Jalan, G.: Functional coverage - driven uvm based jtag verification. In: 2017 IEEE International Conference on Computational Intelligence and Computing Research (ICCIC), pp. 1–7 (2017). https://doi.org/10.1109/ICCIC.2017.8524556

  12. Fine, S., Ziv, A.: Coverage directed test generation for functional verification using Bayesian networks. In: Proceedings of the 40th Design Automation Conference, pp. 286–291 (2003)

    Google Scholar 

  13. Fine, S., Fournier, L., Ziv, A.: Using bayesian networks and virtual coverage to hit hard-to-reach events. Int. J. Softw. Tools Tech. Trans. 11(4), 291–305 (2009)

    Article  Google Scholar 

  14. Foster, H.: The 2020 wilson research group functional verification study part 8 IC/ASIC resource trends. https://blogs.sw.siemens.com/verificationhorizons/2021/01/06/part-8-the-2020-wilson-research-group-functional-verification-study

  15. Friedman, J., Hastie, T., Tibshirani, R.: The Elements of Statistical Learning, vol. 1. Springer Series in Statistics. Springer, Berlin (2001)

    Google Scholar 

  16. Gal, R., Kermany, E., Saleh, B., A.Ziv, Behm, M.L., Hickerson, B.G.: Template aware coverage: Taking coverage analysis to the next level. In: Proceedings of the 54th Design Automation Conference, pp. 36:1–36:6 (2017)

    Google Scholar 

  17. Gal, R., Haber, E., Irwin, B., Saleh, B., Ziv, A.: How to catch a lion in the desert: on the solution of the coverage directed generation (CDG) problem. Optim. Eng. (2020)

    Google Scholar 

  18. Gal, R., Haber, E., Ziv, A.: Using dnns and smart sampling for coverage closure acceleration. In: Proceedings of the 2020 ACM/IEEE Workshop on Machine Learning for CAD, pp. 15–20 (2020). https://doi.org/10.1145/3380446.3430627

  19. Gal, R., Kermany, H., Ivrii, A., Nevo, Z., Ziv, A.: Late breaking results: Friends - finding related interesting events via neighbor detection. In: Proceedings of the 57th Design Automation Conference (2020)

    Google Scholar 

  20. Gal, R., Simchoni, G., Ziv, A.: Using machine learning clustering to find large coverage holes. In: 2020 ACM/IEEE 2nd Workshop on Machine Learning for CAD (MLCAD), pp. 139–144 (2020). https://doi.org/10.1145/3380446.3430621

  21. Gal, R., Haber, E., Ibraheem, W., Irwin, B., Nevo, Z., Ziv, A.: Automatic scalable system for the coverage-directed generation (CDG) problem. In: Proceedings of the Design, Automation and Test in Europe Conference (2021)

    Google Scholar 

  22. Gal, R., Haber, E., Irwin, B., Mouallem, M., Saleh, B., Ziv, A.: Using deep neural networks and derivative free optimization to accelerate coverage closure. In: Proceedings of the 2021 ACM/IEEE Workshop on Machine Learning for CAD (2021)

    Google Scholar 

  23. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016)

    Google Scholar 

  24. Grinwald, R., Harel, E., Orgad, M., Ur, S., Ziv, A.: User defined coverage - a tool supported methodology for design verification. In: Proceedings of the 35th Design Automation Conference, pp. 158–165 (1998)

    Google Scholar 

  25. Hajjar, A., Chen, T., Munn, I., Andrews, A., Bjorkman, M.: High quality behavioral verification using statistical stopping criteria. In: Proceedings of the 2001 Design, Automation and Test in Europe Conference, pp. 411–418 (2001)

    Google Scholar 

  26. Hsiou-Wen, H., Eder, K.: Test directive generation for functional coverage closure using inductive logic programming. In: Proceedings of the High-Level Design Validation and Test Workshop, pp. 11–18 (2006)

    Google Scholar 

  27. IBM—ILOG: https://www.ibm.com/products/ilog-cplex-optimization-studio. [Online; accessed 5-August-2015]

  28. Imková, M., Kotásek, Z.: Automation and optimization of coverage-driven verification. In: 2015 Euromicro Conference on Digital System Design, pp. 87–94 (2015). https://doi.org/10.1109/DSD.2015.34

  29. James, G., Witten, D., Hastie, T., Tibshirani, R.: An Introduction to Statistical Learning with Applications in R. Springer Text in Statistics. Springer (2013)

    Book  MATH  Google Scholar 

  30. Karnaugh, M.: The map method for synthesis of combinational logic circuits. Trans. Am. Inst. Electr. Eng. 72(9), 593–599 (1953)

    MathSciNet  Google Scholar 

  31. Khan, S.S., Madden, M.G.: A survey of recent trends in one class classification. In: Coyle, L., Freyne, J. (eds.) Artificial Intelligence and Cognitive Science, pp. 188–197. Springer, Berlin, Heidelberg (2010)

    Chapter  Google Scholar 

  32. Lachish, O., Marcus, E., Ur, S., Ziv, A.: Hole analysis for functional coverage data. In: Proceedings of the 39th Design Automation Conference, pp. 807–812 (2002)

    Google Scholar 

  33. Laeufer, K., Koenig, J., Kim, D., Bachrach, J., Sen, K.: RFUZZ: coverage-directed fuzz testing of RTL on fpgas. In: Proceedings of the International Conference on Computer-Aided Design, pp. 1–8 (2018)

    Google Scholar 

  34. Lee, D.D., Seung, H.S.: Algorithms for non-negative matrix factorization. In: Leen, T.K., Dietterich, T.G., Tresp, V. (eds.) Advances in Neural Information Processing Systems, vol. 13, pp. 556–562. MIT Press (2001)

    Google Scholar 

  35. Marick, B.: The Craft of Software Testing, Subsystem Testing Including Object-Based and Object-Oriented Testing. Prentice-Hall (1985)

    Google Scholar 

  36. Mishra, P., Dutt, N.: Automatic functional test program generation for pipelined processors using model checking. In: Seventh Annual IEEE International Workshop on High-Level Design Validation and Test, pp. 99–103 (2002)

    Google Scholar 

  37. Piziali, A.: Functional Verification Coverage Measurement and Analysis. Springer (2004)

    Google Scholar 

  38. Pyne, S., Rao, B.P., Rao, S.: Big Data Analytics Methods and Applications. Springer (2016)

    Google Scholar 

  39. Reddy, C.K., Aggarwal, C.C.: Data Clustering. Chapman and Hall/CRC (2016)

    Google Scholar 

  40. Schubert, K.D., Roesner, W., Ludden, J.M., Jackson, J., Buchert, J., Paruthi, V., Behm, M., Ziv, A., Schumann, J., Meissner, C., Koesters, J., Hsu, J., Brock, B.: Functional verification of the IBM POWER7 microprocessor and POWER7 multiprocessor systems. IBM J. Res. Dev. 55(3), 308–324 (2011)

    Article  Google Scholar 

  41. Settles, B.: Active learning literature survey. Computer sciences technical report 1648, University of Wisconsin–Madison (2009)

    Google Scholar 

  42. Smith, J., Bartley, M., Fogarty, T.: Microprocessor design verification by two-phase evolution of variable length tests. In: Proceedings of the 1997 IEEE Conference on Evolutionary Computation, pp. 453–458 (1997)

    Google Scholar 

  43. Tasiran, S., Fallah, F., Chinnery, D.G., Weber, S.J., Keutzer, K.: A functional validation technique: biased-random simulation guided by observability-based coverage. In: Proceedings of the 2001 International Conference on Computer Design, pp. 82–88 (2001)

    Google Scholar 

  44. Ur, S., Yadin, Y.: Micro-architecture coverage directed generation of test programs. In: Proceedings of the 36th Design Automation Conference, pp. 175–180 (1999)

    Google Scholar 

  45. Wagner, I., Bertacco, V., Austin, T.: Microprocessor verification via feedback-adjusted Markov models. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 26(6), 1126–1138 (2007)

    Article  Google Scholar 

  46. Wile, B., Goss, J.C., Roesner, W.: Comprehensive Functional Verification - The Complete Industry Cycle. Elsevier (2005)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. IBM Research, Haifa, Haifa, Israel

    Raviv Gal, Wesam Ibraheem, Ziv Nevo, Bilal Saleh & Avi Ziv

Authors
  1. Raviv Gal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wesam Ibraheem
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ziv Nevo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bilal Saleh
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Avi Ziv
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Avi Ziv .

Editor information

Editors and Affiliations

  1. Silicon Valley Polytechnic Institute, Los Altos Hills, CA, USA

    Ali Iranmanesh

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Gal, R., Ibraheem, W., Nevo, Z., Saleh, B., Ziv, A. (2023). Data Analytics and Machine Learning for Coverage Closure. In: Iranmanesh, A. (eds) Frontiers of Quality Electronic Design (QED). Springer, Cham. https://doi.org/10.1007/978-3-031-16344-9_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-16344-9_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-16343-2

  • Online ISBN: 978-3-031-16344-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation