Skip to main content
SpringerLink
Log in

Machine Learning in the Service of Hardware Functional Verification

  • Chapter
  • First Online:
Machine Learning Applications in Electronic Design Automation

  • 1742 Accesses

Abstract

Modern hardware functional verification employs many different tools alongside large compute farms to ensure that the hardware’s implementation matches its specification. This produces a lot of data that can be used to better monitor and control the verification process. Data science in general and machine learning specifically are disciplines in computer science that deal with extracting patterns and information from datasets. This chapter shows how these technologies can be integrated into the verification process in a holistic manner and become an integral part of the verification process backbone. The chapter begins with examples on the use of machines learning in specific verification tools. This is followed by a description of how to connect the various verification tools and data sources and create a unified data repository in a data warehouse that is optimized for data retrieval. This connection allows the use of advanced data science techniques that improve the quality of the entire verification process.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 119.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    To avoid tedious repetitions, we use the general term tests from here on, unless the distinction is important.

  2. 2.

    We assume that simulation is deterministic. Verification tools work very hard to maintain this assumption because of the difficulty in debugging non-deterministic simulations.

  3. 3.

    The exception is when a search for one event accidentally hit another, unrelated event with a high enough probability, making the search for the event unnecessary.

References

  1. Aarts, E., Korst, J., Michiels, W.: Simulated annealing. In: Burke, E.K., Kendall, G. (eds.) Search Methodologies: Introductory Tutorials in Optimization and Decision Support Techniques, pp. 187–210. Springer US, Boston, MA (2005). https://doi.org/10.1007/0-387-28356-0_7

    Chapter  Google Scholar 

  2. Amrani, M., Lúcio, L., Bibal, A.: ML + FV = ? A survey on the application of machine learning to formal verification. CoRR abs/1806.03600 (2018). https://arxiv.org/abs/1806.03600

  3. Angell, R., Oztalay, B., DeOrio, A.: A topological approach to hardware bug triage. In: 2015 16th International Workshop on Microprocessor and SOC Test and Verification (MTV), pp. 20–25 (2015). https://doi.org/10.1109/MTV.2015.10

  4. Arar, M., Behm, M., Boni, O., Gal, R., Goldin, A., Ilyaev, M., Kermany, E., Reysa, J., Saleh, B., Schubert, K.D., Shurek, G., Ziv, A.: The verification cockpit—creating the dream playground for data analytics over the verification process. In: Proceedings of the 11th Haifa Verification Conference, pp. 104–119 (2015)

    Google Scholar 

  5. Azatchi, H., Fournier, L., Marcus, E., Ur, S., Ziv, A., Zohar, K.: Advanced analysis techniques for cross-product coverage. IEEE Trans. Comput. 55(11), 1367–1379 (2006)

    Article  Google Scholar 

  6. Baras, D., Fine, S., Fournier, L., Geiger, D., Ziv, A.: Automatic boosting of cross-product coverage using Bayesian networks. Int. J. Softw. Tools Technol. Transfer 13(3), 247–261 (2011)

    Article  Google Scholar 

  7. Biere, A., Heule, M., van Maaren, H., Walsch, T.: Handbook of Satisfiability. IOS Press, New York (2008)

    Google Scholar 

  8. Bin, E., Emek, R., Shurek, G., Ziv, A.: Using a constraint satisfaction formulation and solution techniques for random test program generation. IBM Syst. J. 41(3), 386–402 (2002)

    Article  Google Scholar 

  9. Birnbaum, A., Fournier, L., Mittermaier, S., Ziv, A.: Reverse coverage analysis. In: Eder, K., Lourenço, J., Shehory, O. (eds.) Hardware and Software: Verification and Testing, pp. 190–202. Springer, Berlin (2012)

    Chapter  Google Scholar 

  10. Blackmore, T., Hodson, R., Schaal, S.: Novelty-driven verification: Using machine learning to identify novel stimuli and close coverage. In: Proceedings of the design and verification conference and exhibition US (DVCon) (2021)

    Google Scholar 

  11. Blum, A., Hopcroft, J., Kannan, R.: Foundations of Data Science. Cambridge University Press, Cambridge (2020)

    Book  MATH  Google Scholar 

  12. Campenhout, D.V., Mudge, T., Hayes, J.P.: High-level test generation for design verification of pipelined microprocessors. In: Proceedings of the 36th Design Automation Conference, pp. 185–188 (1999)

    Google Scholar 

  13. Chatfield, C., Xing, H.: The Analysis of Time Series: An Introduction with R. CRC Press, New York (2019)

    Book  MATH  Google Scholar 

  14. Clarke, E.M., Grumberg, O., Peled, D.A.: Model Checking. MIT-Press, New York (1999)

    MATH  Google Scholar 

  15. Clerc, M.: Particle Swarm Optimization. ISTE. Wiley, New York (2010). https://books.google.co.il/books?id=Slee72idZ8EC

  16. Conn, A., Scheinberg, K., Vicente, L.: Introduction to Derivative-Free Optimization. SIAM, Philadelphia (2009)

    Book  MATH  Google Scholar 

  17. Copty, S., Fine, S., Ur, S., Yom-Tov, E., Ziv, A.: A probabilistic alternative to regression suites. Theor. Comput. Sci. 404(3), 219–234 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  18. Davis, M., Putnam, H.: A computing procedure for quantification theory. J. ACM 7(3), 201–215 (1960). https://doi.org/10.1145/321033.321034

    Article  MathSciNet  MATH  Google Scholar 

  19. Efendioglu, M., Sen, A., Koroglu, Y.: Bug prediction of systemC models using machine learning. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 38(3), 419–429 (2019). https://doi.org/10.1109/TCAD.2018.2878193

    Article  Google Scholar 

  20. Fine, S., Freund, A., Jaeger, I., Mansour, Y., Naveh, Y., Ziv, A.: Harnessing machine learning to improve the success rate of stimuli generation. IEEE Trans. Comput. 55(11), 1344–1355 (2006)

    Article  Google Scholar 

  21. Fine, S., Fournier, L., Ziv, A.: Using Bayesian networks and virtual coverage to hit hard-to-reach events. Int. J. Softw. Tools Technol. Transfer 11(4), 291–305 (2009)

    Article  Google Scholar 

  22. Fine, S., Ziv, A.: Coverage directed test generation for functional verification using Bayesian networks. In: Proceedings of the 40th Design Automation Conference, pp. 286–291 (2003)

    Google Scholar 

  23. Foster, H.: The 2020 Wilson research group functional verification study part 8 IC/ASIC resource trends. https://blogs.sw.siemens.com/verificationhorizons/2021/01/06/part-8-the-2020-wilson-research-group-functional-verification-study

  24. Gal, R., Kermany, E., Saleh, B., A.Ziv, Behm, M.L., Hickerson, B.G.: Template aware coverage: Taking coverage analysis to the next level. In: Proceedings of the 54th Design Automation Conference, pp. 36:1–36:6 (2017)

    Google Scholar 

  25. Gal, R., Shurek, G., Simchoni, G., Ziv, A.: Risk analysis based on design version control data. In: 2019 ACM/IEEE 1st Workshop on Machine Learning for CAD (MLCAD), pp. 1–6 (2019). https://doi.org/10.1109/MLCAD48534.2019.9142105

  26. Gal, R., Haber, E., Ziv, A.: Using dnns and smart sampling for coverage closure acceleration. In: Proceedings of the 2020 ACM/IEEE Workshop on Machine Learning for CAD, pp. 15–20 (2020). https://doi.org/10.1145/3380446.3430627

  27. Gal, R., Kermany, H., Ivrii, A., Nevo, Z., Ziv, A.: Late breaking results: Friends—finding related interesting events via neighbor detection. In: 2020 57th ACM/IEEE Design Automation Conference (DAC), pp. 1–2 (2020). https://doi.org/10.1109/DAC18072.2020.9218685

  28. Gal, R., Simchoni, G., Ziv, A.: Using machine learning clustering to find large coverage holes. In: Proceedings of the 2020 ACM/IEEE Workshop on Machine Learning for CAD, MLCAD ’20, pp. 139–144. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3380446.3430621

  29. Gal, R., Haber, E., Ibraheem, W., Irwin, B., Nevo, Z., Ziv, A.: Automatic scalable system for the coverage-directed generation (CDG) problem. In: Proceedings of the Design, Automation and Test in Europe Conference (2021)

    Google Scholar 

  30. Gal, R., Haber, E., Irwin, B., Mouallem, M., Saleh, B., Ziv, A.: Using deep neural networks and derivative free optimization to accelerate coverage closure. In: Proceedings of the 2021 ACM/IEEE Workshop on Machine Learning for CAD (2021)

    Google Scholar 

  31. Gal, R., Haber, E., Irwin, B., Saleh, B., Ziv, A.: How to catch a lion in the desert: on the solution of the coverage directed generation (CDG) problem. Optim. Eng. 22(1), 217–245 (2021). https://doi.org/10.1007/s11081-020-09507-w

    Article  MathSciNet  MATH  Google Scholar 

  32. Geng, H.: Semiconductor Manufacturing Handbook. McGraw-Hill Education, New York (2017)

    Google Scholar 

  33. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, New York (2016). http://www.deeplearningbook.org

    MATH  Google Scholar 

  34. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, New York (2016)

    MATH  Google Scholar 

  35. Goyal, A., Sardana, N.: Empirical analysis of ensemble machine learning techniques for bug triaging. In: 2019 Twelfth International Conference on Contemporary Computing (IC3), pp. 1–6 (2019). https://doi.org/10.1109/IC3.2019.8844876

  36. Grinwald, R., Harel, E., Orgad, M., Ur, S., Ziv, A.: User defined coverage—a tool supported methodology for design verification. In: Proceedings of the 35th Design Automation Conference, pp. 158–165 (1998)

    Google Scholar 

  37. Guo, Q., Chen, T., Shen, H., Chen, Y., Hu, W.: On-the-fly reduction of stimuli for functional verification. In: 2010 19th IEEE Asian Test Symposium, pp. 448–454 (2010). https://doi.org/10.1109/ATS.2010.82

  38. Guo, Q., Chen, T., Chen, Y., Wang, R., Chen, H., Hu, W., Chen, G.: Pre-silicon bug forecast. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 33(3), 451–463 (2014). https://doi.org/10.1109/TCAD.2013.2288688

    Article  Google Scholar 

  39. Guzey, O., Wang, L.C., Levitt, J.R., Foster, H.: Increasing the efficiency of simulation-based functional verification through unsupervised support vector analysis. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 29(1), 138–148 (2010). https://doi.org/10.1109/TCAD.2009.2034347

    Article  Google Scholar 

  40. Haim, S., Walsh, T.: Restart strategy selection using machine learning techniques. In: Kullmann, O. (ed.) Theory and Applications of Satisfiability Testing—SAT 2009, pp. 312–325. Springer, Berlin (2009)

    Chapter  Google Scholar 

  41. Hajjar, A., Chen, T., Munn, I., Andrews, A., Bjorkman, M.: High quality behavioral verification using statistical stopping criteria. In: Proceedings of the 2001 Design, Automation and Test in Europe Conference, pp. 411–418 (2001)

    Google Scholar 

  42. Hamilton, W.L.: Graph Representation Learning. Morgan & Claypool Publishers, Los Altos (2020)

    Book  MATH  Google Scholar 

  43. Hsiou-Wen, H., Eder, K.: Test directive generation for functional coverage closure using inductive logic programming. In: Proceedings of the High-Level Design Validation and Test Workshop, pp. 11–18 (2006)

    Google Scholar 

  44. Ibm cognos analytics. https://www.ibm.com/products/cognos-analytics

  45. IBM DB2 with BLU Acceleration. https://www.redbooks.ibm.com/abstracts/tips1204.html

  46. Iwashita, H., Kowatari, S., Nakata, T., Hirose, F.: Automatic test program generation for pipelined processors. In: Proceedings of the International Conference on Computer Aided Design, pp. 580–583 (1994)

    Google Scholar 

  47. James, G., Witten, D., Hastie, T., Tibshirani, R.: An Introduction to Statistical Learning with Applications in R. Springer Text in Statistics. Springer, Berlin (2013)

    Google Scholar 

  48. Karnaugh, M.: The map method for synthesis of combinational logic circuits. Trans. Am. Inst. Electr. Eng. 72(9), 593–599 (1953)

    MathSciNet  Google Scholar 

  49. Kelley, C.: Implicit Filtering. SIAM, Philadelphia (2011)

    Book  MATH  Google Scholar 

  50. Lachish, O., Marcus, E., Ur, S., Ziv, A.: Hole analysis for functional coverage data. In: Proceedings of the 39th Design Automation Conference, pp. 807–812 (2002)

    Google Scholar 

  51. Lagoudakis, M.G., Littman, M.L.: Learning to select branching rules in the DPLL procedure for satisfiability. Electron Notes Discrete Math. 9, 344–359 (2001). https://doi.org/10.1016/S1571-0653(04)00332-4. https://www.sciencedirect.com/science/article/pii/S1571065304003324. LICS 2001 Workshop on Theory and Applications of Satisfiability Testing (SAT 2001)

  52. Lee, D.D., Seung, H.S.: Algorithms for non-negative matrix factorization. In: Leen, T.K., Dietterich, T.G., Tresp, V. (eds.) Advances in Neural Information Processing Systems, vol. 13, pp. 556–562 (2001)

    Google Scholar 

  53. Lewis, C., Ou, R.: Bug prediction at Google (2011). http://google-engtools.blogspot.sg/2011/12/bug-prediction-at-google.html. [Online; accessed 25-Oct-2019]

  54. Liang, J., Ganesh, V., Poupart, P., Czarnecki, K.: Exponential recency weighted average branching heuristic for sat solvers. Proceedings of the AAAI Conference on Artificial Intelligence 30(1) (2016). https://ojs.aaai.org/index.php/AAAI/article/view/10439

  55. Lyu, M.: The Handbook of Software Reliability Engineering. McGraw Hill, New York (1996)

    Google Scholar 

  56. Malka, Y., Ziv, A.: Design reliability—estimation through statistical analysis of bug discovery data. In: Proceedings of the 35th Design Automation Conference, pp. 644–649 (1998)

    Google Scholar 

  57. Mishra, P., Dutt, N.: Automatic functional test program generation for pipelined processors using model checking. In: Seventh Annual IEEE International Workshop on High-Level Design Validation and Test, pp. 99–103 (2002)

    Google Scholar 

  58. Mockus, J.: Bayesian Approach to Global Optimization: Theory and Applications. Springer, Berlin (1989)

    Book  MATH  Google Scholar 

  59. Molitor, P., Mohnke, J.: Equivalence Checking of Digital Circuits: Fundamentals, Principles, Methods. Kluwer Academic Publishers, Dordrecht (2004)

    MATH  Google Scholar 

  60. Nudelman, E., Leyton-Brown, K., Hoos, H.H., Devkar, A., Shoham, Y.: Understanding random SAT: Beyond the clauses-to-variables ratio. In: Wallace, M. (ed.) Principles and Practice of Constraint Programming—CP 2004, pp. 438–452. Springer, Berlin (2004)

    Chapter  Google Scholar 

  61. Parizy, M., Takayama, K., Kanazawa, Y.: Software defect prediction for LSI designs. In: 2014 IEEE International Conference on Software Maintenance and Evolution, pp. 565–568 (2014). https://doi.org/10.1109/ICSME.2014.96

  62. Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Network of Plausible Inference. Morgan Kaufmann, Los Altos (1988)

    MATH  Google Scholar 

  63. PinDown—an Automatic Debugger of Regression Failures. https://verifyter.com/technology/debug. [Online; accessed 25-Oct-2021]

  64. Piziali, A.: Functional Verification Coverage Measurement and Analysis. Springer, Berlin (2004)

    Google Scholar 

  65. Ponniah, P.: Data Warehousing Fundamentals for IT Professionals, 2nd edn. Wiley, Hoboken (2010)

    Book  Google Scholar 

  66. Poulos, Z., Veneris, A.: Exemplar-based failure triage for regression design debugging. In: 2015 16th Latin-American Test Symposium (LATS), pp. 1–6 (2015). https://doi.org/10.1109/LATW.2015.7102521

  67. Pyne, S., Rao, B.P., Rao, S.: Big Data Analytics Methods and Applications. Springer, Berlin (2016)

    Google Scholar 

  68. Reddy, C.K., Aggarwal, C.C.: Data Clustering. Chapman and Hall/CRC, New York (2016)

    Google Scholar 

  69. Rolfsnes, T., Alesio, S.D., Behjati, R., Moonen, L., Binkley, D.W.: Generalizing the analysis of evolutionary coupling for software change impact analysis. In: IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), pp. 201–212 (2016)

    Google Scholar 

  70. Roy, R., Duvedi, C., Godil, S., Williams, M.: Deep predictive coverage collection. In: Proceedings of the Design and Verification Conference and Exhibition US (DVCon) (2018)

    Google Scholar 

  71. Roy, R., Benipal, M., Godil, S.: Dynamically optimized test generation using machine learning. In: Proceedings of the Design and Verification Conference and Exhibition US (DVCon) (2021)

    Google Scholar 

  72. Schubert, K.D., Roesner, W., Ludden, J.M., Jackson, J., Buchert, J., Paruthi, V., Behm, M., Ziv, A., Schumann, J., Meissner, C., Koesters, J., Hsu, J., Brock, B.: Functional verification of the IBM POWER7 microprocessor and POWER7 multiprocessor systems. IBM J. Res. Dev. 55(3), 308–324 (2011)

    Article  Google Scholar 

  73. Schubert, K., et al.: Solutions to IBM POWER8 verification challenges. IBM J. Res. Dev. 59(1), 1–17 (2015)

    Article  Google Scholar 

  74. Shalev-Shwartz, S., Ben-David, S.: Understanding Machine Learning: From Theory to Algorithms. Cambridge University, Cambridge (2014)

    Book  MATH  Google Scholar 

  75. Shawe-Taylor, J., Cristianini, N.: Kernel methods: An overview. In: Kernel Methods for Pattern Analysis, pp. 25–44. Cambridge University, Cambridge (2004)

    Google Scholar 

  76. Shewhart, W.: Statistical Method from the Viewpoint of Quality Control. In: Dover Books on Mathematics. Dover Publications, New York (2012)

    Google Scholar 

  77. Sokorac, S.: Optimizing random test constraints using machine learning algorithms. In: Proceedings of the Design and Verification Conference and Exhibition US (DVCon) (2017)

    Google Scholar 

  78. Thompson, M.P., Hamann, J.D., Sessions, J.: Selection and penalty strategies for genetic algorithms designed to solve spatial forest planning problems. International Journal of Forestry Research 2009, 1–14 (2009). https://doi.org/10.1155/2009/527392

    Article  Google Scholar 

  79. Ur, S., Yadin, Y.: Micro-architecture coverage directed generation of test programs. In: Proceedings of the 36th Design Automation Conference, pp. 175–180 (1999)

    Google Scholar 

  80. Vasudevan, S., Sheridan, D., Patel, S., Tcheng, D., Tuohy, B., Johnson, D.: Goldmine: Automatic assertion generation using data mining and static analysis. In: 2010 Design, Automation Test in Europe Conference Exhibition (DATE 2010), pp. 626–629 (2010). https://doi.org/10.1109/DATE.2010.5457129

  81. Vasudevan, S., Jiang, W., Bieber, D., Singh, R., Hamid Shojaei, Ho, R., Sutton, C.: Learning semantic representations to verify hardware designs. In: Advances in Neural Information Processing Systems 34 (NeurIPS 2021) (2021)

    Google Scholar 

  82. Wagner, I., Bertacco, V., Austin, T.: Microprocessor verification via feedback-adjusted Markov models. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 26(6), 1126–1138 (2007)

    Article  Google Scholar 

  83. Wikipedia: Product lifecycle—Wikipedia, the free encyclopedia. https://en.wikipedia.org/wiki/Product_lifecycle. [Online; accessed 25-Oct-2021]

  84. Wikipedia: Representational state transfer—Wikipedia, the free encyclopedia. https://en.wikipedia.org/wiki/Representational_state_transfer. [Online; accessed 25-Oct-2021]

  85. Wile, B., Goss, J.C., Roesner, W.: Comprehensive Functional Verification—The Complete Industry Cycle. Elsevier, Amsterdam (2005)

    Google Scholar 

  86. Wu, B.H., Huang, C.Y.: A robust constraint solving framework for multiple constraint sets in constrained random verification. In: 2013 50th ACM/EDAC/IEEE Design Automation Conference (DAC), pp. 1–7 (2013)

    Google Scholar 

  87. XGBoost Documentation. https://xgboost.readthedocs.io/en/latest/. [Online; accessed 25-Oct-2021]

  88. Xu, L., Hutter, F., Hoos, H.H., Leyton-Brown, K.: SATzilla: Portfolio-based algorithm selection for SAT. J. Artif. Intell. Res. 32(3), 201–215 (2008). https://doi.org/10.1613/jair.2490

    MATH  Google Scholar 

  89. Zhou, Z.H.: Ensemble Methods: Foundations and Algorithms. CRC Press, New York (2012)

    Book  Google Scholar 

  90. Zimmermann, T., Zeller, A., Weissgerber, P., Diehl, S.: Mining version histories to guide software changes. IEEE Trans. Softw. Eng. 31(6), 429–445 (2005)

    Article  Google Scholar 

  91. Zuse, H.: Software Complexity—Measures and Methods. Walter de Gruyter, Berlin (2019)

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. IBM Research, Haifa, Israel

    Raviv Gal & Avi Ziv

Authors
  1. Raviv Gal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Avi Ziv
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Avi Ziv .

Editor information

Editors and Affiliations

  1. NVIDIA, Austin, TX, USA

    Haoxing Ren

  2. Texas A&M University, College Station, TX, USA

    Jiang Hu

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Gal, R., Ziv, A. (2022). Machine Learning in the Service of Hardware Functional Verification. In: Ren, H., Hu, J. (eds) Machine Learning Applications in Electronic Design Automation. Springer, Cham. https://doi.org/10.1007/978-3-031-13074-8_14

Download citation

Publish with us

Policies and ethics

Search

Navigation