Skip to main content
SpringerLink
Log in

A Comparative Study of Assertion Mining Algorithms in GoldMine

  • Chapter
  • First Online:
Machine Learning in VLSI Computer-Aided Design

Abstract

GoldMine automatically generates assertions for register transfer level (RTL) designs using a combination of static analysis and machine learning algorithms. We compare four different machine learning algorithms implemented in GoldMine with respect to metrics that determine quality of their results. Metrics include assertion complexity, coverage, expectedness, predictive accuracy, and number of assertions. We introduce a new statically guided assertion mining algorithm, the best-gain decision forest (BGDF) algorithm here. In our comparative study, two of the algorithms have been invented by us for assertion generation, while the two others are standard machine learning algorithms. The best-gain decision forest and coverage guided association mining algorithms developed by us are compared with the decision tree and PRISM algorithms. We provide an extensive comparative study of assertion mining algorithms with experimental results on realistic designs. We also hereby provide access to these implementations in GoldMine that can be downloaded from [64] for research purposes.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    GoldMine generates assertions in linear temporal logic (LTL) of the form A => C where the antecedent A and consequent C can be propositional or temporal.

  2. 2.

    We use the terms decision forest and acronym BGDF interchangeably through this chapter.

  3. 3.

    X refers to a “don’t care” value since the output does not affect the input space coverage.

References

  1. ActiveProp assertion-based verification system, http://www.jasper-da.com/products/activeprop-assertion-based-verification-system

  2. V. Athavale, S. Ma, S. Hertz, S. Vasudevan, Code coverage of assertions using RTL source code analysis, in The 51st Annual Design Automation Conference 2014, DAC ‘14, San Francisco, CA, June 1–5, 2014 (2014), pp. 61:1–61:6

    Google Scholar 

  3. R. Agrawal, T. Imielinski, A. Swami, Mining association rules between sets of items in large databases, in Proceedings of SIGMOD (1993), pp. 207–216

    Google Scholar 

  4. G. Ammons, R. Bodk, J.R. Larus, Mining specifications, in POPL (2002), pp. 4–16

    Google Scholar 

  5. Assertion synthesis, http://www.atrenta.com/about-assertion_based-verification.htm5

  6. V. Athavale, S. Hertz, S. Vasudevan, Evaluating code coverage of assertions by static analysis of RTL. Technical report uilu-eng-11-2209 (2011)

    Google Scholar 

  7. S. Bensalem, Y. Lakhnech, H. Sadi, Powerful techniques for the automatic generation of invariants, in Proceedings of CAV (1996), pp. 323–335

    Google Scholar 

  8. A. Biere, E.M. Clarke, R. Raimi, Y. Zhu, Verifying safety properties of a power PC microprocessor using symbolic model checking without BDDs, in Computer Aided Verification, 11th International Conference, CAV ‘99, Trento, July 6–10, 1999, Proceedings (1999), pp. 60–71

    Google Scholar 

  9. L. Breiman, Random forests. Mach. Learn. 45(1), 5–32 (2001)

    Article  Google Scholar 

  10. M. Caplain, Finding invariant assertions for proving programs, in Proceedings of the International Conference on Reliable Software (ACM, New York, 1975), pp. 165–171

    Google Scholar 

  11. J. Cendrowska, Prism: an algorithm for inducing modular rules. Int. J. Man-Mach. Stud. 27(4), 349–370 (1987)

    Article  Google Scholar 

  12. P.-H. Chang, L.-C. Wang, Automatic assertion extraction via sequential data mining of simulation traces, in Proceedings of ASPDAC (2010), pp. 607–612

    Google Scholar 

  13. H. Chao, H. Li, X. Song, T. Wang, X. Li, On evaluating and constraining assertions using conflicts in absent scenarios, in 2017 IEEE 26th Asian Test Symposium (ATS), Nov 2017, pp. 195–200

    Google Scholar 

  14. X. Cheng, M.S. Hsiao, Simulation-directed invariant mining for software verification, in Proceedings of DATE (2008), pp. 682–687

    Google Scholar 

  15. C.-N. Chung, C.-W. Chang, K.-H. Chang, S.-Y. Kuo, Applying verification intention for design customization via property mining under constrained testbenches, in Proceedings of ICCD (2011), pp. 84–89

    Google Scholar 

  16. E.M. Clarke, M. Fujita, S.P. Rajan, T.W. Reps, S. Shankar, T. Teitelbaum, Program slicing of hardware description languages, in Correct Hardware Design and Verification Methods, 10th IFIP WG 10.5 Advanced Research Working Conference, CHARME ‘99, Bad Herrenalb, September 27–29, 1999, Proceedings (1999), pp. 298–312

    Google Scholar 

  17. E.M. Clarke, A. Biere, R. Raimi, Y. Zhu, Bounded model checking using satisfiability solving. Formal Methods Syst. Des. 19(1), 7–34 (2001)

    Article  Google Scholar 

  18. A.M. Cruz, R.B. Fernández, H.M. Lozano, M.A. Ramírez Salinas, L.A. Villa Vargas, Automated functional test generation for digital systems through a compact binary differential evolution algorithm. J. Electron. Test. 31(4), 361–380 (2015)

    Article  Google Scholar 

  19. A. Danese, F. Filini, G. Pravadelli, A time-window based approach for dynamic assertions mining on control signals, in 2015 IFIP/IEEE International Conference on Very Large Scale Integration (VLSI-SoC), Oct 2015, pp. 246–251

    Google Scholar 

  20. A. DeOrio, A. Bauserman, V. Bertacco, B. Isaksen, Inferno: streamlining verification with inferred semantics. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 28(5), 728–741 (2009)

    Article  Google Scholar 

  21. M.D. Ernst, J.H. Perkins, P.J. Guo, S. McCamant, C. Pacheco, M.S. Tschantz, C. Xiao, The daikon system for dynamic detection of likely invariants. J. Sci. Comput. Program. 69, 35–45 (2007)

    Article  MathSciNet  Google Scholar 

  22. F. Farahmandi, R. Morad, A. Ziv, Z. Nevo, P. Mishra, Cost-effective analysis of post-silicon functional coverage events, in Design, Automation & Test in Europe Conference & Exhibition, DATE 2017, Lausanne, March 27–31, 2017 (2017), pp. 392–397

    Google Scholar 

  23. H.D. Foster, A.C. Krolnik, D.J. Lacey, Assertion-Based Design, 2nd edn. (Springer Publishing Company, New York, 2010)

    Google Scholar 

  24. T. Ghasempouri, G. Pravadelli, On the estimation of assertion interestingness, in 2015 IFIP/IEEE International Conference on Very Large Scale Integration (VLSI-SoC), Oct 2015, pp. 325–330

    Google Scholar 

  25. O. Grumberg, H. Veith (eds.), 25 Years of Model Checking: History, Achievements, Perspectives (Springer, Berlin, 2008)

    MATH  Google Scholar 

  26. J. Han, M. Kamber, Data Mining: Concepts and Techniques (Morgan Kaufmann Publishers Inc., San Francisco, 2000)

    MATH  Google Scholar 

  27. M. Hanafy, H. Said, A.M. Wahba, Complete properties extraction from simulation traces for assertions auto-generation, in 2015 IEEE 24th North Atlantic Test Workshop, May 2015, pp. 1–6

    Google Scholar 

  28. M. Hanafy, H. Said, A.M. Wahba, New methodology for complete properties extraction from simulation traces guided with static analysis. J. Electron. Test. 32(6), 705–719 (2016)

    Article  Google Scholar 

  29. S. Hangal, N. Chandra, S. Narayanan, S. Chakravorty, Iodine: a tool to automatically infer dynamic invariants for hardware designs, in Proceedings of DAC (2005), pp. 775–778

    Google Scholar 

  30. A. Hekmatpour, A. Salehi, Block-based schema-driven assertion generation for functional verification, in Proceedings of ATS (2005), pp. 34–39

    Google Scholar 

  31. S. Hertz, Enhancing quality of assertion generation: methods for automatic assertion generation and evaluation. PhD thesis, The University of Illinois at Urbana-Champaign, 2013

    Google Scholar 

  32. S. Hertz, D. Sheridan, S. Vasudevan, Mining hardware assertions with guidance from static analysis. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 32, 952–965 (2013)

    Article  Google Scholar 

  33. T.K. Ho, The random subspace method for constructing decision forests. IEEE Trans. Pattern Anal. Mach. Intell. 20(8), 832–844 (1998)

    Article  Google Scholar 

  34. C.A.R. Hoare, An axiomatic basis for computer programming. Commun. ACM 12(10), 576–580 (1969)

    Article  Google Scholar 

  35. IEEE standard for verilog hardware description language. IEEE Std 1364-2005 (Revision of IEEE Std 1364-2001) (2006), pp. 1–560

    Google Scholar 

  36. R.P. Kurshan, Computer-Aided Verification of Coordinating Processes: The Automata-Theoretic Approach (Princeton University Press, Princeton, 1994)

    MATH  Google Scholar 

  37. L. Liu, D. Sheridan, V. Athavale, S. Vasudevan, Automatic generation of assertions from system level design using data mining, in Proceedings of MEMOCODE (2011), pp. 191–200

    Google Scholar 

  38. L. Liu, D. Sheridan, W. Tuohy, S. Vasudevan, Towards coverage closure: using GoldMine assertions for generating design validation stimulus, in Proceedings of DATE (2011), pp. 173–178

    Google Scholar 

  39. L. Liu, C.-H. Lin, S. Vasudevan, Word level feature discovery to enhance quality of assertion mining, in Proceedings of ICCAD (2012), pp. 210–217

    Google Scholar 

  40. L. Liu, D. Sheridan, W. Tuohy, S. Vasudevan, A technique for test coverage closure using goldmine. IEEE Trans. CAD Integr. Circuits Syst. 31(5), 790–803 (2012)

    Article  Google Scholar 

  41. L. Liu, X. Zhong, X. Chen, S. Vasudevan, Diagnosing root causes of system level performance violations, in Proceedings of ICCAD, 2013

    Google Scholar 

  42. J. Malburg, T. Flenker, G. Fey, Property mining using dynamic dependency graphs, in 22nd Asia and South Pacific Design Automation Conference, ASP-DAC 2017, Chiba, January 16–19, 2017 (2017), pp. 244–250

    Google Scholar 

  43. K. McGarry, A survey of interestingness measures for knowledge discovery. Knowl. Eng. Rev. 20(1), 39–61 (2005)

    Article  Google Scholar 

  44. J. Misra, Prospects and limitations of automatic assertion generation for loop programs. SIAM J. Comput. 6(4), 718–729 (1977)

    Article  MathSciNet  Google Scholar 

  45. T.M. Mitchell, Machine Learning. McGraw Hill Series in Computer Science (McGraw-Hill, New York, 1997)

    Google Scholar 

  46. J.W. Nimmer, M.D. Ernst, Automatic generation of program specifications, in ISSTA (2002), pp. 229–239

    Google Scholar 

  47. Opencore web page, http://www.opencores.org

  48. M.S. Pang-Ning Tan, V. Kumar, Introduction to Data Mining (Pearson, London, 2005)

    Google Scholar 

  49. C.S. Pasareanu, W. Visser, Verification of java programs using symbolic execution and invariant generation, in Proceedings of SPIN (2004), pp. 164–181

    Google Scholar 

  50. G. Pinter, I. Majzik, Automatic generation of executable assertions for runtime checking temporal requirements, in Proceedings of HASE 2005 (2005), pp. 111–120

    Google Scholar 

  51. A. Pnueli, The temporal logic of programs, in Proceedings of the 18th Annual Symposium on Foundations of Computer Science (1977), pp. 46–57

    Google Scholar 

  52. J.R. Quinlan, Induction of decision trees. Mach. Learn. 1(1), 81–106 (1986)

    Google Scholar 

  53. J.R. Quinlan, C4.5: Programs for Machine Learning (Morgan Kaufmann Publishers, San Francisco, 1993)

    Google Scholar 

  54. F. Rogin, T. Klotz, G. Fey, R. Drechsler, S. Rülke, Automatic generation of complex properties for hardware designs, in Proceedings of DATE (2008), pp. 545–548

    Google Scholar 

  55. C.E. Shannon, A mathematical theory of communication. SIGMOBILE Mob. Comput. Commun. Rev. 5(1), 3–55 (2001)

    Article  MathSciNet  Google Scholar 

  56. D. Sheridan, H.-S. Kim, L. Liu, J. Han, S. Vasudevan, Guided association rule learning for assertion generation. Technical report (2012)

    Google Scholar 

  57. D. Sheridan, L. Liu, H. Kim, S. Vasudevan, A coverage guided mining approach for automatic generation of succinct assertions, in 2014 27th International Conference on VLSI Design and 2014 13th International Conference on Embedded Systems, Mumbai, January 5–9, 2014 (2014), pp. 68–73

    Google Scholar 

  58. A. Silberschatz, A. Tuzhilin, What makes patterns interesting in knowledge discovery systems. IEEE Trans. Knowl. Data Eng. 8(6), 970–974 (1996)

    Article  Google Scholar 

  59. Systemverilog, 2012

    Google Scholar 

  60. P. Taatizadeh, N. Nicolici, Emulation-based selection and assessment of assertion checkers for post-silicon validation, in 2015 33rd IEEE International Conference on Computer Design (ICCD), Oct 2015, pp. 46–53

    Google Scholar 

  61. P. Taatizadeh, N. Nicolici, Automated selection of assertions for bit-flip detection during post-silicon validation. IEEE Trans. CAD Integr. Circuits Syst. 35(12), 2118–2130 (2016)

    Article  Google Scholar 

  62. P. Taatizadeh, N. Nicolici, Emulation infrastructure for the evaluation of hardware assertions for post-silicon validation. IEEE Trans. VLSI Syst. 25(6), 1866–1880 (2017)

    Article  Google Scholar 

  63. P. Tan, V. Kumar, J. Srivastava, Selecting the right interestingness measure for association patterns, in Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, July 23–26, 2002, Edmonton, AB (2002), pp. 32–41

    Google Scholar 

  64. The GoldMine Website, http://goldmine.csl.illinois.edu/

  65. A. Tiwari, H. Rue, H. Sadi, N. Shankar, A technique for invariant generation, in Proceedings of TACAS (2001), pp. 113–127

    Google Scholar 

  66. S. Vasudevan, D. Sheridan, S.J. Patel, D. Tcheng, W. Tuohy, D.R. Johnson, Goldmine: automatic assertion generation using data mining and static analysis, in Design, Automation and Test in Europe, DATE 2010, Dresden, March 8–12, 2010 (2010), pp. 626–629

    Google Scholar 

  67. L.-C. Wang, M.S. Abadir, N. Krishnamurthy, Automatic generation of assertions for formal verification of powerpc microprocessor arrays using symbolic trajectory evaluation, in Proceedings of DAC (1998), pp. 534–537

    Google Scholar 

  68. H. Zhao, A. Sinha, An efficient algorithm for generating generalized decision forests. IEEE Trans. Syst. Man Cybern. A: Syst. Hum. 35(5), 754–762 (2005)

    Article  Google Scholar 

Download references

Acknowledgement

We thank Debjit Pal for his assistance with editing this chapter.

Author information

Authors and Affiliations

  1. Electrical and Computer Engineering Department, University of Illinois at Urbana-Champaign, Champaign, IL, USA

    Shobha Vasudevan, Lingyi Liu & Samuel Hertz

Authors
  1. Shobha Vasudevan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lingyi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Samuel Hertz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shobha Vasudevan .

Editor information

Editors and Affiliations

  1. Department of Electrical and Computer Engineering and Center for Cyber Physical Systems, Khalifa University, Abu Dhabi, United Arab Emirates

    Ibrahim (Abe) M. Elfadel

  2. Massachusetts Institute of Technology, Cambridge, MA, USA

    Duane S. Boning

  3. Department of Electrical and Computer Engineering, Duke University, Durham, NC, USA

    Xin Li

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Vasudevan, S., Liu, L., Hertz, S. (2019). A Comparative Study of Assertion Mining Algorithms in GoldMine. In: Elfadel, I., Boning, D., Li, X. (eds) Machine Learning in VLSI Computer-Aided Design. Springer, Cham. https://doi.org/10.1007/978-3-030-04666-8_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-04666-8_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-04665-1

  • Online ISBN: 978-3-030-04666-8

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation