A Comparative Study of Assertion Mining Algorithms in GoldMine

  • Shobha VasudevanEmail author
  • Lingyi Liu
  • Samuel Hertz


GoldMine automatically generates assertions for register transfer level (RTL) designs using a combination of static analysis and machine learning algorithms. We compare four different machine learning algorithms implemented in GoldMine with respect to metrics that determine quality of their results. Metrics include assertion complexity, coverage, expectedness, predictive accuracy, and number of assertions. We introduce a new statically guided assertion mining algorithm, the best-gain decision forest (BGDF) algorithm here. In our comparative study, two of the algorithms have been invented by us for assertion generation, while the two others are standard machine learning algorithms. The best-gain decision forest and coverage guided association mining algorithms developed by us are compared with the decision tree and PRISM algorithms. We provide an extensive comparative study of assertion mining algorithms with experimental results on realistic designs. We also hereby provide access to these implementations in GoldMine that can be downloaded from [64] for research purposes.



We thank Debjit Pal for his assistance with editing this chapter.


  1. 1.
  2. 2.
    V. Athavale, S. Ma, S. Hertz, S. Vasudevan, Code coverage of assertions using RTL source code analysis, in The 51st Annual Design Automation Conference 2014, DAC ‘14, San Francisco, CA, June 1–5, 2014 (2014), pp. 61:1–61:6Google Scholar
  3. 3.
    R. Agrawal, T. Imielinski, A. Swami, Mining association rules between sets of items in large databases, in Proceedings of SIGMOD (1993), pp. 207–216Google Scholar
  4. 4.
    G. Ammons, R. Bodk, J.R. Larus, Mining specifications, in POPL (2002), pp. 4–16Google Scholar
  5. 5.
  6. 6.
    V. Athavale, S. Hertz, S. Vasudevan, Evaluating code coverage of assertions by static analysis of RTL. Technical report uilu-eng-11-2209 (2011)Google Scholar
  7. 7.
    S. Bensalem, Y. Lakhnech, H. Sadi, Powerful techniques for the automatic generation of invariants, in Proceedings of CAV (1996), pp. 323–335Google Scholar
  8. 8.
    A. Biere, E.M. Clarke, R. Raimi, Y. Zhu, Verifying safety properties of a power PC microprocessor using symbolic model checking without BDDs, in Computer Aided Verification, 11th International Conference, CAV ‘99, Trento, July 6–10, 1999, Proceedings (1999), pp. 60–71Google Scholar
  9. 9.
    L. Breiman, Random forests. Mach. Learn. 45(1), 5–32 (2001)CrossRefGoogle Scholar
  10. 10.
    M. Caplain, Finding invariant assertions for proving programs, in Proceedings of the International Conference on Reliable Software (ACM, New York, 1975), pp. 165–171Google Scholar
  11. 11.
    J. Cendrowska, Prism: an algorithm for inducing modular rules. Int. J. Man-Mach. Stud. 27(4), 349–370 (1987)CrossRefGoogle Scholar
  12. 12.
    P.-H. Chang, L.-C. Wang, Automatic assertion extraction via sequential data mining of simulation traces, in Proceedings of ASPDAC (2010), pp. 607–612Google Scholar
  13. 13.
    H. Chao, H. Li, X. Song, T. Wang, X. Li, On evaluating and constraining assertions using conflicts in absent scenarios, in 2017 IEEE 26th Asian Test Symposium (ATS), Nov 2017, pp. 195–200Google Scholar
  14. 14.
    X. Cheng, M.S. Hsiao, Simulation-directed invariant mining for software verification, in Proceedings of DATE (2008), pp. 682–687Google Scholar
  15. 15.
    C.-N. Chung, C.-W. Chang, K.-H. Chang, S.-Y. Kuo, Applying verification intention for design customization via property mining under constrained testbenches, in Proceedings of ICCD (2011), pp. 84–89Google Scholar
  16. 16.
    E.M. Clarke, M. Fujita, S.P. Rajan, T.W. Reps, S. Shankar, T. Teitelbaum, Program slicing of hardware description languages, in Correct Hardware Design and Verification Methods, 10th IFIP WG 10.5 Advanced Research Working Conference, CHARME ‘99, Bad Herrenalb, September 27–29, 1999, Proceedings (1999), pp. 298–312Google Scholar
  17. 17.
    E.M. Clarke, A. Biere, R. Raimi, Y. Zhu, Bounded model checking using satisfiability solving. Formal Methods Syst. Des. 19(1), 7–34 (2001)CrossRefGoogle Scholar
  18. 18.
    A.M. Cruz, R.B. Fernández, H.M. Lozano, M.A. Ramírez Salinas, L.A. Villa Vargas, Automated functional test generation for digital systems through a compact binary differential evolution algorithm. J. Electron. Test. 31(4), 361–380 (2015)CrossRefGoogle Scholar
  19. 19.
    A. Danese, F. Filini, G. Pravadelli, A time-window based approach for dynamic assertions mining on control signals, in 2015 IFIP/IEEE International Conference on Very Large Scale Integration (VLSI-SoC), Oct 2015, pp. 246–251Google Scholar
  20. 20.
    A. DeOrio, A. Bauserman, V. Bertacco, B. Isaksen, Inferno: streamlining verification with inferred semantics. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 28(5), 728–741 (2009)CrossRefGoogle Scholar
  21. 21.
    M.D. Ernst, J.H. Perkins, P.J. Guo, S. McCamant, C. Pacheco, M.S. Tschantz, C. Xiao, The daikon system for dynamic detection of likely invariants. J. Sci. Comput. Program. 69, 35–45 (2007)MathSciNetCrossRefGoogle Scholar
  22. 22.
    F. Farahmandi, R. Morad, A. Ziv, Z. Nevo, P. Mishra, Cost-effective analysis of post-silicon functional coverage events, in Design, Automation & Test in Europe Conference & Exhibition, DATE 2017, Lausanne, March 27–31, 2017 (2017), pp. 392–397Google Scholar
  23. 23.
    H.D. Foster, A.C. Krolnik, D.J. Lacey, Assertion-Based Design, 2nd edn. (Springer Publishing Company, New York, 2010)Google Scholar
  24. 24.
    T. Ghasempouri, G. Pravadelli, On the estimation of assertion interestingness, in 2015 IFIP/IEEE International Conference on Very Large Scale Integration (VLSI-SoC), Oct 2015, pp. 325–330Google Scholar
  25. 25.
    O. Grumberg, H. Veith (eds.), 25 Years of Model Checking: History, Achievements, Perspectives (Springer, Berlin, 2008)zbMATHGoogle Scholar
  26. 26.
    J. Han, M. Kamber, Data Mining: Concepts and Techniques (Morgan Kaufmann Publishers Inc., San Francisco, 2000)zbMATHGoogle Scholar
  27. 27.
    M. Hanafy, H. Said, A.M. Wahba, Complete properties extraction from simulation traces for assertions auto-generation, in 2015 IEEE 24th North Atlantic Test Workshop, May 2015, pp. 1–6Google Scholar
  28. 28.
    M. Hanafy, H. Said, A.M. Wahba, New methodology for complete properties extraction from simulation traces guided with static analysis. J. Electron. Test. 32(6), 705–719 (2016)CrossRefGoogle Scholar
  29. 29.
    S. Hangal, N. Chandra, S. Narayanan, S. Chakravorty, Iodine: a tool to automatically infer dynamic invariants for hardware designs, in Proceedings of DAC (2005), pp. 775–778Google Scholar
  30. 30.
    A. Hekmatpour, A. Salehi, Block-based schema-driven assertion generation for functional verification, in Proceedings of ATS (2005), pp. 34–39Google Scholar
  31. 31.
    S. Hertz, Enhancing quality of assertion generation: methods for automatic assertion generation and evaluation. PhD thesis, The University of Illinois at Urbana-Champaign, 2013Google Scholar
  32. 32.
    S. Hertz, D. Sheridan, S. Vasudevan, Mining hardware assertions with guidance from static analysis. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 32, 952–965 (2013)CrossRefGoogle Scholar
  33. 33.
    T.K. Ho, The random subspace method for constructing decision forests. IEEE Trans. Pattern Anal. Mach. Intell. 20(8), 832–844 (1998)CrossRefGoogle Scholar
  34. 34.
    C.A.R. Hoare, An axiomatic basis for computer programming. Commun. ACM 12(10), 576–580 (1969)CrossRefGoogle Scholar
  35. 35.
    IEEE standard for verilog hardware description language. IEEE Std 1364-2005 (Revision of IEEE Std 1364-2001) (2006), pp. 1–560Google Scholar
  36. 36.
    R.P. Kurshan, Computer-Aided Verification of Coordinating Processes: The Automata-Theoretic Approach (Princeton University Press, Princeton, 1994)zbMATHGoogle Scholar
  37. 37.
    L. Liu, D. Sheridan, V. Athavale, S. Vasudevan, Automatic generation of assertions from system level design using data mining, in Proceedings of MEMOCODE (2011), pp. 191–200Google Scholar
  38. 38.
    L. Liu, D. Sheridan, W. Tuohy, S. Vasudevan, Towards coverage closure: using GoldMine assertions for generating design validation stimulus, in Proceedings of DATE (2011), pp. 173–178Google Scholar
  39. 39.
    L. Liu, C.-H. Lin, S. Vasudevan, Word level feature discovery to enhance quality of assertion mining, in Proceedings of ICCAD (2012), pp. 210–217Google Scholar
  40. 40.
    L. Liu, D. Sheridan, W. Tuohy, S. Vasudevan, A technique for test coverage closure using goldmine. IEEE Trans. CAD Integr. Circuits Syst. 31(5), 790–803 (2012)CrossRefGoogle Scholar
  41. 41.
    L. Liu, X. Zhong, X. Chen, S. Vasudevan, Diagnosing root causes of system level performance violations, in Proceedings of ICCAD, 2013Google Scholar
  42. 42.
    J. Malburg, T. Flenker, G. Fey, Property mining using dynamic dependency graphs, in 22nd Asia and South Pacific Design Automation Conference, ASP-DAC 2017, Chiba, January 16–19, 2017 (2017), pp. 244–250Google Scholar
  43. 43.
    K. McGarry, A survey of interestingness measures for knowledge discovery. Knowl. Eng. Rev. 20(1), 39–61 (2005)CrossRefGoogle Scholar
  44. 44.
    J. Misra, Prospects and limitations of automatic assertion generation for loop programs. SIAM J. Comput. 6(4), 718–729 (1977)MathSciNetCrossRefGoogle Scholar
  45. 45.
    T.M. Mitchell, Machine Learning. McGraw Hill Series in Computer Science (McGraw-Hill, New York, 1997)Google Scholar
  46. 46.
    J.W. Nimmer, M.D. Ernst, Automatic generation of program specifications, in ISSTA (2002), pp. 229–239Google Scholar
  47. 47.
    Opencore web page,
  48. 48.
    M.S. Pang-Ning Tan, V. Kumar, Introduction to Data Mining (Pearson, London, 2005)Google Scholar
  49. 49.
    C.S. Pasareanu, W. Visser, Verification of java programs using symbolic execution and invariant generation, in Proceedings of SPIN (2004), pp. 164–181Google Scholar
  50. 50.
    G. Pinter, I. Majzik, Automatic generation of executable assertions for runtime checking temporal requirements, in Proceedings of HASE 2005 (2005), pp. 111–120Google Scholar
  51. 51.
    A. Pnueli, The temporal logic of programs, in Proceedings of the 18th Annual Symposium on Foundations of Computer Science (1977), pp. 46–57Google Scholar
  52. 52.
    J.R. Quinlan, Induction of decision trees. Mach. Learn. 1(1), 81–106 (1986)Google Scholar
  53. 53.
    J.R. Quinlan, C4.5: Programs for Machine Learning (Morgan Kaufmann Publishers, San Francisco, 1993)Google Scholar
  54. 54.
    F. Rogin, T. Klotz, G. Fey, R. Drechsler, S. Rülke, Automatic generation of complex properties for hardware designs, in Proceedings of DATE (2008), pp. 545–548Google Scholar
  55. 55.
    C.E. Shannon, A mathematical theory of communication. SIGMOBILE Mob. Comput. Commun. Rev. 5(1), 3–55 (2001)MathSciNetCrossRefGoogle Scholar
  56. 56.
    D. Sheridan, H.-S. Kim, L. Liu, J. Han, S. Vasudevan, Guided association rule learning for assertion generation. Technical report (2012)Google Scholar
  57. 57.
    D. Sheridan, L. Liu, H. Kim, S. Vasudevan, A coverage guided mining approach for automatic generation of succinct assertions, in 2014 27th International Conference on VLSI Design and 2014 13th International Conference on Embedded Systems, Mumbai, January 5–9, 2014 (2014), pp. 68–73Google Scholar
  58. 58.
    A. Silberschatz, A. Tuzhilin, What makes patterns interesting in knowledge discovery systems. IEEE Trans. Knowl. Data Eng. 8(6), 970–974 (1996)CrossRefGoogle Scholar
  59. 59.
    Systemverilog, 2012Google Scholar
  60. 60.
    P. Taatizadeh, N. Nicolici, Emulation-based selection and assessment of assertion checkers for post-silicon validation, in 2015 33rd IEEE International Conference on Computer Design (ICCD), Oct 2015, pp. 46–53Google Scholar
  61. 61.
    P. Taatizadeh, N. Nicolici, Automated selection of assertions for bit-flip detection during post-silicon validation. IEEE Trans. CAD Integr. Circuits Syst. 35(12), 2118–2130 (2016)CrossRefGoogle Scholar
  62. 62.
    P. Taatizadeh, N. Nicolici, Emulation infrastructure for the evaluation of hardware assertions for post-silicon validation. IEEE Trans. VLSI Syst. 25(6), 1866–1880 (2017)CrossRefGoogle Scholar
  63. 63.
    P. Tan, V. Kumar, J. Srivastava, Selecting the right interestingness measure for association patterns, in Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, July 23–26, 2002, Edmonton, AB (2002), pp. 32–41Google Scholar
  64. 64.
    The GoldMine Website,
  65. 65.
    A. Tiwari, H. Rue, H. Sadi, N. Shankar, A technique for invariant generation, in Proceedings of TACAS (2001), pp. 113–127Google Scholar
  66. 66.
    S. Vasudevan, D. Sheridan, S.J. Patel, D. Tcheng, W. Tuohy, D.R. Johnson, Goldmine: automatic assertion generation using data mining and static analysis, in Design, Automation and Test in Europe, DATE 2010, Dresden, March 8–12, 2010 (2010), pp. 626–629Google Scholar
  67. 67.
    L.-C. Wang, M.S. Abadir, N. Krishnamurthy, Automatic generation of assertions for formal verification of powerpc microprocessor arrays using symbolic trajectory evaluation, in Proceedings of DAC (1998), pp. 534–537Google Scholar
  68. 68.
    H. Zhao, A. Sinha, An efficient algorithm for generating generalized decision forests. IEEE Trans. Syst. Man Cybern. A: Syst. Hum. 35(5), 754–762 (2005)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Electrical and Computer Engineering DepartmentUniversity of Illinois at Urbana-ChampaignChampaignUSA

Personalised recommendations