Skip to main content
SpringerLink
Log in

Formal Verification of the rank Algorithm for Succinct Data Structures

  • Conference paper
  • First Online:
Formal Methods and Software Engineering (ICFEM 2016)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 10009))

Included in the following conference series:

Abstract

Succinct data structures are designed to use a minimal amount of computer memory in a time-efficient way. Their correct implementation is essential to big data analysis. Yet, succinct data structures are difficult to verify because they rely on bit-level manipulations better achieved with low-level languages. In this paper, we report on the formal verification of the standard Jacobson rank algorithm using the Coq proof-assistant and extract an OCaml implementation from it. This requires overcoming the mismatch between Coq being a purely functional programming language and succinct data structures being inherently imperative. To enjoy the best of both worlds, we propose to use code extraction from Coq to OCaml but with an original (tested but unverified) implementation of bitstrings. We can then use Coq to formalize correctness, including important claims about storage requirements, and still obtain efficient native code. To the best of our knowledge, this is the first application of formal verification to succinct data structures.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The function bcount is not intended to be extracted as it is but replaced by a more efficient function. It could be tabulated as explained in Sect. 2.2, but in this paper, it will be replaced by a single gcc built-in operation (see Sect. 4.2).

  2. 2.

    Let s be a bitstring of length n. bsize s is O(n), bnth i s is O(i), bcount b i l s is \(O(i + l)\). bcount requires an additional O(i) because of the drop function (see Sect. 3.1).

  3. 3.

    Currently, bytes is the same as string; OCaml plans to change string to immutable.

  4. 4.

    The OCaml definitions below belong to the module Pbits; the prefix Pbits. is omitted when no confusion is possible.

  5. 5.

    This function is implemented in C using gcc’s__builtin_clzl [6], which counts the number of leading zeros in a long value. gcc generates LZCNT instructions (since Intel AVX2 [8]).

  6. 6.

    In this case, w1 and w2 become 0 and our word array cannot distinguish an empty array and non-empty array.

References

  1. Affeldt, R., Marti, N.: An approach to formal verification of arithmetic functions in assembly. In: Okada, M., Satoh, I. (eds.) ASIAN 2006. LNCS, vol. 4435, pp. 346–360. Springer, Heidelberg (2008)

    Google Scholar 

  2. Agarwal, R., Khandelwal, A., Stoica, I.: Succinct: enabling queries on compressed data. In: NSDI 2015, pp. 337–350. USENIX Association (2015). Technical report: http://people.eecs.berkeley.edu/~rachit/succinct-techreport.pdf

  3. Armand, M., Grégoire, B., Spiwack, A., Théry, L.: Extending Coq with imperative features and its application to SAT verification. In: Kaufmann, M., Paulson, L.C. (eds.) ITP 2010. LNCS, vol. 6172, pp. 83–98. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  4. Clark, D.: Compact pat trees. Doctoral dissertation, University of Waterloo (1996)

    Google Scholar 

  5. The Coq Development Team: Reference Manual. Version 8.5. INRIA (2004–2016). http://coq.inria.fr

  6. Free Software Foundation: GCC 4.9.2 Manual (2014). http://gcc.gnu.org/onlinedocs/gcc-4.9.2/gcc

  7. Gonthier, G., Mahboubi, A., Tassi, E.: A small scale reflection extension for the Coq system. Version 16. Technical report RR-6455, INRIA (2015)

    Google Scholar 

  8. Intel Advanced Vector Extensions Programming Reference, June 2011

    Google Scholar 

  9. Intel 64 and IA-32 Architectures Optimization Reference Manual, September 2015

    Google Scholar 

  10. Intel SSE4 Programming Reference, April 2007

    Google Scholar 

  11. Jacobson, G.: Succinct static data structures. Doctoral dissertation, Carnegie Mellon University (1988)

    Google Scholar 

  12. Jones, R.W.M.: A beginners guide to OCaml internals (2009). https://rwmj.wordpress.com/2009/08/04/ocaml-internals

  13. Kim, D.-K., Na, J.C., Kim, J.E., Park, K.: Efficient implementation of rank and select functions for succinct representation. In: Nikoletseas, S.E. (ed.) WEA 2005. LNCS, vol. 3503, pp. 315–327. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  14. Klein, G., Andronick, J., Elphinstone, K., Heiser, G., Cock, D., Derrin, P., Elkaduwe, D., Engelhardt, K., Kolanski, R., Norrish, M., Sewell, T., Tuch, H., Winwood, S.: seL4: formal verification of an operating-system kernel. Commun. ACM 53(6), 107–115 (2010)

    Article  Google Scholar 

  15. Nipkow, T.: Amortized complexity verified. In: Urban, C., Zhang, X. (eds.) ITP 2015. LNCS, vol. 9236, pp. 310–324. Springer, Berlin (2015)

    Google Scholar 

  16. SDSL: Succinct Data Structure Library. https://github.com/simongog/sdsl-lite

  17. OUnit: Unit test framework for OCaml. http://ounit.forge.ocamlcore.org/

  18. Okanohara, D.: The world of fast character string analysis. Iwanami Shoten (2012). (in Japanese)

    Google Scholar 

  19. Tanaka, A., Affeldt, R., Garrigue, J.: Formal Verification of the Rank Function for Succinct Data Structures. https://staff.aist.go.jp/tanaka-akira/succinct/index.html

Download references

Acknowledgments

The authors are grateful to the anonymous reviewers for their helpful comments. This work is partially supported by a JSPS Grant-in-Aid for Scientific Research (Project Number: 15K12013).

Author information

Authors and Affiliations

  1. National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba, Japan

    Akira Tanaka & Reynald Affeldt

  2. Nagoya University, Nagoya, Japan

    Jacques Garrigue

Authors
  1. Akira Tanaka
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Reynald Affeldt
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jacques Garrigue
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Akira Tanaka .

Editor information

Editors and Affiliations

  1. School of Information Science, Japan Advanced Institute of Science and Technology (JAIST), Nomi, Japan

    Kazuhiro Ogata

  2. Department of Computing and Software, McMaster University, Hamilton, Ontario, Canada

    Mark Lawford

  3. Department of Computer Science, Hosei University, Tokyo, Japan

    Shaoying Liu

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing AG

About this paper

Cite this paper

Tanaka, A., Affeldt, R., Garrigue, J. (2016). Formal Verification of the rank Algorithm for Succinct Data Structures. In: Ogata, K., Lawford, M., Liu, S. (eds) Formal Methods and Software Engineering. ICFEM 2016. Lecture Notes in Computer Science(), vol 10009. Springer, Cham. https://doi.org/10.1007/978-3-319-47846-3_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-47846-3_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-47845-6

  • Online ISBN: 978-3-319-47846-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation