Advertisement

Using Coq in Specification and Program Extraction of Hadoop MapReduce Applications

  • Kosuke Ono
  • Yoichi Hirai
  • Yoshinori Tanabe
  • Natsuko Noda
  • Masami Hagiya
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7041)

Abstract

Hadoop MapReduce is a framework for distributed computation on key-value pairs. The goal of this research is to verify actual running code of MapReduce applications. We first constructed an abstract model of MapReduce computation with the proof assistant Coq. In the model, mappers and reducers in MapReduce computation are modeled as functions in Coq, and a specification of a MapReduce application is expressed in terms of invariants among functions involving its mapper and reducer. The model also provides modular proofs of lemmas that do not depend on applications. To achieve the goal, we investigated the feasibility of two approaches. In one approach, we transformed verified mapper and reducer functions into Haskell programs and executed them under Hadoop Streaming. In the other approach, we verified JML annotations on Java programs of the mapper and reducer using Krakatoa, translated them into Coq axioms, and proved Coq specifications from them. In either approach, we were able to verify correctness of MapReduce applications that actually run on the Hadoop MapReduce framework.

Keywords

Reducer Function Proof Obligation Java Modeling Language MapReduce Framework Program Extraction 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bertot, Y., Casteran, P.: Interactive Theorem Proving and Program Development. Springer, Heidelberg (2004)CrossRefzbMATHGoogle Scholar
  2. 2.
    Chalin, P., Kiniry, J., Leavens, G., Poll, E.: Beyond assertions: Advanced specification and verification with JML and ESC/Java2. In: de Boer, F., Bonsangue, M., Graf, S., de Roever, W.P. (eds.) FMCO 2005. LNCS, vol. 4111, pp. 342–363. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  3. 3.
    Chrząszcz, J.: Implementing modules in the Coq system. In: Basin, D., Wolff, B. (eds.) TPHOLs 2003. LNCS, vol. 2758, pp. 270–286. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  4. 4.
    Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 51, 107–113 (2008)CrossRefGoogle Scholar
  5. 5.
    Detlefs, D., Nelson, G., Saxe, J.B.: Simplify: a theorem prover for program checking. J. ACM 52, 365–473 (2005)CrossRefzbMATHGoogle Scholar
  6. 6.
    Dörre, J., Apel, S., Lengauer, C.: Static type checking of Hadoop MapReduce. In: MapReduce 2011. ACM, New York (to appear, 2011)Google Scholar
  7. 7.
    Hübel, T.: The Holumbus Framework. Master’s thesis, Wedel University of Applied Sciences (2008)Google Scholar
  8. 8.
    Jimmy Lin, C.D.: Data-Intensive Text Processing with MapReduce. Morgan and Claypool (2010)Google Scholar
  9. 9.
    Lämmel, R.: Google’s MapReduce programming model – revisited. Science of Computer Programming 70(1), 1–30 (2008)CrossRefzbMATHGoogle Scholar
  10. 10.
    Leavens, G.T., Poll, E., Clifton, C., Cheon, Y., Ruby, C., Cok, D., Muller, P., Kiniry, J., Chalin, P., Zimmerman, D.M.: JML reference manual (2011), http://www.eecs.ucf.edu/~leavens/JML/documentation.shtml
  11. 11.
    Marché, C., Paulin-Mohring, C., Urbain, X.: The KRAKATOA tool for certificationof JAVA/JAVACARD programs annotated in JML. Journal of Logic and Algebraic Programming 58(1-2), 89–106 (2004)CrossRefzbMATHGoogle Scholar
  12. 12.
    Marlow, S.: Haskell 2010 language report (2010), http://www.haskell.org/onlinereport/haskell2010/
  13. 13.
    de Moura, L., Bjøner, N.: Z3: An efficient SMT solver. In: Ramakrishnan, C., Rehof, J. (eds.) TACAS 2008. LNCS, vol. 4963, pp. 337–340. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  14. 14.
    White, T.: Hadoop: The Definitive Guide. O’Reilly Media, Inc., Sebastopol (2009)Google Scholar
  15. 15.
    Yang, F., Su, W., Zhu, H., Li, Q.: Formalizing MapReduce with CSP. In: Proceedings of the 2010 17th IEEE International Conference and Workshops on the Engineering of Computer-Based Systems, ECBS 2010, pp. 358–367. IEEE, Los Alamitos (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Kosuke Ono
    • 1
  • Yoichi Hirai
    • 1
  • Yoshinori Tanabe
    • 2
  • Natsuko Noda
    • 3
  • Masami Hagiya
    • 1
  1. 1.University of TokyoJapan
  2. 2.National Institute of InformaticsJapan
  3. 3.NEC CorporationJapan

Personalised recommendations