Using Coq in Specification and Program Extraction of Hadoop MapReduce Applications
Hadoop MapReduce is a framework for distributed computation on key-value pairs. The goal of this research is to verify actual running code of MapReduce applications. We first constructed an abstract model of MapReduce computation with the proof assistant Coq. In the model, mappers and reducers in MapReduce computation are modeled as functions in Coq, and a specification of a MapReduce application is expressed in terms of invariants among functions involving its mapper and reducer. The model also provides modular proofs of lemmas that do not depend on applications. To achieve the goal, we investigated the feasibility of two approaches. In one approach, we transformed verified mapper and reducer functions into Haskell programs and executed them under Hadoop Streaming. In the other approach, we verified JML annotations on Java programs of the mapper and reducer using Krakatoa, translated them into Coq axioms, and proved Coq specifications from them. In either approach, we were able to verify correctness of MapReduce applications that actually run on the Hadoop MapReduce framework.
KeywordsReducer Function Proof Obligation Java Modeling Language MapReduce Framework Program Extraction
Unable to display preview. Download preview PDF.
- 6.Dörre, J., Apel, S., Lengauer, C.: Static type checking of Hadoop MapReduce. In: MapReduce 2011. ACM, New York (to appear, 2011)Google Scholar
- 7.Hübel, T.: The Holumbus Framework. Master’s thesis, Wedel University of Applied Sciences (2008)Google Scholar
- 8.Jimmy Lin, C.D.: Data-Intensive Text Processing with MapReduce. Morgan and Claypool (2010)Google Scholar
- 10.Leavens, G.T., Poll, E., Clifton, C., Cheon, Y., Ruby, C., Cok, D., Muller, P., Kiniry, J., Chalin, P., Zimmerman, D.M.: JML reference manual (2011), http://www.eecs.ucf.edu/~leavens/JML/documentation.shtml
- 12.Marlow, S.: Haskell 2010 language report (2010), http://www.haskell.org/onlinereport/haskell2010/
- 14.White, T.: Hadoop: The Definitive Guide. O’Reilly Media, Inc., Sebastopol (2009)Google Scholar
- 15.Yang, F., Su, W., Zhu, H., Li, Q.: Formalizing MapReduce with CSP. In: Proceedings of the 2010 17th IEEE International Conference and Workshops on the Engineering of Computer-Based Systems, ECBS 2010, pp. 358–367. IEEE, Los Alamitos (2010)Google Scholar