Towards Efficient Join Processing over Large RDF Graph Using MapReduce

  • Xiaofei Zhang
  • Lei Chen
  • Min Wang
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7338)


Existing solutions for answering SPARQL queries in a shared-nothing environment using MapReduce failed to fully explore the substantial scalability and parallelism of the computing framework. In this paper, we propose a cost model based RDF join processing solution using MapReduce to minimize the query responding time as much as possible. After transforming a SPARQL query into a sequence of MapReduce jobs, we propose a novel index structure, called All Possible Join tree (APJ-tree), to reduce the searching space for the optimal execution plan of a set of MapReduce jobs. To speed up the join processing, we employ hybrid join and bloom filter for performance optimization. Extensive experiments on real data sets proved the effectiveness of our cost model. Our solution has as much as an order of magnitude time saving compared with the state of art solutions.


Query Processing Cost Model Input Size SPARQL Query Query Processing Time 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Husain, M.F., et al.: Data intensive query processing for large RDF graphs using cloud computing tools. In: CLOUD 2010 (2010)Google Scholar
  2. 2.
    Farhan Husain, M., Doshi, P., Khan, L., Thuraisingham, B.: Storage and Retrieval of Large RDF Graph Using Hadoop and MapReduce. In: Jaatun, M.G., Zhao, G., Rong, C. (eds.) CloudCom 2009. LNCS, vol. 5931, pp. 680–686. Springer, Heidelberg (2009)Google Scholar
  3. 3.
    Myung, J., et al.: Sparql basic graph pattern processing with iterative mapreduce. In: MDAC 2010 (2010)Google Scholar
  4. 4.
    Tanimura, Y., et al.: Extensions to the pig data processing platform for scalable RDF data processing using hadoop. In: 22nd International Conference on Data Engineering Workshops, pp. 251–256 (2010)Google Scholar
  5. 5.
    Chebotko, A., Atay, M., Lu, S., Fotouhi, F.: Relational Nested Optional Join for Efficient Semantic Web Query Processing. In: Dong, G., Lin, X., Wang, W., Yang, Y., Yu, J.X. (eds.) APWeb/WAIM 2007. LNCS, vol. 4505, pp. 428–439. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  6. 6.
    Jaynes, E.T.: Probability theory: The logic of science. Cambridge University Press, Cambridge (2003)zbMATHCrossRefGoogle Scholar
  7. 7.
    Zhang, X., et al.: Towards efficient join processing over large RDF graph using mapreduce. Technical Report (2011)Google Scholar
  8. 8.
  9. 9.
    Blanas, S., et al.: A comparison of join algorithms for log processing in mapreduce. In: SIGMOD 2010 (2010)Google Scholar
  10. 10.
  11. 11.
    Thomas, N., et al.: The RDF-3x engine for scalable management of RDF data. VLDB J. 19(1), 91–113 (2010)CrossRefGoogle Scholar
  12. 12.
    Weiss, C., et al.: Hexastore: sextuple indexing for semantic web data management. Proc. VLDB Endow. (2008)Google Scholar
  13. 13.
    Neumann, T., et al.: Scalable join processing on very large RDF graphs. In: SIGMOD Conference, pp. 627–640 (2009)Google Scholar
  14. 14.
    Abadi, D.J., et al.: Sw-store: a vertically partitioned dbms for semantic web data management. The VLDB Journal 18, 385–406 (2009)CrossRefGoogle Scholar
  15. 15.
  16. 16.
    Newman, A., et al.: A scale-out RDF molecule store for distributed processing of biomedical data. In: Semantic Web for Health Care and Life Sciences Workshop (2008)Google Scholar
  17. 17.
    Newman, A., et al.: Scalable semantics - the silver lining of cloud computing. In: ESCIENCE 2008 (2008)Google Scholar
  18. 18.
    Urbani, J., Kotoulas, S., Oren, E., van Harmelen, F.: Scalable Distributed Reasoning Using MapReduce. In: Bernstein, A., Karger, D.R., Heath, T., Feigenbaum, L., Maynard, D., Motta, E., Thirunarayan, K. (eds.) ISWC 2009. LNCS, vol. 5823, pp. 634–649. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  19. 19.
    McGlothlin, J.P., et al.: Rdfkb: efficient support for RDF inference queries and knowledge management. In: IDEAS 2009 (2009)Google Scholar
  20. 20.
  21. 21.
    Afrati, F.N., et al.: Optimizing joins in a map-reduce environment. In: EDBT 2010 (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Xiaofei Zhang
    • 1
  • Lei Chen
    • 1
  • Min Wang
    • 2
  1. 1.Hong Kong University of Science and TechnologyHong Kong
  2. 2.HP LabsBeijingChina

Personalised recommendations