Skip to main content
SpringerLink
Log in

HDTCat: Let’s Make HDT Generation Scale

  • Conference paper
  • First Online:
The Semantic Web – ISWC 2020 (ISWC 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12507))

Included in the following conference series:

Abstract

Data generation in RDF has been increasing over the last years as a means to publish heterogeneous and interconnected data. RDF is usually serialized in verbose text formats, which is problematic for publishing and managing huge datasets. HDT is a binary serialization of RDF that makes use of compact data structures, making it possible to publish and query highly compressed RDF data. This allows to reduce both the volume needed to store it and the speed at which it can be transferred or queried. However, it moves the burden of dealing with huge amounts of data from the consumer to the publisher, who needs to serialize the text data into HDT. This process consumes a lot of resources in terms of time, processing power, and especially memory. In addition, adding data to a file in HDT format is currently not possible, whether this additional data is in plain text or already serialized into HDT.

In this paper, we present HDTCat, a tool to merge the contents of two HDT files with low memory footprint. Apart from creating an HDT file with the added data of two or more datasets efficiently, this tool can be used in a divide-and-conquer strategy to generate HDT files from huge datasets with low memory consumption.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.w3.org/TR/rdf11-concepts/.

  2. 2.

    https://github.com/rdfhdt/hdt-java.

  3. 3.

    All files retrieved by: wget -r -nc -nH –cut-dirs=1 -np -l1 -A ‘*ttl.bz2’ -A ‘*.owl’-R ‘*unredirected*’–tries 2 http://downloads.dbpedia.org/2016-10/core-i18n/en/, i.e. all files published in the english DBpedia. We exclude the following files: nif_page_structure_en.ttl, raw_tables_en.ttl and page_links_en.ttl since they do not contain typical data used in application relying on DBpedia.

References

  1. Brisaboa, N.R., Cánovas, R., Martínez-Prieto, M.A., Navarro, G.: Compressed string dictionaries. CoRR abs/1101.5506 (2011). http://arxiv.org/abs/1101.5506

  2. Curé, O., Blin, G., Revuz, D., Faye, D.C.: WaterFowl: a compact, self-indexed and inference-enabled immutable RDF store. In: Presutti, V., d’Amato, C., Gandon, F., d’Aquin, M., Staab, S., Tordai, A. (eds.) ESWC 2014. LNCS, vol. 8465, pp. 302–316. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07443-6_21

    Chapter  Google Scholar 

  3. Diefenbach, D., Both, A., Singh, K., Maret, P.: Towards a question answering system over the semantic web. Semant. Web J. 1–19 (2020)

    Google Scholar 

  4. Diefenbach, D., Singh, K., Maret, P.: WDAqua-core0: a question answering component for the research community. In: Dragoni, M., Solanki, M., Blomqvist, E. (eds.) SemWebEval 2017. CCIS, vol. 769, pp. 84–89. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-69146-6_8

    Chapter  Google Scholar 

  5. Fernández, J.D., Beek, W., Martínez-Prieto, M.A., Arias, M.: LOD-a-lot. In: d’Amato, C., et al. (eds.) ISWC 2017. LNCS, vol. 10588, pp. 75–83. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68204-4_7

    Chapter  Google Scholar 

  6. Fernández, J.D., Martínez-Prieto, M.A., Gutiérrez, C., Polleres, A., Arias, M.: Binary RDF representation for publication and exchange (HDT). J. Web Semant. 19(22–41), 00124 (2013)

    Google Scholar 

  7. Giménez-García, J.M., Fernández, J.D., Martínez-Prieto, M.A.: HDT-MR: a scalable solution for RDF compression with HDT and MapReduce. In: Gandon, F., Sabou, M., Sack, H., d’Amato, C., Cudré-Mauroux, P., Zimmermann, A. (eds.) ESWC 2015. LNCS, vol. 9088, pp. 253–268. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-18818-8_16

    Chapter  Google Scholar 

  8. Guo, Y., Pan, Z., Heflin, J.: LUBM: a benchmark for OWL knowledge base systems. J. Web Semant. 3(2), 158–182 (2005)

    Article  Google Scholar 

  9. Gutierrez, C., Hurtado, C.A., Mendelzon, A.O., Pérez, J.: Foundations of semantic web databases. J. Comput. Syst. Sci. 77(3), 520–541 (2011). https://doi.org/10.1016/j.jcss.2010.04.009

    Article  MathSciNet  MATH  Google Scholar 

  10. Hu, T.C., Tucker, A.C.: Optimal computer search trees and variable-length alphabetical codes. Siam J. Appl. Math. 21(4), 514–532 (1971)

    Article  MathSciNet  Google Scholar 

  11. Martínez-Prieto, M., Arias, M., Fernández, J.: Exchange and consumption of huge RDF data. In: Proceeding of ESWC, pp. 437–452 (2012)

    Google Scholar 

  12. Verborgh, R., et al.: Querying Datasets on the Web with High Availability. In: Mika, Peter, Tudorache, Tania, Bernstein, Abraham, Welty, Chris, Knoblock, Craig, Vrandečić, Denny, Groth, Paul, Noy, Natasha, Janowicz, Krzysztof, Goble, Carole (eds.) ISWC 2014. LNCS, vol. 8796, pp. 180–196. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11964-9_12, http://linkeddatafragments.org/publications/iswc2014.pdf

Download references

Acknowledgements

We would like to thank Pedro Migliatti for executing part of the experiments as well as Javier D. Fernández for the helpful discussions with him. We also want to thank Dimitris Nikolopoulos and Wouter Beek from Triply for porting the algorithm to C++. This project has received funding from the European Union’s Horizon 2020 research and innovation program under the Marie Sklodowska-Curie grant agreement No 642795.

Author information

Authors and Affiliations

  1. The QA Company SAS, Saint-Étienne, France

    Dennis Diefenbach

  2. Univ Lyon, UJM-Saint-Étienne, CNRS, Laboratoire Hubert Curien, UMR 5516, 42023, Saint-Étienne, France

    Dennis Diefenbach & José M. Giménez-García

Authors
  1. Dennis Diefenbach
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. José M. Giménez-García
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dennis Diefenbach .

Editor information

Editors and Affiliations

  1. University of Edinburgh, Edinburgh, UK

    Jeff Z. Pan

  2. University of Liverpool, Liverpool, UK

    Valentina Tamma

  3. University of Bari, Bari, Italy

    Claudia d’Amato

  4. University of California, Santa Barbara, Santa Barbara, CA, USA

    Krzysztof Janowicz

  5. California State University, Long Beach, Long Beach, CA, USA

    Bo Fu

  6. Vienna University of Economics and Business, Vienna, Austria

    Axel Polleres

  7. Rensselaer Polytechnic Institute, Troy, NY, USA

    Oshani Seneviratne

  8. Massachusetts Institute of Technology, Cambridge, MA, USA

    Lalana Kagal

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Diefenbach, D., Giménez-García, J.M. (2020). HDTCat: Let’s Make HDT Generation Scale. In: Pan, J.Z., et al. The Semantic Web – ISWC 2020. ISWC 2020. Lecture Notes in Computer Science(), vol 12507. Springer, Cham. https://doi.org/10.1007/978-3-030-62466-8_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-62466-8_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-62465-1

  • Online ISBN: 978-3-030-62466-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation