Skip to main content
View expanded cover

International Semantic Web Conference

ISWC 2020: The Semantic Web – ISWC 2020 pp 3–17Cite as

CASQAD – A New Dataset for Context-Aware Spatial Question Answering

  • 1918 Accesses

Part of the Lecture Notes in Computer Science book series (LNISA,volume 12507)

Abstract

The task of factoid question answering (QA) faces new challenges when applied in scenarios with rapidly changing context information, for example on smartphones. Instead of asking who the architect of the “Holocaust Memorial” in Berlin was, the same question could be phrased as “Who was the architect of the many stelae in front of me?” presuming the user is standing in front of it. While traditional QA systems rely on static information from knowledge bases and the analysis of named entities and predicates in the input, question answering for temporal and spatial questions imposes new challenges to the underlying methods. To tackle these challenges, we present the Context-aware Spatial QA Dataset (CASQAD) with over 5,000 annotated questions containing visual and spatial references that require information about the user’s location and moving direction to compose a suitable query. These questions were collected in a large scale user study and annotated semi-automatically, with appropriate measures to ensure the quality.

Keywords

  • Datasets
  • Benchmark
  • Question answering
  • Knowledge graphs

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-62466-8_1
  • Chapter length: 15 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   109.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-62466-8
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   139.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.

Notes

  1. 1.

    He designed the Memorial to the Murdered Jews of Europe https://www.visitberlin.de/en/memorial-murdered-jews-europe.

  2. 2.

    A visibility engine computes, which objects are visible from the user’s point of view.

  3. 3.

    https://casqad.sda.tech/.

  4. 4.

    A HIT describes the micro tasks a requester posts to the workers on Amazon’s platform, also known as a “project”.

  5. 5.

    Using state-of-the-art models from https://spacy.io/.

  6. 6.

    https://www.google.com/intl/en/streetview/.

  7. 7.

    https://www.visit-hannover.com/en/Sightseeing-City-Tours/Sightseeing/City-tours.

  8. 8.

    https://www.openstreetmap.org.

  9. 9.

    https://en.wikipedia.org/wiki/List_of_states_and_territories_of_the_United_States_by_population.

  10. 10.

    All meta information is provided by Google’s Street View API https://developers.google.com/maps/documentation/streetview/.

  11. 11.

    We removed manually questions such as “Who am I?”.

  12. 12.

    https://www.openstreetmap.org.

  13. 13.

    https://cloud.google.com/maps-platform/places.

  14. 14.

    https://www.wikidata.org/.

  15. 15.

    Even though we instructed the turkers to phrase only one question per input frame, not all followed the instruction.

  16. 16.

    https://www.volkswagenag.com/en/group/research---innovations.html.

References

  1. Auer, S., et al.: DBpedia: a nucleus for a web of open data. In: Aberer, K., et al. (eds.) ASWC/ISWC -2007. LNCS, vol. 4825, pp. 722–735. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-76298-0_52

    CrossRef  Google Scholar 

  2. Bao, J., Duan, N., Yan, Z., Zhou, M., Zhao, T.: Constraint-based question answering with knowledge graph. In: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp. 2503–2514. The COLING 2016 Organizing Committee, Osaka, Japan (2016)

    Google Scholar 

  3. Berant, J., Chou, A., Frostig, R., Liang, P.: semantic parsing on freebase from question-answer pairs. In: Proceedings of EMNLP (October), pp. 1533–1544 (2013)

    Google Scholar 

  4. Bollacker, K., Cook, R., Tufts, P.: Freebase: a shared database of structured general human knowledge. In: Proceedings of the National Conference on Artificial Intelligence, vol. 22, no 2, p. 1962 (2007)

    Google Scholar 

  5. Bordes, A., Chopra, S., Weston, J.: Question answering with subgraph embeddings. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 25–29 October, pp. 615–620 (2014). https://doi.org/10.3115/v1/D14-1067

  6. Bordes, A., Usunier, N., Chopra, S., Weston, J.: Large-scale simple question answering with memory networks (2015). https://doi.org/10.1016/j.geomphys.2016.04.013

  7. Buhrmester, M.D., Talaifar, S., Gosling, S.D.: An evaluation of amazon’s mechanical turk, its rapid rise, and its effective use. Perspect. Psychol. Sci. 13(2), 149–154 (2018). https://doi.org/10.1177/1745691617706516

    CrossRef  Google Scholar 

  8. Bulcaen, C.: Rethinking context: language as an interactive phenomenon. Lang. Lit. 4(1), 61–64 (1995). https://doi.org/10.1177/096394709500400105

    CrossRef  Google Scholar 

  9. Cai, Q., Yates, A.: Large-scale semantic parsing via schema matching and lexicon extension. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 423–433. Association for Computational Linguistics, Sofia, Bulgaria, August 2013

    Google Scholar 

  10. Cheung, J.H., Burns, D.K., Sinclair, R.R., Sliter, M.: Amazon mechanical turk in organizational psychology: an evaluation and practical recommendations. J. Bus. Psychol. 32(4), 347–361 (2016). https://doi.org/10.1007/s10869-016-9458-5

    CrossRef  Google Scholar 

  11. Dhingra, B., Danish, D., Rajagopal, D.: Simple and effective semi-supervised question answering. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 582–587. Association for Computational Linguistics, Stroudsburg, PA, USA (2018). https://doi.org/10.18653/v1/N18-2092

  12. Diefenbach, D., Lopez, V., Singh, K., Maret, P.: Core techniques of question answering systems over knowledge bases: a survey. Knowl. Inf. Syst. 55(3), 529–569 (2017). https://doi.org/10.1007/s10115-017-1100-y

    CrossRef  Google Scholar 

  13. Diefenbach, D., Tanon, T.P., Singh, K., Maret, P.: Question answering benchmarks for Wikidata. In: CEUR Workshop Proceedings, vol. 1963, pp. 3–6 (2017)

    Google Scholar 

  14. Dubey, M., Banerjee, D., Abdelkawi, A., Lehmann, J.: LC-QuAD 2.0: a large dataset for complex question answering over Wikidata and DBpedia. In: Ghidini, C., et al. (eds.) ISWC 2019. LNCS, vol. 11779, pp. 69–78. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30796-7_5

    CrossRef  Google Scholar 

  15. Dunn, M., Sagun, L., Higgins, M., Guney, V.U., Cirik, V., Cho, K.: SearchQA: a new q&a dataset augmented with context from a search engine (2017)

    Google Scholar 

  16. Hara, K., Le, V., Froehlich, J.: Combining crowdsourcing and google street view to identify street-level accessibility problems. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’2013, p. 631 (2013). https://doi.org/10.1145/2470654.2470744

  17. Höffner, K., Walter, S., Marx, E., Usbeck, R., Lehmann, J., Ngonga Ngomo, A.C.: Survey on challenges of question answering in the semantic web. Semant. Web 8(6), 895–920 (2017). https://doi.org/10.3233/SW-160247

    CrossRef  Google Scholar 

  18. Janarthanam, S., et al.: Evaluating a city exploration dialogue system combining question-answering and pedestrian navigation. In: 51st Annual Meeting of the Association of Computational Linguistics (October 2015), pp. 1660–1668 (2013). https://doi.org/10.18411/a-2017-023

  19. Janarthanam, S., et al.: Integrating location, visibility, and question-answering in a spoken dialogue system for pedestrian city exploration. In: Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue (July), pp. 134–136 (2012)

    Google Scholar 

  20. Jia, Z., Abujabal, A., Roy, R.S., Strötgen, J., Weikum, G.: TempQuestions: a benchmark for temporal question answering. In: WWW (Companion Volume), vol. 2, pp. 1057–1062 (2018)

    Google Scholar 

  21. Kočiský, T., et al.: The narrativeQA reading comprehension challenge. Trans. Assoc. Comput. Linguist. 6, 317–328 (2018). https://doi.org/10.1162/tacl_a_00023

    CrossRef  Google Scholar 

  22. Liu, Y., Alexandrova, T., Nakajima, T.: Using stranger [sic] as sensors: temporal and geo-sensitive question answering via social media. In: WWW ’13: Proceedings of the 22nd international conference on World Wide Web, pp. 803–813 (2013). https://doi.org/10.1145/2488388.2488458

  23. Petrochuk, M., Zettlemoyer, L.: SimpleQuestions nearly solved: a new upperbound and baseline approach. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 554–558. Association for Computational Linguistics, Stroudsburg, PA, USA (2018). https://doi.org/10.18653/v1/D18-1051

  24. Rajpurkar, P., Jia, R., Liang, P.: Know what you don’t know: unanswerable questions for SQuAD. In: ACL 2018–56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers) 2, pp. 784–789 (2018)

    Google Scholar 

  25. Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: SQuAD: 100,000+ questions for machine comprehension of text. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (ii), pp. 2383–2392 (2016). https://doi.org/10.18653/v1/D16-1264

  26. Salmen, J., Houben, S., Schlipsing, M.: Google street view images support the development of vision-based driver assistance systems. In: Proceedings of the IEEE Intelligent Vehicles Symposium (June 2012), pp. 891–895 (2012). https://doi.org/10.1109/IVS.2012.6232195

  27. Suchanek, F.M., Kasneci, G., Weikum, G.: Yago: a core of semantic knowledge. In: Proceedings of the 16th International Conference on World Wide Web (2007). https://doi.org/10.1145/1242572.1242667

  28. Trivedi, P., Maheshwari, G., Dubey, M., Lehmann, J.: LC-QuAD: a corpus for complex question answering over knowledge graphs. In: d’Amato, C., et al. (eds.) ISWC 2017. LNCS, vol. 10588, pp. 210–218. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68204-4_22

    CrossRef  Google Scholar 

  29. Unger, C., et al.: Question answering over linked data (QALD-4). In: CEUR Workshop Proceedings, vol. 1180, pp. 1172–1180 (2014)

    Google Scholar 

  30. Unger, C., et al.: Question answering over linked data (QALD-5). In: CLEF, vol. 1180, pp. 1172–1180 (2015)

    Google Scholar 

  31. Usbeck, R., Ngomo, A.C.N., Haarmann, B., Krithara, A., Röder, M., Napolitano, G.: 7th open challenge on question answering over linked data (QALD-7). Commun. Comput. Inf. Sci. 769, 59–69 (2017). https://doi.org/10.1007/978-3-319-69146-6_6

    CrossRef  Google Scholar 

  32. Usbeck, R., et al.: Benchmarking question answering systems. Semant. Web 1, 1–5 (2016)

    Google Scholar 

  33. Yang, M.C., Duan, N., Zhou, M., Rim, H.C.: Joint relational embeddings for knowledge-based question answering. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 645–650 (2014). https://doi.org/10.3115/v1/D14-1071

  34. Yang, Z., Hu, J., Salakhutdinov, R., Cohen, W.: Semi-supervised QA with generative domain-adaptive nets. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1040–1050. Association for Computational Linguistics, Stroudsburg, PA, USA (2017). https://doi.org/10.18653/v1/P17-1096

  35. Yang, Z., et al.: HotpotQA: a dataset for diverse, explainable multi-hop question answering. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369–2380. Association for Computational Linguistics, Stroudsburg, PA, USA (2019). https://doi.org/10.18653/v1/D18-1259

  36. Yatskar, M.: A qualitative comparison of CoQA, SQuAD 2.0 and QuAC. NAACL-HLT, September 2018

    Google Scholar 

  37. Yih, W.T., Chang, M.W., He, X., Gao, J.: Semantic parsing via staged query graph generation: question answering with knowledge base. In: ACL, pp. 1321–1331 (2015). https://doi.org/10.3115/v1/P15-1128

  38. Yin, Z., Goldberg, D.W., Zhang, C., Prasad, S.: An NLP-based question answering framework for spatio-temporal analysis and visualization. In: ACM International Conference Proceeding Series Part, vol. F1482, pp. 61–65 (2019). https://doi.org/10.1145/3318236.3318240

Download references

Author information

Authors and Affiliations

  1. Volkswagen Innovation Center Europe, Wolfsburg, Germany

    Jewgeni Rose

  2. Computer Science III, University of Bonn, Bonn, Germany

    Jens Lehmann

  3. Fraunhofer IAIS, Dresden, Germany

    Jens Lehmann

Authors
  1. Jewgeni Rose
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jens Lehmann
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jewgeni Rose .

Editor information

Editors and Affiliations

  1. University of Edinburgh, Edinburgh, UK

    Jeff Z. Pan

  2. University of Liverpool, Liverpool, UK

    Valentina Tamma

  3. University of Bari, Bari, Italy

    Claudia d’Amato

  4. University of California, Santa Barbara, Santa Barbara, CA, USA

    Krzysztof Janowicz

  5. California State University, Long Beach, Long Beach, CA, USA

    Bo Fu

  6. Vienna University of Economics and Business, Vienna, Austria

    Axel Polleres

  7. Rensselaer Polytechnic Institute, Troy, NY, USA

    Oshani Seneviratne

  8. Massachusetts Institute of Technology, Cambridge, MA, USA

    Lalana Kagal

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Rose, J., Lehmann, J. (2020). CASQAD – A New Dataset for Context-Aware Spatial Question Answering. In: , et al. The Semantic Web – ISWC 2020. ISWC 2020. Lecture Notes in Computer Science(), vol 12507. Springer, Cham. https://doi.org/10.1007/978-3-030-62466-8_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-62466-8_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-62465-1

  • Online ISBN: 978-3-030-62466-8

  • eBook Packages: Computer ScienceComputer Science (R0)