WIP - SKOD: A Framework for Situational Knowledge on Demand

  • Servio PalaciosEmail author
  • K. M. A. SolaimanEmail author
  • Pelin Angin
  • Alina Nesen
  • Bharat Bhargava
  • Zachary Collins
  • Aaron Sipser
  • Michael Stonebraker
  • James Macdonald
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11721)


Extracting relevant patterns from heterogeneous data streams poses significant computational and analytical challenges. Further, identifying such patterns and pushing analogous content to interested parties according to mission needs in real-time is a difficult problem. This paper presents the design of SKOD, a novel Situational Knowledge Query Engine that continuously builds a multi-modal relational knowledge base using SQL queries; SKOD pushes dynamic content to relevant users through triggers based on modeling of users’ interests. SKOD is a scalable, real-time, on-demand situational knowledge extraction and dissemination framework that processes streams of multi-modal data utilizing publish/subscribe stream engines. The initial prototype of SKOD uses deep neural networks and natural language processing techniques to extract and model relevant objects from video streams and topics, entities and events from unstructured text resources such as Twitter and news articles. Through its extensible architecture, SKOD aims to provide a high-performance, generic framework for situational knowledge on demand, supporting effective information retrieval for evolving missions.


Query engine Multi-modal information retrieval Knowledge base Stream processing Targeted information dissemination 


Funding Information

Distribution Statement A: Approved for Public Release; Distribution is Unlimited; #19-1107; Dated 07/18/19.


  1. 1.
    Abu-El-Haija, S., et al.: YouTube-8M: a large-scale video classification benchmark. CoRR abs/1609.08675 (2016).
  2. 2.
    Adjali, O., Hina, M.D., Dourlens, S., Ramdane-Cherif, A.: Multimodal fusion, fission and virtual reality simulation for an ambient robotic intelligence. In: ANT/SEIT. Procedia Computer Science, vol. 52, pp. 218–225. Elsevier (2015)Google Scholar
  3. 3.
    Bienvenu, M., Bourgaux, C., Goasdoué, F.: Query-driven repairing of inconsistent DL-Lite knowledge bases. In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, 9–15 July 2016, pp. 957–964 (2016).
  4. 4.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003). Scholar
  5. 5.
    Chen, Y., Wang, D.Z.: Knowledge expansion over probabilistic knowledge bases. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, SIGMOD 2014, pp. 649–660. ACM, New York (2014).
  6. 6.
    Cugola, G., Margara, A.: Processing flows of information: from data stream to complex event processing. ACM Comput. Surv. 44(3), 15:1–15:62 (2012). Scholar
  7. 7.
    Dong, X., et al.: Knowledge vault: a web-scale approach to probabilistic knowledge fusion. In: KDD, pp. 601–610. ACM (2014)Google Scholar
  8. 8.
    Foresti, G.L., Farinosi, M., Vernier, M.: Situational awareness in smart environments: socio-mobile and sensor data fusion for emergency response to disasters. J. Ambient Intell. Humanized Comput. 6(2), 239–257 (2015)CrossRefGoogle Scholar
  9. 9.
    Itria, M.L., Daidone, A., Ceccarelli, A.: A complex event processing approach for crisis-management systems. CoRR abs/1404.7551 (2014)Google Scholar
  10. 10.
    Kang, D., Bailis, P., Zaharia, M.: BlazeIt: fast exploratory video queries using neural networks. CoRR abs/1805.01046 (2018)Google Scholar
  11. 11.
    Krishna, R., et al.: Visual genome: connecting language and vision using crowdsourced dense image annotations. CoRR abs/1602.07332 (2016).
  12. 12.
    Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). Scholar
  13. 13.
    Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., McClosky, D.: The Stanford CoreNLP natural language processing toolkit. In: Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 55–60. Association for Computational Linguistics, Baltimore, June 2014.
  14. 14.
    Meditskos, G., Vrochidis, S., Kompatsiaris, I.: Description logics and rules for multimodal situational awareness in healthcare. In: Amsaleg, L., Guðmundsson, G., Gurrin, C., Jónsson, B., Satoh, S. (eds.) MMM 2017, Part I. LNCS, vol. 10132, pp. 714–725. Springer, Cham (2017). Scholar
  15. 15.
    Nguyen, D.B., Abujabal, A., Tran, N.K., Theobald, M., Weikum, G.: Query-driven on-the-fly knowledge base construction. Proc. VLDB Endow. 11(1), 66–79 (2017). Scholar
  16. 16.
    Palacios, S., Santos, V., Barsallo, E., Bhargava, B.: MioStream: a peer-to-peer distributed live media streaming on the edge. Multimedia Tools Appl. (2019). Scholar
  17. 17.
    Poria, S., Cambria, E., Howard, N., Huang, G.B., Hussain, A.: Fusing audio, visual and textual clues for sentiment analysis from multimodal content. Neurocomputing 174(PA), 50–59 (2016). Scholar
  18. 18.
    Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779–788 (2016)Google Scholar
  19. 19.
    Rodríguez, M.E., Goldberg, S., Wang, D.Z.: SigmaKB: multiple probabilistic knowledge base fusion. PVLDB 9(13), 1577–1580 (2016)Google Scholar
  20. 20.
    Wu, Q., Wang, P., Shen, C., Dick, A.R., van den Hengel, A.: Ask me anything: free-form visual question answering based on knowledge from external sources. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, 27–30 June 2016, pp. 4622–4630 (2016).
  21. 21.
    Zhu, Y., Lim, J.J., Fei-Fei, L.: Knowledge acquisition for visual question answering via iterative querying. In: CVPR, pp. 6146–6155. IEEE Computer Society (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Servio Palacios
    • 1
    Email author
  • K. M. A. Solaiman
    • 1
    Email author
  • Pelin Angin
    • 2
  • Alina Nesen
    • 1
  • Bharat Bhargava
    • 1
  • Zachary Collins
    • 3
  • Aaron Sipser
    • 3
  • Michael Stonebraker
    • 3
  • James Macdonald
    • 4
  1. 1.Purdue UniversityWest LafayetteUSA
  2. 2.METUAnkaraTurkey
  3. 3.MIT CSAILCambridgeUSA
  4. 4.Northrop Grumman CorporationFalls ChurchUSA

Personalised recommendations