New Metric Based on SQuAD for Evaluating Accuracy of Enterprise Search Algorithms

  • Harshad Kulkarni
  • Himanshu Gupta
  • Kalpesh Balar
  • Praful KrishnaEmail author
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1130)


Enterprise Search is a continuously evolving and important field, which is seeing a resurgence driven by artificial intelligence. Still, there is no objective, generally accepted way to compare various enterprise search systems. SQuAD is becoming popular for measuring algorithmic reading comprehension (MRC) but is ineffective for quantifying effectiveness of enterprise search in business-use situations. In this paper we modify the SQuAD scoring methodology to propose a scoring system for enterprise search systems that aligns with the real world expectations of users. Further, we use a search system based on Calibrated Quantum Mesh (CQM) to underscore the relevance of this metric.


Enterprise search Scoring system Squad Calibrated Quantum Mesh CQM 


  1. 1.
    Hawking, D.: Challenges in enterprise search. In: Proceedings of the 15th Australasian Database Conference (ADC 2004), vol. 27, pp. 15–24. Australian Computer Society, Inc., Darlinghurst (2004)Google Scholar
  2. 2.
    Page, L., Brin, S., Motwani, R., Winograd, T.: The PageRank citation ranking: bringing order to the web (1999)Google Scholar
  3. 3.
    Guha, R., McCool, R., Miller, E.: Semantic search. In: Proceedings of the 12th International Conference on World Wide Web (WWW 2003), pp. 700–709. ACM, New York (2003)Google Scholar
  4. 4.
    Voigt, C.A., Gordon, D.B., Mayo, S.L.: Trading accuracy for speed: a quantitative comparison of search algorithms in protein sequence design. J. Mol. Biol. 299(3), 789–803 (2000). Edited by J ThorntonCrossRefGoogle Scholar
  5. 5.
    Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: SQuAD: 100,000+ questions for machine comprehension of text. CoRR, abs/1606.05250 (2016)Google Scholar
  6. 6.
    Rajpurkar, P., Jia, R., Liang, P.: Know what you don’t know: unanswerable questions for SQuAD. CoRR, abs/1806.03822 (2018)Google Scholar
  7. 7.
    Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: Squad Rajpurkar - Google Scholar (2019). Accessed 21 May 2019Google Scholar
  8. 8.
    Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: The Stanford Question Answering Dataset (2019). Accessed 21 May 2019Google Scholar
  9. 9.
    Kulkarni, R., Kulkarni, H., Balar, K., Krishna, P.: Cognitive natural language search using calibrated quantum mesh. In: 2018 IEEE 17th International Conference on Cognitive Informatics Cognitive Computing (ICCI*CC), pp. 174–178, July 2018Google Scholar
  10. 10.
    Viswanath, S., Yates, M., Burt, J., Yazell, J., Kuhr, R., Strum, B., Krishna, P., Balar, K., Kulkarni, R., Kulkarni, H., Fennell, J.: An intelligent machine for document preparation. In: AICHE Annual Meeting, October 2018Google Scholar
  11. 11.
    Han, K.H., Park, J.W.: Process-centered knowledge model and enterprise ontology for the development of knowledge management system. Expert Syst. Appl. 36(4), 7441–7447 (2009)CrossRefGoogle Scholar
  12. 12.
    Redfern, D.M.: Natural language meta-search system and method. VI 20 2000. US Patent 6,078,914Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Harshad Kulkarni
    • 1
  • Himanshu Gupta
    • 1
  • Kalpesh Balar
    • 1
  • Praful Krishna
    • 1
    Email author
  1. 1.Arbot Solutions Inc. dba CoseerSan FranciscoUSA

Personalised recommendations