Applying a Convolutional Neural Network to Legal Question Answering

  • Mi-Young KimEmail author
  • Ying Xu
  • Randy Goebel
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10091)


Our legal question answering system combines legal information retrieval and textual entailment, and we describe a legal question answering system that exploits a deep convolutional neural network. We have evaluated our system using the training/test data from the competition on legal information extraction/entailment (COLIEE). The competition focuses on the legal information processing related to answering yes/no questions from Japanese legal bar exams, and it consists of three phases: ad-hoc legal information retrieval, textual entailment, and a learning model-driven combination of the two phases. Phase 1 requires the identification of Japan civil law articles relevant to a legal bar exam query. For that phase, we have implemented a combined TF-IDF and Ranking SVM information retrieval component. Phase 2 requires the system to answer “Yes” or “No” to previously unseen queries, by comparing extracted meanings of queries with relevant articles. Our training of an entailment model focuses on features based on word embeddings, syntactic similarities and identification of negation/antonym relations. We augment our textual entailment component with a convolutional neural network with dropout regularization and Rectified Linear Units. To our knowledge, our study is the first to adapt deep learning for textual entailment. Experimental evaluation demonstrates the effectiveness of the convolutional neural network and dropout regularization. The results show that our deep learning-based method outperforms our baseline SVM-based supervised model and K-means clustering.


Legal question answering Recognizing textual entailment Information retrieval Convolutional neural network 



This research was supported by the Alberta Innovates Centre for Machine Learning (AICML) and the Natural Sciences and Engineering Research Council (NSERC). We are indebted to Ken Satoh of the National Institute for Informatics, who has had the vision to create the COLIEE competition.


  1. 1.
    Jones, K.S.: A statistical interpretation of term specificity and its application in retrieval. In: Willett, P. (ed.) Document Retrieval Systems, pp. 132–142. Taylor Graham Publishing, London (1988)Google Scholar
  2. 2.
    Joachims, T.: Optimizing search engines using clickthrough data. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2002, pp. 133–142. ACM, New York (2002)Google Scholar
  3. 3.
    Maxwell, K.T., Oberlander, J., Croft, W.B.: Feature-based selection of dependency paths in ad hoc information retrieval. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, vol. 1. Long Papers, pp. 507–516. Association for Computational Linguistics, Sofia, Bulgaria, August 2013Google Scholar
  4. 4.
    Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The WEKA data mining software: an update. SIGKDD Explor. 11(1), 10–18 (2009)CrossRefGoogle Scholar
  5. 5.
    Kim, M-Y., Kang, S-J., Lee, J-H.: Resolving ambiguity in inter-chunk dependency parsing. In: Proceedings of 6th Natural Language Processing Pacific Rim Symposium, pp. 263–270 (2001)Google Scholar
  6. 6.
    Bdour, W.N., Gharaibeh, N.K.: Development of yes/no arabic question answering system. Int. J. Artif. Intell. Appl. 4(1), 51–63 (2013)Google Scholar
  7. 7.
    Nielsen, R.D., Ward, W., Martin, J.H.: Toward dependency path based entailment. In: Proceedings of the Second PASCAL Challenges Workshop on RTE (2006)Google Scholar
  8. 8.
    Zanzotto, F.M., Moschitti, A., Pennacchiotti, M., Pazienza, M.T.: Learning textual entailment from examples. In: Proceedings of the Second PASCAL Challenges Workshop on RTE (2006)Google Scholar
  9. 9.
    Harmeling, S.: An extensible probabilistic transformation-based approach to the third recognizing textual entailment challenge. In: Proceedings of ACL PASCAL Workshop on Textual Entailment and Paraphrasing (2007)Google Scholar
  10. 10.
    Marsi, E., Krahmer, E., Bosma, W.: Dependency-based paraphrasing for recognizing textual entailment. In: Proceedings of ACL PASCAL Workshop on Textual Entailment and Paraphrasing (2007)Google Scholar
  11. 11.
    Yu, L., Hermann, K.M., Blunsom, P., Pulman, S.: Deep learning for answer sentence selection. arXiv preprint arXiv:1412.1632 (2014)
  12. 12.
    Kalchbrenner, N., Grefenstette, E., Blunsom, P.: A convolutional neural network for modelling sentences. In: Proceedings of ACL (2014)Google Scholar
  13. 13.
    Socher, R., Huval, B., Manning, C.D., Ng, A.Y.: Semantic compositionality through recursive matrix-vector spaces. In: Proceedings of EMNLP-CoNLL (2012)Google Scholar
  14. 14.
    Hermann, K.M., Blunsom, P.: Multilingual models for compositional distributional semantics. In: Proceedings of ACL (2014)Google Scholar
  15. 15.
    Kim, M-Y., Xu, Y., Goebel, R.: Alberta-KXG: legal question answering using ranking SVM and syntactic/semantic similarity. In: JURISIN Workshop (2014)Google Scholar
  16. 16.
    Yih, W.-T., He, X., Meek, C.: Semantic parsing for single-relation question answering. In: Proceedings of ACL (2014)Google Scholar
  17. 17.
    Dahl, G.E., Sainath, T.N., Hinton, G.E.: Improving deep neural networks for LVCSR using rectified linear units and dropout. In: Proceedings of Acoustics, Speech and Signal Processing (ICASSP), pp. 8609–8613 (2013)Google Scholar
  18. 18.
    Deng, L., Abdel-Hamid, O., Yu, D.: A deep convolutional neural network using heterogeneous pooling for trading acoustic invariance with phonetic confusion. In: Proceedings of Acoustics, Speech and Signal Processing (ICASSP), pp. 6669–6673. IEEE (2013)Google Scholar
  19. 19.
    Bordes, A., Chopra, S., Weston, J.: Question answering with subgraph embeddings. In: Proceedings of EMNLP (2014)Google Scholar
  20. 20.
    Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)Google Scholar
  21. 21.
    Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580 (2012)
  22. 22.
    Kouylekov, M., Magnini, B.: Tree edit distance for recognizing textual entailment: estimating the cost of insertion. In: Proceedings of the Second PASCAL Challenges Workshop on RTE (2006)Google Scholar
  23. 23.
    Kim, M., Goebel, R., Satoh, K.: COLIEE-2015: evaluation of legal question answering. In: Ninth Workshop on Juris-Informatics (JURISIN) (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Alberta Innovates Centre for Machine Learning, Department of Computing ScienceUniversity of AlbertaEdmontonCanada

Personalised recommendations