Using semantic roles to improve text classification in the requirements domain

Original Paper

Abstract

Engineering activities often produce considerable documentation as a by-product of the development process. Due to their complexity, technical analysts can benefit from text processing techniques able to identify concepts of interest and analyze deficiencies of the documents in an automated fashion. In practice, text sentences from the documentation are usually transformed to a vector space model, which is suitable for traditional machine learning classifiers. However, such transformations suffer from problems of synonyms and ambiguity that cause classification mistakes. For alleviating these problems, there has been a growing interest in the semantic enrichment of text. Unfortunately, using general-purpose thesaurus and encyclopedias to enrich technical documents belonging to a given domain (e.g. requirements engineering) often introduces noise and does not improve classification. In this work, we aim at boosting text classification by exploiting information about semantic roles. We have explored this approach when building a multi-label classifier for identifying special concepts, called domain actions, in textual software requirements. After evaluating various combinations of semantic roles and text classification algorithms, we found that this kind of semantically-enriched data leads to improvements of up to 18% in both precision and recall, when compared to non-enriched data. Our enrichment strategy based on semantic roles also allowed classifiers to reach acceptable accuracy levels with small training sets. Moreover, semantic roles outperformed Wikipedia- and WordNET-based enrichments, which failed to boost requirements classification with several techniques. These results drove the development of two requirements tools, which we successfully applied in the processing of textual use cases.

Keywords

Text classification Natural language processing Knowledge representation Semantic enrichment Use case specification 

Notes

Acknowledgements

This work was partially supported by ANPCyT (Argentina) through PICT Project 2015 No. 2565. The authors are grateful to the doctoral students that helped to manually tag the sentences of the case-studies with DAs. The authors would like to make a special mention to Paula Frade, Miguel Ruival, German Attanasio and Rodrigo Gonzalez for testing the DA classifier and helping us to make adjustments to the implementation. The authors also thank the anonymous reviewers for their feedback that helped to improve the quality of the manuscript.

References

  1. Badawi, D., & Altincay, H. (2014). A novel framework for termset selection and weighting in binary text classification. Engineering Applications of Artificial Intelligence, 35, 38–53.  https://doi.org/10.1016/j.engappai.2014.06.012.CrossRefGoogle Scholar
  2. Bai, R., Wang, X., & Liao, J. (2010). Extract semantic information from wordnet to improve text classification performance. In T. H. Kim & H. Adeli (Eds.), Advances in computer science and information technology, Lecture notes in computer science (Vol. 6059, pp. 409–420). Berlin: Springer.  https://doi.org/10.1007/978-3-642-13577-4_36.CrossRefGoogle Scholar
  3. Björkelund, A., Bohnet, B., Hafdell, L., & Nugues, P. (2010). A high-performance syntactic and semantic dependency parser. 23rd International conference on computational linguistics: Demonstrations (COLING ’10) (pp. 33–36). Beijing: Association for Computational Linguistics.Google Scholar
  4. Bloehdorn, S., & Hotho, A. (2006). Boosting for text classification with semantic features. Advances in Web Mining and Web Usage Analysis, 3932, 149–166.  https://doi.org/10.1007/11899402_10.CrossRefGoogle Scholar
  5. Casamayor, A., Godoy, D., & Campo, M. (2012). Functional grouping of natural language requirements for assistance in architectural software design. Knowledge-Based Systems, 30, 78–86.  https://doi.org/10.1016/j.knosys.2011.12.009.CrossRefGoogle Scholar
  6. Compeau, P., & Pevzner, P. (2015). Bioinformatics algorithms: An active learning approach (2nd ed.). San Diego: Active Learning Publishers.Google Scholar
  7. Diamantopoulos, T., Roth, M., Symeonidis, A., & Klein, E. (2017). Software requirements as an application domain for natural language processing. Language Resources and Evaluation.  https://doi.org/10.1007/s10579-017-9381-z.Google Scholar
  8. Egozi, O., Markovitch, S., & Gabrilovich, E. (2011). Concept-based information retrieval using explicit semantic analysis. ACM Transactions on Information Systems (TOIS), 29(2), 8:1–8:34.  https://doi.org/10.1145/1961209.1961211.CrossRefGoogle Scholar
  9. Falessi, D., Cantone, G., & Canfora, G. (2013). Empirical principles and an industrial case study in retrieving equivalent requirements via natural language processing techniques. IEEE Transactions on Software Engineering, 39(1), 18–44.  https://doi.org/10.1109/TSE.2011.122.CrossRefGoogle Scholar
  10. Femmer, H., Fernandez, D. M., Wagner, S., & Eder, S. (2017). Rapid quality assurance with requirements smells. Journal of Systems and Software, 123, 190–213.  https://doi.org/10.1016/j.jss.2016.02.047.CrossRefGoogle Scholar
  11. Huang, L. (2011). Concept-based text clustering. Doctoral thesis, The University of Waikato, HamiltonGoogle Scholar
  12. Huang, L., Milne, D., Frank, E., & Witten, I. H. (2012). Learning a concept-based document similarity measure. Journal of the American Society for Information Science and Technology, 63(8), 1593–1608.  https://doi.org/10.1002/asi.22689.CrossRefGoogle Scholar
  13. Hull, E., Jackson, K., & Dick, J. (2014). Requirements engineering (3rd ed.). Berlin: Springer.Google Scholar
  14. Jurkiewicz, J., & Nawrocki, J. (2015). Automated events identification in use cases. Information and Software Technology, 58, 110–122.  https://doi.org/10.1016/j.infsof.2014.09.011.CrossRefGoogle Scholar
  15. Kamalrudin M., Hosking J. G., & Grundy, J. (2011). Improving requirements quality using essential use case interaction patterns. In ICSE’11, Hawaii (pp. 531–540).  https://doi.org/10.1145/1985793.1985866.
  16. Kang, S., Cho, S., & Kang, P. (2015). Multi-class classification via heterogeneous ensemble of one-class classifiers. Engineering Applications of Artificial Intelligence, 43, 35–43.  https://doi.org/10.1016/j.engappai.2015.04.003.CrossRefGoogle Scholar
  17. Kehagias, A., Petridis, V., Kaburlasos, V. G., & Fragkou, P. (2003). A comparison of word- and sense-based text categorization using several classification algorithms. Journal of Intelligent Information Systems, 21(3), 227–247.  https://doi.org/10.1023/A:1025554732352.CrossRefGoogle Scholar
  18. Kelleher, J. D., Namee, B. M., & D’Arcy, A. (2015). Fundamentals of machine learning for predictive data analytics: Algorithms, worked examples, and case studies (1st ed.). Cambridge: MIT Press.Google Scholar
  19. Li, J. Q., Zhao, Y., & Liu, B. (2012). Exploiting semantic resources for large scale text categorization. Journal of Intelligent Information Systems, 39(3), 763–788.  https://doi.org/10.1007/s10844-012-0211-x.CrossRefGoogle Scholar
  20. Llorens, H., Saquete, E., & Navarro-Colorado, B. (2013). Applying semantic knowledge to the automatic processing of temporal expressions and events in natural language. Information Processing & Management, 49(1), 179–197.  https://doi.org/10.1016/j.ipm.2012.05.005.CrossRefGoogle Scholar
  21. Mahmoud A., & Carver, D. (2015). Exploiting online human knowledge in requirements engineering. In 23rd International requirements engineering conference (RE’15), IEEE (pp. 262–267).  https://doi.org/10.1109/RE.2015.7320434
  22. Mansuy T., & Hilderman, R. J. (2006). A characterization of wordnet features in Boolean models for text classification. In 5th Australasian conference on data mining and analystics (AusDM’06) (Vol. 61, pp. 103–109).Google Scholar
  23. Ménard, P. A., & Ratté, S. (2016). Concept extraction from business documents for software engineering projects. Automated Software Engineering, 23(4), 649–686.  https://doi.org/10.1007/s10515-015-0184-4.CrossRefGoogle Scholar
  24. Mund, J, Fernandez, D, M., Femmer, H., & Eckhardt, J. (2015) Does quality of requirements specifications matter? Combined results of two empirical studies. In ACM/IEEE international symposium on empirical software engineering and measurement (ESEM’15) (pp. 1–10).  https://doi.org/10.1109/ESEM.2015.7321195.
  25. Navigli, R., Faralli, S., Soroa, A., de Lacalle, O., & Agirre, E. (2011). Two birds with one stone: Learning semantic models for text categorization and word sense disambiguation. In 20th ACM international conference on information and knowledge management (CIKM’11) (pp. 2317–2320).  https://doi.org/10.1145/2063576.2063955.
  26. Nazir, F., Butt, W. H., Anwar, M. W., & Khan Khattak, M. A. (2017). The applications of natural language processing (NLP) for software requirement engineering—A systematic literature review (pp. 485–493). Singapore: Springer.  https://doi.org/10.1007/978-981-10-4154-9_56.Google Scholar
  27. Nguyen, T. H., Grundy, J., & Almorsy, M. (2015). Rule-based extraction of goal-use case models from text. In 10th Joint meeting on foundations of software engineering (FSE’2015) (pp. 591–601).  https://doi.org/10.1145/2786805.2786876.
  28. Palmer, M., Gildea, D., & Kingsbury, P. (2005). The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1), 71–106.  https://doi.org/10.1162/0891201053630264.CrossRefGoogle Scholar
  29. Palmer, M., Gildea, D., & Xue, N. (2010). Semantic role labeling. Synthesis lectures on human language technologies. San Rafael: Morgan & Claypool.Google Scholar
  30. Rago, A., Marcos, C., & Diaz-Pace, A. (2013). Uncovering quality-attribute concerns in use case specifications via early aspect mining. Requirements Engineering, 18(1), 67–84.  https://doi.org/10.1007/s00766-011-0142-z.CrossRefGoogle Scholar
  31. Rago, A., Marcos, C., & Diaz-Pace, A. (2016a). Assisting requirements analysts to find latent concerns with REAssistant. Automated Software Engineering, 23(2), 219–252.  https://doi.org/10.1007/s10515-014-0156-0.CrossRefGoogle Scholar
  32. Rago, A., Marcos, C., & Diaz-Pace, A. (2016b). Identifying duplicate functionality in textual use cases by aligning semantic actions. Software and Systems Modeling, 15(2), 579–603.  https://doi.org/10.1007/s10270-014-0431-3.CrossRefGoogle Scholar
  33. Rago, A., Marcos, C., & Diaz-Pace, A. (2016c). Opportunities for analyzing hardware specifications with NLP techniques. In 3rd Workshop on design automation for understanding hardware designs (DUHDe’16), design, automation and test in Europe conference and exhibition (DATE’16), Dresden, Germany.Google Scholar
  34. Rooney, N., Wang, H., & Taylor, P. S. (2014). An investigation into the application of ensemble learning for entailment classification. Information Processing & Management, 50(1), 87–103.  https://doi.org/10.1016/j.ipm.2013.08.002.CrossRefGoogle Scholar
  35. Rosadini, B., Ferrari, A., Gori, G., Fantechi, A., Gnesi, S., Trotta, I., & Bacherini, S. (2017). Using NLP to detect requirements defects: An industrial experience in the railway domain. In chap 23rd International working conference REFSQ 2017, Essen, Germany, February 27–March 2, 2017, Proceedings (pp. 344–360). Springer International Publishing.  https://doi.org/10.1007/978-3-319-54045-0_24.
  36. Roth, M., & Klein, E. (2015). Parsing software requirements with an ontology-based semantic role labeler. In 1st Workshop on language and ontologies at the 11th international conference on computational semantics (IWCS’15) (pp. 15–21). London, United Kingdom.Google Scholar
  37. Roth, M., Diamantopoulos, T., Klein, E., & Symeonidis, A. (2014). Software requirements: A new domain for semantic parsers. In Workshop on semantic parsing at the conference of the association for computational linguistics (ACL’14) (pp. 50–54). Baltimore, MD.Google Scholar
  38. Selvaretnam, B., & Belkhatir, M. (2016). A linguistically driven framework for query expansion via grammatical constituent highlighting and role-based concept weighting. Information Processing & Management, 52(2), 174–192.  https://doi.org/10.1016/j.ipm.2015.04.002.CrossRefGoogle Scholar
  39. Sengupta, S., Ramnani, R. R., Das, S., & Chandran, A. (2015). Verb-based semantic modelling and analysis of textual requirements. In 8th India software engineering conference (ISEC’15) (pp. 30–39).  https://doi.org/10.1145/2723742.2723745.
  40. Sinha, A., Paradkar, A., Takeuchi, H., & Nakamura, T. (2010). Extending automated analysis of natural language use cases to other languages. In 18th IEEE international requirements engineering conference (RE’10) (pp. 364–369).  https://doi.org/10.1109/RE.2010.52
  41. Sokolova, M., & Lapalme, G. (2009). A systematic analysis of performance measures for classification tasks. Information Processing & Management, 45(4), 427–437.  https://doi.org/10.1016/j.ipm.2009.03.002.CrossRefGoogle Scholar
  42. Szu-ting, Y. (2015). Robust semantic role labeling. United States: LAP Lambert Academic Publishing.Google Scholar
  43. Tommasel, A., & Godoy, D. (2014). Semantic grounding of social annotations for enhancing resource classification in folksonomies. Journal of Intelligent Information Systems.  https://doi.org/10.1007/s10844-014-0339-y.Google Scholar
  44. Tsoumakas, G., Katakis, I., & Vlahavas, I. (2010). Mining multi-label data. In O. Maimon & L. Rokach (Eds.), Data mining and knowledge discovery handbook (pp. 667–685). Boston, MA: Springer.  https://doi.org/10.1007/978-0-387-09823-4_34.
  45. Uysal, A. K., & Gunal, S. (2014). The impact of preprocessing on text classification. Information Processing & Management, 50(1), 104–112.  https://doi.org/10.1016/j.ipm.2013.08.006.CrossRefGoogle Scholar
  46. Wang, P., & Domeniconi, C. (2008). Building semantic kernels for text classification using wikipedia. In 14th ACM SIGKDD international conference on knowledge discovery and data mining (KDD’08) (pp. 713–721).  https://doi.org/10.1145/1401890.1401976.
  47. Wang, P., Hu, J., Zeng, H. J., & Chen, Z. (2009). Using wikipedia knowledge to improve text classification. Knowledge and Information Systems, 19(3), 265–281.  https://doi.org/10.1007/s10115-008-0152-4.CrossRefGoogle Scholar
  48. Wiegers, K., & Beatty, J. (2013). Software requirements (3rd ed.). Developer best practices. Redmond, WA: Microsoft Press.Google Scholar
  49. Zhang, H. (2004). The optimality of naive bayes. In V. Barr & Z. Markov (Eds.), 17th International Florida Artificial Intelligence Research Society conference (FLAIRS 2004) (pp. 562–567). Miami Beach, FL: AAAI Press.Google Scholar
  50. Zhang, M. L., & Zhou, Z. H. (2014). A review on multi-label learning algorithms. IEEE Transactions on Knowledge and Data Engineering, 26(8), 1819–1837.  https://doi.org/10.1109/TKDE.2013.39.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V., part of Springer Nature 2017

Authors and Affiliations

  1. 1.ISISTAN Research InstituteUNICEN UniversityTandilArgentina
  2. 2.CONICETBuenos AiresArgentina
  3. 3.CICBuenos AiresArgentina

Personalised recommendations