Automated Software Engineering

, Volume 24, Issue 4, pp 839–861 | Cite as

NLCI: a natural language command interpreter

  • Mathias LandhäußerEmail author
  • Sebastian Weigelt
  • Walter F. Tichy


Natural language interfaces are becoming more and more common, because they are powerful and easy to use. Examples of such interfaces are voice controlled navigation devices, Apple’s personal assistant Siri, Google Voice Search, and translation services. However, such interfaces are extremely difficult to build, to maintain, and to port to new domains. We present an approach for building and porting such interfaces quickly. NLCI is a natural language command interpreter that accepts action commands in English and translates them into executable code. The core component is an ontology that models an API. Once the API is “ontologized”, NLCI translates input sentences into sequences of API calls that implement the intended actions. Two radically different APIs were ontologized: openHAB for home automation and Alice for building 3D animations. Construction of the ontology can be automated if the API uses descriptive names for its components. In that case, the language interface can be generated completely automatically. Recall and precision of NLCI on a benchmark of 50 input scripts are 67 and 78 %, resp. Though not yet acceptable for practical use, the results indicate that the approach is feasible. NLCI accepts typed input only. Future work will use a speech front-end to test spoken input.


Programming in natural language End-user programming Knowledge-based software engineering Program synthesis Natural language processing for software engineering 


  1. Androutsopoulos, I., Ritchie, G., Thanisch, P.: Natural language interfaces to databases—an introduction. Nat. Lang. Eng. 1(01), 29–81 (1995). doi: 10.1017/S135132490000005X.
  2. Ballard, B.W., Biermann, A.W.: Programming in natural language: NLC as a prototype. In: Proceedings of the 1979 Annual Conference, ACM ’79, pp. 228–237. ACM, New York, NY, USA (1979). doi: 10.1145/800177.810072
  3. Bellegarda, J.R.: Spoken language understanding for natural interaction: the Siri experience. In: Mariani, J., Rosset, S., Garnier-Rizet, M., Devillers, L. (eds.) Natural Interaction with Robots, Knowbots and Smartphones, pp. 3–14. Springer, New York (2014)CrossRefGoogle Scholar
  4. Biermann, A.W., Ballard, B.W., Sigmon, A.H.: An experimental study of natural language programming. Int. J. Man Mach. Stud. 18(1), 71–87 (1983). doi: 10.1016/S0020-7373(83)80005-4.
  5. Chomsky, N.: The Minimalist Program. MIT Press, Cambridge (1995)zbMATHGoogle Scholar
  6. Conway, M.J.: Alice: Easy-to-learn 3D scripting for novices. PhD Thesis, Faculty of the School of Engineering and Applied Science, University of Virginia (1997).
  7. Damljanovic, D., Agatonovic, M., Cunningham, H.: Natural language interfaces to ontologies: combining syntactic analysis and ontology-based lookup through the user interaction. In: Aroyo, L., Antoniou, G., Hyvönen, E., ten Teije, A., Stuckenschmidt, H., Cabral, L., Tudorache, T. (eds.) The Semantic Web: Research and Applications, Lecture Notes in Computer Science, vol. 6088, pp. 106–120. Springer, Berlim (2010). doi: 10.1007/978-3-642-13486-9_8 CrossRefGoogle Scholar
  8. de Marneffe, M.C., Manning, C.D.: The Stanford typed dependencies representation. In: Coling 2008: Proceedings of the Workshop on Cross-Rramework and Cross-Domain Parser Evaluation, CrossParser ’08, pp. 1–8. Association for Computational Linguistics, Stroudsburg, PA, USA (2008).
  9. de Marneffe, M.C., Manning, C.D.: Stanford typed dependencies manual. Tech. Rep. (2015).
  10. Ferrucci, D., Brown, E., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A.A., Lally, A., Murdock, J.W., Nyberg, E., Prager, J., Schlaefer, N., Welty, C.: Building Watson: an overview of the DeepQA project. AI Mag. 31(3), 59–79 (2010). doi: 10.1609/aimag.v31i3.2303.
  11. Goodman, D.: Danny Goodman’s AppleScript Handbook, 2nd edn., Incorporated (1998)Google Scholar
  12. Gruber, T.R.: A translation approach to portable ontology specifications. Knowl. Acquis. 5(2), 199–220 (1993). doi: 10.1006/knac.1993.1008.
  13. Guzzoni, D., Baur, C., Cheyer, A.: Modeling human-agent interaction with active ontologies. In: Interaction Challenges for Intelligent Assistants, Papers from the 2007 AAAI Spring Symposium, Technical Report SS-07-04, Stanford, California, USA, March 26–28, 2007, pp. 52–59. AAAI (2007).
  14. Knöll, R., Gasiunas, V., Mezini, M.: Naturalistic types. In: Proceedings of the 10th SIGPLAN Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, Onward! 2011, pp. 33–48. ACM, New York, NY, USA (2011). doi: 10.1145/2048237.2048243
  15. Knöll, R., Mezini, M.: Pegasus: First steps toward a naturalistic programming language. In: Companion to the 21st ACM SIGPLAN Symposium on Object-Oriented Programming Systems, Languages, and Applications, OOPSLA ’06, pp. 542–559. ACM, Portland, Oregon, USA (2006). doi: 10.1145/1176617.1176628
  16. Knublauch, H., Fergerson, R.W., Noy, N.F., Musen, M.A.: The Protégé OWL Plugin: an open development environment for semantic web applications. In: McIlraith, S.A., Plexousakis, D., van Harmelen, F. (eds.) The semantic Web—ISWC 2004. Lecture Notes in Computer Science, vol. 3298, pp. 229–243. Springer, BerlinGoogle Scholar
  17. Körner, S.J., Brumm, T.: Natural language specification improvement with ontologies. Int. J. Semant. Comput. (IJSC) 03(04), 445–470 (2010). doi: 10.1142/S1793351X09000872.
  18. Landhäußer, M., Hey, T., Tichy, W.F.: Deriving timelines from texts. In: Proceedings of the 3rd International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering, pp. 45–51 (2014). doi: 10.1145/2593801.2593809
  19. Landhäußer, M., Hug, R.: Text understanding for programming in natural language: control structures. In: Proceedings of the 4th International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (2015). doi: 10.1109/RAISE.2015.9
  20. Le, V., Gulwani, S., Su, Z.: Smartsynth: synthesizing smartphone automation scripts from natural language. In: MobSys’13, vol. 2, p. 5 (2013)Google Scholar
  21. Lieberman, H., Paternò, F., Wulf, V. (eds.): End-User Development, Human-Computer Interaction Series, vol. 9. Springer, The Netherlands (2006)Google Scholar
  22. Liu, H., Lieberman, H.: Toward a programmatic semantics of natural language. In: Visual Languages and Human Centric Computing, 2004 IEEE Symposium on, pp. 281–282 (2004). doi: 10.1109/VLHCC.2004.59
  23. Liu, H., Lieberman, H.: Programmatic semantics for natural language interfaces. In: CHI ’05 extended abstracts on human factors in computing systems, CHI ’05, pp. 1597–1600. ACM, Portland, OR, USA (2005). doi: 10.1145/1056808.1056975
  24. Lopez, V., Fernández, M., Motta, E., Stieler, N.: PowerAqua: supporting users in querying and exploring the semantic web. Semant. Web 3(3), 249–265 (2012). doi: 10.3233/SW-2011-0030 Google Scholar
  25. Lopez, V., Motta, E., Uren, V.: PowerAqua: fishing the semantic web. In: Sure, Y., Domingue, J. (eds.) The Semantic Web: Research and Applications, No. 4011 in Lecture Notes in Computer Science, pp. 393–410. Springer, Berlin (2006). doi: 10.1007/11762256_30 Google Scholar
  26. Manning, C.D., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S.J., McClosky, D.: The Stanford CoreNLP natural language processing toolkit. In: Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 55–60. Association for Computational Linguistics, Baltimore, Maryland (2014)Google Scholar
  27. Manshadi, M.H., Gildea, D., Allen, J.F.: Integrating programming by example and natural language programming (2013).
  28. Martinez-Barco, P., Métais, E., Llopis, F., Moreda, P.: Editorial: an overview of the applications of natural language to information systems. Data Knowl. Eng. 88, 109–112 (2013). doi: 10.1016/j.datak.2013.08.001.
  29. Mihalcea, R., Liu, H., Lieberman, H.: NLP (natural language processing) for NLP (natural language programming). In: Gelbukh, A. (ed.) Proceedings of the 7th International Conference, CICLing 2006, Mexico City, Mexico, February 19–25, 2006, Lecture Notes in Computer Science, vol. 3878, pp. 319–330. Springer, Berlin (2006). doi: 10.1007/11671299_34
  30. Miller, J.E.: A Critical Introduction to Syntax. Continuum Critical Introductions to Linguistics. Bloomsbury (2011).
  31. Ortiz, C.L.: The road to natural conversational speech interfaces. Internet Comput. IEEE 18(2), 74–78 (2014). doi: 10.1109/MIC.2014.36 CrossRefGoogle Scholar
  32. Pane, J.F., Myers, B.A.: Usability issues in the design of novice programming systems. Technical Report CMU-CS-96-132, School of Computer Science, Carnegie Mellon University, Pittsburgh, Pennsylvania (1996).
  33. Pane, J.F., Myers, B.A., Miller, L.: Using HCI techniques to design a more usable programming system. In: IEEE 2002 Symposia on Human Centric Computing Languages and Environments, 2002. Proceedings, pp. 198–206 (2002). doi: 10.1109/HCC.2002.1046372
  34. Pane, J.F., Ratanamahatana, C., Myers, B.A.: Studying the language and structure in non-programmers’ solutions to programming problems. Int. J. Hum. Comput. Stud. 54(2), 237–264 (2001). doi: 10.1006/ijhc.2000.0410.
  35. Price, D., Rilofff, E., Zachary, J., Harvey, B.: NaturalJava: a natural language interface for programming in Java. In: Proceedings of the 5th International Conference on Intelligent User Interfaces, IUI ’00, pp. 207–211. ACM, New Orleans, Louisiana, USA (2000). doi: 10.1145/325737.325845
  36. Sammet, J.E.: The use of English as a programming language. Commun. ACM 9(3), 228–230 (1966). doi: 10.1145/365230.365274 MathSciNetCrossRefGoogle Scholar
  37. Thummalapenta, S., Devaki, P., Sinha, S., Chandra, S., Gnanasundaram, S., Nagaraj, D., Kumar, S., Kumar, S.: Efficient and change-resilient test automation: an industrial case study. In: 2013 35th International Conference on Software Engineering (ICSE), pp. 1002–1011 (2013). doi: 10.1109/ICSE.2013.6606650
  38. Thummalapenta, S., Sinha, S., Singhania, N., Chandra, S.: Automating test automation. In: 2012 34th International Conference on Software Engineering (ICSE), pp. 881–891 (2012). doi: 10.1109/ICSE.2012.6227131
  39. Unger, C., Cimiano, P.: Pythia: Compositional meaning construction for ontology-based question answering on the semantic web. In: Natural Language Processing and Information Systems—16th International Conference on Applications of Natural Language to Information Systems, NLDB 2011, Alicante, Spain, June 28–30 (2011) Proceedings, pp. 153–160 (2011). doi: 10.1007/978-3-642-22327-3_15
  40. Vadas, D., Curran, J.R.: Programming with unrestricted natural language. In: Proceedings of the Australasian Language Technology Workshop, pp. 191–199. Sydney, Australia (2005)Google Scholar
  41. Waltz, D.: Natural language access to a large data base: an engineering approach. In: Proceedings of the 4th International Joint Conference on Artificial Intelligence, IJCAI’75, vol. 1, pp. 868–872. Morgan Kaufmann Publishers Inc., Tblisi, USSR (1975).
  42. Wang, C., Xiong, M., Zhou, Q., Yu, Y.: PANTO: a portable natural language interface to ontologies. In: Franconi, E., Kifer, M., May, W. (eds.) The Semantic Web: Research and Applications, Lecture Notes in Computer Science, vol. 4519, pp. 473–487. Springer, Berlin (2007). doi: 10.1007/978-3-540-72667-8_34 CrossRefGoogle Scholar
  43. Yang, H., Cui, Z., O’Brien, P.: Extracting ontologies from legacy systems for understanding and re-engineering. In: Computer Software and Applications Conference, 1999. COMPSAC ’99. Proceedings. The Twenty-Third Annual International, pp. 21–26 (1999). doi: 10.1109/CMPSAC.1999.812512
  44. Zhang, Y., Witte, R., Rilling, J., Haarslev, V.: An ontology-based approach for traceability recovery. In: 3rd International Workshop on Metamodels, Schemas, Grammars, and Ontologies for Reverse Engineering (ATEM 2006), pp. 36–43 (2006)Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Mathias Landhäußer
    • 1
    Email author
  • Sebastian Weigelt
    • 1
  • Walter F. Tichy
    • 1
  1. 1.Karlsruhe Institute of Technology, Institute for Program Structures and Data OrganizationKarlsruheGermany

Personalised recommendations