Skip to main content
SpringerLink
Log in

NLCI: a natural language command interpreter

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

Natural language interfaces are becoming more and more common, because they are powerful and easy to use. Examples of such interfaces are voice controlled navigation devices, Apple’s personal assistant Siri, Google Voice Search, and translation services. However, such interfaces are extremely difficult to build, to maintain, and to port to new domains. We present an approach for building and porting such interfaces quickly. NLCI is a natural language command interpreter that accepts action commands in English and translates them into executable code. The core component is an ontology that models an API. Once the API is “ontologized”, NLCI translates input sentences into sequences of API calls that implement the intended actions. Two radically different APIs were ontologized: openHAB for home automation and Alice for building 3D animations. Construction of the ontology can be automated if the API uses descriptive names for its components. In that case, the language interface can be generated completely automatically. Recall and precision of NLCI on a benchmark of 50 input scripts are 67 and 78 %, resp. Though not yet acceptable for practical use, the results indicate that the approach is feasible. NLCI accepts typed input only. Future work will use a speech front-end to test spoken input.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. Available on GitHub: https://javaparser.github.io/javaparser/, visited 05/17/2016.

  2. https://svn.ipd.kit.edu/trac/AliceNLP/wiki/Papers/LWT2016, visited 05/18/2016.

  3. http://www.openhab.org/features/introduction.html, visited 05/18/2016.

References

  • Androutsopoulos, I., Ritchie, G., Thanisch, P.: Natural language interfaces to databases—an introduction. Nat. Lang. Eng. 1(01), 29–81 (1995). doi:10.1017/S135132490000005X. http://journals.cambridge.org/article_S135132490000005X

  • Ballard, B.W., Biermann, A.W.: Programming in natural language: NLC as a prototype. In: Proceedings of the 1979 Annual Conference, ACM ’79, pp. 228–237. ACM, New York, NY, USA (1979). doi:10.1145/800177.810072

  • Bellegarda, J.R.: Spoken language understanding for natural interaction: the Siri experience. In: Mariani, J., Rosset, S., Garnier-Rizet, M., Devillers, L. (eds.) Natural Interaction with Robots, Knowbots and Smartphones, pp. 3–14. Springer, New York (2014)

    Chapter  Google Scholar 

  • Biermann, A.W., Ballard, B.W., Sigmon, A.H.: An experimental study of natural language programming. Int. J. Man Mach. Stud. 18(1), 71–87 (1983). doi:10.1016/S0020-7373(83)80005-4. http://www.sciencedirect.com/science/article/pii/S0020737383800054

  • Chomsky, N.: The Minimalist Program. MIT Press, Cambridge (1995)

    MATH  Google Scholar 

  • Conway, M.J.: Alice: Easy-to-learn 3D scripting for novices. PhD Thesis, Faculty of the School of Engineering and Applied Science, University of Virginia (1997). http://alice.org/publications/ConwayDissertation.PDF

  • Damljanovic, D., Agatonovic, M., Cunningham, H.: Natural language interfaces to ontologies: combining syntactic analysis and ontology-based lookup through the user interaction. In: Aroyo, L., Antoniou, G., Hyvönen, E., ten Teije, A., Stuckenschmidt, H., Cabral, L., Tudorache, T. (eds.) The Semantic Web: Research and Applications, Lecture Notes in Computer Science, vol. 6088, pp. 106–120. Springer, Berlim (2010). doi:10.1007/978-3-642-13486-9_8

    Chapter  Google Scholar 

  • de Marneffe, M.C., Manning, C.D.: The Stanford typed dependencies representation. In: Coling 2008: Proceedings of the Workshop on Cross-Rramework and Cross-Domain Parser Evaluation, CrossParser ’08, pp. 1–8. Association for Computational Linguistics, Stroudsburg, PA, USA (2008). http://dl.acm.org/citation.cfm?id=1608858.1608859

  • de Marneffe, M.C., Manning, C.D.: Stanford typed dependencies manual. Tech. Rep. (2015). http://nlp.stanford.edu/software/dependencies_manual.pdf

  • Ferrucci, D., Brown, E., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A.A., Lally, A., Murdock, J.W., Nyberg, E., Prager, J., Schlaefer, N., Welty, C.: Building Watson: an overview of the DeepQA project. AI Mag. 31(3), 59–79 (2010). doi:10.1609/aimag.v31i3.2303. http://www.aaai.org/ojs/index.php/aimagazine/article/view/2303

  • Goodman, D.: Danny Goodman’s AppleScript Handbook, 2nd edn. iUniverse.com, Incorporated (1998)

  • Gruber, T.R.: A translation approach to portable ontology specifications. Knowl. Acquis. 5(2), 199–220 (1993). doi:10.1006/knac.1993.1008. http://www.sciencedirect.com/science/article/pii/S1042814383710083

  • Guzzoni, D., Baur, C., Cheyer, A.: Modeling human-agent interaction with active ontologies. In: Interaction Challenges for Intelligent Assistants, Papers from the 2007 AAAI Spring Symposium, Technical Report SS-07-04, Stanford, California, USA, March 26–28, 2007, pp. 52–59. AAAI (2007). http://www.aaai.org/Library/Symposia/Spring/2007/ss07-04-009.php

  • Knöll, R., Gasiunas, V., Mezini, M.: Naturalistic types. In: Proceedings of the 10th SIGPLAN Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, Onward! 2011, pp. 33–48. ACM, New York, NY, USA (2011). doi:10.1145/2048237.2048243

  • Knöll, R., Mezini, M.: Pegasus: First steps toward a naturalistic programming language. In: Companion to the 21st ACM SIGPLAN Symposium on Object-Oriented Programming Systems, Languages, and Applications, OOPSLA ’06, pp. 542–559. ACM, Portland, Oregon, USA (2006). doi:10.1145/1176617.1176628

  • Knublauch, H., Fergerson, R.W., Noy, N.F., Musen, M.A.: The Protégé OWL Plugin: an open development environment for semantic web applications. In: McIlraith, S.A., Plexousakis, D., van Harmelen, F. (eds.) The semantic Web—ISWC 2004. Lecture Notes in Computer Science, vol. 3298, pp. 229–243. Springer, Berlin

  • Körner, S.J., Brumm, T.: Natural language specification improvement with ontologies. Int. J. Semant. Comput. (IJSC) 03(04), 445–470 (2010). doi:10.1142/S1793351X09000872. http://www.worldscinet.com/ijsc/

  • Landhäußer, M., Hey, T., Tichy, W.F.: Deriving timelines from texts. In: Proceedings of the 3rd International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering, pp. 45–51 (2014). doi:10.1145/2593801.2593809

  • Landhäußer, M., Hug, R.: Text understanding for programming in natural language: control structures. In: Proceedings of the 4th International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (2015). doi:10.1109/RAISE.2015.9

  • Le, V., Gulwani, S., Su, Z.: Smartsynth: synthesizing smartphone automation scripts from natural language. In: MobSys’13, vol. 2, p. 5 (2013)

  • Lieberman, H., Paternò, F., Wulf, V. (eds.): End-User Development, Human-Computer Interaction Series, vol. 9. Springer, The Netherlands (2006)

    Google Scholar 

  • Liu, H., Lieberman, H.: Toward a programmatic semantics of natural language. In: Visual Languages and Human Centric Computing, 2004 IEEE Symposium on, pp. 281–282 (2004). doi:10.1109/VLHCC.2004.59

  • Liu, H., Lieberman, H.: Programmatic semantics for natural language interfaces. In: CHI ’05 extended abstracts on human factors in computing systems, CHI ’05, pp. 1597–1600. ACM, Portland, OR, USA (2005). doi:10.1145/1056808.1056975

  • Lopez, V., Fernández, M., Motta, E., Stieler, N.: PowerAqua: supporting users in querying and exploring the semantic web. Semant. Web 3(3), 249–265 (2012). doi:10.3233/SW-2011-0030

    Google Scholar 

  • Lopez, V., Motta, E., Uren, V.: PowerAqua: fishing the semantic web. In: Sure, Y., Domingue, J. (eds.) The Semantic Web: Research and Applications, No. 4011 in Lecture Notes in Computer Science, pp. 393–410. Springer, Berlin (2006). doi:10.1007/11762256_30

    Google Scholar 

  • Manning, C.D., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S.J., McClosky, D.: The Stanford CoreNLP natural language processing toolkit. In: Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 55–60. Association for Computational Linguistics, Baltimore, Maryland (2014)

  • Manshadi, M.H., Gildea, D., Allen, J.F.: Integrating programming by example and natural language programming (2013). http://www.aaai.org/ocs/index.php/AAAI/AAAI13/paper/viewFile/6477/7230

  • Martinez-Barco, P., Métais, E., Llopis, F., Moreda, P.: Editorial: an overview of the applications of natural language to information systems. Data Knowl. Eng. 88, 109–112 (2013). doi:10.1016/j.datak.2013.08.001. http://dx.doi.org/10.1016/j.datak.2013.08.001

  • Mihalcea, R., Liu, H., Lieberman, H.: NLP (natural language processing) for NLP (natural language programming). In: Gelbukh, A. (ed.) Proceedings of the 7th International Conference, CICLing 2006, Mexico City, Mexico, February 19–25, 2006, Lecture Notes in Computer Science, vol. 3878, pp. 319–330. Springer, Berlin (2006). doi:10.1007/11671299_34

  • Miller, J.E.: A Critical Introduction to Syntax. Continuum Critical Introductions to Linguistics. Bloomsbury (2011). http://books.google.com/books?id=a0UQHXUzR9AC

  • Ortiz, C.L.: The road to natural conversational speech interfaces. Internet Comput. IEEE 18(2), 74–78 (2014). doi:10.1109/MIC.2014.36

    Article  Google Scholar 

  • Pane, J.F., Myers, B.A.: Usability issues in the design of novice programming systems. Technical Report CMU-CS-96-132, School of Computer Science, Carnegie Mellon University, Pittsburgh, Pennsylvania (1996). http://www.cs.cmu.edu/~pane/ftp/CMU-CS-96-132.pdf

  • Pane, J.F., Myers, B.A., Miller, L.: Using HCI techniques to design a more usable programming system. In: IEEE 2002 Symposia on Human Centric Computing Languages and Environments, 2002. Proceedings, pp. 198–206 (2002). doi:10.1109/HCC.2002.1046372

  • Pane, J.F., Ratanamahatana, C., Myers, B.A.: Studying the language and structure in non-programmers’ solutions to programming problems. Int. J. Hum. Comput. Stud. 54(2), 237–264 (2001). doi:10.1006/ijhc.2000.0410. http://www.sciencedirect.com/science/article/B6WGR-458NDXY-M/2/e785863fea0552236a5587e1fbe7a57f

  • Price, D., Rilofff, E., Zachary, J., Harvey, B.: NaturalJava: a natural language interface for programming in Java. In: Proceedings of the 5th International Conference on Intelligent User Interfaces, IUI ’00, pp. 207–211. ACM, New Orleans, Louisiana, USA (2000). doi:10.1145/325737.325845

  • Sammet, J.E.: The use of English as a programming language. Commun. ACM 9(3), 228–230 (1966). doi:10.1145/365230.365274

    Article  MathSciNet  Google Scholar 

  • Thummalapenta, S., Devaki, P., Sinha, S., Chandra, S., Gnanasundaram, S., Nagaraj, D., Kumar, S., Kumar, S.: Efficient and change-resilient test automation: an industrial case study. In: 2013 35th International Conference on Software Engineering (ICSE), pp. 1002–1011 (2013). doi:10.1109/ICSE.2013.6606650

  • Thummalapenta, S., Sinha, S., Singhania, N., Chandra, S.: Automating test automation. In: 2012 34th International Conference on Software Engineering (ICSE), pp. 881–891 (2012). doi:10.1109/ICSE.2012.6227131

  • Unger, C., Cimiano, P.: Pythia: Compositional meaning construction for ontology-based question answering on the semantic web. In: Natural Language Processing and Information Systems—16th International Conference on Applications of Natural Language to Information Systems, NLDB 2011, Alicante, Spain, June 28–30 (2011) Proceedings, pp. 153–160 (2011). doi:10.1007/978-3-642-22327-3_15

  • Vadas, D., Curran, J.R.: Programming with unrestricted natural language. In: Proceedings of the Australasian Language Technology Workshop, pp. 191–199. Sydney, Australia (2005)

  • Waltz, D.: Natural language access to a large data base: an engineering approach. In: Proceedings of the 4th International Joint Conference on Artificial Intelligence, IJCAI’75, vol. 1, pp. 868–872. Morgan Kaufmann Publishers Inc., Tblisi, USSR (1975). http://dl.acm.org/citation.cfm?id=1624626.1624762

  • Wang, C., Xiong, M., Zhou, Q., Yu, Y.: PANTO: a portable natural language interface to ontologies. In: Franconi, E., Kifer, M., May, W. (eds.) The Semantic Web: Research and Applications, Lecture Notes in Computer Science, vol. 4519, pp. 473–487. Springer, Berlin (2007). doi:10.1007/978-3-540-72667-8_34

    Chapter  Google Scholar 

  • Yang, H., Cui, Z., O’Brien, P.: Extracting ontologies from legacy systems for understanding and re-engineering. In: Computer Software and Applications Conference, 1999. COMPSAC ’99. Proceedings. The Twenty-Third Annual International, pp. 21–26 (1999). doi:10.1109/CMPSAC.1999.812512

  • Zhang, Y., Witte, R., Rilling, J., Haarslev, V.: An ontology-based approach for traceability recovery. In: 3rd International Workshop on Metamodels, Schemas, Grammars, and Ontologies for Reverse Engineering (ATEM 2006), pp. 36–43 (2006)

Download references

Author information

Authors and Affiliations

  1. Karlsruhe Institute of Technology, Institute for Program Structures and Data Organization, Karlsruhe, Germany

    Mathias Landhäußer, Sebastian Weigelt & Walter F. Tichy

Authors
  1. Mathias Landhäußer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sebastian Weigelt
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Walter F. Tichy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mathias Landhäußer.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Landhäußer, M., Weigelt, S. & Tichy, W.F. NLCI: a natural language command interpreter. Autom Softw Eng 24, 839–861 (2017). https://doi.org/10.1007/s10515-016-0202-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10515-016-0202-1

Keywords

Search

Navigation