Natural Notation for the Domestic Internet of Things

  • Charith Perera
  • Saeed Aghaee
  • Alan Blackwell
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9083)


This study explores the use of natural language to give instructions that might be interpreted by Internet of Things (IoT) devices in a domestic ‘smart home’ environment. We start from the proposition that reminders can be considered as a type of end-user programming, in which the executed actions might be performed either by an automated agent or by the author of the reminder. We conducted an experiment in which people wrote sticky notes specifying future actions in their home. In different conditions, these notes were addressed to themselves, to others, or to a computer agent. We analyse the linguistic features and strategies that are used to achieve these tasks, including the use of graphical resources as an informal visual language. The findings provide a basis for design guidance related to end-user development for the Internet of Things.


Internet of things Smart home End-User programming User study 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Blackwell, A.F., Rode, J.A., Toye, E.F.: How do we program the home? gender, attention investment, and the psychology of programming at home. International Journal of Human-Computer Studies 67, 324–341 (2009)CrossRefGoogle Scholar
  2. 2.
    Dey, A.K., Sohn, T., Streng, S., Kodama, J.: icap: interactive prototyping of context-aware applications. In: Fishkin, K.P., Schiele, B., Nixon, P., Quigley, A. (eds.) PERVASIVE 2006. LNCS, vol. 3968, pp. 254–271. Springer, Heidelberg (2006) CrossRefGoogle Scholar
  3. 3.
    Ur, B., McManus, E., Pak Yong Ho, M., Littman, M.L.: Practical trigger-action programming in the smart home. In: Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems, pp. 803–812 (2014)Google Scholar
  4. 4.
    Myers, B.A., Pane, J.F., Ko, A.: Natural programming languages and environments. Communications of the ACM 47, 47–52 (2004)CrossRefGoogle Scholar
  5. 5.
    Tolmie, P., Pycock, J., Diggins, T., MacLean, A., Karsenty, A.: Unremarkable computing. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 399–406 (2002)Google Scholar
  6. 6.
    Rode, J.A., Toye, E.F., Blackwell, A.F.: The fuzzy felt ethnographyunderstanding the programming patterns of domestic appliances. Personal and Ubiquitous Computing 8, 161–176 (2004)CrossRefGoogle Scholar
  7. 7.
    Graham, C., Rouncefield, M., Gibbs, M., Vetere, F., Cheverst, K.: How probes work. In: Proceedings of the 19th Australasian Conference on Computer-Human Interaction: Entertaining User Interfaces, pp. 29–37 (2007)Google Scholar
  8. 8.
    Hutchinson, H., Mackay, W., Westerlund, B., Bederson, B.B., Druin, A., Plaisant, C., Beaudouin-Lafon, M., Conversy, S., Evans, H., Hansen, H., et al.: Technology probes: inspiring design for and with families. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 17–24 (2003)Google Scholar
  9. 9.
    Taylor, A.S., Swan, L., Eardley, R., Sellen, A., Hodges, S., Wood, K.: Augmenting refrigerator magnets: why less is sometimes more. In: Proceedings of the 4th Nordic Conference on Human-Computer Interaction: Changing Roles, pp. 115–124 (2006)Google Scholar
  10. 10.
    Tarkan, S., Sazawal, V., Druin, A., Golub, E., Bonsignore, E.M., Walsh, G., Atrash, Z.: Toque: designing a cooking-based programming language for and with children. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2417–2426 (2010)Google Scholar
  11. 11.
    Mistry, P., Maes, P.: Augmenting sticky notes as an i/o interface. In: Universal Access in Human-Computer Interaction. Intelligent and Ubiquitous Interaction Environments. Springer, pp. 547–556 (2009)Google Scholar
  12. 12.
    Petrick, S.R.: On natural language based computer systems. IBM Journal of Research and Development 20, 314–325 (1976)CrossRefzbMATHGoogle Scholar
  13. 13.
    Dijkstra, E.W.: On the foolishness of “natural language programming”. In: Bauer, F.L., Broy, M., Dijkstra, E.W., Gerhart, S.L., Gries, D., Griffiths, M., Guttag, J.V., Horning, J.J., Owicki, S.S., Pair, C., Partsch, H., Pepper, P., Wirsing, M., Wössner, H. (eds.) Program Construction, vol. 69, pp. 51–53. Springer, Heidelberg (1979)CrossRefGoogle Scholar
  14. 14.
    Blackwell, A., Green, T.: Notational systems-the cognitive dimensions of notations framework. In: Carroll, J.M. (ed.) HCI Models, Theories and Frameworks: Toward a Multidisciplinary Science, pp. 103–134. Morgan Kaufmann, San Francisco (2003)CrossRefGoogle Scholar
  15. 15.
    Stolcke, A.: Srilm - an extensible language modeling toolkit, pp. 901–904 (2002)Google Scholar
  16. 16.
    Speer, R., Havasi, C.: Representing general relational knowledge in conceptnet 5. In: LREC, pp. 3679–3686 (2012)Google Scholar
  17. 17.
    Searle, J.R.: A classification of illocutionary acts. Language in society 5, 1–23 (1976)CrossRefzbMATHGoogle Scholar
  18. 18.
    Bertin, J.: Semiology of graphics: diagrams, networks, maps (1983)Google Scholar
  19. 19.
    Engelhardt, Y.: The language of graphics: A framework for the analysis of syntax and meaning in maps, charts and diagrams. Unpublished Ph.D. thesis, Institute for Logic, Language and Computation, University of Amsterdam, The Netherlands (2002)Google Scholar
  20. 20.
    Blackwell, A.: Visual representation. Soegaard, M., Dam, R.F. (eds.) The Encyclopedia of Human-Computer Interaction, 2nd Ed. Aarhus, Denmark: The Interaction Design Foundation (2013).
  21. 21.
    Jespersen, O.: The philosophy of grammar. University of Chicago Press (1992)Google Scholar
  22. 22.
    Aghaee, S., Nowak, M., Pautasso, C.: Reusable decision space for mashup tool design. In: Proceedings of the 4th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, pp. 211–220 (2012)Google Scholar
  23. 23.
    Lucci, G., Paternò, F.: Understanding end-user development of context-dependent applications in smartphones. In: Sauer, S., Bogdan, C., Forbrig, P., Bernhaupt, R., Winckler, M. (eds.) HCSE 2014. LNCS, vol. 8742, pp. 182–198. Springer, Heidelberg (2014) Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.Research School of Computer Science (RSCS)The Australian National UniversityCanberraAustralia
  2. 2.Faculty of Maths, Computing and TechnologyThe Open UniversityMilton KeynesUK
  3. 3.Computer LaboratoryUniversity of CambridgeCambridgeUK

Personalised recommendations