Programming IoT Devices by Demonstration Using Mobile Apps

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10303)

Abstract

The revolutionary advances of Internet of Things (IoT) devices and applications have helped IoT emerge as an increasingly important domain for end-user development (EUD). Past research has shown that end users desire to create various customized automations, which would often utilize multiple IoT devices. Many solutions exist to support EUD across multiple IoT devices, but they are limited to devices from the same manufacturer, within the same “eco-system” or supporting a common API. We present Epidosite, a mobile programming-by-demonstration system that addresses this limitation by leveraging the smartphone as a hub for IoT automation. It enables the creation of automations for most consumer IoT devices on smartphones by demonstrating the desired behaviors through directly manipulating the corresponding smartphone app for each IoT device. Epidosite also supports using the smartphone app usage context and external web services as triggers and data for automations, enabling the creation of highly context-aware IoT applications.

Keywords

Internet of Things Programming by demonstration End user development 

Notes

Acknowledgement

This work was supported in part by the Yahoo! InMind project.

References

  1. 1.
    Argall, B.D., et al.: A survey of robot learning from demonstration. Robot. Auton. Syst. 57(5), 469–483 (2009)CrossRefGoogle Scholar
  2. 2.
    Barricelli, B.R., Valtolina, S.: A visual language and interactive system for end-user development of internet of things ecosystems. J. Vis. Lang. Comput. (2017, in press)Google Scholar
  3. 3.
    Barricelli, B.R., Valtolina, S.: Designing for end-user development in the Internet of Things. In: Díaz, P., Pipek, V., Ardito, C., Jensen, C., Aedo, I., Boden, A. (eds.) IS-EUD 2015. LNCS, vol. 9083, pp. 9–24. Springer, Cham (2015). doi:10.1007/978-3-319-18425-8_2 CrossRefGoogle Scholar
  4. 4.
    Billard, A., et al.: Robot programming by demonstration. In: Siciliano, B., Khatib, O. (eds.) Springer Handbook of Robotics, pp. 1371–1394. Springer, Heidelberg (2008)Google Scholar
  5. 5.
    Blackstock, M., Lea, R.: IoT interoperability: a hub-based approach. In: 2014 International Conference on the Internet of Things (IOT), pp. 79–84. IEEE (2014)Google Scholar
  6. 6.
    Böhmer, M., et al.: What’s in the apps for context? Extending a sensor for studying app usage to informing context-awareness. In: Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication, pp. 1423–1426. ACM, New York (2013)Google Scholar
  7. 7.
    Coutaz, J., Crowley, J.L.: A first-person experience with end-user development for smart homes. IEEE Pervasive Comput. 15(2), 26–39 (2016)CrossRefGoogle Scholar
  8. 8.
    Cypher, A., Halbert, D.C.: Watch What I Do: Programming by Demonstration. MIT Press, Cambridge (1993)Google Scholar
  9. 9.
    Danado, J., Paternò, F.: Puzzle: a visual-based environment for end user development in touch-based mobile phones. In: Winckler, M., Forbrig, P., Bernhaupt, R. (eds.) HCSE 2012. LNCS, vol. 7623, pp. 199–216. Springer, Heidelberg (2012). doi:10.1007/978-3-642-34347-6_12 CrossRefGoogle Scholar
  10. 10.
    De Russis, L., Corno, F.: HomeRules: a tangible end-user programming interface for smart homes. In: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, pp. 2109–2114. ACM, New York (2015)Google Scholar
  11. 11.
    Demeure, A., Caffiau, S., Elias, E., Roux, C.: Building and using home automation systems: a field study. In: Díaz, P., Pipek, V., Ardito, C., Jensen, C., Aedo, I., Boden, A. (eds.) IS-EUD 2015. LNCS, vol. 9083, pp. 125–140. Springer, Cham (2015). doi:10.1007/978-3-319-18425-8_9 CrossRefGoogle Scholar
  12. 12.
    Dey, A.K., et al.: A CAPpella: programming by demonstration of context-aware applications. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 33–40. ACM (2004)Google Scholar
  13. 13.
    Dey, A.K., et al.: iCAP: interactive prototyping of context-aware applications. In: Fishkin, K.P., et al. (eds.) Pervasive Computing, pp. 254–271. Springer, Berlin Heidelberg (2006)CrossRefGoogle Scholar
  14. 14.
    Edwards, W.K., Grinter, R.E.: At home with ubiquitous computing: seven challenges. In: Abowd, Gregory D., Brumitt, B., Shafer, S. (eds.) UbiComp 2001. LNCS, vol. 2201, pp. 256–272. Springer, Heidelberg (2001). doi:10.1007/3-540-45427-6_22 CrossRefGoogle Scholar
  15. 15.
    Fox, A., et al.: Integrating information appliances into an interactive workspace. IEEE Comput. Graph. Appl. 20(3), 54–65 (2000)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Gama, K., et al.: Combining heterogeneous service technologies for building an Internet of Things middleware. Comput. Commun. 35(4), 405–417 (2012)CrossRefGoogle Scholar
  17. 17.
    González García, C., et al.: Midgar: generation of heterogeneous objects interconnecting applications. a domain specific language proposal for Internet of Things scenarios. Comput. Netw. 64, 143–158 (2014)Google Scholar
  18. 18.
    Grabler, F., et al.: Generating photo manipulation tutorials by demonstration. In: ACM SIGGRAPH 2009 Papers. pp. 66:1–66:9. ACM, New York (2009)Google Scholar
  19. 19.
    Greenberg, S.: Context as a dynamic construct. Hum.-Comput. Interact. 16(2), 257–268 (2001)CrossRefGoogle Scholar
  20. 20.
    Guinard, D., et al.: Towards physical mashups in the web of things. In: 2009 Sixth International Conference on Networked Sensing Systems (INSS), pp. 1–4. IEEE (2009)Google Scholar
  21. 21.
    Guinard, D., Trifa, V.: Towards the web of things: web mashups for embedded devices. In: Workshop on Mashups, Enterprise Mashups and Lightweight Composition on the Web (MEM 2009), in Proceedings of WWW (International World Wide Web Conferences), Madrid, Spain (2009)Google Scholar
  22. 22.
    Humble, J., Crabtree, A., Hemmings, T., Åkesson, K.-P., Koleva, B., Rodden, T., Hansson, P.: “Playing with the bits” user-configuration of ubiquitous domestic environments. In: Dey, Anind K., Schmidt, A., McCarthy, Joseph F. (eds.) UbiComp 2003. LNCS, vol. 2864, pp. 256–263. Springer, Heidelberg (2003). doi:10.1007/978-3-540-39653-6_20 CrossRefGoogle Scholar
  23. 23.
    Jesdabodi, C., Maalej, W.: Understanding usage states on mobile devices. In: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 1221–1225. ACM, New York (2015)Google Scholar
  24. 24.
    Katasonov, A., et al.: Smart semantic middleware for the Internet of Things. ICINCO-ICSO 8, 169–178 (2008)Google Scholar
  25. 25.
    Kubitza, T., Schmidt, A.: Towards a toolkit for the rapid creation of smart environments. In: Díaz, P., Pipek, V., Ardito, C., Jensen, C., Aedo, I., Boden, A. (eds.) IS-EUD 2015. LNCS, vol. 9083, pp. 230–235. Springer, Cham (2015). doi:10.1007/978-3-319-18425-8_21 CrossRefGoogle Scholar
  26. 26.
    Lee, T.Y., et al.: Towards understanding human mistakes of programming by example: an online user study. In: Proceedings of the 22nd International Conference on Intelligent User Interfaces, pp. 257–261. ACM, New York (2017)Google Scholar
  27. 27.
    Leshed, G., et al.: CoScripter: automating and sharing how-to knowledge in the enterprise. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1719–1728. ACM, New York (2008)Google Scholar
  28. 28.
    Li, T.J.-J., et al.: Designing a conversational interface for a multimodal smartphone programming-by-demonstration agent. In: Conversational UX Design CHI 2017 Workshop, Denver, CO (2017, in press)Google Scholar
  29. 29.
    Li, T.J.-J., et al.: SUGILITE: creating multimodal smartphone automation by demonstration. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, Denver (2017, in press)Google Scholar
  30. 30.
    Lieberman, H.: Your Wish is My Command: Programming by Example. Morgan Kaufmann, San Francisco (2001)Google Scholar
  31. 31.
    McDaniel, R.G., Myers, B.A.: Gamut: demonstrating whole applications. In: Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology, pp. 81–82 ACM, New York (1997)Google Scholar
  32. 32.
    McDaniel, R.G., Myers, B.A.: Getting more out of programming-by-demonstration. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 442–449 ACM, New York (1999)Google Scholar
  33. 33.
    Mennicken, S., et al.: From today’s augmented houses to tomorrow’s smart homes: new directions for home automation research. In: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 105–115. ACM, New York (2014)Google Scholar
  34. 34.
    Modugno, F., Myers, B.A.: Pursuit: graphically representing programs in a demonstrational visual shell. In: Conference Companion on Human Factors in Computing Systems, pp. 455–456. ACM, New York (1994)Google Scholar
  35. 35.
    Myers, B.A.: Demonstrational interfaces: a step beyond direct manipulation. Computer 25(8), 61–73 (1992)CrossRefGoogle Scholar
  36. 36.
    Myers, B.A.: Visual programming, programming by example, and program visualization: a taxonomy. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 59–66. ACM, New York (1986)Google Scholar
  37. 37.
    Pintus, A., et al.: The anatomy of a large scale social web for internet enabled objects. In: Proceedings of the Second International Workshop on Web of Things. pp. 6:1–6:6. ACM, New York (2011)Google Scholar
  38. 38.
    Raffle, H., et al.: Beyond record and play: backpacks: tangible modulators for kinetic behavior. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 681–690. ACM, New York (2006)Google Scholar
  39. 39.
    Ricquebourg, V., et al.: The smart home concept: our immediate future. In: 2006 1st IEEE International Conference on E-Learning in Industrial Electronics, pp. 23–28 (2006)Google Scholar
  40. 40.
    Schmidt, A.: Programming ubiquitous computing environments. In: Díaz, P., Pipek, V., Ardito, C., Jensen, C., Aedo, I., Boden, A. (eds.) IS-EUD 2015. LNCS, vol. 9083, pp. 3–6. Springer, Cham (2015). doi:10.1007/978-3-319-18425-8_1 CrossRefGoogle Scholar
  41. 41.
    Song, Z., et al.: Semantic middleware for the Internet of Things. In: 2010 Internet of Things (IOT), pp. 1–8 (2010)Google Scholar
  42. 42.
    Ur, B., et al.: Practical trigger-action programming in the smart home. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 803–812. ACM, New York (2014)Google Scholar
  43. 43.
    Wiese, J.S.: Evolving the Ecosystem of Personal Behavioral Data (2015)Google Scholar
  44. 44.
    Zhang, T., Brügge, B.: Empowering the user to build smart home applications. In: Proceedings of 2nd International Conference on Smart Homes and Health Telematics (ICOST2004), Singapore (2004)Google Scholar
  45. 45.
    Zhao, S., et al.: Discovering different kinds of smartphone users through their application usage behaviors. In: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 498–509. ACM, New York (2016)Google Scholar
  46. 46.
    Zhong, Y., et al.: Smart home on smart phone. In: Proceedings of the 13th International Conference on Ubiquitous Computing, pp. 467–468. ACM, New York (2011)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Human-Computer Interaction InstituteCarnegie Mellon UniversityPittsburghUSA
  2. 2.School of Electronics Engineering and Computer SciencePeking UniversityBeijingChina

Personalised recommendations