Skip to main content

Understanding Concepts, Methods and Tools for End-User Control of Automations in Ecosystems of Smart Objects and Services

  • Conference paper
  • First Online:
End-User Development (IS-EUD 2023)

Abstract

The continuously increasing number of connected objects and sensors is opening up the possibility of introducing automations in many domains to better support people in their activities. However, such automations to be effective should be under the user control. Unfortunately, people often report difficulties in understanding the surrounding automations and how to modify them. The goal of this paper is to provide a multi-perspective view of what has been done in terms of design, tools, and evaluation in the area of end-user control of automations in ecosystems of smart objects and services. For each aspect we introduce the main challenge, the current possible approaches to address it, and the issues that still need further investigation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.statista.com/statistics/1183457/iot-connected-devices-worldwide/, last accessed 2023/04/12.

  2. 2.

    https://www.upod.io/datasets.html, last accessed 2023/04/12.

  3. 3.

    https://www-users.cse.umn.edu/~fengqian/ifttt_measurement/, last accessed 2023/04/12.

  4. 4.

    https://doi.org/10.5281/zenodo.5572861, last accessed 2023/04/12.

  5. 5.

    https://github.com/andrematt/trigger_action_rules, last accessed 2023/04/12.

References

  1. Andrao, M., Desolda, G., Greco, F., Manfredi, R., Treccani, B., Zancanaro, M.: End-user programming and math teachers: an initial study. In: Proceedings of the 2022 International Conference on Advanced Visual Interfaces, pp. 1–3 (2022)

    Google Scholar 

  2. Andrao, M., Treccani, B., Zancanaro, M.: Therapists as designers: an initial investigation of end-user programming of a tangible tool for therapeutic interventions. In: Proceedings of the 2nd International Workshop on Empowering People in Dealing with Internet of Things Ecosystems co-located with INTERACT 2021, Bari, Italy, Online / Bari, Italy, September 30, 2021(CEUR Workshop Proceedings, vol. 3053). CEUR-WS.org, pp. 38–42 (2021). http://ceur-ws.org/Vol-3053/paper_8.pdf

  3. Ardito, C., Desolda, G., Lanzilotti, R., Malizia, A., Matera, M.: Analysing trade-offs in frameworks for the design of smart environments. Behav. Inf. Technol. 39(1), 47–71 (2020)

    Article  Google Scholar 

  4. Ardito, C., et al.: User-defined semantics for the design of IoT systems enabling smart interactive experiences. Pers. Ubiquit. Comput. 24(6), 781–796 (2020). https://doi.org/10.1007/s00779-020-01457-5

    Article  Google Scholar 

  5. Ariano, R., Manca, M., Paternò, F., Santoro, C.: Smartphone-based augmented reality for end-user creation of home automations. Behav. Inf. Techno. 42(1), 124–140 (2023)

    Article  Google Scholar 

  6. Balducci, F., Buono, P., Desolda, G., Impedovo, D., Piccinno, A.: Improving smart interactive experiences in cultural heritage through pattern recognition techniques. Pattern Recogn. Lett. 131, 142–149 (2020)

    Article  Google Scholar 

  7. Bellucci, A., Vianello, A., Florack, Y., Micallef, L., Jacucci, G.: Augmenting objects at home through programmable sensor tokens: a design journey. Int. J. Hum. Comput Stud. 122, 211–231 (2019)

    Article  Google Scholar 

  8. Brackenbury, W., et al.: How users interpret bugs in trigger-action programming. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI 2019). Association for Computing Machinery, New York, NY, USA, Article Paper 552, p. 12 (2019). https://doi.org/10.1145/3290605.330078

  9. Breve, B., Cimino, G., Deufemia, V.: Towards explainable security for ECA rules. In: Proceedings of the 3rd International Workshop on Empowering End-Users in Dealing with Internet of Things Ecosystems (EMPATHY), CEUR-WS, vol. 3172, pp. 26–30 (2022)

    Google Scholar 

  10. Breve, B., Cimino, G., Deufemia, V.: Identifying security and privacy violation rules in trigger-action IoT platforms with NLP models. IEEE Internet Things J. 10(6), 5607–5622 (2023)

    Article  Google Scholar 

  11. Brich, J., Walch, M., Rietzler, M., Weber, M., Schaub, F.: Exploring end user programming needs in home automation. ACM Trans. Comput. Human Interact. (TOCHI) 24(2), 1–35 (2017)

    Article  Google Scholar 

  12. Cabitza, F., Fogli, D., Lanzilotti, R., Piccinno, A.: Rule-based tools for the configuration of ambient intelligence systems: a comparative user study. Multimed. Tools Appl. 76(4), 5221–5241 (2016). https://doi.org/10.1007/s11042-016-3511-2

    Article  Google Scholar 

  13. Casati, F., Castano, S., Fugini, M., Mirbel, I., Pernici, B.: Using patterns to design rules in workflows. IEEE Trans. Softw. Eng. 26(8), 760–785 (2000)

    Article  Google Scholar 

  14. Cena, F., et al.: Incorporating personality traits in user modeling for EUD. In: 3rd International Workshop on Empowering People in Dealing with Internet of Things Ecosystems, CEUR Workshop Proceedings, vol. 3172, pp. 41–48 (2022)

    Google Scholar 

  15. Chen, X., et al.: Fix the leaking tap: a survey of trigger-action programming (TAP) security issues, detection techniques and solutions. Comput. Secur. 102812 (2022)

    Google Scholar 

  16. Clark, M., Newman, M.W., Dutta, P.: ARticulate: one-shot interactions with intelligent assistants in unfamiliar smart spaces using augmented reality. Proc. ACM Interact. Mob. Wearabl. Ubiquit. Technol. 6(1), 1–24 (2022)

    Article  Google Scholar 

  17. Cobb, C., et al.: How risky are real users’ IFTTT applets? In: Proceedings of the Sixteenth Symposium on Usable Privacy and Security (SOUPS 2020), pp. 505–529 (2020)

    Google Scholar 

  18. Coppers, S., Vanacken, D., Luyten, K.: FortClash: predicting and mediating unintended behavior in home automation. Proc. ACM Human Comput. Interact. 6(EICS), 1–20 (2022). https://doi.org/10.1145/3532204

    Article  Google Scholar 

  19. Coppers, S., Vanacken, D., Luyten, K.: Fortniot: Intelligible predictions to improve user understanding of smart home behavior. Proc. ACM Interact. Mob. Wearabl. Ubiquit. Technol. 4(4), 1–24 (2020)

    Article  Google Scholar 

  20. Corcella, L., Manca, M., Paternò, F.: Personalizing a student home behaviour. In: Barbosa, S., Markopoulos, P., Paternò, F., Stumpf, S., Valtolina, S. (eds.) End-User Development. LNCS, vol. 10303, pp. 18–33. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-58735-6_2

    Chapter  Google Scholar 

  21. Corno, F., De Russis, L., Roffarello, A.M.: A high-level semantic approach to end-user development in the Internet of Things. Int. J. Hum Comput Stud. 125, 41–54 (2019)

    Article  Google Scholar 

  22. Corno, F., De Russis, L., Roffarello, A.M.: Empowering end users in debugging trigger-action rules. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–13 (2019)

    Google Scholar 

  23. Corno, F., De Russis, L., Roffarello, A.M.: RecRules: recommending IF-THEN rules for end-user development. ACM Trans. Intell. Syst. Technol. 10(5), 1–27 (2019). https://doi.org/10.1145/3344211

    Article  Google Scholar 

  24. Corno, F., De Russis, L., Roffarello, A.M.: HeyTAP: bridging the gaps between users’ needs and technology in IF-THEN rules via conversation. In: Proceedings of the International Conference on Advanced Visual Interfaces, pp. 1–9 (2020)

    Google Scholar 

  25. Corno, F., De Russis, L., Roffarello, A.M.: TAPrec: supporting the composition of trigger-action rules through dynamic recommendations. In: Proceedings of the 25th International Conference on Intelligent User Interfaces, pp. 579–588 (2020)

    Google Scholar 

  26. Corno, F., De Russis, L., Roffarello, A.M.: Devices, information, and people: abstracting the internet of things for end-user personalization. In: Fogli, D., Tetteroo, D., Barricelli, B.R., Borsci, S., Markopoulos, P., Papadopoulos, G.A. (eds.) End-User Development. LNCS, vol. 12724, pp. 71–86. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-79840-6_5

    Chapter  Google Scholar 

  27. Cypher, A.: Eager: programming repetitive tasks by example. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 33–39 (1991)

    Google Scholar 

  28. De Russis, L., Roffarello, A.M.: A debugging approach for trigger-action programming. In: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (2018). https://doi.org/10.1145/3170427.3188641

  29. de Vega, M., Rinck, M., Diaz, J.M., León, I.: Figure and ground in temporal sentences: the role of the adverbs when and while. Discourse Process. 43(1), 1–23 (2007). https://doi.org/10.1080/01638530709336891

    Article  Google Scholar 

  30. Desolda, G., Ardito, C., Matera, M.: Empowering end users to customize their smart environments: model, composition paradigms, and domain-specific tools. ACM Trans. Comput. Human Interact. (TOCHI) 24(2), 1–52 (2017)

    Article  Google Scholar 

  31. Desolda, G., Greco, F., Guarnieri, F., Mariz, N., Zancanaro, M.: SENSATION: an authoring tool to support event–state paradigm in end-user development. In: Ardito, C., et al. (eds.) Human-Computer Interaction – INTERACT 2021. LNCS, vol. 12933, pp. 373–382. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-85616-8_22

    Chapter  Google Scholar 

  32. Epifania, F., Cremonesi, P.: User-centered evaluation of recommender systems with comparison between short and long profile. In: 2012 Sixth International Conference on Complex, Intelligent, and Software Intensive Systems, pp. 204–211. IEEE (2012)

    Google Scholar 

  33. Fitzgerald, S., et al.: Debugging: finding, fixing and flailing, a multi-institutional study of novice debuggers. Comput. Sci. Educ. 18(2), 93–116 (2008)

    Article  Google Scholar 

  34. Fitzgerald, S., McCauley, R., Hanks, B., Murphy, L., Simon, B., Zander, C.: Debugging from the student perspective. IEEE Trans. Educ. 53(3), 390–396 (2010). https://doi.org/10.1109/TE.2009.2025266

    Article  Google Scholar 

  35. Fogli, D., Peroni, M., Stefini, C.: ImAtHome: Making trigger-action programming easy and fun. J. Vis. Lang. Comput. 42, 60–75 (2017)

    Article  Google Scholar 

  36. Gallitto, G., Treccani, B., Zancanaro, M.: If when is better than if (and while might help): on the importance of influencing mental models in EUD (a pilot study). In: Proceedings of the 1st International Workshop on Empowering People in Dealing with Internet of Things Ecosystems co-located with International Conference on Advanced Visual Interfaces (AVI), Ischia Island, Italy, 2020, pp. 7–11 (2020)

    Google Scholar 

  37. Gallo, S., Paterno, F.: A conversational agent for creating flexible daily automation. In: Proceedings of the 2022 International Conference on Advanced Visual Interfaces, pp. 1–8 (2022)

    Google Scholar 

  38. Gallo, S., Manca, M., Mattioli, A., Paternò, F., Santoro, C.: Comparative analysis of composition paradigms for personalization rules in IoT settings. In: Fogli, D., Tetteroo, D., Barricelli, B.R., Borsci, S., Markopoulos, P., Papadopoulos, G.A. (eds.) End-User Development. LNCS, vol. 12724, pp. 53–70. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-79840-6_4

    Chapter  Google Scholar 

  39. Gao, L., Bai, X.: A unified perspective on the factors influencing consumer acceptance of internet of things technology. Asia Pac. J. Market. Logist. 26(2), 211–231 (2014). https://doi.org/10.1108/APJML-06-2013-0061

    Article  Google Scholar 

  40. Gennari, R., Matera, M., Morra, D., Melonio, A., Rizvi, M.: Design for social digital well-being with young generations: Engage them and make them reflect. Int. J. Human Comput. Stud. 173, 103006 (2023). https://doi.org/10.1016/j.ijhcs.2023.103006

    Article  Google Scholar 

  41. Ghiani, G., Manca, M., Paternò, F., Santoro, C.: Personalization of context-dependent applications through trigger-action rules. ACM Trans. Computer-Human Interaction (TOCHI) 24(2), 1–33 (2017)

    Article  Google Scholar 

  42. Grigoreanu, V., Burnett, M., Wiedenbeck, S., Cao, J., Rector, K., Kwan, I.: End-user debugging strategies: a sensemaking perspective. ACM Trans. Comput. Human Interact. (TOCHI) 19(1), 1–28 (2012)

    Article  Google Scholar 

  43. Gunawardana, A., Shani, G., Yogev, S.: Evaluating recommender systems. In: Recommender systems handbook, pp. 547–601. Springer US, New York, NY (2012)

    Google Scholar 

  44. Huang, J., Cakmak, M.: Supporting mental model accuracy in trigger-action programming. In: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing - UbiComp 2015, pp. 215–225. ACM Press, Osaka, Japan (2015). https://doi.org/10.1145/2750858.2805830

  45. Jain, M., Kumar, P., Kota, R., Patel, S.N.: Evaluating and informing the design of chatbots. In: Proceedings of the 2018 Designing Interactive Systems Conference, pp. 895–906 (2018)

    Google Scholar 

  46. Knijnenburg, B.P., Willemsen, M.C., Gantner, Z., Soncu, H., Newell, C.: Explaining the user experience of recommender systems. User Model. User-Adap. Inter. 22, 441–504 (2012)

    Article  Google Scholar 

  47. Ko, A., Myers, B.: Designing the whyline: a debugging interface for asking questions about program behavior. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 151–158. ACM (2004)

    Google Scholar 

  48. Li, C., Chan, E., Denny, P., Luxton-Reilly, A., Tempero, E.: Towards a framework for teaching debugging. In: Proceedings of the Twenty-First Australasian Computing Education Conference on - ACE 2019 (2019). https://doi.org/10.1145/3286960.3286970

  49. Liao, Q.V., Gruen, D., Miller, S.: Questioning the AI: informing design practices for explainable AI user experiences. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–15 (2020)

    Google Scholar 

  50. Manca, M., Paternò, F., Santoro, C.: Remote monitoring of end-user created automations in field trials. J. Ambient Intell. Human. Comput. 13(12), 5669–5697 (2021). https://doi.org/10.1007/s12652-021-03239-0

    Article  Google Scholar 

  51. Manca, M., Paternò, F., Santoro, C., Corcella, L.: Supporting end-user debugging of trigger-action rules for IoT applications. Int. J. Hum. Comput. Stud. 123, 56–69 (2019)

    Article  Google Scholar 

  52. Manfredi, R., Andrao, M., Greco, F., Desolda, G., Treccani, B. Zancanaro, M.: Toward a better understanding of end-user debugging strategies: a pilot study. In Proceedings of the 3rd international workshop on empowering people in dealing with Internet of Things ecosystems co-located with AVI 2022, Frascati, Rome, Italy, June 06, 2022. (CEUR Workshop Proceedings, vol. 3172). CEUR-WS.org, pp. 31–35 (2022). https://ceur-ws.org/Vol-3172/short6.pdf

  53. Masui, T., Nakayama, K.: Repeat and predict—two keys to efficient text editing. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 118–130 (1994)

    Google Scholar 

  54. Mattioli, A., Paternò, F.: A visual environment for end-user creation of IoT customization rules with recommendation support. In: Proceedings of the International Conference on Advanced Visual Interfaces, pp. 1–5 (2020)

    Google Scholar 

  55. Mattioli, A., Paternò, F.: Recommendations for creating trigger-action rules in a block-based environment. Behav. Inf. Technol. 40(10), 1024–1034 (2021)

    Article  Google Scholar 

  56. Mi, X., Qian, F., Zhang, Y., Wang, X.F.: An empirical characterization of IFTTT: ecosystem, usage, and performance. In: Proceedings of the 2017 Internet Measurement Conference, pp. 398–404 (2017)

    Google Scholar 

  57. Paci, F., Bianchin, D., Quintarelli, E., Zannone, N.: IFTTT privacy checker. In: Saracino, A., Mori, P. (eds.) Emerging Technologies for Authorization and Authentication. LNCS, vol. 12515, pp. 90–107. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-64455-0_6

    Chapter  Google Scholar 

  58. Paternò, F., Santoro, C.: End-user development for personalizing applications, things, and robots. Int. J. Hum Comput Stud. 131, 120–130 (2019)

    Article  Google Scholar 

  59. Pianesi, F., Varzi, A.C.: Events and event talk: an introduction. In: Speaking of Events, pp. 3–47. Oxford University Press, New York, NY (2000)

    Google Scholar 

  60. Resnick, M., et al.: Scratch: programming for all. Commun. ACM 52(11), 60–67 (2009). https://doi.org/10.1145/1592761.1592779

    Article  Google Scholar 

  61. Ruvini, J.-D., Dony, C.: Learning users’ habits to automate repetitive tasks. In: Your Wish is My Command, pp. 271-XIV. Morgan Kaufmann (2001) https://doi.org/10.1016/B978-155860688-3/50015-4

  62. Saeidi, M., Calvert, M., Au, A.W., Sarma, A., Bobba, R.B.: If this context then that concern: exploring users’ concerns with IFTTT applets. In: Proceedings on Privacy Enhancing Technologies, vol. 2022(1), pp. 166–186 (2021)

    Google Scholar 

  63. Salovaara, A., Bellucci, A., Vianello, A., Jacucci, G.: Programmable smart home toolkits should better address households’ social needs. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–14 (2021)

    Google Scholar 

  64. Seiger, R., Kühn, R., Korzetz, M., Aßmann, U.: HoloFlows: modelling of processes for the Internet of Things in mixed reality. Softw. Syst. Model. 20(5), 1465–1489 (2021). https://doi.org/10.1007/s10270-020-00859-6

    Article  Google Scholar 

  65. Soares, D., Dias, J.P., Restivo, A., Ferreira, H.S.: Programming IoT-spaces: a user-survey on home automation rules. In: Paszynski, M., Kranzlmüller, D., Krzhizhanovskaya, V.V., Dongarra, J.J., Sloot, P.M.A. (eds.) Computational Science – ICCS 2021. LNCS, vol. 12745, pp. 512–525. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-77970-2_39

    Chapter  Google Scholar 

  66. Srinivasan, V., Koehler, C., Jin, H.: RuleSelector: Selecting conditional action rules from user behavior patterns. Proc. ACM Interact. Mobile. Wearabl. Ubiquit. Technol. 2(1), 1–34 (2018)

    Article  Google Scholar 

  67. Stefanidi, E., et al.: MagiPlay: an augmented reality serious game allowing children to program intelligent environments. Trans. Comput. Sci. XXXVII: Spec. Issue Comput. Graph. 144–169 (2020)

    Google Scholar 

  68. Surbatovich, M., Aljuraidan, J., Bauer, L., Das, A., Jia, L.: Some recipes can do more than spoil your appetite: analysing the security and privacy risks of IFTTT recipes. In: Proceedings of the 26th International Conference on World Wide Web (WWW 2017), pp. 1501–1510 (2017)

    Google Scholar 

  69. Ur, B., McManus, E., Yong Ho, M.P., Littman, M.L.: Practical trigger-action programming in the smart home. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 803–812 (2014)

    Google Scholar 

  70. Ur, B., et al.: Trigger-action programming in the wild: An analysis of 200,000 IFTTT recipes. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 3227–3231 (2016)

    Google Scholar 

  71. Valtolina, S., Barricelli, B.R., Di Gaetano, S.: Communicability of traditional interfaces VS chatbots in healthcare and smart home domains. Behav. Inf. Technol. 39(1), 108–132 (2020)

    Google Scholar 

  72. Wang, Q., Datta, P., Yang, W., Liu, S., Bates, A., Gunter, C.A.: Charting the attack surface of trigger-action IoT platforms. In: Proceedings of the ACM Conference on Computer and Communications Security, pp. 1439–1453 (2019)

    Google Scholar 

  73. Yang, F., Kalloori, S., Chalumattu, R., Gross, M.: Personalized information retrieval for touristic attractions in augmented reality. In: Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pp. 1613–1616 (2022)

    Google Scholar 

  74. Yu, H., Hua, J., Julien, C.: Analysis of IFTTT recipes to study how humans use internet-of-things (iot) devices. In: Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, pp. 537–541 (2021)

    Google Scholar 

  75. Yusuf, I.N.B., Jamal, D.B.A., Jiang, L., Lo, D.: RecipeGen++: an automated trigger action programs generator. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1672–1676, November 2022

    Google Scholar 

  76. Zancanaro, M., Gallitto, G., Dina, Y., Treccani, B.: Improving mental models in IoT end-user development. Human-centric Comput. Inf. Sci. 2, 48 (2022)

    Google Scholar 

  77. Zheng, S., Apthorpe, N., Chetty, M., Feamster, N.: User perceptions of smart home IoT privacy. In: Proceedings of the ACM on Human-Computer Interaction, vol. 2, CSCW, pp. 1–20 (2018)

    Google Scholar 

Download references

Acknowledgements

This work has been supported by the PRIN 2017 “EMPATHY: Empowering People in Dealing with Internet of Things Ecosystems”, https://www.empathy-project.eu/. Balducci acknowledges the support by the REFIN grant, POR Puglia FESR FSE 2014–2020 “Gestione di oggetti intelligenti per migliorare le esperienze di visita di siti di interesse culturale”.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Fabio Paternò .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Andrao, M. et al. (2023). Understanding Concepts, Methods and Tools for End-User Control of Automations in Ecosystems of Smart Objects and Services. In: Spano, L.D., Schmidt, A., Santoro, C., Stumpf, S. (eds) End-User Development. IS-EUD 2023. Lecture Notes in Computer Science, vol 13917. Springer, Cham. https://doi.org/10.1007/978-3-031-34433-6_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-34433-6_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-34432-9

  • Online ISBN: 978-3-031-34433-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics