Skip to main content
SpringerLink
Log in

Interactive Journaling with AI: Probing into Words and Language as Interaction Design Materials

  • Conference paper
  • First Online:
Chatbot Research and Design (CONVERSATIONS 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13815))

Included in the following conference series:

  • 1138 Accesses

Abstract

Conversational Agents (CAs) are making human-computer interaction more collaborative and conversational through using natural language. The HCI and interaction design communities, have been experimenting with and exploring the area of designing conversational interactions. Furthermore, interaction designers may need to acquire new skills for designing, prototyping, and evaluating artifacts that embody AI technologies in general, and CAs in particular. This paper builds upon a previous study on principles of designing interactive journaling experiences with CA and explores the practice of designing such experiences, using words, language, and conversations as design materials. We present a prototype for interactive and reflective journaling interaction with CA and the result of a Wizard of Oz experiment. Our findings suggest that designing interactions with CA challenges designers to use materials with inherently different natures and qualities. Despite this challenge, words appear to have unique characteristics to support designers to externalize and iterate on ideas, e.g., tone and intent. Hence, we suggest considering words, language, and conversations as the primary design materials, and the AI’s predictability, adaptivity, and agency as secondary materials, while designing human interactions with Conversational Agents.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Blake, T.K.: Journaling; an active learning technique. Int. J. Nurs. Educ. Scholarsh. 2 (2005). https://doi.org/10.2202/1548-923X.1116

  2. Kocielnik, R., Xiao, L., Avrahami, D., Hsieh, G.: Reflection companion: a conversational system for engaging users in reflection on physical activity. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 70:1–70:26 (2018). https://doi.org/10.1145/3214273

  3. Hubbs, D.L., Brand, C.F.: The paper mirror: understanding reflective journaling. J. Exp. Educ. 28, 60–71 (2005). https://doi.org/10.1177/105382590502800107

    Article  Google Scholar 

  4. Hughes, H.W., Kooy, M., Kanevsky, L.: Dialogic reflection and journaling. Clearing House: J. Educ. Strat. Issues Ideas 70, 187–190 (1997). https://doi.org/10.1080/00098655.1997.10544193

    Article  Google Scholar 

  5. Deibel, D., Evanhoe, R.: Conversations with Things: UX Design for Chat and Voice. Rosenfeld Media (2021)

    Google Scholar 

  6. Yang, X., Aurisicchio, M., Baxter, W.: Understanding affective experiences with conversational agents. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–12. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3290605.3300772

  7. Amershi, S., et al.: Guidelines for human-AI interaction. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–13. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3290605.3300233

  8. Kocielnik, R., Avrahami, D., Marlow, J., Lu, D., Hsieh, G.: Designing for workplace reflection: a chat and voice-based conversational agent. In: Proceedings of the 2018 Designing Interactive Systems Conference, pp. 881–894 (2018)

    Google Scholar 

  9. Luger, E., Sellen, A.: “Like having a really bad PA” the gulf between user expectation and experience of conversational agents. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 5286–5297 (2016)

    Google Scholar 

  10. Norman, D.A.: How might people interact with agents. Commun. ACM 37, 68–71 (1994)

    Article  Google Scholar 

  11. Rozendaal, M.C., Ghajargar, M., Pasman, G., Wiberg, M.: Giving form to smart objects: exploring intelligence as an interaction design material. In: Filimowicz, M., Tzankova, V. (eds.) New Directions in Third Wave Human-Computer Interaction: Volume 1 - Technologies. HIS, pp. 25–42. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-73356-2_3

    Chapter  Google Scholar 

  12. Wiberg, M.: The Materiality of Interaction: Notes on the Materials of Interaction Design. The MIT Press, Cambridge (2018)

    Book  Google Scholar 

  13. Ghajargar, M., Bardzell, J., Lagerkvist, L.: A redhead walks into a bar: experiences of writing fiction with artificial intelligence. In: 25th International Academic Mindtrek Conference, pp. 230–241. Association for Computing Machinery, New York (2022). https://doi.org/10.1145/3569219.3569418

  14. Wärnestål, P.: Design av AI-drivna tjänster. Studentlitteratur AB (2021)

    Google Scholar 

  15. Ghajargar, M., Wiberg, M., Stolterman, E.: Designing IoT systems that support reflective thinking: a relational approach. Int. J. Des. 12, 21–35 (2018)

    Google Scholar 

  16. Pirzadeh, A., He, L., Stolterman, E.: Personal informatics and reflection: a critical examination of the nature of reflection. In: CHI’13 Extended Abstracts on Human Factors in Computing Systems, pp. 1979–1988 (2013)

    Google Scholar 

  17. Terzimehić, N., Schött, S.Y., Bemmann, F., Buschek, D.: MEMEories: internet memes as means for daily journaling. In: Designing Interactive Systems Conference 2021, pp. 538–548. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3461778.3462080

  18. Ong, L.T.R., Pemberton, R.: Enhancing classroom learning through computer-mediated reflective writing and peer feedback. J. Mod. Lang. 19, 99–120 (2009)

    Google Scholar 

  19. Tholander, J., Normark, M.: Crafting personal information - resistance, imperfection, and self-creation in bullet journaling. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–13. Association for Computing Machinery, New York (2020)

    Google Scholar 

  20. Zhang, Y., Parker, A.G.: Eat4Thought: a design of food journaling. In: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–8. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3334480.3383044

  21. Ghajargar, M., et al.: From “explainable AI” to “graspable AI.” In: Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction, Salzburg, Austria, pp. 1–4. Association for Computing Machinery (2021). https://doi.org/10.1145/3430524.3442704

  22. van Berkel, N., Skov, M.B., Kjeldskov, J.: Human-AI interaction: intermittent, continuous, and proactive. Interactions 28, 67–71 (2021). https://doi.org/10.1145/3486941

    Article  Google Scholar 

  23. Kulesza, T., Burnett, M., Wong, W.-K., Stumpf, S.: Principles of explanatory debugging to personalize interactive machine learning. In: Proceedings of the 20th International Conference on Intelligent User Interfaces, pp. 126–137 (2015)

    Google Scholar 

  24. Cronholm, S., Göbel, H.: Design principles for human-centred AI. In: ECIS 2022 Research Papers (2022)

    Google Scholar 

  25. Angenius, M., Ghajargar, M.: Design principles for interactive and reflective journaling with AI. In: Computing Conference 2023. Springer, London (2023)

    Google Scholar 

  26. Hall, E.: Conversational Desigjn. A Book Apart (2018)

    Google Scholar 

  27. Allen, J.F., Byron, D.K., Dzikovska, M., Ferguson, G., Galescu, L., Stent, A.: Toward conversational human-computer interaction. AI Mag. 22, 27 (2001). https://doi.org/10.1609/aimag.v22i4.1590

    Article  Google Scholar 

  28. Janlert, L.-E., Stolterman, E.: Faceless Interaction—a conceptual examination of the notion of interface: past, present, and future. Hum.-Comput. Interact. 30, 507–539 (2015). https://doi.org/10.1080/07370024.2014.944313

    Article  Google Scholar 

  29. Gaver, W.: What should we expect from research through design? In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Austin, Texas, USA, pp. 937–946. Association for Computing Machinery (2012). https://doi.org/10.1145/2207676.2208538

  30. Angenius, M.: Talkus AI-relius: An Interactive Journaling Artifact That Supports Reflection Through Conversation (2022)

    Google Scholar 

  31. Lee, M., Ackermans, S., van As, N., Chang, H., Lucas, E., IJsselsteijn, W.: Caring for vincent: a chatbot for self-compassion. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–13. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3290605.3300932

  32. Sanders, E., Stappers, P.: Convivial Toolbox: Generative Research for the Front End of Design. BIS, Amsterdam (2012)

    Google Scholar 

  33. Lenz, E., Diefenbach, S., Hassenzahl, M.: Exploring relationships between interaction attributes and experience. In: Proceedings of the 6th International Conference on Designing Pleasurable Products and Interfaces, pp. 126–135. Association for Computing Machinery, New York (2013). https://doi.org/10.1145/2513506.2513520

  34. Petersen, M.G., Iversen, O.S., Krogh, P.G., Ludvigsen, M.: Aesthetic interaction: a pragmatist’s aesthetics of interactive systems. In: Proceedings of the 5th Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques, pp. 269–276 (2004)

    Google Scholar 

  35. Lim, Y., Stolterman, E., Jung, H., Donaldson, J.: Interaction gestalt and the design of aesthetic interactions. In: Proceedings of the 2007 Conference on Designing Pleasurable Products and Interfaces, pp. 239–254. Association for Computing Machinery, New York (2007). https://doi.org/10.1145/1314161.1314183

  36. Hornbæk, K., Oulasvirta, A.: What is interaction? In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 5040–5052. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3025453.3025765

  37. Fleck, R., Fitzpatrick, G.: Reflecting on reflection: framing a design landscape. In: Proceedings of the 22nd Conference of the Computer-Human Interaction Special Interest Group of Australia on Computer-Human Interaction, pp. 216–223. Association for Computing Machinery, New York (2010). https://doi.org/10.1145/1952222.1952269

  38. Wheel Decide. https://wheeldecide.com/. Accessed 15 Aug 2022

  39. Navas, E., Gallagher, O., Burrough, X.: Keywords in Remix Studies. Routledge, Milton Park (2017)

    Google Scholar 

  40. Houde, S., Hill, C.: What do prototypes prototype? In: Helander, M.G., Landauer, T.K., and Prabhu, P.V. (eds.) Handbook of Human-Computer Interaction, 2nd edn, pp. 367–381. North-Holland, Amsterdam (1997). https://doi.org/10.1016/B978-044481862-1.50082-0

  41. Buchenau, M., Suri, J.F.: Experience prototyping. In: Proceedings of the 3rd Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques, pp. 424–433. Association for Computing Machinery, New York (2000). https://doi.org/10.1145/347642.347802

  42. van Boeijen, A., Daalhuizen, J., Zijlstra, J.: Delft Design Guide: Perspectives, Models, Approaches, Methods. BIS Publishers, Amsterdam (2020)

    Google Scholar 

  43. Nakamura, K., Feng, H., Priss, S., Mei, H.: Designing for night-time reflection: how to support night-time reflection through non-digital means. In: The 39th ACM International Conference on Design of Communication, pp. 386–388 (2021)

    Google Scholar 

  44. Ghajargar, M., Wiberg, M.: Thinking with interactive artifacts: reflection as a concept in design outcomes. Des. Issues 34, 48–63 (2018). https://doi.org/10.1162/DESI_a_00485

    Article  Google Scholar 

  45. Ghajargar, M., De Marco, A., Montagna, F.: Wise things”: when smart objects stimulate reflection. In: Proceedings of 11th International Conference on Interfaces and Human Computer Interaction, pp. 233–238 (2017)

    Google Scholar 

  46. Baumer, E.P.S., Khovanskaya, V., Matthews, M., Reynolds, L., Schwanda Sosik, V., Gay, G.: Reviewing reflection: on the use of reflection in interactive system design. In: Proceedings of the 2014 Conference on Designing Interactive Systems, pp. 93–102. Association for Computing Machinery, New York (2014). https://doi.org/10.1145/2598510.2598598

  47. Schön, D.A.: The Reflective Practitioner : How Professionals Think in Action. Routledge, Milton Park (1983). https://doi.org/10.4324/9781315237473

  48. World Leaders in Research-Based User Experience: 10 Usability Heuristics for User Interface Design, https://www.nngroup.com/articles/ten-usability-heuristics/. Accessed 09 Sept 2022

  49. Hutchins, E.L., Hollan, J.D., Norman, D.A.: Direct manipulation interfaces. Hum.-Comput. Interact. 1, 311–338 (1985). https://doi.org/10.1207/s15327051hci0104_2

    Article  Google Scholar 

  50. Norman, D.: The Design of Everyday Things: Revised and Expanded. Basic Books, New York (2013)

    Google Scholar 

Download references

Acknowledgments

We would like to show appreciation for the people who participated in the user studies.

Author information

Authors and Affiliations

  1. School of Arts and Communication, Malmö University, Malmö, Sweden

    Max Angenius & Maliheh Ghajargar

  2. Internet of Things and People Research Center, Malmö University, Malmö, Sweden

    Maliheh Ghajargar

Authors
  1. Max Angenius
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maliheh Ghajargar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Max Angenius .

Editor information

Editors and Affiliations

  1. SINTEF, Oslo, Norway

    Asbjørn Følstad

  2. University of Amsterdam, Amsterdam, The Netherlands

    Theo Araujo

  3. CERTH-ITI, Thessaloniki, Greece

    Symeon Papadopoulos

  4. Durham University, Durham, UK

    Effie L.-C. Law

  5. University of Edinburgh, Edinburgh, UK

    Ewa Luger

  6. University of Agder, Grimstad, Norway

    Morten Goodwin

  7. University of Oslo, Oslo, Norway

    Petter Bae Brandtzaeg

Appendix

Appendix

1.1 Appendix A. Examples of CA’S Key Behaviors and Prompts

The situation

CA’s behavior

CA’s prompt

Meeting someone for the first time

Greet the user, introduce themselves, and explain a short version of what can be expected

“Hello there, my name is Talkus AI-relius, and I am your journaling companion. What’s your name?”

Talking with someone familiar

Greet by name and express happiness about seeing them again

“Welcome back [name], it is good to see you again. Today it’s 10th of May 2022. Would you like to journal?”

Asked to help with something it can do

Proceed to explain to the user if needed

“Ok, great. You can journal in two different ways, the first way is by using the pen to write in the journal, and the second way is to speak to me using your voice. Are you ready to start our first journaling session together?”

Asked to help with something it can’t do

Apologize and inform them that it can’t do it and suggest an alternative

“I apologize, this is not an action I can perform. Do you want me to suggest an alternative action?”

Interrupted

Stop what it is doing, wait, and listen to what the user is saying and act on suggested new task

“…do you want me to do [name of task recognized from user interaction]?”

Mistaken

Apologize and ask for clarification or repetition

“I am sorry, I didn’t understand that. Do you mind clarifying or repeating?”

Correcting someone

Ask the user if they meant “x”

“Do you mean [correction]?”

Asked a question it couldn’t answer

Apologize and inform the user that it is a question it can’t answer

“I am sorry, I don’t have answers to that question.”

Asked a personal question

Explain that it itself doesn’t have an opinion but that it could bring in an outside perspective if the user wants

“I don’t have personal opinions. I am here to help you process your opinions, thoughts, and emptions. But if you want, I can bring in an outside perspective.”

Asked an inappropriate question

Explain that this is not something it will discuss but encourages the user to explore their thoughts, feelings, and point of view

“I don’t discuss these kinds of topics, but I encourage you to explore your thoughts, feelings, and point of view in your journal.”

1.2 Appendix B. The Interaction Design Wheel(s) of Fortune

figure a

1.3 Appendix C: Combinations of Elements of Interaction

Sketch No.

Combination of elements

24

Fast, tool, continuity, control, secondary orality, R2 dialogic reflection, continuous human-ai interaction

25

Fluent, media, movement, tool use, oral culture, R3 transformative reflection, continuous human-ai interaction

26

Targeted, dialogue partner, state, embodiment, literate culture, R0 descriptive reflection, proactive human-ai interaction

27

Uniform, aesthetic experience, pace, experience, secondary orality, R1 reflective description, proactive human-ai interaction

28

Constant, tool, connectivity, embodiment, oral culture, R0 description, proactive human-ai interaction

29

Approximate, dialogue partner, movement, control, secondary orality, R4 critical reflection, intermittent human-ai interaction

30

Powerful, aesthetic experience, pace, control, literate culture, R2 reflective description (forgot to include human-ai interaction)

31

Slow, aesthetic experience, proximity, transmission, literate culture, R0 description, proactive human-ai interaction

32

Instant, aesthetic experience, continuity, optimal behavior, oral culture, R1. Reflective description, proactive human-ai interaction

33

Approximate, dialogue partner, speed, control, secondary orality, R2 dialogic reflection, intermittent human-ai interaction

34

Inconstant, system, movement, optimal behavior, literate culture, R0 description, intermittent human-ai interaction

35

Precise, system, connectivity, optimal behavior, oral culture, R2. Dialogic reflection, proactive human-ai interaction

36

Incidental, system, state, control, oral culture, R2 dialogic reflection, intermittent human-ai interaction

37

Gentle, media, time-depth, embodiment, literate culture, R4 critical reflection, proactive human-ai interaction

38

Delayed, media, directness, optimal behavior, literate culture, R0 description, continuous human-ai interaction

39

Fast, tool, continuity, control, secondary orality, R2 dialogic reflection, continuous human-ai interaction

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Angenius, M., Ghajargar, M. (2023). Interactive Journaling with AI: Probing into Words and Language as Interaction Design Materials. In: Følstad, A., et al. Chatbot Research and Design. CONVERSATIONS 2022. Lecture Notes in Computer Science, vol 13815. Springer, Cham. https://doi.org/10.1007/978-3-031-25581-6_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25581-6_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25580-9

  • Online ISBN: 978-3-031-25581-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation