Skip to main content

A Gesture Elicitation Study to Generate Interaction Design Insights for Self-reporting of Mental and Body States Using a Portable Device

  • 1820 Accesses

Part of the Lecture Notes in Networks and Systems book series (LNNS,volume 260)


Self-report of mental or body states through Ecological Momentary Assessment is recommended over asynchronous methods, however, in contexts where users lack time for self-reporting, as with factory operators, this is a challenge. Furthermore, interaction designers lack references and guidelines to design these interactions. We designed a speculative prototype of a portable device that could recognise gestures corresponding to certain mental and body states and we conducted a gesture elicitation study with the prototype with eight participants. The gesture data were analysed and represented graphically to guide and inspire interaction designers. These visualisations were discussed and iterated on through two focus groups with a group of three interaction and communication designers. The study allowed to elicit visual gesture representations which may be useful to inspire and inform interaction designers and suggests that, if representations are aesthetically appealing to designers, they will be encouraged to spend time understanding complex data.


  • Gesture elicitation
  • Interaction design
  • Self-report
  • Information visualisation

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-80829-7_109
  • Chapter length: 8 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   299.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-80829-7
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   379.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.


  1. 1.


  1. Doherty, K., Balaskas, A., Doherty, G.: The design of ecological momentary assessment technologies. Interact. Comput. 32(3), 257–278 (2020)

    CrossRef  Google Scholar 

  2. Shimon, S., Lutton, C., Xu, Z., Morrison-Smith, S., Boucher, C., Ruiz, J.: Exploring non-touchscreen gestures for smartwatches. In: 2016 CHI Conference on Human Factors in Computing Systems (CHI 2016), pp. 3822–3833. Association for Computing Machinery, New York, NY, USA (2016)

    Google Scholar 

  3. Vuletic, T., Duffy, A., Hay, L., McTeague, C., Campbell, G., Grealy, M.: Systematic literature review of hand gestures used in human computer interaction interfaces. Int. J. Hum. Comput. Stud. 129(2019), 7–94 (2019)

    Google Scholar 

  4. Wobbrock, J., Morris, M., Wilson, A.: User-defined gestures for surface computing. In: 27th International Conference on Human Factors for Computing Systems (CHI 2009) (2009)

    Google Scholar 

  5. Kipp, M., Martin, J.-C.: Gesture and emotion: can basic gestural form features discriminate emotions? In: International Conference on Affective Computing and Intelligent Interaction (ACII-2009), IEEE Press (2009)

    Google Scholar 

  6. Lee, U., Tanaka, J.: Finger identification and hand gesture recognition techniques for natural user interface. In: 11th Asia Pacific Conference on Computer Human Interaction (APCHI 2013), pp. 274–279 (2013)

    Google Scholar 

  7. Morris, M., et al.: Mobile therapy: Case study evaluations of a cell phone application for emotional self-awareness. J. Med. Internet Res. 12(2), e10 (2010)

    CrossRef  Google Scholar 

  8. Rodriguez, I., Herskovic, V., Fuentes, C., Campos, M.: B-ePain: a wearable interface to self-report pain and emotions. In: 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct (UbiComp 2016), pp. 1120–1125 (2016)

    Google Scholar 

  9. Younis, E.M.G., Kanjo, E., Chamberlain, A.: Designing and evaluating mobile self-reporting techniques: crowdsourcing for citizen science. Pers. Ubiquit. Comput. 23(2), 329–338 (2019)

    CrossRef  Google Scholar 

  10. Hemmert, F., Löwe, M., Wohlauf, A., Joost, G.: Animate mobiles: proxemically reactive posture actuation as a means of relational interaction with mobile phones. In: 7th International Conference on Tangible, Embedded and Embodied Interaction (TEI 2013), pp. 267–270 (2013)

    Google Scholar 

Download references


Project OPERATOR (NORTE-01-0247-FEDER-045910) leading to this work is co-financed by the ERDF - European Regional Development Fund through the North Portugal Regional Operational Program and Lisbon Regional Operational Program and by the Portuguese Foundation for Science and Technology, under the MIT Portugal Program (2019 Open Call for Flagship projects).

Author information

Authors and Affiliations


Corresponding author

Correspondence to Ana Correia de Barros .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Correia de Barros, A., Dantas, C., Ricaldoni, F. (2021). A Gesture Elicitation Study to Generate Interaction Design Insights for Self-reporting of Mental and Body States Using a Portable Device. In: Shin, C.S., Di Bucchianico, G., Fukuda, S., Ghim , YG., Montagna, G., Carvalho, C. (eds) Advances in Industrial Design. AHFE 2021. Lecture Notes in Networks and Systems, vol 260. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-80828-0

  • Online ISBN: 978-3-030-80829-7

  • eBook Packages: EngineeringEngineering (R0)