, Volume 22, Issue 1, pp 143–169 | Cite as

Towards pervasive geospatial affect perception

  • Muneeba Raja
  • Anja Exler
  • Samuli Hemminki
  • Shin’ichi Konomi
  • Stephan Sigg
  • Sozo Inoue


Due to the enormous penetration of connected computing devices with diverse sensing and localization capabilities, a good fraction of an individual’s activities, locations, and social connections can be sensed and spatially pinpointed. We see significant potential to advance the field of personal activity sensing and tracking beyond its current state of simple activities, at the same time linking activities geospatially. We investigate the detection of sentiment from environmental, on-body and smartphone sensors and propose an affect map as an interface to accumulate and interpret data about emotion and mood from diverse set of sensing sources. In this paper, we first survey existing work on affect sensing and geospatial systems, before presenting a taxonomy of large-scale affect sensing. We discuss model relationships among human emotions and geo-spaces using networks, apply clustering algorithms to the networks and visualize clusters on a map considering space, time and mobility. For the recognition of emotion and mood, we report from two studies exploiting environmental and on-body sensors. Thereafter, we propose a framework for large-scale affect sensing and discuss challenges and open issues for future work.


Geospatial mapping Emotion recognition Device-free sensing Activity recognition 



We are thankful to Christoph Klebsattel for his support in conducting the experiments on the mood induction and recognition. We would further like to thank Dr. Andrea Schankin for the fruitful discussions about affective states, their characteristics and assessment methods. We are also thankful to Syed Safi Ali Shah for sharing his knowledge in DFAR research.


  1. 1.
    Abdelnasser H, Youssef M, Harras KA (2015) WiGest: A Ubiquitous WiFi-based Gesture Recognition System. ArXiv e-printsGoogle Scholar
  2. 2.
    Adib F, Katabi D (2013) See through Walls with Wifi!. SIGCOMM Comput Commun Rev 43(4):75–86CrossRefGoogle Scholar
  3. 3.
    Nguyen A-T, Chen W, Rauterberg M (2012) The role of human body expression in affect detection: A review. In: 10th Asia Pacific Conference on Computer Human Interaction (APCHI 2012)Google Scholar
  4. 4.
    Ark WS, Dryer DC, Lu DJ (1999) The emotion mouse. In: HCI (1), pp 818–823Google Scholar
  5. 5.
    Bachmann A, Klebsattel C, Budde M, Riedel T, Beigl M, Reichert M, Santangelo P, Ebner-Priemer U (2015) How to use smartphones for less obtrusive ambulatory mood assessment and mood recognition. In: Adjunct Proceedings of Ubicomp’15. ACM, pp 693–702Google Scholar
  6. 6.
    Bachmann A, Klebsattel C, Schankin A, Riedel T, Beigl M, Reichert M, Santangelo P, Ebner-Priemer U (2015) Leveraging smartwatches for unobtrusive mobile ambulatory mood assessment. In: Adjunct Proceedings of Ubicomp’15. ACM, pp 1057–1062Google Scholar
  7. 7.
    Bao X, Fan S, Varshavsky A, Li K, Roy Choudhury R (2013) Your reactions suggest you liked the movie: Automatic content rating via reaction sensing. In: UbiCompGoogle Scholar
  8. 8.
    Jackson B, Brennis L-W (2000) The pupillary system. Cambridge University Press, pp 142–162Google Scholar
  9. 9.
    Beedie CJ, Terry PC, Lane AM, Devonport TJ (2011) Differential assessment of emotions and moods: Development and validation of the emotion and mood components of anxiety questionnaire. Personal Individ Differ 50(2):228–233CrossRefGoogle Scholar
  10. 10.
    Bideaux A, Anastasopoulou P, Hey S, Canadas A, Fernandez A (2014) Mobile monitoring of epileptic patients using a reconfigurable cyberphysical system that handles multi-parametric data acquisition and analysis. In: 2014 EAI 4th International Conference on Wireless Mobile Communication and Healthcare (Mobihealth). IEEE, pp 377–380Google Scholar
  11. 11.
    Bradley MM, Lang PJ (1994) Measuring Emotion: the Self-Assessment Manikin and the Semantic Differential. J Behav Ther Exp Psychiatry 25(1):49–59CrossRefGoogle Scholar
  12. 12.
    Burgoon JK, Guerrero LK, Floyd K (2010) Nonverbal communication. Allyn & Bacon, Boston, MAGoogle Scholar
  13. 13.
    Calvo MG, Lang PJ (2004) Gaze patterns when looking at emotional pictures: Motivationally biased attention. Motiv Emot 28(3):221–243CrossRefGoogle Scholar
  14. 14.
    Cambria E, Schuller B, Xia Y, Havasi C (2013) New avenues in opinion mining and sentiment analysis. IEEE Intelligent Systems, pp. 1Google Scholar
  15. 15.
    Canzian L, Musolesi M (2015) Trajectories of depression: Unobtrusive monitoring of depressive states by means of smartphone mobility traces analysis. International Joint Conference on Pervasive and Ubiquitous Computing, pp 1293–1304Google Scholar
  16. 16.
    Cardone G, Cirri A, Corradi A, Foschini L, Ianniello R, Montanari R (2014) Crowdsensing in urban areas for city-scale mass gathering management: Geofencing and activity recognition. Sensors pp. 4185–4195Google Scholar
  17. 17.
    Chen M, Gonzalez S, Vasilakos A, Cao H, Leung VC (2011) Body area networks: A survey. Mob Netw Appl 16(2):171–193CrossRefGoogle Scholar
  18. 18.
    Coan JA, Allen JJ (2007) Handbook of emotion elicitation and assessment. Oxford University PressGoogle Scholar
  19. 19.
    Collet C, Vernet-Maury E, Delhomme G, Dittmar A (1997) Autonomic nervous system response patterns specificity to basic emotions. J Autonom Nerv Syst 62(1-2):45–57CrossRefGoogle Scholar
  20. 20.
    Coviello L, Sohn Y, Kramer ADI, Marlow C, Franceschetti M, Christakis NA, Fowler JH Detecting emotional contagion in massive social networks. PLoS ONE (2014). doi: 10.1371/journal.pone.0090315
  21. 21.
    Crane E, Gross M (2007) Motion capture and emotion: Affect detection in whole body movement. In: Affective computing and intelligent interaction. Springer, pp 95–101Google Scholar
  22. 22.
    Dalgleish T, Power MJ (1999) Handbook of cognition and emotion. Wiley OnlineGoogle Scholar
  23. 23.
    Dawson ME, Schell AM, Bohmelt AH (2008) Startle modification: Implications for neuroscience, cognitive science, and clinical science. Cambridge University PressGoogle Scholar
  24. 24.
    DePasquale JP, Geller E, Clarke SW, Littleton LC (2001) Measuring road rage: development of the propensity for angry driving scale. J Saf Res 32(1):1–16. doi: 10.1016/S0022-4375(00)00050-5 CrossRefGoogle Scholar
  25. 25.
    Dichter GS, Tomarken AJ, Baucom BR (2002) Startle modulation before, during and after exposure to emotional stimuli. Int J Psychophysiol 43(2):191–196CrossRefGoogle Scholar
  26. 26.
    Van Dongen S (2000) Graph clustering by flow simulation. Ph.D. thesis, University of UtrechtGoogle Scholar
  27. 27.
    Ekman P (1992) Are there basic emotions?Google Scholar
  28. 28.
    El Mawass N, Kanjo E (2013) A supermarket stress map. In: Ubicomp’13 adjunct Proceedings. ACM, pp 1043–1046Google Scholar
  29. 29.
    Exler A, Klebsattel C, Schankin A, Beigl M (2016) A wearable system for mood assessment considering smartphone features and mobile ecg measurements. In: Adjunct Proceedings of Ubicomp’16. ACM. To appearGoogle Scholar
  30. 30.
    Fisher CD, To ML (2012) Using experience sampling methodology in organizational behavior. J Organ Behav 33(7):865–877CrossRefGoogle Scholar
  31. 31.
    Eyben F, W?llmer M, Poitschke T (2010) Emotion on the road?necessity, acceptance, and feasibility of affective computing in the car. Advances in Human-Computer Interaction p 17. doi: 10.1155/2010/263593
  32. 32.
    De Gelder B, Hortensius R (2009) Why bodies? Twelve reasons for including bodily expressions in affective neuroscience. Philos Trans Royal Soc 364:3475–3484CrossRefGoogle Scholar
  33. 33.
    Geller T (2014) How do you feel?: Your computer knows. Commun ACM 57 (1):24–26CrossRefGoogle Scholar
  34. 34.
    Giaccardi E, Fogli D (2008) Affective Geographies: Toward Richer Cartographic Semantics for the Geospatial Web. In: Proceedings of the working conference on Advanced visual interfaces (AVI ’08), pp 173–180Google Scholar
  35. 35.
    Granholm E, Steinhauer SR (2004) Int J Psychophysiol 52(1):1–6. Pupillometric Measures of Cognitive and Emotional ProcessesCrossRefGoogle Scholar
  36. 36.
    Grunerbl A, Muaremi A, Osmani V, Bahle G, Ohler S, Troster G, Mayora O, Haring C, Lukowicz P (2014) Smartphone-based recognition of states and state changes in bipolar disorder patients. IEEE J Biomed Health Inf 19(1):140–148CrossRefGoogle Scholar
  37. 37.
    Hatfield E, Cacioppo J, Rapson R (1993) Emotional contagion. Current directions. Psychol Sci 2:96–99. doi: 10.1111/1467-8721.ep10770953 Google Scholar
  38. 38.
    Healey J, Picard R (2000) Smartcar: Detecting driver stress. In: Pattern Recognition. 15th International Conference on 2000. Proceedings, vol 4, pp 218–221. doi: 10.1109/ICPR.2000.902898
  39. 39.
    Hemminki S, Zhao K, Ding AY, Rannanjarvi M, Tarkoma S, Nurmi P (2013) Cosense: A collaborative sensing platform for mobile devices. Embedded Networked Sensor Systems, p 34Google Scholar
  40. 40.
    Herdem KC (2012) Reactions: Twitter based mobile application for awareness of friends’ emotions. In: UbiComp’12. ACMGoogle Scholar
  41. 41.
    Hernandez J, McDuff DJ, Picard RW (2015) Biophone: Physiology monitoring from peripheral smartphone motions. Engineering in Medicine and Biology SocietyGoogle Scholar
  42. 42.
    Jercic P, Astor PJ, Adam MTP, Hilborn O, Schaaff K, Lindley C, Sennersten C, Eriksson J (2012) A serious game using physiological interfaces for emotion regulation training in the context of financial decision-making. In: ECIS, p 207Google Scholar
  43. 43.
    Joint M (1995) Road rageGoogle Scholar
  44. 44.
    Knighten J, McMillan S, Chambers T, Payton J (2015) Recognizing social gestures with a Wrist-Worn Smartband. Pervasive Computing and Communication, pp 544–549Google Scholar
  45. 45.
    Konomi S (2011) Colocation networks: exploring the use of social andgeographical patterns in context-aware services. In: UbiComp’11 adjunct proceedings. ACM, pp 565–566Google Scholar
  46. 46.
    Koolagudi S, Rao K (2012) Emotion recognition from speech: a Review. Int. J. Speech Technol. 15(2):99–117CrossRefGoogle Scholar
  47. 47.
    Kramer AD, Guillory JE, Hancock JT (2014) Experimental evidence of massive-scale emotional contagion through social networks. Proceedings of the National Academy of Sciences, p 201320040Google Scholar
  48. 48.
    Kramer ADI, Guillory JE, Hancock JT (2014) Experimental evidence of massive-scale emotional contagion through social networks. Proc Natl Acad Sci 111(24)Google Scholar
  49. 49.
    Lane N, Miluzzo E, Lu H, Peebles D, Choudhury T, Campbell A (2010) A survey of mobile phone sensin. Commun Mag 48:140–150CrossRefGoogle Scholar
  50. 50.
    Lang PJ, Greenwald MK, Bradley MM, Hamm AO (1993) Looking at pictures: Affective, Facial, Visceral, and behavioral reactions. Psychophysiology, pp 161–273Google Scholar
  51. 51.
    Lee H, Choi YS, Lee S, Park I (2012) Towards unobtrusive emotion recognition for affective social communication. In: CCNC’12Google Scholar
  52. 52.
    Leng H, Lin Y, Zanzi LA (2007) An experimental study on physiological parameters toward driver emotion recognition. Springer Berlin Heidelberg, Berlin, Heidelberg, pp 237–246. doi: 10.1007/978-3-540-73333-1_30 Google Scholar
  53. 53.
    LiKamWa R, Liu Y, Lane ND, Zhong L (2013) Moodscope: Building a mood sensor from smartphone usage patterns. In: MobiSys 2013, MobiSys ’13. ACMGoogle Scholar
  54. 54.
    Mackerron G, Mourato S (2013) Happiness is greater in natural environments. Glob Environ Chang 23(5):992–1000CrossRefGoogle Scholar
  55. 55.
    Mehrabian A (1996) Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament. Curr Psychol 14(4):261–292CrossRefGoogle Scholar
  56. 56.
    Mody RN, Willis KS, Kerstein R (2009) Wimo: location-based emotion tagging. Mobile and Ubiquitous MultimediaGoogle Scholar
  57. 57.
    Myrtek M, Aschenbrenner E, Brügner G. (2005) Emotions in everyday life: An ambulatory monitoring study with female students. Biol Psychol 68(3):237–55CrossRefGoogle Scholar
  58. 58.
    Nardelli M, Valenza G, Greco A, Lanata A, Scilingo E (2015) Recognizing emotions induced by affective sounds through heart rate variability. IEEE Trans Affect Comput PP(99):1–1Google Scholar
  59. 59.
    Nold C (2004) Bio mappingGoogle Scholar
  60. 60.
    Nurmi P, Koolwaaij J (2006) Identifying meaningful locations. Mobile and Ubiquitous Systems: Networks and Services, pp 1–8Google Scholar
  61. 61.
    De Oliveira THM (2015) The emotion-aware city: using ambient geographic information (agi) in order to understand emotion and stress within smart cities. In: AGILE PhD SchoolGoogle Scholar
  62. 62.
    Paltoglou G, Thelwall M (2012) Twitter, myspace, digg: Unsupervised sentiment analysis in social media. ACM Trans Intell Syst Technol 3(4):66:1–66:19CrossRefGoogle Scholar
  63. 63.
    Pánek J., Pászto V., Marek L (2017) Mapping emotions: Spatial distribution of safety perception in the city of Olomouc. Springer International Publishing, Cham, pp 211–224Google Scholar
  64. 64.
    Picard RW (2015) Recognizing stress, engagement, and positive emotion. In: 20th International Conference on Intelligent User Interfaces. ACM, pp 3–4Google Scholar
  65. 65.
    Prusa JD, Khoshgoftaar TM, Dittmann DJ (2015) Impact of feature selection techniques for tweet sentiment classification. In: Twenty-Eighth International Florida Artificial Intelligence Research Society ConferenceGoogle Scholar
  66. 66.
    Pu Q, Gupta S, Gollakota S, Patel S (2013) Whole-home gesture recognition using wireless signals. In: 19th Annual International Conference on Mobile Computing and Networking, MobiCom ’13, pp 27–38Google Scholar
  67. 67.
    Purves D, Augustine GJ, Fitzpatrick D, Katz LC, LaMantia AS, McNamara JO, Williams SM (2001) Physiological changes associated with emotionGoogle Scholar
  68. 68.
    Raja M, Sigg S (2016) Applicability of rf-based methods for emotion recognition: A survey. In: IEEE PerCom’16 adjunct proceedingsGoogle Scholar
  69. 69.
    Russell JA (1980) A circumplex model of affect. J Pers Soc Psychol 39(6):1161CrossRefGoogle Scholar
  70. 70.
    Samuli H, Petteri N, Sasu T (2013) Accelerometer-based transportation mode detection on smartphones. Embedded Networked Sensor Systems (SenSys)Google Scholar
  71. 71.
    Scherer KR (2005) What are emotions? And how can they be measured? Soc Sci Inf 44(4):695–729CrossRefGoogle Scholar
  72. 72.
    Schimmack U, Grob A (2000) Dimensional models of core affect: A quantitative comparison by means of structural equation modeling. Eur J Personal 14(4):325–345CrossRefGoogle Scholar
  73. 73.
    Schroeder M, Cowie R (2006) Issues in emotion-oriented computing - towards a shared understanding. In: Proceedings Workshop on Emotion and Computing at KI 2006Google Scholar
  74. 74.
    Sekimoto Y, Shibasaki R, Kanasugi H, Usui T, Shimazaki Y (2011) PFlow: Reconstructing people flow recycling large-scale social survey data. IEEE Pervasive Comput 10(4):27–35CrossRefGoogle Scholar
  75. 75.
    Sigg S, Blanke U, Troster G (2014) The telepathic phone: Frictionless activity recognition from wifi-rssi. In: 2014 IEEE International Conference on Pervasive Computing and Communications (PerCom), pp 148–155Google Scholar
  76. 76.
    Sigg S, Scholz M, Shi S, Ji Y, Beigl M (2014) Rf-sensing of activities from non-cooperative subjects in device-free recognition systems using ambient and local signals. IEEE Trans Mob Comput 13(4):907–920CrossRefGoogle Scholar
  77. 77.
    Stanners RF, Coulter M, Sweet AW, Murphy P (1979) The pupillary response as an indicator of arousal and cognition. Motiv Emot 3(4):319–340CrossRefGoogle Scholar
  78. 78.
    Steyer R, Schwenkmezger P, Notz P, Eid M (1997) Mdbf-mehrdimensionaler befindlichkeitsfragebogen [multidimensional mood questionnaire]Google Scholar
  79. 79.
    Sun FT, Kuo C, Cheng HT, Buthpitiya S, Collins P, Griss M (2012) Activity-aware mental stress detection using physiological sensors. Lect Notes Inst Comput Sci Soc Inf 72:282–301Google Scholar
  80. 80.
    Tan CSS, Schöning J, Luyten K, Coninx K (2013) Informing intelligent user interfaces by inferring affective states from body postures in ubiquitous computing environments. In: International Conference on Intelligent user interfaces. ACM, pp 235–246Google Scholar
  81. 81.
    Valenza G, Citi L, Lanatá A, Scilingo EP, Barbieri R (2014) Revealing real-time emotional responses: A personalized assessment based on heartbeat dynamics. Scientific reports, p 4Google Scholar
  82. 82.
    Venture G, Kadone H, Zhang T, Grèzes J., Berthoz A, Hicheur H (2014) Recognizing emotions conveyed by Human Gait. International Journal of Social RoboticsGoogle Scholar
  83. 83.
    Verheyen C, Göritz A.S. (2009) Plain texts as an online mood-induction procedure. Soc Psychol 40(1):6–15CrossRefGoogle Scholar
  84. 84.
    Wang H, Zhang D, Ma J, Wang Y, Wang Y, Wu D, Gu T (2016) Human respiration detection with commodity wifi devices: Do user location and body orientation matter? In: Ubicomp 2016. ACMGoogle Scholar
  85. 85.
    Wang W, Chen L, Thirunarayan K, Sheth A Harnessing twitter ‘big data’ for automatic emotion identificationGoogle Scholar
  86. 86.
    Wang W, Liu A, Shahzad M (2016) Gait recognition using wifi signals. In: Ubicomp 2016. ACMGoogle Scholar
  87. 87.
    Wilhelm P, Schoebi D (2007) Assessing mood in daily life. Eur J Psychol Assess 23(4):258–267CrossRefGoogle Scholar
  88. 88.
    Wilhelm P, Schoebi D (2007) Assessing mood in daily life: Structural validity, Sensitivity to change, and reliability of a short-scale to measure three basic dimensions of mood. Eur J Psychol Assess 23(4):258CrossRefGoogle Scholar
  89. 89.
    Wu D, Zhang D, Xu C, Wang Y, Wang H (2016) Widir: Walking direction estimation using wireless signals. In: Ubicomp 2016. ACMGoogle Scholar
  90. 90.
    Zheng BS, Murugappan M, Yaacob S (2013) Fcm clustering of emotional stress using ecg features. In: 2013 International Conference on Communications and Signal Processing (ICCSP). IEEE, pp 305–309Google Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.Aalto UniversityEspooFinland
  2. 2.Karlsruhe Insitute of TechnologyKarlsruheGermany
  3. 3.Helsinki UniversityHelsinkiFinland
  4. 4.The University of TokyoTokyoJapan
  5. 5.Kyushu Institute of TechnologyFukuokaJapan

Personalised recommendations