Skip to main content
SpringerLink
Log in

Affect sensing from smartphones through touch and motion contexts

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Affect state of a person has an impact on the intellectual processes that control human behavior. Experiencing negative affect escalates mental problems, and experiencing positive affect states improve imaginative reasoning and thereby enhances one’s behavior and discipline. Hence, this work centers around affect acknowledgment from typing-based context data during the pandemic. In this paper, we present a novel sensing scheme that perceives one’s affect state from their unique contexts. We also aim to study how affect states vary in smartphone users during the pandemic. We collected data from 52 participants over 2 months with an Android application. We exploited the Circumplex Model of Affect (CMA) to infer 25 affect states, leveraging built-in motion and touch sensors on smartphones. We conducted comprehensive experiments by developing machine learning models to predict 25 states. Through our study, we observe that the states of users are heavily pertinent to one’s typing and motion contexts. A thorough evaluation shows that affect prediction model yields an F1-score of 0.90 utilizing diverse contexts. To the best of our knowledge, our work predicts the highest number of affect states (25 states) with better performance compared to state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

The dataset generated and analyzed during the current study are not publicly available due to the privacy concerns of the participants but the anonymized data are available from the corresponding author on reasonable request.

References

  1. Gillan, C.M., Rutledge, R.B.: Smartphones and the neuroscience of mental health. Ann. Rev. Neurosci. 44, 129–151 (2021)

    Article  Google Scholar 

  2. Lex, E., Kowald, D., Seitlinger, P., Tran, T.N.T., Felfernig, A., Schedl, M.: Psychology-Informed Recommender Systems. Now Publishers, Hanover (2021)

    Book  Google Scholar 

  3. Sher, L.: The impact of the COVID-19 pandemic on suicide rates. QJM: Int. J. Med. 113(10), 707–712 (2020)

    Article  Google Scholar 

  4. Rudokaite, D., Indriuniene, V.: Effectiveness of psychodrama for mitigating school fears among senior secondary school students. Zeitschrift für Psychodrama und Soziometrie 18(2), 369–385 (2019)

    Article  Google Scholar 

  5. Huang, P.H.: Boost: improving mindfulness, thinking, and diversity. Wm. & Mary Bus. L. Rev. 10, 139 (2018)

    Google Scholar 

  6. Wu, W., Chen, L., Zhao, Y.: Personalizing recommendation diversity based on user personality. User Model. User Adapt. Interact. 28(3), 237–276 (2018)

    Article  Google Scholar 

  7. Trisha, M., Guhan, P., Bhattacharya, U., Chandra, R., Bera, A., Manocha, D.: EmotiCon: context-aware multi-modal emotion recognition using Frege’s principle. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14234–14243 (2020)

  8. Alexandros, Z., Khan, A., Kalogridis, G., Vatsikas, S., Lewis, T., Sooriyabandara, M.: HealthyOffice: mood recognition at work using smartphones and wearable sensors. In: 2016 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops), pp. 1–6. IEEE (2016)

  9. Díaz-García, A., González-Robles, A., Mor, S., Mira, A., Quero, S., García-Palacios, A., Botella, C.: Positive and Negative Affect Schedule (PANAS): psychometric properties of the online Spanish version in a clinical sample with emotional disorders. BMC Psychiatry 20(1), 1–13 (2020)

    Article  Google Scholar 

  10. Russell, J.A.: A circumplex model of affect. J. Personal. Soc. Psychol. 39(6), 1161 (1980)

    Article  Google Scholar 

  11. Samuel, O., Walker, G., Salmon, P., Filtness, A., Stevens, N., Mulvihill, C., Stanton, N.: Riding the emotional roller-coaster: using the circumplex model of affect to model motorcycle riders’ emotional state-changes at intersections. Transp. Res. Part F Traffic Psychol. Behav. 66, 139–150 (2019)

    Article  Google Scholar 

  12. Stanisławski, K., Cieciuch, J., Strus, W.: Ellipse rather than a circumplex: a systematic test of various circumplexes of emotions. Personal. Individ. Diff. 181, 111052 (2021)

    Article  Google Scholar 

  13. Girardi, D., Lanubile, F., Novielli, N.: Emotion detection using noninvasive low cost sensors. In: 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 125–130. IEEE (2017)

  14. LiKamWa, R., Liu, Y., Lane, N. D., Zhong, L.: Moodscope: building a mood sensor from smartphone usage patterns. In: Proceeding of the 11th Annual International Conference on Mobile Systems, Applications, and Services, pp. 389–402 (2013)

  15. Evmenenko, A., Teixeira, D.S.: The circumplex model of affect in physical activity contexts: a systematic review. Int. J. Sport Exerc. Psychol. 20(1), 168–201 (2022)

    Article  Google Scholar 

  16. Fernández-Caballero, A., Martínez-Rodrigo, A., Pastor, J.M., Castillo, J.C., Lozano-Monasor, E., López, M.T., Fernández-Sotos, A.: Smart environment architecture for emotion detection and regulation. J. Biomed. Inform. 64, 55–73 (2016)

    Article  Google Scholar 

  17. Morshed, M.B., Saha, K., Li, R., D’Mello, S.K., De Choudhury, M., Abowd, G.D., Plötz, T.: Prediction of mood instability with passive sensing. Proc. ACM Interact. Mobile Wearable Ubiquitous Technol. 3(3), 1–21 (2019)

    Article  Google Scholar 

  18. Piskioulis, O., Tzafilkou, K., Economides, A.: Emotion detection through smartphone’s accelerometer and gyroscope sensors. In: Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization, pp. 130–137 (2021)

  19. Buda, T.S., Khwaja, M., Matic, A.: Outliers in smartphone sensor data reveal outliers in daily happiness. Proc. ACM Interact. Mobile Wearable Ubiquitous Technol. 5(1), 1–19 (2021)

    Article  Google Scholar 

  20. Wampfler, R., Klingler, S., Solenthaler, B., Schinazi, V. R., Gross, M.: Affective state prediction based on semi-supervised learning from smartphone touch data. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–13 (2020). https://doi.org/10.1145/3313831.3376504

  21. Hashmi, M.A., Riaz, Q., Zeeshan, M., Shahzad, M., Fraz, M.M.: Motion reveal emotions: identifying emotions from human walk using chest mounted smartphone. IEEE Sens. J. 20(22), 13511–13522 (2020)

    Article  Google Scholar 

  22. Balducci, F., Impedovo, D., Macchiarulo, N., Pirlo, G.: Affective states recognition through touch dynamics. Multimed. Tools Appl. 79(47), 35909–35926 (2020)

    Article  Google Scholar 

  23. Wang, P., Dong, L., Liu, W., Jing, N.: Clustering-based emotion recognition micro-service cloud framework for mobile computing. IEEE Access 8, 49695–49704 (2020)

    Article  Google Scholar 

  24. Ruensuk, M., Oh, H., Cheon, E., Oakley, I., Hong, H.: Detecting negative emotions during social media use on smartphones. In: Proceedings of Asian CHI Symposium 2019: Emerging HCI Research Collection, pp. 73–79 (2019)

  25. Tikadar, S., Bhattacharya, S.: A novel method to build and validate an affective state prediction model from touch-typing. In: IFIP Conference on Human-Computer Interaction, pp. 99–119. Springer, Cham (2019)

  26. Ghandeharioun, A., McDuff, D., Czerwinski, M., Rowan, K.: Emma: an emotion-aware well-being chatbot. In: 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 1–7. IEEE (2019). https://doi.org/10.1109/ACII.2019.8925455

  27. Ghosh, S., Sahu, S., Ganguly, N., Mitra, B., De, P.: EmoKey: an emotion-aware smartphone keyboard for mental health monitoring. In: 2019 11th International Conference on Communication Systems & Networks (COMSNETS), pp. 496–499. IEEE (2019). https://doi.org/10.1109/COMSNETS.2019.8711078

  28. Zhang, X., Li, W., Chen, X., Lu, S.: Moodexplorer: towards compound emotion detection via smartphone sensing. Proc. ACM Interact. Mobile Wearable Ubiquitous Technol. 1(4), 1–30 (2018). https://doi.org/10.1145/3161414

    Article  Google Scholar 

  29. Gloor, P.: Consistent excitement correlates with happiness-Predicting mood through body sensing with smartwatches. ICKN White Pap (2017)

  30. Ma, Y., Xu, B., Bai, Y., Sun, G., Zhu, R.: Daily mood assessment based on mobile phone sensing. In: 2012 Ninth International Conference on Wearable and Implantable Body Sensor Networks, pp. 142–147. IEEE (2012) https://doi.org/10.1109/BSN.2012.3

  31. Yang, N., Samuel, A.: Context-rich detection of user’s emotions using a smartphone. Microsoft Research Internship Report (2011)

  32. Sultana, M., Al-Jefri, M., Lee, J.: Using machine learning and smartphone and smartwatch data to detect emotional states and transitions: exploratory study. JMIR mHealth uHealth 8(9), e17818 (2020)

    Article  Google Scholar 

  33. Budner, P., Eirich, J., Gloor, P.A.: Making you happy makes me happy–Measuring Individual Mood with Smartwatches (2017) arXiv preprint arXiv:1711.06134

  34. Cortes, C., Vapnik, V.: Mach. Learn. Support-vector networks 20(3), 273–297 (1995)

    Google Scholar 

  35. Ali, J., Khan, R., Ahmad, N., Maqsood, I.: Random forests and decision trees. Int. J. Comput. Sci. Issues (IJCSI) 9(5), 272 (2012)

    Google Scholar 

  36. Ghosh, S., Hiware, K., Ganguly, N., Mitra, B., De, P.: Emotion detection from touch interactions during text entry on smartphones. Int. J. Hum. Comput. Stud. 130, 47–57 (2019)

    Article  Google Scholar 

  37. Salido Ortega, M.G., Rodríguez, L.F., Gutierrez-Garcia, J.O.: Towards emotion recognition from contextual information using machine learning. J. Amb. Intell. Humaniz. Comput. 11(8), 3187–3207 (2020). https://doi.org/10.1007/s12652-019-01485-x

    Article  Google Scholar 

  38. https://gs.statcounter.com/os-market-share/mobile/worldwide. Accessed 3 Oct 2020

  39. Breazeal, C.: Social robots for health applications. In: 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 5368–5371. IEEE (2011)

  40. Riva, G., Calvo, R A., Lisetti, C.: Cyberpsychology and affective computing. In: The Oxford Handbook of Affective Computing, pp 547–558 (2015)

  41. Marshall, J., Wang, D.: Mood-sensitive truth discovery for reliable recommendation systems in social sensing. In: Proceedings of the 10th ACM conference on recommender systems, pp 167–174 (2016)

  42. Álvarez, P., Zarazaga-Soria, F.J., Baldassarri, S.: Mobile music recommendations for runners based on location and emotions: the dj-running system. Pervasive Mobile Comput. 67, 101242 (2020)

    Article  Google Scholar 

Download references

Funding

The authors have not received any funding for the execution of this work.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, SCMS School of Engineering and Technology, Karukutty, Kerala, India

    Susmi Jacob, Arjun Subramanian & Varun G. Menon

  2. APJ Abdul Kalam Technological University, Thiruvananthapuram, Kerala, India

    Susmi Jacob, Arjun Subramanian & Varun G. Menon

  3. Department of Computer Applications, Cochin University of Science and Technology, Cochin, Kerala, India

    P. Vinod

  4. Department of Mathematics, University of Padua, Padua, Italy

    P. Vinod

Authors
  1. Susmi Jacob
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. P. Vinod
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Arjun Subramanian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Varun G. Menon
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

SJ and PV wrote, revised, and corrected the main manuscript. AS and SJ did data collection, experiments, and analysis. PV. and VGM supervised the conceptualization, ideation, and implementation of the work. All authors reviewed the manuscript

Corresponding author

Correspondence to Susmi Jacob.

Ethics declarations

Conflicts of interest

The authors have no financial or proprietary interest in any material discussed in this article. The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Communicated by B. Bao.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jacob, S., Vinod, P., Subramanian, A. et al. Affect sensing from smartphones through touch and motion contexts. Multimedia Systems 29, 2495–2509 (2023). https://doi.org/10.1007/s00530-023-01142-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-023-01142-6

Keywords

Search

Navigation