Advertisement

Enhancing Text Using Emotion Detected from EEG Signals

  • Akash Gupta
  • Harsh Sahu
  • Nihal Nanecha
  • Pradeep Kumar
  • Partha Pratim Roy
  • Victor Chang
Article
  • 29 Downloads

Abstract

Often people might not be able to express themselves properly on social media, like not being able to think of appropriate words representative of their emotional state. In this paper, we propose an end to end system which aims to enhance user-input sentence according to his/her current emotional state. It works by a) detecting the emotion of the user and b) enhancing the input sentence by inserting emotive words to make the sentence more representative of the emotional state of the user. The emotional state of the user is recognized by analyzing the Electroencephalogram (EEG) signals from the brain. For text enhancement, we modify the words corresponding to the detected emotion using correlation finder scheme. Next, the verification of the sentence correctness has been performed using Long Short Term Memory (LSTM) Networks based Language Modeling framework. An accuracy of 74.95% has been recorded for the classification of five emotional states in a dataset of 25 participants using EEG signals. Similarly, promising results have been obtained for the task text enhancement and overall end-to-end system. To the best of our knowledge, this work is the first of its kind to enhance text according to the emotional state detected by EEG brainwaves. The system also releases an individual from thinking and typing words, which might be a complicated procedure sometimes.

Keywords

Electroencephalography (EEG) LSTM Language modeling Knowledge discovery Emotion 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Notes

Acknowledgments

We wish to acknowledge Mr. Harvineet Singh from Adobe Systems India for his guidance towards solving the problem.

References

  1. 1.
    Agarwal, A., Xie, B., Vovsha, I., Rambow, O., Passonneau, R.: Sentiment analysis of twitter data. In: Proceedings of the workshop on languages in social media. Association for Computational Linguistics, pp. 30–38 (2011)Google Scholar
  2. 2.
    Almehmadi, A., Bourque, M., El-Khatib, K.: A Tweet of the Mind: Automated Emotion Detection for Social Media Using Brain Wave Pattern Analysis. In: International Conference on Social Computing, pp. 987–991 (2013)Google Scholar
  3. 3.
    Amarasinghe, K., Sivils, P., Manic, M.: Eeg Feature Selection for Thought Driven Robots Using Evolutionary Algorithms. In: 9th International Conference on Human System Interactions, pp. 355–361 (2016)Google Scholar
  4. 4.
    Becker, K., Moreira, V.P., dos Santos, A.G.: Multilingual emotion classification using supervised learning: Comparative experiments. Inf. Process. Manag. 53(3), 684–704 (2017)CrossRefGoogle Scholar
  5. 5.
    Bird, S., Klein, E., Loper, E: Natural language processing with Python: analyzing text with the natural language toolkit. ”O’Reilly Media Inc.” (2009)Google Scholar
  6. 6.
    Blaiech, H., Neji, M., Wali, A., Alimi, A.M.: Emotion Recognition by Analysis of Eeg Signals. In: 13th International Conference on Hybrid Intelligent Systems, pp. 312–318 (2013)Google Scholar
  7. 7.
    Boldrini, E., Balahur Dobrescu, A., Martínez-barco, P., Montoyo, A., et al.: Emotiblog: a fine-grained model for emotion detection in non-traditional textual genres (2009)Google Scholar
  8. 8.
    Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., Robinson, T.: One billion word benchmark for measuring progress in statistical language modeling. arXiv:1312.3005 (2013)
  9. 9.
    Cherry, K.: The Everything Psychology Book: Explore the human psyche and understand why we do the things we do. Simon and Schuster, New York (2010)Google Scholar
  10. 10.
    De Belder, J., Moens, M.F.: Text Simplification for Children. In: Prroceedings of the SIGIR Workshop on Accessible Search Systems, pp. 19–26 (2010)Google Scholar
  11. 11.
    Derczynski, L., Ritter, A., Clark, S., Bontcheva, K.: Twitter Part-Of-Speech Tagging for All: Overcoming Sparse and Noisy Data. In: RANLP, pp. 198–206 (2013)Google Scholar
  12. 12.
    Fattouh, A., Albidewi, I., Baterfi, B.: Eeg-Based Emotion Recognition of Quran Listeners. In: 3rd International Conference on Computing for Sustainable Global Development, pp. 1338–1342 (2016)Google Scholar
  13. 13.
    Furuta, R., Plaisant, C., Shneiderman, B.: Automatically transforming regularly structured linear documents into hypertext. Electron. Publ. 2(4), 211–229 (1989)Google Scholar
  14. 14.
    Gers, F.A., Schmidhuber, E.: Lstm recurrent networks learn simple context-free and context-sensitive languages. IEEE Trans. Neural Netw. 12(6), 1333–1340 (2001)CrossRefGoogle Scholar
  15. 15.
    Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Netw. 18(5), 602–610 (2005)CrossRefGoogle Scholar
  16. 16.
    Graves, A., Schmidhuber, J.: Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks. In: Advances in Neural Information Processing Systems, pp. 545–552 (2009)Google Scholar
  17. 17.
    Higuchi, T.: Approach to an irregular time series on the basis of the fractal theory. Physica D: Nonlinear Phenom. 31(2), 277–283 (1988)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Hochreiter, S., Heusel, M., Obermayer, K.: Fast model-based protein homology detection without alignment. Bioinformatics 23(14), 1728–1736 (2007)CrossRefGoogle Scholar
  19. 19.
    Jozefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., Wu, Y.: Exploring the limits of language modeling. arXiv:1602.02410 (2016)
  20. 20.
    Kaur, B., Singh, D., Roy, P.P.: A novel framework of eeg- based user identification by analyzing music-listening behavior. Multimedia Tools and Applications 76(24), 1–22 (2016)Google Scholar
  21. 21.
    Kim, K.H., Bang, S.W., Kim, S.R.: Emotion recognition system using short-term monitoring of physiological signals. Med. Biol. Eng. Comput. 42(3), 419–427 (2004)CrossRefGoogle Scholar
  22. 22.
    Koelstra, S., Muhl, C., Soleymani, M., Lee, J.S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., Patras, I.: Deap: a database for emotion analysis; using physiological signals. IEEE Trans. Affect. Comput. 3(1), 18–31 (2012)CrossRefGoogle Scholar
  23. 23.
    Kumar, P., Roy, P.P., Dogra, D.P.: Independent bayesian classifier combination based sign language recognition using facial expression. Inform. Sci. 428, 30–48 (2018)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Kumar, P., Saini, R., Roy, P.P., Dogra, D.P.: A bio-signal based framework to secure mobile devices. J. Netw. Comput. Appl. 89, 62–71 (2017)CrossRefGoogle Scholar
  25. 25.
    Li, M., Lu, B.L.: Emotion Classification Based on Gamma-Band Eeg. In: International Conference of the Engineering in Medicine and Biology Society, pp. 1223–1226 (2009)Google Scholar
  26. 26.
    Liaw, A., Wiener, M., et al.: Classification and regression by randomforest. R News 2(3), 18–22 (2002)Google Scholar
  27. 27.
    Lin, Y.P., Wang, C.H., Jung, T.P., Wu, T.L., Jeng, S.K., Duann, J.R., Chen, J.H.: Eeg-based emotion recognition in music listening. IEEE Trans. Biomed. Eng. 57(7), 1798–1806 (2010)CrossRefGoogle Scholar
  28. 28.
    Liu, F., Weng, F., Wang, B., Liu, Y.: Insertion, Deletion, Or Substitution?: Normalizing Text Messages without Pre-Categorization Nor Supervision. In: 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers. Association for Computational Linguistics, Vol. 2, pp. 71–76 (2011)Google Scholar
  29. 29.
    Liu, Y., Sourina, O., Nguyen, M.K.: Real-Time Eeg-Based Human Emotion Recognition and Visualization. In: International Conference on Cyberworlds, pp. 262–269. IEEE (2010)Google Scholar
  30. 30.
    Liu, Y., Sourina, O., Nguyen, M.K.: Real-Time Eeg-Based Emotion Recognition and Its Applications. In: Transactions on Computational Science XII. Springer, pp. 256–277 (2011)Google Scholar
  31. 31.
    Matlovic, T., Gaspar, P., Moro, R., Simko, J., Bielikova, M.: Emotions Detection Using Facial Expressions Recognition and Eeg. In: 11th International Workshop on Semantic and Social Media Adaptation and Personalization, pp. 18–23 (2016)Google Scholar
  32. 32.
    Mohammad, S.M., Kiritchenko, S.: Using hashtags to capture fine emotion categories from tweets. Comput. Intell. 31(2), 301–326 (2015)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Mohammad, S.M., Turney, P.D.: Crowdsourcing a word–emotion association lexicon. Comput. Intell. 29(3), 436–465 (2013)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Parapar, J., Bellogín, A., Castells, P., Barreiro, Á.: Relevance-based language modelling for recommender systems. Inf. Process. Manag. 49(4), 966–980 (2013)CrossRefGoogle Scholar
  35. 35.
    Petrantonakis, P.C., Hadjileontiadis, L.J.: Emotion recognition from eeg using higher order crossings. IEEE Trans. Inf. Technol. Biomed. 14(2), 186–197 (2010)CrossRefGoogle Scholar
  36. 36.
    Roy, R.S., Agarwal, S., Ganguly, N., Choudhury, M.: Syntactic complexity of web search queries through the lenses of language models, networks and users. Inf. Process. Manag. 52(5), 923–948 (2016)CrossRefGoogle Scholar
  37. 37.
    Roy, R.S., Padmakumar, A., Jeganathan, G.P., Kumaraguru, P.: Automated Linguistic Personalization of Targeted Marketing Messages Mining User-Generated Text on Social Media. In: International Conference on Intelligent Text Processing and Computational Linguistics. Springer, pp. 203–224 (2015)Google Scholar
  38. 38.
    Saini, R., Kaur, B., Singh, P., Kumar, P., Roy, P.P., Raman, B., Singh, D.: Don’t just sign use brain too: A novel multimodal approach for user identification and verification. Information Sciences (2017)Google Scholar
  39. 39.
    Savitzky, A., Golay, M.J.: Smoothing and differentiation of data by simplified least squares procedures. Anal. Chem. 36(8), 1627–1639 (1964)CrossRefGoogle Scholar
  40. 40.
    Schmidhuber, J., Wierstra, D., Gomez, F.J.: Evolino: Hybrid Neuroevolution/Optimal Linear Search for Sequence Prediction. In: 19th International Joint Conference on Artificial Intelligence (2005)Google Scholar
  41. 41.
    Schuller, B., Reiter, S., Muller, R., Al-Hames, M., Lang, M., Rigoll, G.: Speaker Independent Speech Emotion Recognition by Ensemble Classification. In: International Conference on Multimedia and Expo. IEEE, pp. 864–867 (2005)Google Scholar
  42. 42.
    Shaver, P., Schwartz, J., Kirson, D., O’connor, C.: Emotion knowledge: further exploration of a prototype approach. J. Personal. Soc. Psychol. 52(6), 1061 (1987)CrossRefGoogle Scholar
  43. 43.
    Smedt, T.D., Daelemans, W.: Pattern for python. J. Mach. Learn. Res. 13(Jun), 2063–2067 (2012)zbMATHGoogle Scholar
  44. 44.
    Soleymani, M., Pantic, M., Pun, T.: Multimodal emotion recognition in response to videos. IEEE Trans. Affect. Comput. 3(2), 211–223 (2012)CrossRefGoogle Scholar
  45. 45.
    Teplan, M., et al.: Fundamentals of eeg measurement. Measur. Sci. Rev. 2(2), 1–11 (2002)Google Scholar
  46. 46.
    Thuy, P.T.T., Lee, Y.K., Lee, S.: Dtd2owl: Automatic Transforming Xml Documents into Owl Ontology. In: 2nd International Conference on Interaction Sciences: Information Technology, Culture and Human, pp. 125–131 (2009)Google Scholar
  47. 47.
    Toutanova, K., Manning, C.D.: Enriching the Knowledge Sources Used in a Maximum Entropy Part-Of-Speech Tagger. In: Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora: Held in Conjunction with the 38th Annual Meeting of the Association for Computational Linguistics, pp. 63–70 (2000)Google Scholar
  48. 48.
    Tsou, M.H.: Research challenges and opportunities in mapping social media and big data. Cartogr. Geogr. Inf. Sci. 42(sup1), 70–74 (2015)CrossRefGoogle Scholar
  49. 49.
    Vogt, T., André, E., Bee, N.: Emovoice—a framework for online recognition of emotions from voice. Perception in Multimodal Dialogue Systems, 188–199 (2008)Google Scholar
  50. 50.
    Wang, S., Gwizdka, J., Chaovalitwongse, W.A.: Using wireless eeg signals to assess memory workload in the n-back task. IEEE Trans. Human-Mach. Syst. 46(3), 424–435 (2016)CrossRefGoogle Scholar
  51. 51.
    Wang, W., Chen, L., Thirunarayan, K., Sheth, A.P.: Harnessing Twitter” Big Data” for Automatic Emotion Identification. In: International Conference on Privacy, Security, Risk and Trust and International Confernece on Social Computing (Socialcom), pp. 587–592 (2012)Google Scholar
  52. 52.
    Wu, X., Zhu, X., Wu, G.Q., Ding, W.: Data mining with big data. IEEE Trans. Knowl. Data Eng. 26(1), 97–107 (2014)CrossRefGoogle Scholar
  53. 53.
    Yang, C., Lin, K.H.Y., Chen, H.H.: Building Emotion Lexicon from Weblog Corpora. In: 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, pp. 133–136 (2007)Google Scholar

Copyright information

© Springer Nature B.V. 2018

Authors and Affiliations

  1. 1.Department of Computer Science & Engineering, IIT RoorkeeRoorkeeIndia
  2. 2.International Business School Suzhou and Research Institute of Big Data AnalyticsXi’an Jiaotong-Liverpool UniversitySuzhouChina

Personalised recommendations