Enhancing Text Using Emotion Detected from EEG Signals

Abstract

Often people might not be able to express themselves properly on social media, like not being able to think of appropriate words representative of their emotional state. In this paper, we propose an end to end system which aims to enhance user-input sentence according to his/her current emotional state. It works by a) detecting the emotion of the user and b) enhancing the input sentence by inserting emotive words to make the sentence more representative of the emotional state of the user. The emotional state of the user is recognized by analyzing the Electroencephalogram (EEG) signals from the brain. For text enhancement, we modify the words corresponding to the detected emotion using correlation finder scheme. Next, the verification of the sentence correctness has been performed using Long Short Term Memory (LSTM) Networks based Language Modeling framework. An accuracy of 74.95% has been recorded for the classification of five emotional states in a dataset of 25 participants using EEG signals. Similarly, promising results have been obtained for the task text enhancement and overall end-to-end system. To the best of our knowledge, this work is the first of its kind to enhance text according to the emotional state detected by EEG brainwaves. The system also releases an individual from thinking and typing words, which might be a complicated procedure sometimes.

This is a preview of subscription content, access via your institution.

References

  1. 1.

    Agarwal, A., Xie, B., Vovsha, I., Rambow, O., Passonneau, R.: Sentiment analysis of twitter data. In: Proceedings of the workshop on languages in social media. Association for Computational Linguistics, pp. 30–38 (2011)

  2. 2.

    Almehmadi, A., Bourque, M., El-Khatib, K.: A Tweet of the Mind: Automated Emotion Detection for Social Media Using Brain Wave Pattern Analysis. In: International Conference on Social Computing, pp. 987–991 (2013)

  3. 3.

    Amarasinghe, K., Sivils, P., Manic, M.: Eeg Feature Selection for Thought Driven Robots Using Evolutionary Algorithms. In: 9th International Conference on Human System Interactions, pp. 355–361 (2016)

  4. 4.

    Becker, K., Moreira, V.P., dos Santos, A.G.: Multilingual emotion classification using supervised learning: Comparative experiments. Inf. Process. Manag. 53(3), 684–704 (2017)

    Article  Google Scholar 

  5. 5.

    Bird, S., Klein, E., Loper, E: Natural language processing with Python: analyzing text with the natural language toolkit. ”O’Reilly Media Inc.” (2009)

  6. 6.

    Blaiech, H., Neji, M., Wali, A., Alimi, A.M.: Emotion Recognition by Analysis of Eeg Signals. In: 13th International Conference on Hybrid Intelligent Systems, pp. 312–318 (2013)

  7. 7.

    Boldrini, E., Balahur Dobrescu, A., Martínez-barco, P., Montoyo, A., et al.: Emotiblog: a fine-grained model for emotion detection in non-traditional textual genres (2009)

  8. 8.

    Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., Robinson, T.: One billion word benchmark for measuring progress in statistical language modeling. arXiv:1312.3005 (2013)

  9. 9.

    Cherry, K.: The Everything Psychology Book: Explore the human psyche and understand why we do the things we do. Simon and Schuster, New York (2010)

    Google Scholar 

  10. 10.

    De Belder, J., Moens, M.F.: Text Simplification for Children. In: Prroceedings of the SIGIR Workshop on Accessible Search Systems, pp. 19–26 (2010)

  11. 11.

    Derczynski, L., Ritter, A., Clark, S., Bontcheva, K.: Twitter Part-Of-Speech Tagging for All: Overcoming Sparse and Noisy Data. In: RANLP, pp. 198–206 (2013)

  12. 12.

    Fattouh, A., Albidewi, I., Baterfi, B.: Eeg-Based Emotion Recognition of Quran Listeners. In: 3rd International Conference on Computing for Sustainable Global Development, pp. 1338–1342 (2016)

  13. 13.

    Furuta, R., Plaisant, C., Shneiderman, B.: Automatically transforming regularly structured linear documents into hypertext. Electron. Publ. 2(4), 211–229 (1989)

    Google Scholar 

  14. 14.

    Gers, F.A., Schmidhuber, E.: Lstm recurrent networks learn simple context-free and context-sensitive languages. IEEE Trans. Neural Netw. 12(6), 1333–1340 (2001)

    Article  Google Scholar 

  15. 15.

    Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Netw. 18(5), 602–610 (2005)

    Article  Google Scholar 

  16. 16.

    Graves, A., Schmidhuber, J.: Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks. In: Advances in Neural Information Processing Systems, pp. 545–552 (2009)

  17. 17.

    Higuchi, T.: Approach to an irregular time series on the basis of the fractal theory. Physica D: Nonlinear Phenom. 31(2), 277–283 (1988)

    MathSciNet  Article  MATH  Google Scholar 

  18. 18.

    Hochreiter, S., Heusel, M., Obermayer, K.: Fast model-based protein homology detection without alignment. Bioinformatics 23(14), 1728–1736 (2007)

    Article  Google Scholar 

  19. 19.

    Jozefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., Wu, Y.: Exploring the limits of language modeling. arXiv:1602.02410 (2016)

  20. 20.

    Kaur, B., Singh, D., Roy, P.P.: A novel framework of eeg- based user identification by analyzing music-listening behavior. Multimedia Tools and Applications 76(24), 1–22 (2016)

    Google Scholar 

  21. 21.

    Kim, K.H., Bang, S.W., Kim, S.R.: Emotion recognition system using short-term monitoring of physiological signals. Med. Biol. Eng. Comput. 42(3), 419–427 (2004)

    Article  Google Scholar 

  22. 22.

    Koelstra, S., Muhl, C., Soleymani, M., Lee, J.S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., Patras, I.: Deap: a database for emotion analysis; using physiological signals. IEEE Trans. Affect. Comput. 3(1), 18–31 (2012)

    Article  Google Scholar 

  23. 23.

    Kumar, P., Roy, P.P., Dogra, D.P.: Independent bayesian classifier combination based sign language recognition using facial expression. Inform. Sci. 428, 30–48 (2018)

    MathSciNet  Article  Google Scholar 

  24. 24.

    Kumar, P., Saini, R., Roy, P.P., Dogra, D.P.: A bio-signal based framework to secure mobile devices. J. Netw. Comput. Appl. 89, 62–71 (2017)

    Article  Google Scholar 

  25. 25.

    Li, M., Lu, B.L.: Emotion Classification Based on Gamma-Band Eeg. In: International Conference of the Engineering in Medicine and Biology Society, pp. 1223–1226 (2009)

  26. 26.

    Liaw, A., Wiener, M., et al.: Classification and regression by randomforest. R News 2(3), 18–22 (2002)

    Google Scholar 

  27. 27.

    Lin, Y.P., Wang, C.H., Jung, T.P., Wu, T.L., Jeng, S.K., Duann, J.R., Chen, J.H.: Eeg-based emotion recognition in music listening. IEEE Trans. Biomed. Eng. 57(7), 1798–1806 (2010)

    Article  Google Scholar 

  28. 28.

    Liu, F., Weng, F., Wang, B., Liu, Y.: Insertion, Deletion, Or Substitution?: Normalizing Text Messages without Pre-Categorization Nor Supervision. In: 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers. Association for Computational Linguistics, Vol. 2, pp. 71–76 (2011)

  29. 29.

    Liu, Y., Sourina, O., Nguyen, M.K.: Real-Time Eeg-Based Human Emotion Recognition and Visualization. In: International Conference on Cyberworlds, pp. 262–269. IEEE (2010)

  30. 30.

    Liu, Y., Sourina, O., Nguyen, M.K.: Real-Time Eeg-Based Emotion Recognition and Its Applications. In: Transactions on Computational Science XII. Springer, pp. 256–277 (2011)

  31. 31.

    Matlovic, T., Gaspar, P., Moro, R., Simko, J., Bielikova, M.: Emotions Detection Using Facial Expressions Recognition and Eeg. In: 11th International Workshop on Semantic and Social Media Adaptation and Personalization, pp. 18–23 (2016)

  32. 32.

    Mohammad, S.M., Kiritchenko, S.: Using hashtags to capture fine emotion categories from tweets. Comput. Intell. 31(2), 301–326 (2015)

    MathSciNet  Article  Google Scholar 

  33. 33.

    Mohammad, S.M., Turney, P.D.: Crowdsourcing a word–emotion association lexicon. Comput. Intell. 29(3), 436–465 (2013)

    MathSciNet  Article  Google Scholar 

  34. 34.

    Parapar, J., Bellogín, A., Castells, P., Barreiro, Á.: Relevance-based language modelling for recommender systems. Inf. Process. Manag. 49(4), 966–980 (2013)

    Article  Google Scholar 

  35. 35.

    Petrantonakis, P.C., Hadjileontiadis, L.J.: Emotion recognition from eeg using higher order crossings. IEEE Trans. Inf. Technol. Biomed. 14(2), 186–197 (2010)

    Article  Google Scholar 

  36. 36.

    Roy, R.S., Agarwal, S., Ganguly, N., Choudhury, M.: Syntactic complexity of web search queries through the lenses of language models, networks and users. Inf. Process. Manag. 52(5), 923–948 (2016)

    Article  Google Scholar 

  37. 37.

    Roy, R.S., Padmakumar, A., Jeganathan, G.P., Kumaraguru, P.: Automated Linguistic Personalization of Targeted Marketing Messages Mining User-Generated Text on Social Media. In: International Conference on Intelligent Text Processing and Computational Linguistics. Springer, pp. 203–224 (2015)

  38. 38.

    Saini, R., Kaur, B., Singh, P., Kumar, P., Roy, P.P., Raman, B., Singh, D.: Don’t just sign use brain too: A novel multimodal approach for user identification and verification. Information Sciences (2017)

  39. 39.

    Savitzky, A., Golay, M.J.: Smoothing and differentiation of data by simplified least squares procedures. Anal. Chem. 36(8), 1627–1639 (1964)

    Article  Google Scholar 

  40. 40.

    Schmidhuber, J., Wierstra, D., Gomez, F.J.: Evolino: Hybrid Neuroevolution/Optimal Linear Search for Sequence Prediction. In: 19th International Joint Conference on Artificial Intelligence (2005)

  41. 41.

    Schuller, B., Reiter, S., Muller, R., Al-Hames, M., Lang, M., Rigoll, G.: Speaker Independent Speech Emotion Recognition by Ensemble Classification. In: International Conference on Multimedia and Expo. IEEE, pp. 864–867 (2005)

  42. 42.

    Shaver, P., Schwartz, J., Kirson, D., O’connor, C.: Emotion knowledge: further exploration of a prototype approach. J. Personal. Soc. Psychol. 52(6), 1061 (1987)

    Article  Google Scholar 

  43. 43.

    Smedt, T.D., Daelemans, W.: Pattern for python. J. Mach. Learn. Res. 13(Jun), 2063–2067 (2012)

    MATH  Google Scholar 

  44. 44.

    Soleymani, M., Pantic, M., Pun, T.: Multimodal emotion recognition in response to videos. IEEE Trans. Affect. Comput. 3(2), 211–223 (2012)

    Article  Google Scholar 

  45. 45.

    Teplan, M., et al.: Fundamentals of eeg measurement. Measur. Sci. Rev. 2(2), 1–11 (2002)

    Google Scholar 

  46. 46.

    Thuy, P.T.T., Lee, Y.K., Lee, S.: Dtd2owl: Automatic Transforming Xml Documents into Owl Ontology. In: 2nd International Conference on Interaction Sciences: Information Technology, Culture and Human, pp. 125–131 (2009)

  47. 47.

    Toutanova, K., Manning, C.D.: Enriching the Knowledge Sources Used in a Maximum Entropy Part-Of-Speech Tagger. In: Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora: Held in Conjunction with the 38th Annual Meeting of the Association for Computational Linguistics, pp. 63–70 (2000)

  48. 48.

    Tsou, M.H.: Research challenges and opportunities in mapping social media and big data. Cartogr. Geogr. Inf. Sci. 42(sup1), 70–74 (2015)

    Article  Google Scholar 

  49. 49.

    Vogt, T., André, E., Bee, N.: Emovoice—a framework for online recognition of emotions from voice. Perception in Multimodal Dialogue Systems, 188–199 (2008)

  50. 50.

    Wang, S., Gwizdka, J., Chaovalitwongse, W.A.: Using wireless eeg signals to assess memory workload in the n-back task. IEEE Trans. Human-Mach. Syst. 46(3), 424–435 (2016)

    Article  Google Scholar 

  51. 51.

    Wang, W., Chen, L., Thirunarayan, K., Sheth, A.P.: Harnessing Twitter” Big Data” for Automatic Emotion Identification. In: International Conference on Privacy, Security, Risk and Trust and International Confernece on Social Computing (Socialcom), pp. 587–592 (2012)

  52. 52.

    Wu, X., Zhu, X., Wu, G.Q., Ding, W.: Data mining with big data. IEEE Trans. Knowl. Data Eng. 26(1), 97–107 (2014)

    Article  Google Scholar 

  53. 53.

    Yang, C., Lin, K.H.Y., Chen, H.H.: Building Emotion Lexicon from Weblog Corpora. In: 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, pp. 133–136 (2007)

Download references

Acknowledgments

We wish to acknowledge Mr. Harvineet Singh from Adobe Systems India for his guidance towards solving the problem.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Akash Gupta.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Gupta, A., Sahu, H., Nanecha, N. et al. Enhancing Text Using Emotion Detected from EEG Signals. J Grid Computing 17, 325–340 (2019). https://doi.org/10.1007/s10723-018-9462-2

Download citation

Keywords

  • Electroencephalography (EEG)
  • LSTM
  • Language modeling
  • Knowledge discovery
  • Emotion