Wisdom Web of Things pp 121-153 | Cite as
Ontology-Based Model for Mining User’s Emotions on the Wisdom Web
Abstract
The task of automatically detecting emotion on a web is challenging. This is due to the fact that a traditional web cannot directly interpret the meaning of semantic concepts or assess users emotions. We describe an ontology-based mining model for representation and integration of affect-related knowledge and apply it to detect user’s emotions. This application is a typical use case of the broad-based Wisdom Web of Things (W2T) methodology. The model (named BIO-EMOTION) acts as an integrated framework for: (1) representation and interpretation of affect-related knowledge, including user profile, bio-signal data, situation and environment factors, and (2) supporting intelligent reasoning on users’ emotions. To evaluate the effectiveness of the mining model, we conduct an experiment on a public dataset DEAP and capture a semantic knowledge base expressing both known and deduced knowledge. Evaluation shows that the model not only reaches higher accuracy than other emotion detection results from the same dataset but also achieves a comprehensive affect-related knowledge base which could represent things from both social world, physical world and cyber world in semantics. The ultimate goal of present research is to provide active, transparent, safe and reliable services to web users through their inner emotion. The model implements crucial sub-processes of W2T data cycle: from Things (acquisition of things in the hyper world) to Wisdom (performing intelligent reasoning on web users’ emotion). A long-term goal is to achieve the whole W2T data cycle and to realize a holistic intelligent mining model used on the Wisdom Web.
Keywords
Emotion Recognition Inference Rule Inference Engine Reasoning Rule Emotion DetectionNotes
Acknowledgments
This work was supported by the National Basic Research Program of China (973 Program) (No.2014CB744600), the National Natural Science Foundation of China (No.60973138 and No.61003240), the International Cooperation Project of Ministry of Science and Technology (No.2013DFA11140), the National Basic Research Program of China (973 Program) (No.2011CB711000), and Gansu Provincial Science & Technology Department (No. 1208RJYA015). The authors would like to acknowledge European Community’s Seventh Framework Program (FP7/2007–2011) for their public DEAP database.
References
- 1.Aduna. Sesame (2012). http://www.openrdf.org
- 2.L. Aftanas, N. Reva, A. Varlamov, S. Pavlov, V. Makhnev, Analysis of evoked EEG synchronization and desynchronization in conditions of emotional activation in humans: temporal and topographic characteristics. Neurosci. Behav. Physiol. 34(8), 859–867 (2004)CrossRefGoogle Scholar
- 3.D. Ariely, G.S. Berns, Neuromarketing: the hope and hype of neuroimaging in business. Nat. Rev. Neurosci. 11(4), 284–292 (2010)CrossRefGoogle Scholar
- 4.G.M.M Aurup. User Preference Extraction from Bio-signals: An Experimental Study (Concordia University, 2011)Google Scholar
- 5.Author Anderson Business Consultation, Zukai knowledge management (Toyo Keizai, Tokyo, 1999)Google Scholar
- 6.K.-I. Benta, A. Raru, M. Cremene, Ontology based affective context representation, in Proceedings of the 2007 Euro American Conference on Telematics and Information Systems (ACM, 2007), p. 46Google Scholar
- 7.L.F. Barrett, Are emotions natural kinds? Perspect. Psychol. Sci. 1(1), 28–58 (2006)CrossRefGoogle Scholar
- 8.L.F. Barrett, B. Mesquita, K.N. Ochsner, J.J. Gross, The experience of emotion. Ann. Rev. Psychol. 58, 373 (2007)CrossRefGoogle Scholar
- 9.N. Bourdaud, R. Chavarriaga, F. Galn, J. del R. Millan, Characterizing the EEG correlates of exploratory behavior. IEEE Trans. Neural Syst. Rehabil. Eng. 16(6), 549–556 (2008)Google Scholar
- 10.D. Brickley, R.V. Guha, RDF Vocabulary Description Language 1.0: RDF Schema (2004). http://www.w3.org/TR/rdf-schema/
- 11.A.J. Caas, G. Hill, J. Lott, Support for Constructing Knowledge Models in CmapTools. Technical Report No. IHMC CmapTools 2003-02) (Institute for Human and Machine Cognition, Pensacola, FL, 2003)Google Scholar
- 12.G.A. Calvert, M.J. Brammer, Predicting consumer behavior: using novel mind-reading approaches. IEEE Pulse 3(3), 38–41 (2012)CrossRefGoogle Scholar
- 13.Y. Cao, Z. Cai, E. Shen, W. Shen, X. Chen, G. Gu, T. Shou, Quantitative analysis of brain optical images with 2D C0 complexity measure. J. Neurosci. Methods 159(1), 181–186 (2007)CrossRefGoogle Scholar
- 14.I. Cearreta, J.M. Lpez, N. Garay-Vitoria, Modelling multimodal context-aware affective interaction, in Proceedings of the Doctoral Consortium of the Second international Conference on ACII (2007), pp. 57–64Google Scholar
- 15.G. Chanel, J. Kronegg, D. Grandjean, T. Pun, Emotion assessment: arousal evaluation using EEGs and peripheral physiological signals, in Multimedia Content Representation, Classification and Security (Springer, 2006), pp. 530–537Google Scholar
- 16.H. Chaouchi, The Internet of Things: Connecting Objects (Wiley, 2013)Google Scholar
- 17.F. Chen, J. Xu, F. Gu, X. Yu, X. Meng, Z. Qiu, Dynamic process of information transmission complexity in human brains. Biol. Cybernet. 83(4), 355–366 (2000)CrossRefGoogle Scholar
- 18.Y.-L. Chi, S.-Y. Peng, C.-C. Yang, Creating Kansei engineering-based ontology for annotating and archiving photos database, in Human-Computer Interaction. Interaction Design and Usability (Springer, 2007), pp. 701–710Google Scholar
- 19.S. Claesen, R. Kitney, Estimation of the largest Lyapunov exponent of an RR interval and its use as an indicator of decreased autonomic heart rate control, in Computers in Cardiology 1994 (IEEE, 1994), pp. 133–136Google Scholar
- 20.R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, J.G. Taylor, Emotion recognition in human-computer interaction. IEEE Sign. Process. Mag. 18(1), 32–80 (2001)CrossRefGoogle Scholar
- 21.S.K. D’Mello, A. Graesser, Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Model User-Adap. Inter. 20(2), 147–187 (2010)CrossRefGoogle Scholar
- 22.H.K. Dai, B. Mobasher, Using ontologies to discover domain-level web usage profiles. Semant. Web Min. 35 (2002)Google Scholar
- 23.T. Dalgleish, B.D. Dunn, D. Mobbs, Affective neuroscience: past, present, and future. Emot. Rev. 1(4), 355–368 (2009)CrossRefGoogle Scholar
- 24.T. Dalgleish, M.J. Power, J. Wiley, Handbook of Cognition and Emotion (Wiley Online Library, 1999)Google Scholar
- 25.R.J. Davidson, P. Ekman, C.D. Saron, J.A. Senulis, W.V. Friesen, Approach-withdrawal and cerebral asymmetry: Emotional expression and brain physiology: I. J. Pers. Soc. Psychol. 58(2), 330 (1990)CrossRefGoogle Scholar
- 26.R.J. Davidson, K.R. Scherer, H. Goldsmith, Handbook of Affective Sciences (Oxford University Press, 2003)Google Scholar
- 27.T.S. Dillon, A. Talevski, V. Potdar, E. Chang, Web of things as a framework for ubiquitous intelligence and computing, in Ubiquitous Intelligence and Computing (Springer, 2009), pp. 2–13Google Scholar
- 28.D. Dou, G. Frishkoff, J. Rong, R. Frank, A. Malony, D. Tucker, Development of NeuroElectroMagnetic ontologies (NEMO): a framework for mining brainwave ontologies, in Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2007 (ACM, 2007), pp. 270–279Google Scholar
- 29.M. Eirinaki, M. Vazirgiannis, I. Varlamis, SEWeP: using site semantics and a taxonomy to enhance the Web personalization process, in Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2003 (ACM, 2003), pp. 99–108Google Scholar
- 30.P. Ekman, Emotions in the Human Faces, 2nd edn. (Cambridge University, Press, 1982)Google Scholar
- 31.P. Ekman, Are there Basic Emotions? (1992a)Google Scholar
- 32.P. Ekman, An argument for basic emotions. Cogni. Emot. 6(3–4), 169–200 (1992b)CrossRefGoogle Scholar
- 33.C. Fisher, P. Sanderson, Exploratory sequential data analysis: exploring continuous observational data. Interactions 3(2), 25–34 (1996)CrossRefGoogle Scholar
- 34.V. Francisco, P. Gervs, F. Peinado, Ontological reasoning to configure emotional voice synthesis, in Web Reasoning and Rule Systems (Springer, 2007), pp. 88–102Google Scholar
- 35.P. Fraternali, M. Matera, A. Maurino, Conceptual-level log analysis for the evaluation of web application quality, in Web Congress, 2003. Proceedings. First Latin American, 2003 (IEEE, 2003), pp. 46–57Google Scholar
- 36.V. Galunov, B. Lobanov, N. Zagoruiko, Ontology of the subject domain, in Speech Signals Recognition and Synthesis SPECOM (2004)Google Scholar
- 37.N. Gibbins, S. Harris, N. Shadbolt, Agent-based semantic web services. Web Semant. Sci. Serv. Agents World Wide Web 1(2), 141–154 (2004)CrossRefGoogle Scholar
- 38.P. Grassberger, I. Procaccia, Estimation of the Kolmogorov entropy from a chaotic signal. Phys. Rev. A 28(4), 2591–2593 (1983)CrossRefGoogle Scholar
- 39.F. Hnig, A. Batliner, E. Nth, Real-time recognition of the affective user state with physiological signals, in Proceedings of the Doctoral Consortium, Affective Computing and Intelligent Interaction (2007)Google Scholar
- 40.B. Hjorth, EEG analysis based on time domain properties. Electroencephalogr. Clin. Neurophysiol. 29(3), 306–310 (1970)CrossRefGoogle Scholar
- 41.B. Hjorth, The physical significance of time domain descriptors in EEG analysis. Electroencephalogr. Clin. Neurophysiol. 34(3), 321–325 (1973)CrossRefGoogle Scholar
- 42.T. Inouye, K. Shinosaki, H. Sakamoto, S. Toi, S. Ukai, A. Iyama, Y. Katsuda, M. Hirano, Quantification of EEG irregularity by use of the entropy of the power spectrum. Electroencephalogr. Clin. Neurophysiol. 79(3), 204–210 (1991)CrossRefGoogle Scholar
- 43.Jena. Apache Jena. HP Labs Semantic Web Toolkit (2011). http://jena.sourceforge.net/
- 44.Y. Jing, D. Jeong, D.-K. Baik, SPARQL graph pattern rewriting for OWL-DL inference queries. Knowl. Inf. Syst. 20(2), 243–262 (2009)CrossRefGoogle Scholar
- 45.C.M. Jones, T. Troen, Biometric valence and arousal recognition, in Proceedings of the 19th Australasian Conference on Computer-Human Interaction: Entertaining User Interfaces, 2007 (ACM, 2007), pp. 191–194Google Scholar
- 46.M. Kawasaki, Y. Yamaguchi, Effects of subjective preference of colors on attention-related occipital theta oscillations. Neuroimage 59(1), 808–814 (2012)CrossRefGoogle Scholar
- 47.G. Klyne, J.J. Carroll, Resource Description Framework (RDF): Concepts and Abstract Syntax (2004). http://www.w3.org/TR/rdf-concepts/
- 48.K.-E. Ko, H.-C. Yang, K.-B. Sim, Emotion recognition using EEG signals with relative power values and Bayesian network. Int. J. Control Autom. Syst. 7(5), 865–870 (2009)CrossRefGoogle Scholar
- 49.S. Koelstra, C. Muhl, M. Soleymani, J.-S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, I. Patras, Deap: a database for emotion analysis using physiological signals. IEEE Trans Affect. Comput. 3(1), 18–31 (2012)CrossRefGoogle Scholar
- 50.Kolmogorov. An entropy per unit time as a metric invariant of automorphisms, in Dokl. Akad. Nauk SSSR (1959), pp. 754–755Google Scholar
- 51.M. Kostyunina, M. Kulikov, Frequency characteristics of EEG spectra in the emotions. Neurosci. Behav. Physiol. 26(4), 340–343 (1996)CrossRefGoogle Scholar
- 52.T. Kunii, J. Ma, R. Huang, Hyperworld modeling, in Proceedings of International Conference Visual Information Systems (VIS 96) (1996), pp. 1–8Google Scholar
- 53.J.M. Lpez, R. Gil, R. Garca, I. Cearreta, N. Garay, Towards an ontology for describing emotions, in Emerging Technologies and Information Systems for the Knowledge Society (Springer, 2008), pp. 96–104Google Scholar
- 54.M.D. Lewis, J.M. Haviland-Jones, L.F. Barrett, Handbook of Emotions (Guilford Press, 2010)Google Scholar
- 55.P. Lewis, H. Critchley, P. Rotshtein, R. Dolan, Neural correlates of processing valence and arousal in affective words. Cerebral Cortex 17(3), 742–748 (2007)CrossRefGoogle Scholar
- 56.J. Ma, R. Huang. Improving human interaction with a hyperworld, in Proceedings of the Pacific Workshop on Distributed Multimedia Systems (DMS’96) (1996), pp. 46–50Google Scholar
- 57.A. Maedche, S. Staab, Discovering conceptual relations from text, in Ecai, vol. 325 (2000), p. 27Google Scholar
- 58.A. Maedche, S. Staab, Ontology Learning (Springer, 2004)Google Scholar
- 59.Y.Y. Mathieu, Annotation of emotions and feelings in texts, in Affective Computing and Intelligent Interaction (Springer, 2005), pp. 350–357Google Scholar
- 60.D.L. McGuinness, F. Van Harmelen, OWL Web Ontology Language Overview (W3C Recommendation, 2004)Google Scholar
- 61.A. Mehrabian, J.A. Russell, An Approach to Environmental Psychology (The MIT Press, 1974)Google Scholar
- 62.R. Meo, P.L. Lanzi, M. Matera, R. Esposito, Integrating web conceptual modeling and web usage mining, in Advances in Web Mining and Web Usage Analysis (Springer, 2006), pp. 135–148Google Scholar
- 63.M. Murugappan, M. Rizon, R. Nagarajan, S. Yaacob, D. Hazry, I. Zunaidi, Time-frequency analysis of EEG signals for human emotion detection, in 4th Kuala Lumpur International Conference on Biomedical Engineering 2008 (Springer, 2008), pp. 262–265Google Scholar
- 64.M.A. Nicolaou, H. Gunes, M. Pantic, Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space. IEEE Trans Affect. Comput. 2(2), 92–105 (2011)CrossRefGoogle Scholar
- 65.E. Niedermeyer, The normal EEG of the waking adult, in Electroencephalography: Basic Principles, Clinical Applications, and Related Fields, vol. 167 (2005)Google Scholar
- 66.D. Oberle, B. Berendt, A. Hotho, J. Gonzalez, Conceptual user tracking, in Advances in Web Intelligence (Springer, 2003), pp. 155–164Google Scholar
- 67.R. Ohme, D. Reykowska, D. Wiener, A. Choromanska, Analysis of neurophysiological reactions to advertising stimuli by means of EEG and galvanic skin response measures. J. Neurosci. Psychol. Econ. 2(1), 21 (2009)CrossRefGoogle Scholar
- 68.R. Ohme, D. Reykowska, D. Wiener, A. Choromanska, Application of frontal EEG asymmetry to advertising research. J. Econ. Psychol. 31(5), 785–793 (2010)CrossRefGoogle Scholar
- 69.A.M. Oliveira, M.P. Teixeira, I.B. Fonseca, M. Oliveira, Joint model-parameter validation of self-estimates of valence and arousal: Probing a differential-weighting model of affective intensity, in Proceedings of the 22nd Annual Meeting of the International Society for Psychophysics (2006), pp. 245–250Google Scholar
- 70.M. Pantic, L.J. Rothkrantz, Toward an affect-sensitive multimodal human-computer interaction. Proc. IEEE 91(9), 1370–1390 (2003)CrossRefGoogle Scholar
- 71.H. Peng, B. Hu, Q. Liu, Q. Dong, Q. Zhao, P. Moore, User-centered depression prevention: an EEG approach to pervasive healthcare, in 2011 IEEE 5th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) (2011a), pp. 325–330Google Scholar
- 72.H. Peng, B. Hu, Y. Qi, Q. Zhao, M. Ratcliffe, An improved EEG de-noising approach in electroencephalogram (EEG) for home care, in 2011 IEEE 5th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) (2011b), pp. 469–474Google Scholar
- 73.P.C. Petrantonakis, L.J. Hadjileontiadis, Adaptive extraction of emotion-related EEG segments using multidimensional directed information in time-frequency domain, in Engineering in Medicine and Biology Society (EMBC), 2010 Annual International Conference of the IEEE (IEEE, 2010), pp. 1–4Google Scholar
- 74.R.W. Picard, Affective computing: challenges. Int. J. Hum.-Comput. Stud. 59(1), 55–64 (2003)CrossRefGoogle Scholar
- 75.R.W. Picard, E. Vyzas, J. Healey, Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Pattern Anal. Mach. Intell. 23(10), 1175–1191 (2001)CrossRefGoogle Scholar
- 76.R. Pivik, R. Broughton, R. Coppola, R. Davidson, N. Fox, M. Nuwer, Guidelines for the recording and quantitative analysis of electroencephalographic activity in research contexts. Psychophysiology 30(6), 547–558 (1993)CrossRefGoogle Scholar
- 77.J.R. Quinlan, C4. 5: Programs for Machine Learning, vol. 1 (Morgan kaufmann, 1993)Google Scholar
- 78.M.T. Rosenstein, J.J. Collins, C.J. De Luca, A practical method for calculating largest Lyapunov exponents from small data sets. Physica D: Nonlin. Phenomena 65(1), 117–134 (1993)MathSciNetMATHCrossRefGoogle Scholar
- 79.J.A. Russell, Core affect and the psychological construction of emotion. Psychol. Rev. 110(1), 145 (2003)CrossRefGoogle Scholar
- 80.J.A. Russell, M. Lewicka, T. Niit, A cross-cultural study of a circumplex model of affect. J. Pers. Soc. Psychol. 57(5), 848 (1989)CrossRefGoogle Scholar
- 81.S.J. Russell, P. Norvig, J.F. Canny, J.M. Malik, D.D. Edwards, Artificial Intelligence: A Modern Approach, vol. 74 (Prentice hall Englewood Cliffs, 1995)Google Scholar
- 82.M. Sabeti, R. Boostani, S. Katebi, G. Price, Selection of relevant features for EEG signal classification of schizophrenic patients. Biomed. Sign. Process. Control 2(2), 122–134 (2007)CrossRefGoogle Scholar
- 83.P. Salovey, J.D. Mayer, Emotional intelligence. Imagination Cogn. Pers. 9(3), 185–211 (1989)CrossRefGoogle Scholar
- 84.N. Sebe, I. Cohen, T.S. Huang, Multimodal emotion recognition. Handb. Pattern Recogn. Comput. Vis. 4, 387–419 (2005)CrossRefGoogle Scholar
- 85.M. Shamsfard, A.A. Barforoush, Learning ontologies from natural language texts. Int. J. Hum.-Comput. Stud. 60(1), 17–63 (2004)CrossRefGoogle Scholar
- 86.C.E. Shannon, A mathematical theory of communication. ACM SIGMOBILE Mobile Comput. Commun. Rev. 5(1), 3–55 (2001)MathSciNetCrossRefGoogle Scholar
- 87.C. Stickel, M. Ebner, S. Steinbach-Nordmann, G. Searle, A. Holzinger, Emotion detection: application of the valence arousal space for rapid biological usability testing to enhance universal access, in Universal Access in Human-Computer Interaction. Addressing Diversity (Springer, 2009), pp. 615–624Google Scholar
- 88.V. Stirbu, Towards a restful plug and play experience in the web of things, in 2008 IEEE International Conference on Semantic Computing (IEEE, 2008), pp. 512–517Google Scholar
- 89.A.J. Tomarken, R.J. Davidson, R.E. Wheeler, L. Kinney, Psychometric properties of resting anterior EEG asymmetry: temporal stability and internal consistency. Psychophysiology 29(5), 576–592 (1992)CrossRefGoogle Scholar
- 90.J. Wackermann, Towards a quantitative characterisation of functional states of the brain: from the non-linear methodology to the global linear description. Int. J. Psychophysiol. 34(1), 65–80 (1999)CrossRefGoogle Scholar
- 91.J. Wackermann, C. Allefeld, On the meaning and interpretation of global descriptors of brain electrical activity. Including a reply to X. Pei, et al. Int. J. Psychophysiol. 64(2), 199–210 (2007)CrossRefGoogle Scholar
- 92.T. Wehrle, K.R. Scherer, Towards computational modeling of appraisal theories, in Appraisal Processes in Emotion: Theory, Methods, Research (Oxford University Press, New York, 2001)Google Scholar
- 93.E. Welbourne, L. Battle, G. Cole, K. Gould, K. Rector, S. Raymer, M. Balazinska, G. Borriello, Building the internet of things using RFID: the RFID ecosystem experience. IEEE Internet Comput. 13(3), 48–55 (2009)CrossRefGoogle Scholar
- 94.Y-H. Yang, Y-C. Lin, Y-F, Su, H.H. Chen, Music emotion classification: a regression approach, in 2007 IEEE International Conference on Multimedia and Expo (IEEE, 2007), pp. 208–211Google Scholar
- 95.E.Yokomatsu, S-i. Ito, Y. Mitsukura, J. Cao, M.A. Fukumi, Design of the preference acquisition detection system, in SICE, 2007 Annual Conference (IEEE, 2007), pp. 2804–2807Google Scholar
- 96.C. Yu, P.M. Aoki, A. Woodruff, Detecting User Engagement in Everyday Conversations. arXiv preprint cs/0410027 (2004)Google Scholar
- 97.Z. Zeng, M. Pantic, G.I. Roisman, T.S. Huang, A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31(1), 39–58 (2009)CrossRefGoogle Scholar
- 98.X.-S. Zhang, R.J. Roy, Derived fuzzy knowledge model for estimating the depth of anesthesia. IEEE Trans. Biomed. Eng. 48(3), 312–323 (2001)CrossRefGoogle Scholar
- 99.N. Zhong, J. Liu, Y. Yao, In search of the wisdom web. IEEE Comput. 35(11), 27–31 (2002)CrossRefGoogle Scholar
- 100.N. Zhong, J. Liu, Y. Yao. Web intelligence (WI): a new paradigm for developing the Wisdom Web and social network intelligence, in Web Intelligence (Springer, 2003), pp. 1–16Google Scholar
- 101.N. Zhong, J.H. Ma, R.H. Huang, J.M. Liu, Y.Y. Yao, Y.X. Zhang, J.H. Chen, Research challenges and perspectives on Wisdom Web of Things (W2T). J. Supercomput. 1–21 (2013)Google Scholar