Abstract
Multimedia documents such as pictures, videos, sounds and text provoke emotional responses of different intensity and polarity. These stimuli are stored in affective multimedia databases together with description of their semantics based on keywords from unsupervised glossaries, expected emotion elicitation potential and other important contextual information. Affective multimedia databases are important in many different areas of research, such as affective computing, human-computer interaction and cognitive sciences, where it is necessary to deliberately modulate emotional states of individuals. However, restrictions in the employed semantic data models impair retrieval performance measures thus severely limiting the databases’ overall usability. An experimental evaluation of multi-keyword search in affective multimedia databases, using lift charts as binomial classifiers optimized for retrieval precision or sensitivity, is presented. Suggestions for improving expressiveness and formality of data models are elaborated, as well as introduction of dedicated ontologies which could lead to better data interoperability.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Horvat, M., Popović, S., Bogunović, N., Ćosić, K.: Tagging multimedia stimuli with ontologies. In: Proceedings of the 32nd International Convention MIPRO 2009: Computers in Technical Systems, pp. 203 − 208. Intelligent Systems, Opatija (2009)
Horvat, M., Popović, S., Ćosić, K.: Multimedia stimuli databases usage patterns: a survey report. In: Proceedings of the 36th International Convention MIPRO 2013: Computers in Technical Systems, pp. 993 − 997. Intelligent Systems, Opatija (2013)
Brave, S., Nass, C.: Emotion in human-computer interaction. In: The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, pp. 81 − 96. CRC Press, Taylor & Francis, Florida (2002)
Coan, J.A., Allen, J.J.B.: The Handbook of Emotion Elicitation and Assessment. Oxford University Press Series in Affective Science. Oxford University Press, New York (2007)
Grandjean, D., Sander, D., Scherer, K.R.: Conscious emotional experience emerges as a function of multilevel, appraisal-driven response synchronization. Conscious. Cogn. 17, 484–495 (2008)
Ćosić, K., Popović, S., Horvat, M., Kukolja, D., Dropuljić, B., Kovač, B., Jakovljević, M.: Computer-aided psychotherapy based on multimodal elicitation, estimation and regulation of emotion. Psychiatr. Danub. 25, 340–346 (2013)
Lang, P.J., Bradley, M. M., Cuthbert, B.N.: International affective picture system (IAPS): Affective ratings of pictures and instruction manual. Technical report A − 8. University of Florida, Gainesville, FL (2008)
Lang, P. J., Bradley, M. M.: The International Affective Digitized Sounds (2nd Edition; IADS-2): affective ratings of sounds and instruction manual. Technical report B-3, University of Florida, Gainesville, FL (2007)
Gross, J.J., Levenson, R.W.: Emotion elicitation using films. Cogn. Emot. 9(1), 87–108 (1995)
Villani, D., Riva, G.: Does interactive media enhance the management of stress? Suggestions from a controlled study. Cyberpsychology Behav. Soc. Netw. 15(1), 24–30 (2012)
Marchewka, A., Żurawski, Ł., Jednoróg, K., Grabowska, A.: The nencki affective picture system (NAPS): introduction to a novel, standardized, wide-range, high-quality, realistic picture database. Behav. Res. Methods 46(2), 596–610 (2014)
Bradley, M.M., Lang, P.J.: Measuring emotion: behavior, feeling and physiology. In: Lane, R., Nadel, L. (eds.) Cognitive Neuroscience of Emotion, pp. 242–276. Oxford University Press, New York (2000)
Dan-Glauser, E.S., Scherer, K.R.: The Geneva Affective PicturE Database (GAPED): A new 730 picture database focusing on valence and normative significance. Behav. Res. Methods 43(2), 468–477 (2011)
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., Patras, I.: DEAP: a database for emotion analysis; using physiological signals. IEEE Trans. Affect. Comput. 3(1), 18–31 (2012)
Tottenham, N., Tanaka, J.W., Leon, A.C., McCarry, T., Nurse, M., Hare, T.A., Marcus, D.J., Westerlund, A., Casey, B.J., Nelson, C.: The NimStim set of facial expressions: judgments from untrained research participants. Psychiatry Res. 168(3), 242–249 (2009)
The Paul Ekman Group, LLC. http://www.paulekman.com/product/pictures-of-facial-affect-pofa/
Bradley, M.M., Lang, P.J.: Affective norms for English words (ANEW): stimuli, instruction manual and affective ratings. Technical report C-1. Gainesville, FL. The Center for Research in Psychophysiology. University of Florida (1999)
Baccianella, S., Esuli, A., Sebastiani, F.: SentiWordNet 3.0: an enhanced lexical resource for sentiment analysis and opinion mining. In: Proceedings of LREC-10, 7th Conference on Language Resources and Evaluation, Valletta, MT, pp. 2200–2204 (2010)
Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31(1), 39–58 (2009)
Gross, R.: Face databases. In: Li, S., Jain, A. (eds.) Handbook of Face Recognition. Springer-Verlag, Pitts-burgh (2005). The Robotics Inistitute, Carnegie Mellon University Forbes Avenue
Peter, C., Herbon, A.: Emotion representation and physiology assignments in digital systems. Interact. Comput. 18(2), 139–170 (2006)
Russell, J.A.: A circumplex model of affect. J. Pers. Soc. Psychol. 39, 1161–1178 (1980)
Mehrabian, A.: Pleasure-arousal-dominance: a general framework for describing and measuring individual differences in Temperament. Curr. Psychol. 14(4), 261–292 (1996)
Bradley, M.M., Lang, P.J.: Measuring emotion: the self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 25, 49–59 (1994)
Ekman, P.: Are there basic emotions? Psychol. Rev. 99, 550–553 (1992)
Hliaoutakis, A., Varelas, G., Voutsakis, E., Petrakis, E.G., Milios, E.: Information retrieval by semantic similarity. Int. J. Semant. Web Inf. Syst. (IJSWIS) 2(3), 55–73 (2006)
Navarro, G.: A guided tour to approximate string matching. ACM Comput. Surv. (CSUR) 33(1), 31–88 (2001)
Hanbury, A.: A survey of methods for image annotation. J. Vis. Lang. Comput. 19(5), 617–627 (2008)
Jacobs, P.S. (ed.): Text-Based Intelligent Systems: Current Research and Practice in Information Extraction and Retrieval. Psychology Press, New York (2014)
Hyvönen, E., Saarela, S., Styrman, A., Viljanen, K.: Ontology-based image retrieval. In: Proceedings of the XML Finland 2002 Conference, Helsinki, Finland, pp. 15–27 (2003)
Over, P., Awad, G.M., Fiscus, J., Antonishek, B., Michel, M., Smeaton, A.F., Kraaij, W., Quénot, G.: TRECVID 2010–An overview of the goals, tasks, data, evaluation mechanisms, and metrics (2011)
Franz, T., Troncy, R., Vacura, M.: The core ontology for multimedia. In: Multimedia Semantics: Metadata, Analysis and Interaction, pp. 145–161 (2011)
Naphade, M., Smith, J.R., Tesic, J., Chang, S.F., Hsu, W., Kennedy, L., Hauptmann, A., Curtis, J.: Large-scale concept ontology for multimedia. IEEE Multimedia 13(3), 86–91 (2006)
Suárez-Figueroa, M.C., Atemezing, G.A., Corcho, O.: The landscape of multimedia ontologies in the last decade. Multimedia Tools Appl. 62(2), 377–399 (2013)
Rahman, M.: Search engines going beyond keyword search: a survey. Int. J. Comput. Appl. 75(17), 1–8 (2013)
Vallet, D., Fernández, M., Castells, P.: An ontology-based information retrieval model. In: Gómez-Pérez, A., Euzenat, J. (eds.) ESWC 2005. LNCS, vol. 3532, pp. 455–470. Springer, Heidelberg (2005)
Pease, A., Niles, I., Li, J.: The suggested upper merged ontology: a large ontology for the semantic web and its applications. In: Working Notes of the AAAI-2002 Workshop on Ontologies and the Semantic Web (2002)
Speer, R., Havasi, C.: ConceptNet 5: a large semantic network for relational knowledge. In: Kim, J., Gurevych, I. (eds.) The People’s Web Meets NLP, pp. 161–176. Springer, Heidelberg (2013)
Gangemi, A., Guarino, N., Masolo, C., Oltramari, A., Schneider, L.: Sweetening ontologies with DOLCE. In: Gómez-Pérez, A., Benjamins, V. (eds.) EKAW 2002. LNCS (LNAI), vol. 2473, pp. 166–181. Springer, Heidelberg (2002)
Hare, J.S., Sinclair, P.A.S., Lewis, P.H., Martinez, K., Enser, P.G.B., Sandom, C.J.: Bridging the semantic gap in multimedia information retrieval: top-down and bottom-up approaches. In: Mastering the Gap: From Information Extraction to Semantic Representation/3rd European Semantic Web Conference, Budva (2006)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Horvat, M., Vuković, M., Car, Ž. (2016). Evaluation of Keyword Search in Affective Multimedia Databases. In: Nguyen, N.T., Kowalczyk, R., Rupino da Cunha, P. (eds) Transactions on Computational Collective Intelligence XXI. Lecture Notes in Computer Science(), vol 9630. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-49521-6_3
Download citation
DOI: https://doi.org/10.1007/978-3-662-49521-6_3
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-662-49520-9
Online ISBN: 978-3-662-49521-6
eBook Packages: Computer ScienceComputer Science (R0)