Cognitive Computation

, Volume 4, Issue 4, pp 477–496 | Cite as

Sentic Album: Content-, Concept-, and Context-Based Online Personal Photo Management System

  • Erik Cambria
  • Amir Hussain


The world of online personal photo management has come a long way in the past few years, but today, there are still huge gaps in annotating, organizing, and retrieving online pictures in such a way that they can be easily queried and visualized. Existing content-based image retrieval systems apply statistics, pattern recognition, signal processing, and computer vision techniques but these are still too weak to ‘bridge the semantic gap’ between the low-level data representation and the high-level concepts the user associates with images. Image meta search engines, on the other hand, rely on tags associated with online pictures but results are often too inaccurate since they mainly depend on keyword-based rather than concept-based algorithms. Sentic Album is a novel content-, concept-, and context-based online personal photo management system that exploits both data and metadata of online personal pictures to intelligently annotate, organize, and retrieve them. Many salient features of pictures, in fact, are only noticeable in the viewer’s mind, and the cognitive ability to grasp such features is a key aspect for accordingly analyzing and classifying personal photos. To this end, Sentic Album exploits not just colors and texture of online images (content), but also the cognitive and affective information associated with their metadata (concept), and their relative timestamp, geolocation, and user interaction metadata (context).


Human computer interaction Cognitive and affective information processing Image affect Image classification Image features Emotional semantic image retrieval Sentic computing 



This work has been part-funded by Hewlett–Packard Labs India, the UK Engineering and Physical Sciences Research Council (EPSRC Grant Reference: EP/G501750/1) and Sitekit Solutions Ltd. (UK). We would like to thank Praphul Chandra for providing the application case study and the HP industrial placement opportunity.


  1. 1.
    Bach J, Fuller C, Gupta A, Hampapur A, Horowitz B, Humphrey R, Jain R, Shu C. Virage image search engine: an open framework for image management. In: Sethi I, Jain R, editors. Storage and retrieval for still image and video databases, vol. 2670. Bellingham: SPIE; 1996. p. 76–87.Google Scholar
  2. 2.
    Bianchi-Berthouze N. K-DIME: an affective image filtering system. IEEE Multimedia. 2003; 10(3):103–6.CrossRefGoogle Scholar
  3. 3.
    Bonanno G, Papa A, O’Neill K, Westphal M, Coifman K. The importance of being flexible: the ability to enhance and suppress emotional expressions predicts long-term adjustment. Psychol Sci. 2004; 15:482–7.PubMedCrossRefGoogle Scholar
  4. 4.
    Burke A, Heuer F, Reisberg D. Remembering emotional events. Memory Cogn. 1992; 20:277–90.CrossRefGoogle Scholar
  5. 5.
    Cambria E, Benson T, Eckl C, Hussain A. Sentic PROMs: application of sentic computing to the development of a novel unified framework for measuring health-care quality. Expert Systems with Applications, Elsevier. 2012. doi: 10.1016/j.eswa.2012.02.120.
  6. 6.
    Cambria E, Grassi M, Hussain A, Havasi C. Sentic computing for social media marketing. Multimedia Tools Appl. 2011. doi: 10.1007/s11042-011-0815-0.
  7. 7.
    Cambria E, Hussain A. Sentic computing: techniques, tools, and applications. Berlin: Springer; 2012.Google Scholar
  8. 8.
    Cambria E, Hussain A, Durrani T, Havasi C, Eckl C, Munro J. Sentic computing for patient centered application. In: Proceedings of IEEE ICSP. Beijing; 2010. p. 1279–82.Google Scholar
  9. 9.
    Cambria E, Hussain A, Havasi C, Eckl C. AffectiveSpace: blending common sense and affective knowledge to perform emotive reasoning. In: Proceedings of CAEPIA. Seville; 2009. p. 32–41.Google Scholar
  10. 10.
    Cambria E, Livingstone A, Hussain A. The Hourglass of Emotions. LNCS, Cognitive Behavioral Systems. Springer-Verlag, Berlin Heidelberg; 2012.Google Scholar
  11. 11.
    Cambria E, Mazzocco T, Hussain A, Eckl C. Sentic medoids: organizing affective common sense knowledge in a multi-dimensional vector space. In: Advances in neural networks. Lecture notes in computer science, vol. 6677. Berlin: Springer; 2011. p. 601–10.Google Scholar
  12. 12.
    Cambria E, Olsher D, Kwok K. Sentic panalogy: swapping affective common sense reasoning strategies and foci. In: Proceedings of CogSci. Sapporo; 2012.Google Scholar
  13. 13.
    Cambria E, Olsher D, Kwok K. Sentic activation: a two-level affective common sense reasoning framework. In: Proceedings of AAAI. Toronto; 2012.Google Scholar
  14. 14.
    Chi P, Lieberman H. Intelligent assistance for conversational storytelling using story patterns. In: IUI. Palo Alto 2011.Google Scholar
  15. 15.
    Christianson S, Loftus E. Remembering emotional events: the fate of detailed information. Cogn Emot. 1991;5:81–108.CrossRefGoogle Scholar
  16. 16.
    Damasio A. Descartes’ error: emotion, reason, and the human brain. New York: Grossett/Putnam; 1994.Google Scholar
  17. 17.
    Datta R, Wang J. ACQUINE: aesthetic quality inference engine—real-time automatic rating of photo aesthetics. In: Proceedings of the international conference on multimedia information retrieval. Philadelphia; 2010.Google Scholar
  18. 18.
    Decherchi S, Gastaldo P, Redi J, Zunino R, Cambria E. Circular-ELM for the reduced-reference assessment of perceived image quality. Neurocomputing. (in press).Google Scholar
  19. 19.
    Duda R, Hart P. Pattern classification and scene analysis. New York: Wiley; 1973.Google Scholar
  20. 20.
    Eckart C, Young G. The approximation of one matrix by another of lower rank. Psychometrika. 1936;1(3):211–8.CrossRefGoogle Scholar
  21. 21.
    Elliott CD. The affective reasoner: A process model of emotions in a multi-agent system. Ph.D. thesis, Northwestern University, Evanston; 1992.Google Scholar
  22. 22.
    Fellbaum C. WordNet: An electronic lexical database (language, speech, and communication). Cambridge: The MIT Press; 1998.Google Scholar
  23. 23.
    Flickner M, Sawhney H, Niblack W, Ashley J, Huang Q, Dom B, Gorkani M, Hafner J, Lee D, Petkovic D, Steele D, Yanker P. Query by image and video content: the QBIC system. Computer. 1995;28(9):23–32.CrossRefGoogle Scholar
  24. 24.
    Frankel C, Swain MJ, Athitsos V. WebSeer: an image search engine for the world wide web. Technical report. Chicago: University of Chicago; 1996.Google Scholar
  25. 25.
    Garey M, Johnson D. Computers and intractability: a guide to the theory of NP-completeness. San Francisco: Freeman; 1979.Google Scholar
  26. 26.
    Goertzel B, Silverman K, Hartley C, Bugaj S, Ross M. The baby webmind project. In: Proceedings of AISB. Birmingham; 2000.Google Scholar
  27. 27.
    Grassi M. Developing HEO human emotions ontology. In: Fierrez J, Ortega-Garcia J, Esposito A, Drygajlo A, Faundez-Zanuy M, editors. Biometric ID management and multimodal communication. Lecture notes in computer science, vol. 5707. Berlin: Springer; 2009. p. 244–51.Google Scholar
  28. 28.
    Hanjalic A. Extracting moods from pictures and sounds: towards truly personalized TV. IEEE Signal Process Mag. 2006;23(2):90–100.CrossRefGoogle Scholar
  29. 29.
    Hartigan J, Wong M. Algorithm AS 136: a k-means clustering algorithm. J R Stat Soc. 1979;28(1):100–8.Google Scholar
  30. 30.
    Havasi C, Speer R, Alonso J. ConceptNet 3: a flexible, multilingual semantic network for common sense knowledge. In: Proceedings of RANLP. Borovets; 2007.Google Scholar
  31. 31.
    Havasi C, Speer R, Holmgren J. Automated color selection using semantic knowledge. In: Proceedings of AAAI CSK. Arlington; 2010.Google Scholar
  32. 32.
    Havasi C, Speer R, Pustejovsky J, Lieberman H. Digital intuition: applying common sense using dimensionality reduction. IEEE Intell Syst. 2009;24(4):24–35.CrossRefGoogle Scholar
  33. 33.
    Hu M, Liu B. Mining opinion features in customer reviews. In: Proceedings of AAAI. San Jose; 2004.Google Scholar
  34. 34.
    Huang J, Ravi S, Mitra M, Zhu W, Zabih R. Image indexing using color correlograms. In: Proceedings of IEEE CVPR, 1997. p. 762–8.Google Scholar
  35. 35.
    Itten J. The art of color: the subjective experience and objective rationale of color. New York: Wiley; 1973.Google Scholar
  36. 36.
    Jing F, Wang C, Yao Y, Deng K, Zhang L, Ma WY. IGroup: web image search results clustering. In: Proceedings of ACM Multimedia. Santa Barbara; 2006.Google Scholar
  37. 37.
    Kaufman L, Rousseeuw P. Finding groups in data: an introduction to cluster analysis. London: Wiley; 1990.CrossRefGoogle Scholar
  38. 38.
    Keelan B. (2002) Handbook of image quality. New York: Marcel Dekker; 2002.CrossRefGoogle Scholar
  39. 39.
    Lakoff G. Women, fire, and dangerous things. Chicago: University of Chicago Press; 1990.Google Scholar
  40. 40.
    Laney C, Campbell H, Heuer F, Reisberg D. Memory for thematically arousing events. Memory Cogn. 2004;32(7):1149–59.CrossRefGoogle Scholar
  41. 41.
    Lansdale M, Edmonds E. Using memory for events in the design of personal filing systems. Int J Man Mach Stud. 1992;36(1):97–126.CrossRefGoogle Scholar
  42. 42.
    Lascu A, Cambria E, Grassi M. Human semiotics ontology. In: Proceedings of ICMC. Venice; 2011. p. 152.Google Scholar
  43. 43.
    Lee B, Hendler J, Lassila O. The semantic web. Scientific American 2001.Google Scholar
  44. 44.
    Lempel R, Soffer A. PicASHOW: pictorial authority search by hyperlinks on the web. In: Proceedings of WWW. Hong Kong; 2001.Google Scholar
  45. 45.
    Lew M, Sebe N, Djeraba C, Jain R. Content-based multimedia information retrieval: state of the art and challenges. ACM Trans Multimedia Comput Commun Appl. 2006;2(1):1–19.CrossRefGoogle Scholar
  46. 46.
    Lieberman H, Rosenzweig E, Singh P. ARIA: an agent for annotating and retrieving images. IEEE Comput. 2001;34(7):57–62.CrossRefGoogle Scholar
  47. 47.
    Lieberman H, Selker T. Out of context: computer systems that adapt to, and learn from, context. IBM Syst J. 2000;39(3):617–32.CrossRefGoogle Scholar
  48. 48.
    Lu W, Zeng K, Tao D, Yuan Y, Gao X. No-reference image quality assessment in contourlet domain. Neurocomputing. 2012;73(4–6):784–94.CrossRefGoogle Scholar
  49. 49.
    Machajdik J, Hanbury A. Affective image classification using features inspired by psychology and art theory. In: Proceedings of the international conference on multimedia. Florence; 2010.Google Scholar
  50. 50.
    Minsky M. The emotion machine: commonsense thinking, artificial intelligence, and the future of the human mind. New York: Simon & Schuster; 2006.Google Scholar
  51. 51.
    Motik B, Sattler U, Studer R. Query answering for OWL-DL with rules. 2004. p. 549–63.Google Scholar
  52. 52.
    Nakazato M, Manola L, Huang T. ImageGrouper: Search, annotate and organize images by groups. In: Chang S, Chen Z, Lee S editors. Recent advances in visual information systems. Lecture notes in computer science, vol. 2314. Berlin: Springer; 2002. p. 93–105.Google Scholar
  53. 53.
    Narwaria M, Lin W. Objective image quality assessment based on support vector regression. IEEE Trans Neural Netw. 2010;12(3):515–19.CrossRefGoogle Scholar
  54. 54.
    O’Hare N, Lee H, Cooray S, Gurrin C, Jones G, Malobabic J, O’Connor N, Smeaton A, Uscilowski B. MediAssist: Using content-based analysis and context to manage personal photo collections. In: Proceedings of CIVR, Tempe; 2006. p. 529–32.Google Scholar
  55. 55.
    Ortony A, Clore G, Collins A. The cognitive structure of emotions. Cambridge: Cambridge University Press; 1988.CrossRefGoogle Scholar
  56. 56.
    Pantic M. Affective computing. In: Encyclopedia of multimedia technology and networking, vol. 1. Idea Group Reference; 2005. p. 8–14.Google Scholar
  57. 57.
    Park H, Jun C. A simple and fast algorithm for k-medoids clustering. Exp Syst Appl. 2009;36(2):3336–41.CrossRefGoogle Scholar
  58. 58.
    Plutchik R. The nature of emotions. Am Sci. 2001;89(4):344–50.Google Scholar
  59. 59.
    Porkaew K, Chakrabarti K. Query refinement for multimedia similarity retrieval in MARS. In: Proceedings of ACM international conference on multimedia. New York: ACM; 1999. p. 235–8.Google Scholar
  60. 60.
    Redi J, Gastaldo P, Heynderickx I, Zunino R. Color distribution information for the reduced-reference assessment of perceived image quality. IEEE Trans Circuits Syst Video Technol. 2012;20(12):1757–69.CrossRefGoogle Scholar
  61. 61.
    Reisberg D, Heuer F. Memory for emotional events. Memory Emot. 2004:3–41.Google Scholar
  62. 62.
    Richards J, Butler E, Gross J. Emotion regulation in romantic relationships: the cognitive consequences of concealing feelings. J. Social Pers. Relation. 2003;20:599–620.CrossRefGoogle Scholar
  63. 63.
    Sebe N, Tian Q, Loupias E, Lew MS, Huang TS. Evaluation of salient point techniques. In: Proceedings of the international conference on image and video retrieval. London: Springer; 2002. p. 367–77.Google Scholar
  64. 64.
    Smith J, Chang S. An image and video search engine for the world-wide web. In: Symposium on electronic imaging: science and technology 1997.Google Scholar
  65. 65.
    Somasundaran S, Wiebe J, Ruppenhofer J. Discourse level opinion interpretation. In: Proceedings of COLING. Manchester; 2008.Google Scholar
  66. 66.
    Strapparava C, Valitutti A. WordNet-Affect: An affective extension of WordNet. In: Proceedings of LREC. Lisbon; 2004.Google Scholar
  67. 67.
    Urban J, Jose J, Van Rijsbergen C. An adaptive approach towards content-based image retrieval. Multimedia Tools Appl. 2006;31:1–28.CrossRefGoogle Scholar
  68. 68.
    Urban J, Jose JM. EGO: a personalized multimedia management and retrieval tool. Int J Intell Syst. 2006;21(7):725–45.CrossRefGoogle Scholar
  69. 69.
    Valdez P, Mehrabian A. Effects of color on emotions. J Exp Psychol General. 1994;123(4):394–409.PubMedCrossRefGoogle Scholar
  70. 70.
    Vesterinen E. Affective computing. In: Digital media research seminar. Helsinki; 2001.Google Scholar
  71. 71.
    Wall M, Rechtsteiner A, Rocha L. Singular value decomposition and principal component analysis. In: Berrar D, Dubitzky W, Granzow M (eds) A practical approach to microarray data analysis. Berlin: Springer; 2003. pp. 91–109.CrossRefGoogle Scholar
  72. 72.
    Wang W, He Q. A survey on emotional semantic image retrieval. In: Proceedings of IEEE ICIP, 2008. p. 117–20.Google Scholar
  73. 73.
    Wessel I, Merckelbach H. The impact of anxiety on memory for details in spider phobics. Appl Cogn Psychol. 1997;11:223–31.CrossRefGoogle Scholar
  74. 74.
    Wiebe J, Wilson T, Cardie C. Annotating expressions of opinions and emotions in language. Lang Resour Eval. 2005;39(2):165–210.CrossRefGoogle Scholar
  75. 75.
    Wilson T, Wiebe J, Hoffmann P. Recognizing contextual polarity in phrase-level sentiment analysis. In: Proceedings of HLT/EMNLP. Vancouver; 2005.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  1. 1.Temasek Laboratories National University of SingaporeSingaporeSingapore
  2. 2.Department of Computing Science and MathematicsUniversity of StirlingStirlingUK

Personalised recommendations