A Visualized Communication System Using Cross-Media Semantic Association

  • Xinming Zhang
  • Yang Liu
  • Chao Liang
  • Changsheng Xu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6524)


Can you imagine that two people who have different native languages and cannot understand other’s language are able to communicate with each other without professional interpreter? In this paper, a visualized communication system is designed to facilitate such people chatting with each other via visual information. Differing from the online instant message tools such as MSN, Google talk and ICQ, which are mostly based on textual information, the visualized communication system resorts to the vivid images which are relevant to the conversation context aside from text to jump the language obstacle. The multi-phase visual concept detection strategy is applied to associate the text with the corresponding web images. Then, a re-ranking algorithm attempts to return the most related and highest quality images at top positions. In addition, sentiment analysis is performed to help people understand the emotion of each other to further reduce the language obstacle. A number of daily conversation scenes are implemented in the experiments and the performance is evaluated by user study. The experimental results show that the visualized communication system is able to effectively help people with language obstacle to better understand each other.


Visualized Communication Sentiment Analysis Semantic Concept Detection 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Adams, P.H., Martell, C.H.: Topic Detection and Extraction in Chat. In: 2008 IEEE International Conference on Semantic Computing, pp. 581–588 (2008)Google Scholar
  2. 2.
    Dong, H., Hui, S.C., He, Y.: Structural analysis of chat messages for topic detection. Online Information Review, 496–516 (2006)Google Scholar
  3. 3.
    Wang, L., Jia, Y., Han, W.: Instant message clustering based on extended vector space model. In: Proceedings of the 2nd International Conference on Advances in Computation and Intelligence, pp. 435–443 (2007)Google Scholar
  4. 4.
    Jiang, Y.-G., Yang, J., Ngo, C.-W., Hauptmann, A.G.: Representations of Keypoint-Based Semantic Concept Detection: A Comprehensive Study. IEEE Transitions on Multimedia, 42–53  (2009)Google Scholar
  5. 5.
    Jiang, Y.G., Ngo, C.W., Chang, S.F.: Semantic context transfer across heterogeneous sources for domain adaptive video search. In: Proceedings of the Seventeen ACM International Conference on Multimedia, pp. 155–164 (2009)Google Scholar
  6. 6.
    Snoek, C.G.M., Huurnink, B., Hollink, L., de Rijke, M., Schreiber, G., Worring, M.: Adding semantics to detectors for video retrieval. IEEE Transaction on Multimedia 9(5), 975–986 (2007)CrossRefGoogle Scholar
  7. 7.
    Snoek, C.G.M., Worring, M., Van Gemert, J.C., Geusebroek, J.M., Smeulders, A.W.M.: The challenge problem for automated detection of 101 semantic concepts in multimedia. In: Proceedings of the 14th Annual ACM International Conference on Multimedia, p. 430 (2006)Google Scholar
  8. 8.
    Yanagawa, A., Chang, S.-F., Kennedy, L., Hsu, W.: Columbia university’s baseline detectors for 374 lscom semantic visual concepts. In: Columbia University ADVENT Technical Report #222-2006-8 (2007)Google Scholar
  9. 9.
    Jiang, Y.-G., Ngo, C.-W., Yang, J.: Towards optimal bag-of-features for object categorization and semantic video retrieval. In: Proceedings of the 6th ACM International Conference on Image and Video Retrieval, p. 501 (2007)Google Scholar
  10. 10.
    Ke, Y., Tang, X., Jing, F.: The design of high-level features for photo quality assessment. In: CVPR 2006 (2006)Google Scholar
  11. 11.
    Natsev, A., Haubold, A., Tesic, J., Xie, L., Yan, R.: Semantic concept-based query expansion and re-ranking for multimedia retrieval. In: ACM Multimedia, p. 1000 (2007)Google Scholar
  12. 12.
    Yao, T., Mei, T., Ngo, C.W.: Co-reranking by Mutual Reinforcement for Image Search. In: Proceedings of the 6th ACM International Conference on Image and Video Retrieval (2010)Google Scholar
  13. 13.
  14. 14.
    Shih, J.-L., Chen, L.-H.: Color image retrieval based on primitives of color moments. In: IEE Proceedings-Vision, Image, and Signal Processing, p. 370 (2002)Google Scholar
  15. 15.
    Van de Wouwer, G., Scheunders, P., Dyck, D.V.: Statistical Texture Characterization from Discrete Wavelet Representations. IEEE Transactions on Image Processing, 592-598 (1999)Google Scholar
  16. 16.
    Zhang, J., Marszalek, M., Lazebnik, S., Schmid, C.: Local features and kernels for classification of texture and object categories: A comprehensive study. IJCV, 213–238 (2007)Google Scholar
  17. 17.
    Keshtkar, F., Inkpen, D.: Using Sentiment Orientation Features for Mood Classification in Blogs. IEEE, Los Alamitos (2009)CrossRefGoogle Scholar
  18. 18.
    Pang, B., Lee, L., Vaithyanathan, S.: Thumbs up?: sentiment classification using machine learning techniques. In: Proceedings of the ACL 2002 Conference on Empirical Methods in Natural Language Processing, vol. 10 (2002)Google Scholar
  19. 19.
    Zha, Z.-J., Yang, L., Mei, T., Wang, M., Wang, Z.: Visual query suggestion. In: Proceedings of ACM International Conference on Multimedia, pp. 15–24 (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Xinming Zhang
    • 1
    • 2
  • Yang Liu
    • 1
    • 2
  • Chao Liang
    • 1
    • 2
  • Changsheng Xu
    • 1
    • 2
  1. 1.National Labratory of Pattern RecognitionInstitute of Automation, Chinese Academy of SciencesBeijngChina
  2. 2.China-Singapore Institute of Digital MediaSingapore

Personalised recommendations