Learning Higher-Level Features with Convolutional Restricted Boltzmann Machines for Sentiment Analysis

  • Trung Huynh
  • Yulan He
  • Stefan Rüger
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9022)


In recent years, learning word vector representations has attracted much interest in Natural Language Processing. Word representations or embeddings learned using unsupervised methods help addressing the problem of traditional bag-of-word approaches which fail to capture contextual semantics. In this paper we go beyond the vector representations at the word level and propose a novel framework that learns higher-level feature representations of n-grams, phrases and sentences using a deep neural network built from stacked Convolutional Restricted Boltzmann Machines (CRBMs). These representations have been shown to map syntactically and semantically related n-grams to closeby locations in the hidden feature space. We have experimented to additionally incorporate these higher-level features into supervised classifier training for two sentiment analysis tasks: subjectivity classification and sentiment classification. Our results have demonstrated the success of our proposed framework with 4% improvement in accuracy observed for subjectivity classification and improved the results achieved for sentiment classification over models trained without our higher level features.


Sentiment analysis Convolutional Restricted Boltzmann Machines Stacked Restricted Boltzmann Machine Word embeddings 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Collobert, R., Weston, J.: A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. In: ICML (2008)Google Scholar
  2. 2.
    Dahl, G.E., Adams, R.P., Larochelle, H.: Training Restricted Boltzmann Machines on Word Observations. In: ICML (2012)Google Scholar
  3. 3.
    Kalchbrenner, N., Grefenstette, E., Blunsom, P.: A convolutional Neural Network for Modelling Sentences. In: ACL (2014)Google Scholar
  4. 4.
    Lee, H., Grosse, R., Ranganath, R., Ng, A.Y.: Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations. In: ICML (2009)Google Scholar
  5. 5.
    Lin, C., He, Y., Everson, R.: Sentence Subjectivity Detection with Weakly-Supervised Learning. In: IJCNLP (2011)Google Scholar
  6. 6.
    Mikolov, T., Zweig, G.: Context Dependent Recurrent Neural Networkl Language Model. Tech. rep., Microsoft Research Technical Report (2012)Google Scholar
  7. 7.
    Nakagawa, T., Inui, K., Kurohashi, S.: Dependency Tree-based Sentiment Classification using CRFs with Hidden Variables. In: NAACL (2010)Google Scholar
  8. 8.
    Pang, B., Lee, L.: Opinion Mining and Sentiment Analysis. Foundations and Trends in Information Retrieval 2(12), 1–135 (2008)CrossRefGoogle Scholar
  9. 9.
    Socher, R., Pennington, J., Huang, E.H., Ng, A.Y., Manning, C.D.: Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions. In: EMNLP (2011)Google Scholar
  10. 10.
    Wiebe, J., Riloff, E.: Creating Subjective and Objective Sentence Classifiers from Unannotated Texts. In: Gelbukh, A. (ed.) CICLing 2005. LNCS, vol. 3406, pp. 486–497. Springer, Heidelberg (2005)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Trung Huynh
    • 1
  • Yulan He
    • 2
  • Stefan Rüger
    • 1
  1. 1.Knowledge Media InstituteThe Open UniversityUK
  2. 2.School of Engineering and Applied ScienceAston UniversityUK

Personalised recommendations