Multimodal Approaches in Analysing and Interpreting Big Social Media Data

  • Eugene Ch’ngEmail author
  • Mengdi Li
  • Ziyang Chen
  • Jingbo Lang
  • Simon See


The general consensus towards the definition of Big data is that it is the data that is too big to manage using conventional methods. Yet, the present Big data approaches will eventually become conventional, where non-specialists can conduct their tasks without the need for consultancy services, much like any standard computing platforms today. In this chapter, we approach the topic from a multimodal perspective but are strategically focused on making meaning out of single-source data using multiple modes, with technologies and data accessible to anyone. We gave attention to social media, Twitter particularly, in order to demonstrate the entire process of our multimodal analysis from acquiring data to the Mixed-Reality approaches in the visualisation of data in near real-time for the future of interpretation. Our argument is that Big data research, which in the past were considered accessible only to corporations with large investment models and academic institutions with large funding streams, should no longer be a barrier. Instead, the bigger issue should be the development of multi-modal approaches to contextualising data so as to facilitate meaningful interpretations.



The author acknowledges the financial support from the International Doctoral Innovation Centre, Ningbo Education Bureau, Ningbo Science and Technology Bureau, and the University of Nottingham. This work was also supported by the UK Engineering and Physical Sciences Research Council [grant number EP/L015463/1]. The equipment provided by NVIDIA is greatly appreciated, without which the freedom of exploratory research and innovation may be constraint to only funded projects.


  1. 1.
    Manovich, L.: Trending: the promises and the challenges of big social data. Debate. Digit. Humanit. 460–475 (2011)Google Scholar
  2. 2.
    Boyd, D., Crawford, K.: Six provocations for big data. A decade in internet time: symposium on the dynamics of the internet and society, Soc. Sci. Res. Netw., New York (2011)Google Scholar
  3. 3.
    Marr, B.: The complete beginner’s guide to big data in 2017. Forbes Technology (2017)Google Scholar
  4. 4.
    Carter, J.: Why big data is crude oil – while rich data is refined, and the ultimate in BI. (2015)
  5. 5.
    Gitelman, L.: Raw data is an oxymoron. MIT Press, Cambridge (2013)CrossRefGoogle Scholar
  6. 6.
    Loukides, M.: What is data science? O’Reilly Media, Inc., Newton (2011)Google Scholar
  7. 7.
    Lindstrom, M.: Small data: the tiny clues that uncover huge trends. St. Martin’s Press, New York (2016)Google Scholar
  8. 8.
    Veinot, T.C.: ‘The eyes of the power company’: workplace information practices of a vault inspector. Libr. Q. 77(2), 157–179 (2007)CrossRefGoogle Scholar
  9. 9.
    Avoyan, A.H.: Machine learning is creating a demand for new skills. Forbes Technology (2017)Google Scholar
  10. 10.
    Peshkin, A.: The nature of interpretation in qualitative research. Educ. Res. 29(9), 5–9 (2000)CrossRefGoogle Scholar
  11. 11.
    Roy, A.K.: Applied big data analytics. Paperback. Create space Independent Publishing Platform, ISBN-10, vol. 1512347183 (2015)Google Scholar
  12. 12.
    Jacobs, A.: The pathologies of big data. Commun. ACM. 52(8), 36–44 (2009)CrossRefGoogle Scholar
  13. 13.
    Ch’ng, E.: The value of using big data technology in computational social science. In: The 3rd ASE big data science 2014, Tsinghua University 4–7 August, pp. 1–4 (2014)Google Scholar
  14. 14.
    Mergel, A.: A framework for interpreting social media interactions in the public sector. Gov. Inf. Q. 30(4), 327–334 (2013)CrossRefGoogle Scholar
  15. 15.
    Stieglitz, S., Dang-Xuan, L.: Social media and political communication: a social media analytics framework. Soc. Netw. Anal. Min. 3(4), 1277–1291 (2013)CrossRefGoogle Scholar
  16. 16.
    Koto, F., Adriani, M.: A comparative study on twitter sentiment analysis: which features are good?, vol. 6177 (2010)Google Scholar
  17. 17.
    Barbosa, L., Junlan, F.: Robust sentiment detection on twitter from biased and noisy data. no. August, pp. 36–44 (2010)Google Scholar
  18. 18.
    Kouloumpis, E., Wilson, T., Moore, J.: Twitter sentiment analysis: the good the bad and the omg!. Proceedings of the Fifth International AAAI Conference on Weblogs and Social Media (ICWSM 11), pp. 538–541 (2011)Google Scholar
  19. 19.
    Pak, A., Paroubek, P.: Twitter as a corpus for sentiment analysis and opinion mining. LREc. 1320–1326 (2010)Google Scholar
  20. 20.
    Li, M., Ch’ng, E., Chong, A., See, S.: The new eye of smart city: novel citizen sentiment analysis in Twitter. In: The 5th International Conference on Audio, Language and Image Processing (2016)Google Scholar
  21. 21.
    Li, M., Ch, E., Chong, A., See, S.: Twitter sentiment analysis of the 2016 U. S. Presidential Election using an emoji training heuristic. In: Applied Informatics and Technology Innovation Conference (AITIC) (2016)Google Scholar
  22. 22.
    McMillan, D.W., Chavis, D.M.: Sense of community: a definition and theory. J. Community Psychol. 14(1), 6–23 (1986)CrossRefGoogle Scholar
  23. 23.
    McMillan, D.: Sense of community: an attempt at definition. George Peabody College for Teachers, Nashville (1976)Google Scholar
  24. 24.
    Maloney-Krichmar, D., Preece, J.: A multilevel analysis of sociability, usability, and community dynamics in an online health community. ACM Trans. Comput. Hum. Interact. 12(2), 201–232 (2005)CrossRefGoogle Scholar
  25. 25.
    Bloch, M.: The historian’s craft. Manchester University Press (1992).Google Scholar
  26. 26.
    Gurin, J.: Big data and open data: what’s what and why does it matter? [Online]. (2014). Accessed 28 Aug 2017
  27. 27.
    Davis, D.: How big data is transforming architecture. (2015)
  28. 28.
    Khan, A., Hornbæk, K.: Big data from the built environment. In: Proceedings of the 2nd international workshop on research in the large, pp. 29–32 (2011)Google Scholar
  29. 29.
    Maeda, J.: Artists and scientists: more alike than different. Sci. Am. 2016 (2013)Google Scholar
  30. 30.
    Kritzer, H.M.: The data puzzle: the nature of interpretation in quantitative research. Am. J. Polit. Sci. 40, 1–32 (1996)CrossRefGoogle Scholar
  31. 31.
    Taylor, A.: Philosophical Papers: Volume 2, Philosophy and the Human Sciences, vol. 2. Cambridge University Press (1985)Google Scholar
  32. 32.
    T. F. F. B. D. Commission: Demystifying big data: a practical guide to transforming the business of government [Online]. (2012)
  33. 33.
  34. 34.
    Pavalanathan, U., Eisenstein, J.: Emoticons vs. emojis on twitter: a causal inference approach. AAAI Spring Symposium on Observational Studies Through Social Media and Other Human-Generated Content (2016)Google Scholar
  35. 35.
    Kalman, Y.M., Gergle, D.: Letter repetitions in computer-mediated communication: a unique link between spoken and online language. Comput. Hum. Behav. 34, 187–193 (2014)CrossRefGoogle Scholar
  36. 36.
    Dresner, E., Herring, S.C.: Functions of the nonverbal in CMC: emoticons and illocutionary force. Commun. Theory. 20(3), 249–268 (2010)CrossRefGoogle Scholar
  37. 37.
    Miller, H., Thebault-Spieker, J., Chang, S., Johnson, I., Terveen, L., Hecht, B.: ‘Blissfully happy’ or ‘ready to fight’: varying interpretations of emoji. GroupLens Research, University of Minnesota (2015)Google Scholar
  38. 38.
    Kelly, R., Watts, L.: Characterising the inventive appropriation of emoji as relationally meaningful in mediated close personal relationships. Experiences of technology appropriation: unanticipated users, usage, circumstances, and design (2015)Google Scholar
  39. 39.
    Mosendz, P.: Why the white house is using emoji [Online]. (2014). Accessed 05 Aug 2017
  40. 40.
    Steinmetz, K.: Oxford’s 2015 word of the year is this emoji. TIME [Online]. (2016) Accessed 12 Jun 2016
  41. 41.
    Chen, Z., Lu, X., Shen, S., Ai, W., Liu, X., Mei, Q.: Through a gender lens: an empirical study of emoji usage over large-scale android users, pp. 1–20 (2017)Google Scholar
  42. 42.
    Vidal, L., Ares, G., Jaeger, S.R.: Use of emoticon and emoji in tweets for food-related emotional expression. Food Qual. Prefer. 49, 119–128 (2016)CrossRefGoogle Scholar
  43. 43.
    Novak, P.K., Smailovic, J., Sluban, B., Mozetic, I.: Sentiment of emojis. PLoS One. 10(12), 1–19 (2015)Google Scholar
  44. 44.
    He, W., Wu, H., Yan, G., Akula, V., Shen, J.: A novel social media competitive analytics framework with sentiment benchmarks. Inf. Manag. 52(7), 801–812 (2015)CrossRefGoogle Scholar
  45. 45.
    Park, J., Baek, Y.M., Cha, M.: Cross-cultural comparison of nonverbal cues in emoticons on twitter: EVIDENCE from big data analysis. J. Commun. 64(2), 333–354 (2014)CrossRefGoogle Scholar
  46. 46.
    Lim, S.L., Bentley, P.J., Kanakam, N., Ishikawa, F., Honiden, S.: Investigating country differences in mobile app user behavior and challenges for software engineering. IEEE Trans. Softw. Eng. 41(1), 40–64 (2015)CrossRefGoogle Scholar
  47. 47.
    Reinecke, K., Bernstein, A.: Improving performance, perceived usability, and aesthetics with culturally adaptive user interfaces. ACM Trans. Comput. Hum. Interact. 18(2), 8 (2011)CrossRefGoogle Scholar
  48. 48.
    Jack, R.E., Blais, C., Scheepers, C., Schyns, P.G., Caldara, R.: Cultural confusions show that facial expressions are not universal. Curr. Biol. 19(18), 1543–1548 (2009)CrossRefGoogle Scholar
  49. 49.
    Lu, X., Ai, W., Liu, X., Li, Q., Wang, N., Huang, G., Mei, Q.: Learning from the ubiquitous language: an empirical analysis of emoji usage of smartphone users. In: Proceedings of Ubicomp, pp. 770–780. ACM (2016)Google Scholar
  50. 50.
    Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space, arXiv preprint arXiv:1301.3781 (2013)Google Scholar
  51. 51.
    Hofstede, G.: Hofstede insights [Online]. (2017). Accessed 05 Aug 2017
  52. 52.
    Medhat, W., Hassan, A., Korashy, H.: Sentiment analysis algorithms and applications: a survey. Ain Shams Eng. J. 5(4), 1093–1113 (2014)CrossRefGoogle Scholar
  53. 53.
    Yu, L.C., Wu, J.L., Chang, P.C., Chu, H.S.: Using a contextual entropy model to expand emotion words and their intensity for the sentiment classification of stock market news. Knowl.-Based Syst. 41, 89–97 (2013)CrossRefGoogle Scholar
  54. 54.
    Xu, T., Peng, Q., Cheng, Y.: Identifying the semantic orientation of terms using S-HAL for sentiment analysis. Knowl.-Based Syst. 35, 279–289 (2012)CrossRefGoogle Scholar
  55. 55.
    Liu, A.: Sentiment analysis and opinion mining. Synth. Lect. Hum. Lang. Technol. 5(1), 1–167 (2012)CrossRefGoogle Scholar
  56. 56.
    Nakov, P., Ritter, A., Rosenthal, S., Stoyanov, V., Sebastiani, F.:{SemEval}-2016task 4: sentiment analysis in {T}witter. Proceedings of the 10th International Workshop on Semantic Evaluation, pp. 1–18 (2016)Google Scholar
  57. 57.
    Ch’ng, V., Gaffney, L., Chapman, H. P.: From product to process: new directions in digital heritage. In: Din, H., Wu, S. (eds.) Digital Heritage and Culture: Strategy and Implementation, 1st ed. World Scientific, pp. 219–243. (2014)Google Scholar
  58. 58.
    Frankel, F., Reid, R.: Big data: distilling meaning from data. Nature. 455(7209), 30 (2008)CrossRefGoogle Scholar
  59. 59.
    Ch’ng, E.: The bottom-up formation and maintenance of a twitter community: analysis of the # FreeJahar twitter community. Ind. Manag. Data Syst. 115(4), 612–624 (2015)CrossRefGoogle Scholar
  60. 60.
    Ch’ng, E.: Social information landscapes: automated mapping of large multimodal, longitudinal social networks. Ind. Manag. Data Syst. 115(9), 1724–1751 (2015)CrossRefGoogle Scholar
  61. 61.
    Ch’ng, E.: Local interactions and the emergence and maintenance of a twitter small-world network. Soc. Networking. 4(2), 33–40 (2015)CrossRefGoogle Scholar
  62. 62.
    Mitchell, L., Frank, M.R., Harris, K.D., Dodds, P.S., Danforth, C.M.: The geography of happiness: connecting twitter sentiment and expression, demographics, and objective characteristics of place. PLoS One. 8(5), e64417 (2013)CrossRefGoogle Scholar
  63. 63.
    Li, M., Ch’ng, E., Li, B., Zhai, S.: Social-cultural monitoring of smart cities using big data methods: alcohol. In: The Third International Conference on Smart Sustainable City and Big Data (ICSSC) (2015)Google Scholar
  64. 64.
    Schatz, M., Trapnell, C.: Fast exact string matching on the GPU. Center for Bioinformatics and Computational Biology (2007)Google Scholar
  65. 65.
    Jacob, N., Brodley, C.: Offloading IDS computation to the GPU. In: Proceedings of the 22nd Annual Computer Security Applications Conference IEEE ACSAC’06, pp. 371–380 (2006)Google Scholar
  66. 66.
    Vasiliadis, A., Antonatos, S., Polychronakis, M., Markatos, E.P., Ioannidis, S.: Gnort: high performance network intrusion detection using graphics processors. In: Proceedings of the 11th International Symposium on Recent Advances in Intrusion Detection, pp. 116–134 (2008)Google Scholar
  67. 67.
    Zhang, Y., Mueller, F., Cui, X., Potok, T.: GPU-accelerated text mining. In: Workshop on exploiting parallelism using GPUs and other hardware-assisted methods, pp. 1–6 (2009)Google Scholar
  68. 68.
    Onsjo, M., Aono, Y., Watanabe, O.: Online approximate string matching with CUDA, [2009-12-18]. 07/26/57/pattmatch-report.pdf (2009)
  69. 69.
    Kouzinopoulos, A.S., Margaritis, K.G.: String matching on a multicore GPU using CUDA. In: Informatics. PCI’09. 13th Panhellenic Conference on, 2009, pp. 14–18 (2009)Google Scholar
  70. 70.
    Ch’ng, E., Chen, Z., See, S.: Real-time GPU-accelerated social media sentiment processing and visualization. In: The 21st IEEE International Symposium on Distributed Simulation and Real Time Applications (DS-RT). October 18–20, 2017Google Scholar
  71. 71.
    ILS: Twitter usage statistics. (2017)

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Eugene Ch’ng
    • 1
    Email author
  • Mengdi Li
    • 1
    • 2
  • Ziyang Chen
    • 1
  • Jingbo Lang
    • 1
  • Simon See
    • 3
  1. 1.NVIDIA Joint-Lab on Mixed RealityUniversity of Nottingham Ningbo ChinaNingboChina
  2. 2.International Doctoral Innovation CentreUniversity of Nottingham Ningbo ChinaNingboChina
  3. 3.NVIDIA AI Technology CentreGalaxis (West Lobby)Singapore

Personalised recommendations