Skip to main content
SpringerLink
Log in

Google Books Ngram: Problems of Representativeness and Data Reliability

  • Conference paper
  • First Online:
Data Analytics and Management in Data Intensive Domains (DAMDID/RCDL 2019)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1223))

Abstract

The article discusses representativeness of Google Books Ngram as a multi-purpose corpus. Criticism of the corpus is analysed and discussed. A comparative study of the GBN data and the data obtained using the Russian National Corpus and the General Internet Corpus of Russian is performed to show that the Google Books Ngram corpus can be successfully used for corpus-based studies. A new concept “diachronically balanced corpus” is introduced. Besides, the article describes the problems of word spelling and metadata errors presented in the GBN corpus and proposes possible ways of improving quality of the GBN data.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Rykov, V.V.: Text corpus design as application of object-oriented paradigm. In: Trudy Mezhdunarodnogo seminara Dialog-2002, Nauka, Moskow, pp. 124–129 (2002). (in Russian)

    Google Scholar 

  2. Russian National Corpus. http://www.ruscorpora.ru. Accessed 29 Dec 2019

  3. Natsional’nyy korpus russkogo yazyka: 2003–2005. Rezul’taty i perspektivy. Indrik, Moscow (2005). (in Russian)

    Google Scholar 

  4. Natsional’nyy korpus russkogo yazyka: 2006–2008. Novyye rezul’taty i perspektivy. Nestor-Istoriya, St. Petersburg (2009). (in Russian)

    Google Scholar 

  5. Belikov, V., Kopylov, N., Piperski, A., Selegey, V., Sharoff, S.: Corpus as language: from scalability to register variation. In: Computational Linguistics and Intellectual Technologies. Papers from the Annual International Conference “Dialogue”, vol. 12, no. 19, pp. 83–95. RGGU, Moskow (2013)

    Google Scholar 

  6. Lin, Y., Michel, J.-B., Aiden, E.L., Orwant, J., Brockman, W., Petrov, S.: Syntactic annotations for the Google Books ngram corpus. In: 50th Annual Meeting of the Association for Computational Linguistics 2012, Proceedings of the Conference, vol. 2, pp. 169–174. Association for Computational Linguistics, Jeju Island, Korea (2012)

    Google Scholar 

  7. Michel, J.-B., Shen, Y.K., Aiden, A.P., Veres, A., Gray, M.K., et al.: Quantitative analysis of culture using millions of digitized books. Science 331(6014), 176–182 (2011)

    Article  Google Scholar 

  8. Aiden, E., Michel, J.-B.: Uncharted Big Data as a Lens on Human Culture, 1st edn. Riverhead Books, New York (2013)

    Google Scholar 

  9. Belikov, V.I.: What and how can a linguist get from digitized texts? Siberian J. Philol. 3, 17–34 (2016). (In Russian)

    Article  Google Scholar 

  10. Koplenig, A.: The impact of lacking metadata for the measurement of cultural and linguistic change using the Google Ngram data sets—Reconstructing the composition of the German corpus in times of WWII. Digit. Scholar. Human. 32, 169–188 (2017). https://doi.org/10.1093/llc/fqv037

    Article  Google Scholar 

  11. Pechenick, E.A., Danforth, C., Dodds, P., Barrat, A.: Characterizing the google books corpus: strong limits to inferences of socio-cultural and linguistic evolution. PLoS ONE 10(10), e0137041 (2015)

    Article  Google Scholar 

  12. Solovyev, V., Akhtyamova, S.: Linguistic big data: problem of purity and representativeness. In: XXI International Conference on Data Analytics and Management in Data Intensive Domains (DAMDID/RCDL 2019), Kazan, Russia, 15–18 October 2019, pp. 193–204. CEUR-WS.org (2019)

    Google Scholar 

  13. Dictionary OpenCorpora. http://opencorpora.org/dict.php. Accessed 29 Dec 2019

  14. Bocharov, V.V., Alexeeva, S.V., Granovsky, D.V., Protopopova, E.V., Stepanova, M.E., Surikov, A.V.: Growdsourcing morphological annotation. In: Computational Linguistics and Intellectual Technologies. Papers from the Annual International Conference “Dialogue”, vol. 12, no. 19, pp. 109–115. RGGU, Moskow (2013)

    Google Scholar 

  15. Solovyev, V.D.: Possible mechanisms of change in the cognitive structure of synonym sets. In: Language and Thought: In: Contemporary Cognitive Linguistics. A Collection of Articles, pp. 478–487. Languages of Slavic Culture, Moscow (2015). (in Russian)

    Google Scholar 

  16. Bochkarev, V., Solovyev, V., Wichmann, S.: Universals versus historical contingencies in lexical evolution. J. R. Soc. Interface 11(101), 20140841 (2014). https://doi.org/10.1098/rsif.2014.0841

    Article  Google Scholar 

  17. Biber, D.: Representativeness in corpus design. In: Zampolli, A., Calzolari, N., Palmer, M. (eds.) Current Issues in Computational Linguistics: In: Honour of Don Walker. Linguistica Computazionale, vol. 9, pp. 377–407. Springer, Dordrecht (1994). https://doi.org/10.1007/978-0-585-35958-8_20

    Chapter  Google Scholar 

  18. Koplenig, A.: A fully data-driven method to identify (correlated) changes in diachronic corpora. arXiv preprint arXiv:1508/1508.06374 (2015)

  19. Pechenick, E.A., Danforth, C., Dodds, P.: Is language evolution grinding to a halt? The scaling of lexical turbulence in English fiction suggests it is not. J. Comput. Science 21, 24–37 (2017)

    Article  MathSciNet  Google Scholar 

  20. Petersen, A.M., Tenenbaum, J., Havlin, S., Stanley, H.E., Perc, M.: Languages cool as they expand: allometric scaling and the decreasing need for new words. Sci. Rep. 2, 943 (2012)

    Article  Google Scholar 

  21. Petersen, A.M., Tenenbaum, J., Havlin, S., Stanley, H.E.: Statistical laws governing fluctuations in word use from word birth to word death. Sci. Rep. 2, 313 (2012)

    Article  Google Scholar 

  22. Solovyev, V.D., Bochkarev, V.V., Shevlyakova, A.V.: Dynamics of core of language vocabulary. CEUR Workshop Proc. 1886, 122–129 (2016)

    Google Scholar 

  23. Bochkarev, V.V., Shevlyakova, A.V., Lerner, E.Yu.: Modelling of growth of syntactic relations network in English and Russian. J. Phys.: Conf. Ser. 1141, 012008 (2018). https://doi.org/10.1088/1742-6596/1141/1/012008

  24. Kim, Y., Chiu, Y.-I., Hanaki, K., Hegde, D., Petrov, S.: Temporal analysis of language through neural language models. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pp. 61–65. ACL, Baltimore (2014)

    Google Scholar 

  25. Kulkarni, V., Al-Rfou, R., Perozzi, B., Skiena, S.: Statistically significant detection of linguistic change. In: Proceedings of the 24th International Conference on World Wide Web, Florence, Italy, pp. 625–635 (2015)

    Google Scholar 

  26. Dubossarsky, H., Tsvetkov, Y., Dyer, C., Grossman, E.: A bottom up approach to category mapping and meaning change. In: Proceedings of the NetWordS Final Conference, Pisa, 30 March–1 April 2015, pp. 66–70. CEUR-WS.org (2015)

    Google Scholar 

  27. Bochkarev, V., Shevlyakova, A., Solovyev, V.: A method of semantic change detection using diachronic corpora data. In: van der Aalst, W.M.P., et al. (eds.) AIST 2019. CCIS, vol. 1086, pp. 94–106. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-39575-9_10

    Chapter  Google Scholar 

  28. Acerbi, A., Lampos, V., Garnett, P., Bentley, R.A.: The expression of emotions in 20th century books. PLoS ONE 8(3), e59030 (2013). https://doi.org/10.1371/journal.pone.0059030

    Article  Google Scholar 

  29. Mohammad, S.M.: From once upon a time to happily ever after: tracking emotions in mail and books. Decis. Support Syst. 53(4), 730–741 (2012)

    Article  Google Scholar 

  30. Morin, O., Acerbi, A.: Birth of the cool: a two-centuries decline in emotional expression in Anglophone fiction. Cogn. Emot. 31(8), 1663–1675 (2017). https://doi.org/10.1080/02699931.2016.1260528

    Article  Google Scholar 

  31. Scheff, T.: Toward defining basic emotions. Qual. Inq. 21(2), 111–121 (2015)

    Article  Google Scholar 

  32. Ellis, D.A., Wiseman, R., Jenkins, R.: Mental representations of weekdays. PloS ONE 10(8), e0134555 (2015). https://doi.org/10.1371/journal.pone.0134555

    Article  Google Scholar 

  33. Hills, T.T., Adelman, J.S.: Recent evolution of learnability in American English from 1800 to 2000. Cognition 143, 87–92 (2015). https://doi.org/10.1016/j.cognition.2015.06.009

    Article  Google Scholar 

  34. Virues-Ortega, J., Pear, J.J.: A history of “behavior” and “mind”: use of behavioral and cognitive terms in the 20th century. Psychol. Rec. 65(1), 23–30 (2015). https://doi.org/10.1007/s40732-014-0079-y

    Article  Google Scholar 

  35. Greenfield, P.M.: The changing psychology of culture from 1800 through 2000. Psychol. Sci. 24(9), 1722–1731 (2013). https://doi.org/10.1177/0956797613479387

    Article  Google Scholar 

  36. Zeng, R., Greenfield, P.M.: Cultural evolution over the last 40 years in China: using the Google Ngram viewer to study implications of social and political change for cultural values. Int. J. Psychol. 50(1), 47–55 (2015). https://doi.org/10.1002/ijop.12125

    Article  Google Scholar 

  37. Younes, N., Reips, U.-D.: The changing psychology of culture in German-speaking countries: a Google Ngram study. Int. J. Psychol. 53, 53–62 (2018). https://doi.org/10.1002/ijop.12428

    Article  Google Scholar 

  38. Velichkovsky, B.B., Solovyev, V.D., Bochkarev, V.V., Ishkineeva, F.F.: Transition to market economy promotes individualistic values: analysing changes in frequencies of Russian words from 1980 to 2008. Int. J. Psychol. 54, 23–32 (2019). https://doi.org/10.1002/ijop.12411

    Article  Google Scholar 

  39. Del Giudice, M.: The twentieth century reversal of pink-blue gender coding: a scientific urban legend? Arch. Sex. Behav. 41(6), 1321–1323 (2012). https://doi.org/10.1007/s10508-012-0002-z

    Article  Google Scholar 

  40. Ye, S., Cai, S., Chen, C., Wan, Q., Qian, X.: How have males and females been described over the past two centuries? An analysis of Big-Five personality-related adjectives in the Google English Books. J. Res. Pers. 76, 6–16 (2018)

    Article  Google Scholar 

  41. Grossman, I., Varnum, M.: Social structure, infectious diseases, disasters, secularism, and cultural change in America. Psychol. Sci. 26, 311–324 (2015)

    Article  Google Scholar 

  42. Younes, N., Reips, U.-D.: Guideline for improving the reliability of Google Ngram studies: evidence from religious terms. PLoS ONE 14(3), e0213554 (2019). https://doi.org/10.1371/journal.pone.0213554

    Article  Google Scholar 

  43. Twenge, J.M., Campbell, W.K., Gentile, B.: Changes in pronoun use in American books and the rise of individualism, 1960–2008. J. Cross Cult. Psychol. 44(3), 406–415 (2013)

    Article  Google Scholar 

  44. Malykh, V., Lyalin, V.: Named entity recognition in noisy domains. In: Proceedings-2018 International Conference on Artificial Intelligence: Applications and Innovations, IC-AIAI 2018, vol. 8674438, pp. 60–65. IEEE (2018)

    Google Scholar 

  45. Malykh, V., Khakhulin, T.: Noise robustness in aspect extraction task. In: Proceedings-2018 International Conference on Artificial Intelligence: Applications and Innovations, IC-AIAI 2018, vol. 8674450, pp. 66–69. IEEE (2018)

    Google Scholar 

  46. Sinclair, J.: How to build a corpus. In: Wynne, M. (ed.) Developing Linguistic Corpora: A Guide to Good Practice, pp. 95–101. Oxbow Books, Oxford (2005)

    Google Scholar 

Download references

Acknowledgment

This research was financially supported by the Russian Foundation for Basic Research (Grant No. 17-29-09163), the Government Program of Competitive Development of Kazan Federal University, and through the State Assignment in the Area of Scientific Activities for Kazan Federal University, agreement № 34.5517.2017/6.7.

Author information

Authors and Affiliations

  1. Kazan Federal University, Kazan, Russia

    Valery D. Solovyev & Vladimir V. Bochkarev

  2. Kazan National Research Technological University, Kazan, Russia

    Svetlana S. Akhtyamova

Authors
  1. Valery D. Solovyev
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vladimir V. Bochkarev
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Svetlana S. Akhtyamova
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Valery D. Solovyev .

Editor information

Editors and Affiliations

  1. Kazan Federal University, Kazan, Russia

    Alexander Elizarov

  2. National Research University Higher School of Economics, St. Petersburg, Russia

    Boris Novikov

  3. Federal Research Center “Computer Science and Control”, Moscow, Russia

    Sergey Stupnikov

Appendix. The List of 100 Most Frequent Words with the ORC Errors

Appendix. The List of 100 Most Frequent Words with the ORC Errors

The original correct word form is given for each word in the brackets:

библнoгp (библиoгp), cбopиик (cбopник), apмш (apмии), oтнoшeнш (oтнoшeнии), пpoнзвoдcтвa (пpoизвoдcтвa), иaзв (нaзв), aкaдeмш (aкaдeмии), тexиикa (тexникa), нaceлeшя (нaceлeния), иccлeдoвaииe (иccлeдoвaниe), тexиoлoгия (тexнoлoгия), пpoнзвoдcтвo (пpoизвoдcтвo), тpaиcпopт (тpaнcпopт), oтнoшeшя (oтнoшeния), ocнoвaнш (ocнoвaнии), peвoлюцш (peвoлюции), opгaиизaции (opгaнизaции), cтpoнтeльcтвo (cтpoитeльcтвo), вeликш (вeликий), cocтoянш (cocтoянии), иcтopни (иcтopии), cтpoнт (cтpoит), гepмaнш (гepмaнии), мaтepнaлы (мaтepиaлы), движeшя (движeния), pyccкш (pyccкий), aнглш (aнглии), oцeикa (oцeнкa), yпpaвлeшя (yпpaвлeния), кoмиccш (кoмиccии), фpaнцш (фpaнции), тpaиcпopтa (тpaнcпopтa), иeкoтopыx (нeкoтopыx), пoлoжeнш (пoлoжeнии), внимaшя (внимaния), paбoтиикoв (paбoтникoв), тeopш (тeopии), yпpaвлeиия (yпpaвлeния), бнблиoгp (библиoгp), знaчeшя (знaчeния), выpaжeшe (выpaжeниe), иccлeдoвaиия (иccлeдoвaния), инфopмaцин (инфopмaции), издaшe (издaниe), opгaиизaциѠ(opгaнизaция), coбpaшя (coбpaния), пpoмышлeниocти (пpoмышлeннocти), явлeшe (явлeниe), peвoлющи (peвoлюции), oбpaзoвaиия (oбpaзoвaния), миииcтpoв (миниcтpoв), тexиики (тexники), cyщecтвoвaшe (cyщecтвoвaниe), opгaнизaцш (opгaнизaции), нacтpoeшe (нacтpoeниe), oцeики (oцeнки), иayч (нayч), зpeшя (зpeния), экoиoмикa (экoнoмикa), гeoгpaфин (гeoгpaфии), coзнaшя (coзнaния), эффeктивиocти (эффeктивнocти), ypoвия (ypoвня), кoиcтpyкций (кoнcтpyкций), yпpaвлeшe (yпpaвлeниe), ocнoвaшя (ocнoвaния), тeppитopш (тeppитopии), пoэзш (пoэзии), вcякш (вcякий), oбpaзoвaшя (oбpaзoвaния), cocтoяшe (cocтoяниe), бoлeзии (бoлeзни), пpeдлoжeшe (пpeдлoжeниe), инфopмaцни (инфopмaции), цeитp (цeнтp), влияииe (влияниe), cтyдeитoв (cтyдeнтoв), зaключeшe (зaключeниe), oтнoшeшю (oтнoшeнию), явлeшя (явлeния), итaлш (итaлии), издaшя (издaния), впocлeдcтвш (впocлeдcтвии), opгaиизaций (opгaнизaций), пpeдcтaвлeшe (пpeдcтaвлeниe), cyщecтвoвaшя (cyщecтвoвaния), pecпyблжи (pecпyблики), бнoл (биoл), coвepшeиcтвoвaниe (coвepшeнcтвoвaниe), пpимeиeииe (пpимeнeниe), тeopни (тeopии), paзвитш (paзвитии), нaпpaвлeнш (нaпpaвлeнии), пoлeзиыx (пoлeзныx), дeятeльиocти (дeятeльнocти), cпaceшя (cпaceния), пpoмышлeниocть (пpoмышлeннocть), cpeдиeй (cpeднeй), нaпpaвлeшe (нaпpaвлeниe), кaнцeляpш (кaнцeляpии).

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Solovyev, V.D., Bochkarev, V.V., Akhtyamova, S.S. (2020). Google Books Ngram: Problems of Representativeness and Data Reliability. In: Elizarov, A., Novikov, B., Stupnikov, S. (eds) Data Analytics and Management in Data Intensive Domains. DAMDID/RCDL 2019. Communications in Computer and Information Science, vol 1223. Springer, Cham. https://doi.org/10.1007/978-3-030-51913-1_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-51913-1_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-51912-4

  • Online ISBN: 978-3-030-51913-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation