Predicting Flu Epidemics Using Twitter and Historical Data
Recently there has been a growing attention on the use of web and social data to improve traditional prediction models in politics, finance, marketing and health, but even though a correlation between observed phenomena and related social data has been demonstrated in many cases, yet the effectiveness of the latter for long-term or even mid-term predictions has not been shown. In epidemiological surveillance, the problem is compounded by the fact that infectious diseases models (such as susceptible-infected-recovered-susceptible, SIRS) are very sensitive to current conditions, such that small changes can produce remarkable differences in future outcomes. Unfortunately, current or nearly-current conditions keep changing as data are collected and updated by the epidemiological surveillance organizations. In this paper we show that the time series of Twitter messages reporting a combination of symptoms that match the influenza-like-illness (ILI) case definition represent a more stable and reliable information on “current conditions”, to the point that they can replace, rather than simply integrate, official epidemiological data. We estimate the effectiveness of these data at predicting current and past flu seasons (17 seasons overall), in combination with official historical data on past seasons, obtaining an average correlation of 0.85 over a period of 17 weeks covering the flu season.
KeywordsTwitter mining epidemiological surveillance predictability of health-related phenomena
Unable to display preview. Download preview PDF.
- 1.Yu, S., Kak, S.: A Survey of Prediction using Social Media (2012), http://arxiv.org/ftp/arxiv/papers/1203/1203.1647.pdf (retrieved)
- 2.Asur, S., Huberman, B.H.: Predicting the future with social media, http://www.hpl.hp.com/research/scl/papers/socialmedia/socialmedia.pdf (retrieved)
- 3.Radinsky, K., Horvitz, E.: Mining the web to predict future events. In: WSDM 2013, pp. 255–264 (2013)Google Scholar
- 4.Ruiz, E.J., Hristidis, V., Castillo, C., Gionis, A., Jaimes, A.: Correlating financial time series with micro-blogging activity. In: Proceedings of the Fifth ACM International Conference on Web Search and Data Mining, WSDM 2012, pp. 513–522. ACM, New York (2012)Google Scholar
- 5.Carrifere-Swallow, Y., Labbfe, F.: Nowcasting with Google Trends in an Emerging Market. Central Bank of Chile, Working Papers n. 588 (2010)Google Scholar
- 7.Althouse, B.M., Ng, Y.Y., Cummings, D.A.T.: Prediction of Dengue Incidence Using Search Query Surveillance. PLoS Negl. Trop. Dis. 5(8) (2011)Google Scholar
- 8.Xu, D., Liu, Y., Zhang, M., Ma, S., Ciu, A., Ru, L.: Predicting Epidemic Tendency through Search Behaviour Analysis. In: Proc. of 22nd IJCAI (2011)Google Scholar
- 9.Lamb, A., Paul, M.J., Dredze, M.: Separating Fact from Fear: Tracking Flu Infections on Twitter. In: NAACL (2013)Google Scholar
- 10.Achrekar, H., Gandhe, A., Lazarus, R., Yu, S., Liu, B.: Predicting Flu Trends using Twitter Data. In: CPNS 2011 (2011)Google Scholar
- 12.Gesualdo, F., Stilo, G., Agricola, E., Gonfiantini, M.V., Pandolfi, E., Velardi, P., Tozzi, A.E.: Influenza-like illness surveillance on Twitter through automated learning of naïve language. PloS One Public Library of Science One, Journal (2013)Google Scholar
- 13.Velardi, P., Stilo, G., Tozzi, A.E., Gesualdo, F.: Twitter mining for fine-grained syndromic surveillance. Artificial Intelligence in Medicine, Special Issue on Text Mining and Information Analysis (in press, 2014)Google Scholar
- 14.Stilo, G., De Vincenzi, M., Tozzi, A.E., Velardi, P.: Automated Learning of Everyday Patients’ Language for Medical Blog Analytics. In: Proceedings of the Recent Advances in Natural Language Processing (RANLP 2013) Hissar, Hissar (Bulgaria), September 9-11 (2013)Google Scholar