Promise and Perils of Big Data Science for Intelligence Community

  • Karan P. JaniEmail author
  • Anmol Soni
Part of the Advanced Sciences and Technologies for Security Applications book series (ASTSA)


Collecting, processing, and analyzing digital data in bulk holds critical importance today more than it has at any period in past, or, arguably, in future. Big data science is influencing the global financial, industrial, academic as well as defense sectors. With the exponential rise of open source data from social media and increasing government monitoring, big data science is now closely aligned with national security policies, and the intelligence community. This chapter reviews the role that big data sciences can play in supporting functions of the intelligence community. A major part of the chapter focuses on the inherent limitations of big data, which can affect and even disrupt the chain of operations of intelligence agencies from gathering information to anticipating surprises. The limiting factors range from technical to ethical issues. The chapter concludes that there is a continuing need for experts with domain knowledge from intelligence community to efficiently guide big data analysis to fill any gaps in knowledge. As a case study on limitations of using big data, work in nuclear intelligence using simple analytics is examined to show why big data analysis in certain cases may lead to unnecessary complications.


  1. Andrieu, C., De Freitas, N., Doucet, A., & Jordan, M. (2003). An introduction to MCMC for machine learning. Machine Learning, 50(1), 5–43.CrossRefGoogle Scholar
  2. Arbesman, S. (2013). Stop Hyping Big Data and Pay Attention to Long Data, WIRED, 2013
  3. Baxter, P. (2015). The False Hope of Nuclear Forensics? Assessing the Timeliness of Forensics Intelligence. FAS
  4. Chandrasekaran, N. (2015). Is data the new currency? World Economic Forum Retrieved April 29, 2017 from
  5. Cox, M. & Ellsworth, D. (1997). Application-controlled demand paging for out-of-core visualization. Retrieved April 29, 2017 from
  6. Defense Science Board Task Force Report. (2014, January). Assessment of Nuclear Monitoring and Verification Technologies,
  7. Detsch, J. (2015). The mixed consequences of Sino-Indian competition in the Indian Ocean, The Diplomat
  8. Diaz, J. (2013). Internet Explorer vs Murder Rate Will Be Your Favorite Chart Today, Gizmodo,
  9. (2014, November). Department of Energy Awards $425 Million for Next Generation Supercomputing Technologies.
  10. Fahey, S. (2012). Big Data Analytics and National Security
  11. Gandomi, A. Haider, M. (2015). Beyond the hype: Big data concepts, methods, and analytics. International Journal of Information Management. 35 2 2015137–144. ISSN 0268-4012,
  12. Gerbaudo, P. (2012). Tweets and the streets: Social media and contemporary activism. London: Pluto Press.Google Scholar
  13. Harbert, T. (2015). IARPA’s New Director Wants You to Surprise Him, IEE Spectrum,
  14. Howard, P. N., Duffy, A., Freelon, D., Hussain, M. M., Mari, W., & Maziad, M. (2011). Opening closed regimes: What was the role of social media during the Arab spring?, SSRN,
  15. Jain, S., Rosenblatt, M., Duke, J. (2014). Is big data the new frontier for academic-industry collaboration?. Journal of the American Medical Association. 2014;311(21):2171–2172. doi:,
  16. Lant, K. (2017). Apple Is Using AI to Unlock Previously Unusable “Dark Data”, Futurism
  17. Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014). The Parable of Google Flu: Traps in Big Data Analysis. Science, 343(6176, March 14), 1203–1205.CrossRefGoogle Scholar
  18. Lee, D. (2017, November 22). Uber concealed huge data breach. BBC News.
  19. Makuch, B. (2014). The Database Tracking the World’s News Could Predict Future Conflicts, Motherboard,
  20. Manjoo, F. (2017). Tech’s frightful five: They’ve got us. New York Times. Retrieved September 16, 2017, from
  21. March, G. & Davis, E. (2014). Eight (No, Nine!) Problems With Big Data. New York Times,
  22. McKinsey (2012). McKinsey Global Institute, Big Data: The next frontier for innovation, competition, and productivity Report, June, 2012.Google Scholar
  23. Metropolis, N. (1987). The Beginning of the Monte Carlo Method. Los Alamos Science, No. 15, p. 125,
  24. MIT Sloan (2017). What’s Your Data Worth? MIT Sloan Management Review. Retrieved September 16, 2017, from
  25. Mitroff, S., Sharpe, B. (2017). Using big data to solve real problems through academic and industry partnerships, Current Opinion in Behavioral Sciences 1891–96, ISSN 2352-1546,
  26. NextGov (2014) Stop Loose Nukes With Big Data And Crowdsourcing, Experts Urge
  27. Norrie J. (2017). Big Data Needs Bigger Security, U.S. News, Scholar
  28. Omand, D., et al. (2012). Introducing social media intelligence (SOCMINT). Intelligence and National Security, 27, 801–823. Scholar
  29. Pomerleau, M. (2015). IC embraces open source intel, even if it is double-edged. Defense Systems,
  30. Preston, J. (2011). Syria restores access to Facebook and YouTube, New York Times,
  31. Richards, N. M. & King, J.H. (2013). Three Paradoxes of Big Data. Stanford Law Review.
  32. Richelson, J. T. (2002). The Wizards of Langley: Inside The Cia’s Directorate Of Science And Technology (p. 386 pp., apps., bibl., index. Boulder, Colo). Westview Press.Google Scholar
  33. Roberto, V. Z. (2013). Big Data: Challenges and Opportunities.
  34. Rosenbach, E. and Peritz, A. (2009). “The role of private corporations in the intelligence community.” Memorandum, “Confrontation or Collaboration? Congress and the intelligence community,” Belfer Center for Science and International Affairs, Harvard Kennedy School, July 2009.Google Scholar
  35. Shane, S. (2017). These are the Ads Russia Bought on Facebook in 2016, New York Times,
  36. SINTEF. (2013). Big Data, for better or worse: 90% of world’s data generated over last two years. ScienceDaily. Retrieved April 29, 2017 from
  37. Soubra D. (2012). The 3Vs that define Big Data, Data Science Central,
  38. Symon, P. and Tarapore, A. (2015). Defense intelligence analysis in the age of big data. Forum of Defense Intelligence and Big Data, 2015.Google Scholar
  39. Temple-Retson, D (2012). Predicting The Future: Fantasy Or A Good Algorithm? NPR
  40. The Wire (2017). FAQ: What the Right to Privacy Judgment Means for Aadhaar and Mass Surveillance, The Wire
  41. U.S. Joint Chiefs of Staff (2013). Joint Intelligence Report
  42. Ward, J. S., & Barker, A. (2013). Undefined by data: A survey of big data definitions. arXiv preprint arXiv:1309.5821.Google Scholar
  43. White House (2014). Big Data: Seizing opportunities, preserving values. Executive Office of the President.
  44. White House (2016). Big Data: A Report on Algorithmic Systems, Opportunity, and Civil Rights. Executive Office of the President.
  45. Wiener J., Facebook (2016). Facebook’s Top Open Data Problems,
  46. World Institute for Nuclear Security (2015). Data Analytics for Nuclear Security,
  47. Young C. (2012). Military intelligence Redefined: Big Data in the Battlefield, Forbes, March 2012

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Georgia Institute of TechnologyAtlantaUSA

Personalised recommendations