Skip to main content
SpringerLink
Log in

Big Data

  • Chapter
  • First Online:
Intelligent Internet of Things

  • 2777 Accesses

Abstract

We live in the data age and big data technologies allow to harness data, uncover hidden patterns and correlations in data, discover insights, and improve decision making. Although the Internet of Things (IoT) and big data have been evolved separately, they are closely intertwined. Indeed, the role of big data in IoT is tremendous as IoT is projected to include billions of connected devices, producing a massive amounts of data within a few years. In this context, big data analytics is a key enabler for unlocking the untapped potential of the IoT and fueling a wide range of data-driven products, services, and business processes. Apache Hadoop is the most prominent and used tool in big data. Hadoop is an open source framework that enables distributed processing of large data sets across commodity clusters. It is designed to scale with built-in high availability and reliability. Additional open source projects have been built around the original Hadoop implementation with the addition of Apache Spark. Spark is the next hype in the industry among the big data tools which was designed to address the shortcomings of Hadoop. Spark provides primitives for in-memory cluster computing which can speed jobs that run on the Hadoop data processing platform. This chapter discusses the details of top Big Data processing frameworks being used today enabling automatic and scalable insight discovery from large and complex data. In particular, we explain the origins of Hadoop and Apache Spark, their functionalities, architectures, and practical applications.

Errors using inadequare data are much less than those using no data at all.

Charles Babbage

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 119.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. https://www.ibmbigdatahub.com/infographic/extracting-business-value-4-vs-big-data

  2. https://spectrum.ieee.org/tech-talk/telecom/internet/popular-internet-of-things-forecast-of-50-billion-devices-by-2020-is-outdated

  3. https://www.visualcapitalist.com/internet-minute-2018/

  4. M. Cafarella, B. Lorica, D. Cutting, The next 10 years of Apache Hadoop. (O’Reilly Media, 2016). https://www.oreilly.com/ideas/the-next-10-years-of-apache-hadoop

  5. G. Sanjay, G. Howard, L. Shun-Tak, The Google file system. SIGOPS Oper. Syst. Rev. 37(5), 29–43 (2003). https://doi.org/10.1145/1165389.945450

    Article  Google Scholar 

  6. J. Dean, S. Ghemawat, MapReduce: Simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008). https://doi.org/10.1145/1327452.1327492

    Article  Google Scholar 

  7. M. Bhandarkar, in 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS). MapReduce programming with apache Hadoop, (Atlanta, GA, 2010), pp. 1–1. https://doi.org/10.1109/IPDPS.2010.5470377

  8. P. Merla, Y. Liang, in 2017 IEEE International Conference on Big Data. Data analysis using Hadoop MapReduce environment, (Boston, MA, 2017), pp. 4783–4785. https://doi.org/10.1109/BigData.2017.8258541

  9. https://hadoop.apache.org/

  10. D. Jeffrey, S. Ghemawat, in OSDI, MapReduce: Simplified data processing on large clusters (2004)

    Google Scholar 

  11. S. Ghemawat, H. Gobioff, S. Leung, The Google file system, in Proceedings of the nineteenth ACM symposium on Operating systems principles (SOSP ‘03), (ACM, New York, NY, USA, 2003), pp. 29–43. https://doi.org/10.1145/945445.945450

    Chapter  Google Scholar 

  12. https://pig.apache.org/

  13. https://hive.apache.org/

  14. https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html

  15. K. Shvachko, H. Kuang, S. Radia, R. Chansler, in 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), The Hadoop Distributed File System, (Incline Village, NV, 2010), pp. 1–10. https://doi.org/10.1109/MSST.2010.5496972

  16. https://www.json.org/

  17. https://avro.apache.org/

  18. http://sqoop.apache.org/

  19. E. Capriolo, D. Wampler, J. Rutherglen, Programming Hive: Data Warehouse and Query Language for Hadoop, 1st edn. (O’Reilly Media, Sebastopol, CA, 2012). ISBN-13: 978-1449319335. ISBN-10: 1449319335

    Google Scholar 

  20. K. Thulasiraman, M.N.S. Swamy, 5.7 Acyclic Directed Graphs, Graphs: Theory and Algorithms (Wiley, New York, 1992), p. 118. ISBN 978-0-471-51356-8

    Book  Google Scholar 

  21. https://zookeeper.apache.org/

  22. https://www.usenix.org/legacy/event/atc10/tech/full_papers/Hunt.pdf

  23. H. Fang, in 2015 IEEE International Conference on Cyber Technology in Automation, Control, and Intelligent Systems (CYBER), Managing data lakes in big data era: What’s a data lake and why has it became popular in data management ecosystem, (Shenyang, 2015), pp. 820–824. https://doi.org/10.1109/CYBER.2015.7288049

  24. https://www.epic.com/

  25. M. Zaharia, M. Chowdhury, M.J. Franklin, S. Shenker, I. Stoica, in Proceedings of the 2nd USENIX conference on Hot topics in cloud computing (HotCloud’10), Spark: Cluster computing with working sets. (USENIX Association, Berkeley, CA, USA, 2010), pp. 10–10

    Google Scholar 

  26. https://spark.apache.org/

  27. https://spark.apache.org/sql/

  28. https://spark.apache.org/streaming/

  29. https://spark.apache.org/mllib/

  30. https://spark.apache.org/grapx

  31. V.J. Srinivas, P. Srikanth, K. Thumati, S.H. Nallamala, in Proceedings International Journal of Computer Science Trends and Technology (IJCST), A review study of Apache Spark in Big Data processing, Vol. 4, Issue 3, May/Jun (2016)

    Google Scholar 

  32. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M.J. Franklin, S. Shenker, I. Stoica, in Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation (NSDI’12), Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. (USENIX Association, Berkeley, CA, USA, 2012), pp. 2–2

    Google Scholar 

  33. https://spark.apache.org/docs/latest/rdd-programming-guide.html

  34. https://spark.apache.org/docs/2.2.0/sql-programming-guide.html

  35. M. Zaharia, B. Chambers, Spark: The Definitive Guide (O’Reilly Media, Sebastopol, CA, 2018)

    Google Scholar 

  36. S. Kumar, in 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom), Evolution of Spark framework for simplifying big data analytics, (New Delhi, 2016), pp. 3597–3602

    Google Scholar 

  37. M. Assefi, E. Behravesh, G. Liu, A.P. Tafti, in 2017 IEEE International Conference on Big Data (Big Data), Big data machine learning using apache spark MLlib (Boston, MA, 2017), pp. 3492–3498. https://doi.org/10.1109/BigData.2017.8258338

  38. D. Siegal, J. Guo, G. Agrawal, in 2016 IEEE International Conference on Cluster Computing (CLUSTER), Smart-MLlib: A high-performance machine-learning library, (Taipei, 2016), pp. 336–345. https://doi.org/10.1109/CLUSTER.2016.49

Download references

Author information

Authors and Affiliations

  1. University of California San Diego, San Diego, CA, USA

    Natasha Balac

Authors
  1. Natasha Balac
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Natasha Balac .

Editor information

Editors and Affiliations

  1. Department of ECE, Duke University, Durham, NC, USA

    Farshad Firouzi

  2. Department of ECE, Duke University, Durham, NC, USA

    Krishnendu Chakrabarty

  3. Radyalis LLC, Austin, TX, USA

    Sani Nassif

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Balac, N. (2020). Big Data. In: Firouzi, F., Chakrabarty, K., Nassif, S. (eds) Intelligent Internet of Things. Springer, Cham. https://doi.org/10.1007/978-3-030-30367-9_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-30367-9_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-30366-2

  • Online ISBN: 978-3-030-30367-9

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation