Skip to main content
SpringerLink
Log in

Towards a Big Data Analytics Framework for IoT and Smart City Applications

  • Chapter
Modeling and Processing for Next-Generation Big-Data Technologies

Part of the book series: Modeling and Optimization in Science and Technologies ((MOST,volume 4))

Abstract

An increasing amount of valuable data sources, advances in Internet of Things and Big Data technologies as well as the availability of a wide range of machine learning algorithms offers new potential to deliver analytical services to citizens and urban decision makers. However, there is still a gap in combining the current state of the art in an integrated framework that would help reducing development costs and enable new kind of services. In this chapter, we show how such an integrated Big Data analytical framework for Internet of Things and Smart City application could look like. The contributions of this chapter are threefold: (1) we provide an overview of Big Data and Internet of Things technologies including a summary of their relationships, (2) we present a case study in the smart grid domain that illustrates the high-level requirements towards such an analytical Big Data framework, and (3) we present an initial version of such a framework mainly addressing the volume and velocity challenge. The findings presented in this chapter are extended results from the EU funded project BIG and the German funded project PEC.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Abadi, D.J., et al.: The Design of the Borealis Stream Processing Engine. In: CIDR, vol. 5, pp. 277–289 (2005)

    Google Scholar 

  2. Atzori, L., Iera, A., Morabito, G.: The internet of things: A survey. Computer Networks 54(15), 2787–2805 (2010)

    Article  MATH  Google Scholar 

  3. Barga, R.S., et al.: Consistent streaming through time: A vision for event stream processing. arXiv preprint cs/0612115 (2006)

    Google Scholar 

  4. Batty, M.: Smart Cities and Big Data, http://www.spatialcomplexity.info/

  5. Bauer, M., Bui, N., Giacomin, P., Gruschka, N., Haller, S., Ho, E., Kernchen, R., Lischka, M., Loof, J.D., Magerkurth, C., Meissner, S., Meyer, S., Nettsträter, A., Lacalle, F.O., Segura, A.S., Serbanati, A., Strohbach, M., Toubiana, V., Walewski, J.W.: IoT-A Project Deliverable D1.2 – Initial Architectural Reference Model for IoT (2011), http://www.iot-a.eu/public/public-documents/d1.2/view (last accessed September 18, 2013)

  6. Bifet, A., Frank, E.: Sentiment knowledge discovery in twitter streaming data. In: Pfahringer, B., Holmes, G., Hoffmann, A. (eds.) DS 2010. LNCS (LNAI), vol. 6332, pp. 1–15. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  7. BIG Project Website, http://www.big-project.eu/ (last accessed September 19, 2013)

  8. Chandrasekaran, S., et al.: TelegraphCQ: continuous dataflow processing. In: ACM SIGMOD International Conference on Management of Data, pp. 668–668. ACM (2003)

    Google Scholar 

  9. Chu, C.-T., Kim, S.K., Lin, Y.A., Yu, Y.Y., Bradski, G., Ng, A.Y., Olukotun, K.: Map-Reduce for Machine Learning on Multicore. In: Schölkopf, B., Platt, J.C., Hoffman, T. (eds.) Advances in Neural Information Processing Systems 19 (NIPS 2006), pp. 281–288. MIT Press, Cambridge (2007)

    Google Scholar 

  10. Correia, Z.P.: Toward a Stakeholder Model for the Co-Production of the Public Sector Information System. Information Research 10(3), paper 228 (2005), http://InformationR.net/ir/10-3/paper228.html (last accessed February 27, 2013)

  11. DataMarket, http://datamarket.com/ (last accessed September 21, 2013)

  12. Data.gov, http://www.data.gov/ (last accessed September 21, 2013)

  13. Davis, J.R., Clodoveu, A., et al.: Inferring the Location of Twitter Messages based on User Relationships. Transactions in GIS 15(6), 735–751 (2011)

    Article  Google Scholar 

  14. Dean, J., Ghemawat, S.: MapReduce: Simplified Data Processing on Large Clusters. Communications of the ACM 51(1), 1–13 (2008), doi:10.1145/1327452.1327492

    Article  Google Scholar 

  15. Directive 2003/98/EC of the European Parliament and of the Council of 17 November 2003 on the re-use of public sector information, http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=CONSLEG:2003L0098:20130717:EN:PDF (last accessed January 13, 2014)

  16. Dohler, M.: Machine-to-Machine Technologies, Applications & Markets. In: 27th IEEE International Conference on Advanced Information Networking and Applications (AINA) (2013)

    Google Scholar 

  17. Dredze, M., Paul, M.J., Bergsma, S., Tran, H.: Carmen: A Twitter Geolocation System with Applications to Public Health (2013)

    Google Scholar 

  18. The Economist, Running out of road (November 2006)

    Google Scholar 

  19. EsperTech, http://esper.codehaus.org (last accessed September 22, 2013)

  20. Etzion, O.: On Off-Line Event Processing. Event Processing Thinking Online Blog (2009), http://epthinking.blogspot.de/2009/02/on-off-line-event-processing.html (last accessed September 17, 2013)

  21. European Open Data Portal, http://open-data.europa.eu/ (last accessed September 21, 2013)

  22. Farroukh, A., Sadoghi, M., Jacobsen, H.-A.: Towards vulnerability-based intrusion detection with event processing. In: 5th ACM International Conference on Distributed Event-based System, pp. 171–182. ACM (2011)

    Google Scholar 

  23. Gazis, V., Strohbach, M., Akiva, N., Walther, M.: A Unified View on Data Path Aspects for Sensing Applications at a Smart City Scale. In: IEEE 27th International Conference onAdvanced Information Networking and Applications Workshops (WAINA 2013), pp. 1283–1288. IEEE Computer Society, Barcelona (2013), doi:10.1109/WAINA.2013.66

    Chapter  Google Scholar 

  24. Gedik, B., Andrade, H., Wu, K.L., Yu, P.S., Doo, M.: SPADE: the system s declarative stream processing engine. In: ACM SIGMOD International Conference on Management of Data, pp. 1123–1134. ACM (2008)

    Google Scholar 

  25. Giraph Project, http://giraph.apache.org/ (last accessed September 21, 2013)

  26. Gubbi, J., Buyya, R., Marusic, S., Palaniswami, M.: Internet of things (IoT): A vision, architectural elements, and future directions. Future Generation Computer Systems (2013)

    Google Scholar 

  27. Hinze, A., Sachs, K., Buchmann, A.: Event-based applications and enabling technologies. In: Third ACM International Conference on Distributed Event-Based Systems (2009)

    Google Scholar 

  28. INFSO D.4 Networked Enterprise & RFID INFSO G.2 Micro & Nanosystems, Internet of Things in 2020 – A roadmap for the Future (September 2008), report available at http://www.smart-systems-integration.org/public/internet-of-things

  29. ITU, The Internet of Things (2005)

    Google Scholar 

  30. Fidler, E., Jacobsen, H.A., Li, G., Mankovski, S.: The PADRES Distributed Publish/Subscribe System. In: FIW, pp. 12–30 (2005)

    Google Scholar 

  31. van Kasteren, T., Ravkin, H., Strohbach, M., Lischka, M., Tinte, M., Pariente, T., Becker, T., Ngonga, A., Lyko, K., Hellmann, S., Morsey, M., Frischmuth, P., Ermilov, I., Martin, M., Zaveri, A., Capadisli, S., Curry, E., Freitas, A., Rakhmawati, N.A., Ul Hassan, U., Iqbal, A.: BIG Project Deliverable D2.2.1 – First Draft of Technical White Papers (2013), http://big-project.eu/deliverables (last accessed September 19, 2013)

  32. Laney, D. 3D Data Management: Controlling Data Volume, Velocity and Variety. Meta Group Research Report (2001), http://blogs.gartner.com/doug-laney/files/2012/01/ad949-3D-Data-Management-Controlling-Data-Volume-Velocity-and-Variety.pdf (last accessed September 21, 2013)

  33. Leeds, D.J.: THE SOFT GRID 2013-2020:Big Data & Utility Analytics for Smart Grid. GTM Research Report (2012), http://www.greentechmedia.com/research/report/the-soft-grid-2013 (last accessed September 21, 2013)

  34. Liu, B.: Sentiment analysis and opinion mining. Synthesis Lectures on Human Language Technologies 5(1), 1–167 (2012)

    Article  Google Scholar 

  35. The London Dahsboard, http://data.london.gov.uk/london-dashboard (last accessed September 21, 2013)

  36. Madden, S.R., Franklin, M.J., Hellerstein, J.M., Hong, W.: TinyDB: An acquisitional query processing system for sensor networks. ACM Transactions on Database Systems 30, 122–173 (2005)

    Article  Google Scholar 

  37. Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C., Hung Byers, A.: Big data: The next frontier for innovation, competition, and productivity. McKinsey Global Institute (2013), http://www.mckinsey.com/insights/business_technology/big_data_the_next_frontier_for_innovation (last accessed August 20, 2013)

  38. Marz, N., Warren, J.: A new paradigm for Big Data. In: Big Data – Principles and Best Practices of Scalable Real-time Data Systems, ch. 1, Manning Publications Co. (to appear), http://www.manning.com/marz/ (last accessed August 16, 2013), ISBN 9781617290343

  39. Miorandi, S., Sicari, F., Pellegrini, D., Chlamtac, I.: Internet of things: Vision, applications and research challenges. Ad Hoc Networks 10(7), 1497–1516 (2012)

    Article  Google Scholar 

  40. Microsoft BI Team, Big Data, Hadoop and StreamInsightTM, http://blogs.msdn.com/b/microsoft_business_intelligence1/archive/2012/02/22/big-data-hadoop-and-streaminsight.aspx (last accessed September 09, 2013)

  41. Rajeev, M., Widom, J., Arasu, A., Babcock, B., Babu, S., Datar, M., Manku, G., Olston, C., Rosenstein, J., Varma, R.: Query processing, approximation, and resource management in a data stream management system. In: CIDR Conference, pp. 1–16 (2002)

    Google Scholar 

  42. Owen, S., Anil, R., Dunning, T., Friedman, E.: Mahout in Action. Manning Publications Co. (2011) ISBN 9781935182689

    Google Scholar 

  43. Neubauer, P.: Neo4j and some graph problems, http://www.slideshare.net/peterneubauer/neo4j-5-cool-graph-examples-4473985?from_search=2 (last accessed September 21, 2013)

  44. Palensky, P., Dietrich, D.: Demand side management: Demand response, intelligent energy systems, and smart loads. IEEE Transactions on Industrial Informatics 7(3), 381–388 (2011)

    Article  Google Scholar 

  45. Pang, B., Lee, L.: Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval 2(1-2), 1–135 (2008)

    Article  Google Scholar 

  46. Peer Energy Cloud project website, http://www.peerenergycloud.de/ (last accessed September 19, 2013)

  47. PredPol, http://www.predpol.com/ (September 21, 2013)

  48. Radoop, http://www.radoop.eu/ (last accessed December 16, 2013)

  49. Redis Project, http://redis.io/topics/faq (last accessed September 17, 2013)

  50. ruleCore, http://www.rulecore.com (last accessed September 22, 2013)

  51. S4 Project, http://incubator.apache.org/s4/ (last accessed August 16, 2013)

  52. Salton, G., Buckley, C.: Term-Weighting Approaches in Automatic Text Retrieval. Information Processing and Management 24(5), 513–523 (1988)

    Article  Google Scholar 

  53. Seo, S., Yoon, E.J., Kim, J., Jin, S., Kim, J.-S., Maeng, S.: HAMA: An Efficient Matrix Computation with the MapReduce Framework. In: 2nd IEEE International Conference on Cloud Computing Technology and Science (CloudCom 2010), pp. 721–726. IEEE Computer Society (2013)

    Google Scholar 

  54. Shigeru, O.: M2M and Big Data to Realize the Smart City. NEC Technical Journal 7(2) (2012)

    Google Scholar 

  55. Siddhi CEP - The Complex Event Processing Engine, http://siddhi.sourceforge.net (last accessed September 22, 2013)

  56. Storm Project, http://storm-project.net/ (last accessed August 16, 2013)

  57. Stonebraker, M.: What Does ‘Big Data’ Mean? (Part 3). BLOG@ACM (2012), http://cacm.acm.org/blogs/blog-cacm/157589-what-does-big-data-mean-part-3/fulltext (last accessed August 16, 2013)

  58. United Nations, World Urbanization Prospects 2011 Revision (2011)

    Google Scholar 

  59. US National Intelligence Council.: Disruptive Civil Technologies: Six Technologies with Potential Impacts on US Interests out to 2025, http://www.fas.org/irp/nic/disruptive.pdf (last accessed December 20, 2013)

  60. Wang, F.-s., Liu, S., Liu, P., Bai, Y.: Bridging physical and virtual worlds: complex event processing for RFID data streams. In: Ioannidis, Y., et al. (eds.) EDBT 2006. LNCS, vol. 3896, pp. 588–607. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  61. Weidlich, M., Ziekow, H., Mendling, J., Günther, O., Weske, M., Desai, N.: Event-based monitoring of process execution violations. In: Rinderle-Ma, S., Toumani, F., Wolf, K. (eds.) BPM 2011. LNCS, vol. 6896, pp. 182–198. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  62. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The WEKA Data Mining Software: An Update. SIGKDD Explorations 11, 10–18 (2009)

    Article  Google Scholar 

  63. White, T.: Hadoop: The Definitive Guide. O’Reilly (2012)

    Google Scholar 

  64. Xively, https://xively.com/ (last accessed September 21, 2013)

  65. Ziekow, H., Doblander, C., Goebel, C., Jacobsen, H.-A.: Forecasting Household Electricity Demand with Complex Event Processing: Insights from a Prototypical Solution. In: Middleware Conference, Beijing, China (2013)

    Google Scholar 

  66. Ziekow, H., Goebel, C., Strüker, J., Jacobsen, H.-A.: The Potential of Smart Home Sensors in Forecasting Household Electricity Demand. In: IEEE International Conference on Smart Grid Communications (SmartGridComm 2013), Vancouver, Canada (2013)

    Google Scholar 

  67. Zillner, S., Rusitschka, S., Munné, R., Lippell, H., Lobillo Vilela, F., Hussain, K., Becker, T., Jung, R., Paradowski, D., Huang, Y.: BIG Project Deliverable D2.3.1 – First Draft of Sector’s Requisites (2013), http://big-project.eu/deliverables (last accessed September 19, 2013)

  68. European Smart Cities project, http://www.smart-cities.eu/model.html

  69. Smart City Week 2012 international conference and exhibition report, October 29-November 2 (2012), http://scw.nikkeibp.co.jp/2013/docs/SCW2012_Conference_Report_vol3.pdf

Download references

Author information

Authors and Affiliations

  1. AGT International, Hilpertstrasse 35, 64295, Darmstadt, Germany

    Martin Strohbach, Holger Ziekow, Vangelis Gazis & Navot Akiva

Authors
  1. Martin Strohbach
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Holger Ziekow
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vangelis Gazis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Navot Akiva
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Martin Strohbach .

Editor information

Editors and Affiliations

  1. Universitat Politècnica de Catalunya, Barcelona, Spain

    Fatos Xhafa

  2. Fukuoka Institute of Technology (FIT), Fukuoka, Fukuoka, Japan

    Leonard Barolli

  3. University of Salerno, Salerno, Italy

    Admir Barolli

  4. Canadian Institute of Technology, Tirana, Albania

    Petraq Papajorgji

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Strohbach, M., Ziekow, H., Gazis, V., Akiva, N. (2015). Towards a Big Data Analytics Framework for IoT and Smart City Applications. In: Xhafa, F., Barolli, L., Barolli, A., Papajorgji, P. (eds) Modeling and Processing for Next-Generation Big-Data Technologies. Modeling and Optimization in Science and Technologies, vol 4. Springer, Cham. https://doi.org/10.1007/978-3-319-09177-8_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-09177-8_11

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-09176-1

  • Online ISBN: 978-3-319-09177-8

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation