Advertisement

On Data Stream Processing in IoT Applications

  • Dmitry NamiotEmail author
  • Manfred Sneps-Sneppe
  • Romass Pauliks
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11118)

Abstract

This article is devoted to the issues of streaming data processing in the Internet of Things applications. Stream processing is a natural fit for the Internet of Things applications. Most of the data models on the Internet Things are exactly the data streams. Accordingly, most applications (business applications) are oriented to processing real-time data streams (e.g., search for anomalies, provide billing features, etc.). The paper considers the architecture of data processing systems, classifies stream processing patterns. Much attention is paid to time management in stream processing systems. The review is conducted from the standpoint of the contents of the master’s course on stream data processing in the Internet of Things and Industrial Internet of Things applications. Also, the paper considers the specific application models and streaming data architecture for the Internet of Things applications as well basic data analysis algorithms that are used in such systems.

Keywords

Internet of Things Stream Data mining 

Notes

Acknowledgement

We would like to thank the reviewers of the EUCNC conference for critical comments on the first versions of this work. Also we are grateful to the employees of the Laboratory of Open Information Technologies of the Lomonosov Moscow State University and Professor V.A. Sukhomlin for valuable discussions.

References

  1. 1.
    Garofalakis, M., Gehrke, J., Rastogi, R. (eds.): Data Stream Management: Processing High-Speed Data Streams. Springer, Heidelberg (2016)Google Scholar
  2. 2.
    EU FP7 CityPulse. http://www.ict-citypulse.eu. Accessed 24 May 2018
  3. 3.
    Tönjes, R., et al.: Real time iot stream processing and large-scale data analytics for smart city applications. Poster session, European Conference on Networks and Communications (2014)Google Scholar
  4. 4.
    Namiot, D., Ventspils, M.S.S., Daradkeh, Y.I.: On Internet of Things education. In: 2017 20th Conference of Open Innovations Association (FRUCT), pp. 309–315. IEEE, April 2017Google Scholar
  5. 5.
    Rose, D.: Enchanted Objects: Design, Human Desire, and the Internet of Things. Simon and Schuster, New York (2014)Google Scholar
  6. 6.
    Namiot, D., Sneps-Sneppe, M.: On Internet of Things and big data in university courses. Int. J. Embed. Real-Time Commun. Syst. (IJERTCS) 8(1), 18–30 (2017)CrossRefGoogle Scholar
  7. 7.
    Namiot, D., Sneps-Sneppe, M.: On data persistence models for mobile crowdsensing applications. In: Kalinichenko, L., Kuznetsov, S., Manolopoulos, Y. (eds.) Data Analytics and Management in Data Intensive Domains. DAMDID/RCDL 2016. Communications in Computer and Information Science, vol. 706, pp. 192–204. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-57135-5_14Google Scholar
  8. 8.
    Lambda architecture. http://lambda-architecture.net. Accessed 24 May 2018
  9. 9.
    Kappa Architecture. http://milinda.pathirage.org/kappa-architecture.com. Accessed 24 May 2018
  10. 10.
    Questioning the Lambda Architecture. https://www.oreilly.com/ideas/questioning-the-lambda-architecture. Accessed 24 May 2018
  11. 11.
    Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., Tzoumas, K.: Apache flink: stream and batch processing in a single engine. Bull. IEEE Comput. Soc. Tech. Committee Data Eng. 36(4), 28–38 (2015)Google Scholar
  12. 12.
    Akidau, T., et al.: The dataflow model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing. Proc. VLDB Endowment 8(12), 1792–1803 (2015)CrossRefGoogle Scholar
  13. 13.
    The world beyond batch: Streaming 101. https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101. Accessed 24 May 2018
  14. 14.
    Zaharia, M., et al.: Fast and interactive analytics over Hadoop data with Spark. USENIX Login 37(4), 45–51 (2012)MathSciNetGoogle Scholar
  15. 15.
  16. 16.
    Sneps-Sneppe, M., Namiot, D.: About M2M standards and their possible extensions. In: 2012 2nd Baltic Congress on Future Internet Communications (BCFIC), pp. 187–193. IEEE, April 2012Google Scholar
  17. 17.
    Namiot, D.: On big data stream processing. Int. J. Open Inf. Technol. 3(8), 48–51 (2015)Google Scholar
  18. 18.
    Golab, L., Özsu, M.T.: Issues in data stream management. ACM SIGMOD Rec. 32(2), 5–14 (2003)CrossRefGoogle Scholar
  19. 19.
    Motwani, R., et al.: Query processing, resource management, and approximation in a data stream management system. In: CIDR, January 2003Google Scholar
  20. 20.
    Gama, J., Gaber, M.M. (eds.): Learning from Data Streams: Processing Techniques in Sensor Networks. Springer, Heidelberg (2007)Google Scholar
  21. 21.
    Kreps, J., Narkhede, N., Rao, J.: Kafka: a distributed messaging system for log processing. In: Proceedings of the NetDB, pp. 1–7, June 2011Google Scholar
  22. 22.
    Gartner says the Internet of Things will transform the data center. http://www.gartner.com/newsroom/id/2684616. Accessed 24 May 2018
  23. 23.
    Standardization Activities of oneM2M. https://www.ntt-review.jp/archive/ntttechnical.php?contents=ntr201408gls.html. Accessed 24 May 2018
  24. 24.
    Gaber, M.M., Zaslavsky, A., Krishnaswamy, S.: Mining data streams: a review. ACM SIGMOD Rec. 34(2), 18–26 (2005)CrossRefGoogle Scholar
  25. 25.
    Rivetti, N., Busnel, Y., Querzoni, L.: Load-aware shedding in stream processing systems. In: Proceedings of the 10th ACM International Conference on Distributed and Event-Based Systems, pp. 61–68. ACM, June 2016Google Scholar
  26. 26.
    Babcock, B., Babu, S., Datar, M., Motwani, R., Widom, J.: Models and issues in data stream systems. In: Proceedings of the Twenty-First ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, pp. 1–16. ACM, June 2002Google Scholar
  27. 27.
    Larsen, K.G., Nelson, J., Nguyên, H.L., Thorup, M.: Heavy hitters via cluster-preserving clustering. In: 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), pp. 61–70. IEEE, October 2016Google Scholar
  28. 28.
    Papadimitriou, S., Sun, J., Faloutsos, C.: Streaming pattern discovery in multiple time-series. In: Proceedings of the 31st International Conference on Very Large Data Bases, pp. 697–708. VLDB Endowment, August 2005Google Scholar
  29. 29.
    Big Data Processing with Apache Spark - Part 3: Spark Streaming. https://www.infoq.com/articles/apache-spark-streaming. Accessed 24 May 2018
  30. 30.
    Hirzel, M., et al.: IBM streams processing language: analyzing big data in motion. IBM J. Res. Dev. 57(3/4), 7:1–7:11 (2013)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Dmitry Namiot
    • 1
    Email author
  • Manfred Sneps-Sneppe
    • 2
  • Romass Pauliks
    • 2
  1. 1.Faculty of Computational Mathematics and CyberneticsLomonosov Moscow State UniversityMoscowRussia
  2. 2.Ventspils International Radio Astronomy CentreVentspils University CollegeVentspilsLatvia

Personalised recommendations