Building a Data Pipeline for the Management and Processing of Urban Data Streams

  • Elarbi BadidiEmail author
  • Nouf El Neyadi
  • Meera Al Saeedi
  • Fatima Al Kaabi
  • Muthucumaru Maheswaran


Urban data streams (UDS) originate from various sensors and Internet of Things (IoT) devices deployed in smart cities as well as social media sources such as Twitter and Facebook. The large volumes of urban data need to be harnessed to help smart city stakeholders and applications make informed decisions on the fly. Furthermore, effective management and governance of smart city components relies on the ability to integrate and federate their data, process urban data streams locally, and use big data analytics. Data integration and interoperability is a challenging problem that smart cities are facing today. Successful data integration is crucial for improved services and governance. This chapter describes a framework that aims to serve in building a data pipeline for the acquisition and processing of urban data streams, urban data analytics, and creation of value-added services. The framework relies on latest technologies for data processing including IoT, edge computing, data integration techniques, cloud computing, and data analytics. The proposed platform will facilitate real-time event detection, notification of alerts, mining the opinions of citizens regarding the governance of their city, and building monitoring dashboards. A prototype of the platform is being implemented using the Kafka messaging platform.


Smart cities Internet of Things Data integration Data Interoperability Data streams processing Messaging Queue 


  1. 1.
    Anastasi G et al. (2013) Urban and social sensing for sustainable mobility in smart cities. in Proc. IFIP/IEEE Int. Conf. Sustainable Internet ICT Sustainability, Palermo, Italy, pp. 1–4.Google Scholar
  2. 2.
    Ciuccarelli P, Lupi G, Simeone L (2014) Visualizing the Data City: Social Media as a Source of Knowledge for Urban Planning and Management. Springer, Heidelberg.CrossRefGoogle Scholar
  3. 3.
    Cugola G. and Margara A (2012) Processing flows of information: from data stream to complex event processing. ACM Comput. Surv. 44 (3) 15:1–15:62.CrossRefGoogle Scholar
  4. 4.
    DataTorrent (2016) Real-Time Event Stream Processing – What are your choices?” Latest access on Dec. 05. 2017.
  5. 5.
    Hashem I A T, Chang V, Anuar N B, Adewole K, Yaqoob I, Gani A, Ahmed E, and Chiroma H (2016) The role of big data in smart city. International Journal of Information Management, vol. 36, no. 5, pp. 748–758.CrossRefGoogle Scholar
  6. 6.
    Jugel U, Jerzak Z, Hackenbroich G, and Markl V (2014) M4 - A Visualization-Oriented Time Series Data Aggregation. PVLDB, Volume 7 Issue 10, pp. 797–808.Google Scholar
  7. 7.
    Margara A, Urbani J, van Harmelen F, and Bal H (2014) Streaming the Web: Reasoning over dynamic data. Web Semantics: Science, Services and Agents on the World Wide Web, vol. 25, pp. 24–44.CrossRefGoogle Scholar
  8. 8.
    Ramparany F and Cao Q H (2016) A semantic approach to IoT data aggregation and interpretation applied to home automation. Internationl Conference on Internet of Things and Applications (IOTA), pp. 23–28.Google Scholar
  9. 9.
    Rosi A et al. (2011) Social sensors and pervasive services: Approaches and perspectives. Proc. IEEE Int. Conf. PERCOM Workshops, Seattle, WA, USA, pp. 525–530.Google Scholar
  10. 10.
    Tim L M V, Birte U, Vignesh S, Maria E. N (2014) Analyzing Tweets to Aid Situational Awareness. Advances in Information Retrieval, Vol. 8416, Lecture Notes in Computer Science, pp. 700–705.CrossRefGoogle Scholar
  11. 11.
    Vaccari A, Liu L, Biderman A, Ratti C, Pereira F, Oliveirinha J, Gerber A (2009) A holistic framework for the study of urban traces and the profiling of urban processes and dynamics. Proc. 12th International IEEE Conference on Intelligent Transportation Systems, IEEE Press, New York, pp. 273–278.Google Scholar
  12. 12.
    Hosebird Client (hbc). Retrieved from:
  13. 13.
    Barnaghi P, Tönjes R, Höller J, Hauswirth M, Sheth A T, Anantharam P. (2015) Citypulse: Real-time iot stream processing and large-scale data analytics for smart city applications. Deliverable D.3.2: Data Federation and Aggregation in Large-Scale Urban Data Streams.Google Scholar
  14. 14.
    Computing (2014) London Westminster City Council introduces smart parking system. Latest access on Dec. 05. 2017.
  15. 15. (2017) Bright Lights. Smart City. San Diego’s pioneering IoT platform. Latest access on Dec. 05. 2017.
  16. 16.
    Fraunhofer FOKUS Institute (2012) FixMyCity. Latest Access on Dec. 05. 2017.
  17. 17.
    Gyrard A and Serrano M (2016) Connected Smart Cities - Interoperability with SEG 3.0 for the Internet of Things. Advanced Information Networking and Applications(AINA 2016) Workshops.Google Scholar
  18. 18.
    Haas L M, Lin E T, and Roth M T (2002) Data integration through database federation. IBM Systems Journal, 41(4), pp. 578–596.CrossRefGoogle Scholar
  19. 19. (2017) IDC FutureScape: Worldwide Internet of Things 2017 Predictions. Latest Access on Dec. 05. 2017.
  20. 20.
    Loshin D (2009) Data Consolidation and Integration. Master Data Management, Elsevier, pp. 177–199.Google Scholar
  21. 21.
    Trilles S, Calia A, Belmonte Ó, Torres-Sospedra J, Montoliu R, and Huerta J (2016) Deployment of an open sensorized platform in a smart city context. Future Generation Computer Systems, vol. 76, pp. 221–233.CrossRefGoogle Scholar
  22. 22. (2015) The 20 Most Bike-Friendly Cities on the Planet. Latest access on Dec. 05. 2017.
  23. 23.
    Aloi G et al. (2016) A Mobile Multi-Technology Gateway to Enable IoT Interoperability. Proc. IEEE First International Conference on Internet-of-Things Design and Implementation (IoTDI), pp. 259–264.Google Scholar
  24. 24.
    Aloi G, Caliciuri G, Fortino G, Gravina R, Pace P, Russo W, Savaglio C (2017) Enabling IoT interoperability through opportunistic smartphone-based mobile gateways. J.Netw.Comput.Appl. 81, pp. 73–83.CrossRefGoogle Scholar
  25. 25.
    Blackstock M and Lea R (2014) IoT interoperability: A hub-based approach. Proc. IEEE International Conference on the Internet of Things (IOT), pp. 79–84.Google Scholar
  26. 26.
    Asensio A, Marco A, Blasco R, Casas R (2014) Protocol and Architecture to Bring Things into Internet of Things. International Journal of Distributed Sensor Networks, Article ID 158252.Google Scholar
  27. 27.
    Blackstock M, Kaviani N, Lea R, and Friday A (2010) MAGIC Broker 2: An open and extensible platform for the Internet of Things. Proc. the 2010 Internet of Things (IOT), pp. 1–8.Google Scholar
  28. 28.
    Perera C, Jayaraman P P, Zaslavsky A, Christen P and Georgakopoulos D (2014) MOSDEN: An Internet of Things Middleware for Resource Constrained Mobile Devices. 47th Hawaii International Conference on System Sciences, Waikoloa, HI, pp. 1053–1062.Google Scholar
  29. 29.
    Feng G, Intizar Ali M, Curry E, and Mileo A (2017) Semantic Discovery and Integration of Urban Data Streams, Future Generation Computer Systems, Volume 76 Issue C, pp. 561–58.Google Scholar
  30. 30.
    Kettouch M, Luca C, Khorief O, Rui Wu and Dascalu S (2017) Semantic data management in Smart Cities. International Conference on Optimization of Electrical and Electronic Equipment (OPTIM 2017) & Intl Aegean Conference on Electrical Machines and Power Electronics (ACEMP 2017), pp. 1126–1131.Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Elarbi Badidi
    • 1
    Email author
  • Nouf El Neyadi
    • 1
  • Meera Al Saeedi
    • 1
  • Fatima Al Kaabi
    • 1
  • Muthucumaru Maheswaran
    • 2
  1. 1.College of Information Technology, United Arab Emirates UniversityAl-AinUnited Arab Emirates
  2. 2.School of Computer Science, McGill UniversityMontrealCanada

Personalised recommendations