Building a Data Pipeline for the Management and Processing of Urban Data Streams
Urban data streams (UDS) originate from various sensors and Internet of Things (IoT) devices deployed in smart cities as well as social media sources such as Twitter and Facebook. The large volumes of urban data need to be harnessed to help smart city stakeholders and applications make informed decisions on the fly. Furthermore, effective management and governance of smart city components relies on the ability to integrate and federate their data, process urban data streams locally, and use big data analytics. Data integration and interoperability is a challenging problem that smart cities are facing today. Successful data integration is crucial for improved services and governance. This chapter describes a framework that aims to serve in building a data pipeline for the acquisition and processing of urban data streams, urban data analytics, and creation of value-added services. The framework relies on latest technologies for data processing including IoT, edge computing, data integration techniques, cloud computing, and data analytics. The proposed platform will facilitate real-time event detection, notification of alerts, mining the opinions of citizens regarding the governance of their city, and building monitoring dashboards. A prototype of the platform is being implemented using the Kafka messaging platform.
KeywordsSmart cities Internet of Things Data integration Data Interoperability Data streams processing Messaging Queue
- 1.Anastasi G et al. (2013) Urban and social sensing for sustainable mobility in smart cities. in Proc. IFIP/IEEE Int. Conf. Sustainable Internet ICT Sustainability, Palermo, Italy, pp. 1–4.Google Scholar
- 4.DataTorrent (2016) Real-Time Event Stream Processing – What are your choices?” https://www.datatorrent.com/blog/real-time-event-stream-processing-what-are-your-choices/. Latest access on Dec. 05. 2017.
- 6.Jugel U, Jerzak Z, Hackenbroich G, and Markl V (2014) M4 - A Visualization-Oriented Time Series Data Aggregation. PVLDB, Volume 7 Issue 10, pp. 797–808.Google Scholar
- 8.Ramparany F and Cao Q H (2016) A semantic approach to IoT data aggregation and interpretation applied to home automation. Internationl Conference on Internet of Things and Applications (IOTA), pp. 23–28.Google Scholar
- 9.Rosi A et al. (2011) Social sensors and pervasive services: Approaches and perspectives. Proc. IEEE Int. Conf. PERCOM Workshops, Seattle, WA, USA, pp. 525–530.Google Scholar
- 11.Vaccari A, Liu L, Biderman A, Ratti C, Pereira F, Oliveirinha J, Gerber A (2009) A holistic framework for the study of urban traces and the profiling of urban processes and dynamics. Proc. 12th International IEEE Conference on Intelligent Transportation Systems, IEEE Press, New York, pp. 273–278.Google Scholar
- 12.Hosebird Client (hbc). Retrieved from: https://github.com/twitter/hbc
- 13.Barnaghi P, Tönjes R, Höller J, Hauswirth M, Sheth A T, Anantharam P. (2015) Citypulse: Real-time iot stream processing and large-scale data analytics for smart city applications. ict-citypulse.eu Deliverable D.3.2: Data Federation and Aggregation in Large-Scale Urban Data Streams.Google Scholar
- 14.Computing (2014) London Westminster City Council introduces smart parking system. https://www.computing.co.uk/ctg/news/2323408/london-westminster-city-council-introduces-smart-parking-system. Latest access on Dec. 05. 2017.
- 15.Diginomica.com (2017) Bright Lights. Smart City. San Diego’s pioneering IoT platform. https://diginomica.com/2017/10/31/bright-lights-smart-city-san-diegos-pioneering-iot-platform/. Latest access on Dec. 05. 2017.
- 16.Fraunhofer FOKUS Institute (2012) FixMyCity. https://www.fokus.fraunhofer.de/04c06110dd0adebc. Latest Access on Dec. 05. 2017.
- 17.Gyrard A and Serrano M (2016) Connected Smart Cities - Interoperability with SEG 3.0 for the Internet of Things. Advanced Information Networking and Applications(AINA 2016) Workshops.Google Scholar
- 19.IDC.com (2017) IDC FutureScape: Worldwide Internet of Things 2017 Predictions. https://www.idc.com/research/viewtoc.jsp?containerId=US40755816. Latest Access on Dec. 05. 2017.
- 20.Loshin D (2009) Data Consolidation and Integration. Master Data Management, Elsevier, pp. 177–199.Google Scholar
- 22.Wired.com (2015) The 20 Most Bike-Friendly Cities on the Planet. https://www.wired.com/2015/06/copenhagenize-worlds-most-bike-friendly-cities/ Latest access on Dec. 05. 2017.
- 23.Aloi G et al. (2016) A Mobile Multi-Technology Gateway to Enable IoT Interoperability. Proc. IEEE First International Conference on Internet-of-Things Design and Implementation (IoTDI), pp. 259–264.Google Scholar
- 25.Blackstock M and Lea R (2014) IoT interoperability: A hub-based approach. Proc. IEEE International Conference on the Internet of Things (IOT), pp. 79–84.Google Scholar
- 26.Asensio A, Marco A, Blasco R, Casas R (2014) Protocol and Architecture to Bring Things into Internet of Things. International Journal of Distributed Sensor Networks, Article ID 158252.Google Scholar
- 27.Blackstock M, Kaviani N, Lea R, and Friday A (2010) MAGIC Broker 2: An open and extensible platform for the Internet of Things. Proc. the 2010 Internet of Things (IOT), pp. 1–8.Google Scholar
- 28.Perera C, Jayaraman P P, Zaslavsky A, Christen P and Georgakopoulos D (2014) MOSDEN: An Internet of Things Middleware for Resource Constrained Mobile Devices. 47th Hawaii International Conference on System Sciences, Waikoloa, HI, pp. 1053–1062.Google Scholar
- 29.Feng G, Intizar Ali M, Curry E, and Mileo A (2017) Semantic Discovery and Integration of Urban Data Streams, Future Generation Computer Systems, Volume 76 Issue C, pp. 561–58.Google Scholar
- 30.Kettouch M, Luca C, Khorief O, Rui Wu and Dascalu S (2017) Semantic data management in Smart Cities. International Conference on Optimization of Electrical and Electronic Equipment (OPTIM 2017) & Intl Aegean Conference on Electrical Machines and Power Electronics (ACEMP 2017), pp. 1126–1131.Google Scholar