Datalyzer: Streaming Data Applications Made Easy
Nowadays, streaming data are continuously generated from thousands of sources, including social networks, mobile apps, sensors, e-commerce transactions, and many more. Hence, it becomes very useful to build applications able to process these data, with the purpose of filtering interesting parts, monitor their run-time evolution, persist valuable chunks, trigger events upon certain conditions are met and provide analytics. While several frameworks and systems have emerged to create this kind of applications, these systems tend to be low-level, based on complicated APIs, challenging to install and configure for end-users, and requiring from high performant hardware for their execution. Our goal is to lower the entry level to develop, deploy and run streaming applications.
To accomplish this goal, we propose Datalyzer, an approach to create streaming data applications on the cloud based on a visual language. This way, Datalyzer provides a facility to describe streaming data sources in an open way, and a visual language to describe the execution flow of the streaming application. Datalyzer is based on model-based development principles, where code is generated automatically, and then compiled, deployed and executed on the cloud. As a proof of concept, we describe a case study in enterprise systems, and how it can be built using our prototype tool.
KeywordsStreaming data Data transformation Data monitoring Cloud-based development environments Model-based development Code generation
Work partially funded by the Spanish MINECO (TIN2014-52129-R) and the R&D programme of the Madrid Region (S2013/ICE-3006).
- 1.Brambilla, M., Cabot, J., Wimmer, M.: Model-Driven Software Engineering in Practice. Synthesis Lectures on Software Engineering, 2nd edn. Morgan & Claypool Publishers, San Rafael (2017)Google Scholar
- 4.Harth, A., Knoblock, C.A., Stadtmüller, S., Studer, R., Szekely, P.A.: On-the-fly integration of static and dynamic sources. In: COLD. CEUR Workshop Proceedings, vol. 1034 (2013)Google Scholar
- 6.Luckham, D.C.: The power of events - an introduction to complex event processing in distributed enterprise systems. ACM (2005)Google Scholar
- 7.Rettig, L., Khayati, M., Cudré-Mauroux, P., Piórkowski, M.: Online anomaly detection over big data streams. In: 2015 IEEE International Conference on Big Data, pp. 1113–1122. IEEE (2015)Google Scholar
- 9.Tatbul, N.: Streaming data integration: challenges and opportunities. In: IEEE 26th International Conference on Data Engineering Workshops (ICDEW 2010), pp. 155–158 (2010)Google Scholar
- 11.W3C. XML Path Language (XPath) 3.1. https://www.w3.org/TR/xpath-31/
- 12.Zhuang, Z., Feng, T., Pan, Y., Ramachandra, H., Sridharan, B.: Effective multi-stream joining in Apache Samza framework. In: 2016 IEEE International Conference on Big Data, pp. 267–274. IEEE Computer Society (2016). See also https://samza.apache.org/