Advertisement

Queuing-Based Processing Platform for Service Delivery in Big Data Environments

  • Florin Stancu
  • Dan Popa
  • Loredana-Marsilia Groza
  • Florin Pop
Conference paper
Part of the Lecture Notes in Business Information Processing book series (LNBIP, volume 247)

Abstract

Service Delivery is one of the most important aspects in every nowadays platforms. Big Data and all analytics processes and services are responsible for new models of service delivery. In this paper we propose an architecture based on message queues for communication between various data sources (e.g. sensors) and a central application, providing stability of delivered services in case of faults: if the central application does not work, messages from the sensors will remain unused in queue and be consumed when the application will be back on-line. Implementation was achieved with RabbitMQ. Also, we have proposed a web application that will generate statistics based on a large volume of data. When we add a new filter (that will generate new statistics), considered as a new task, it must be taken up by a scheduler. The interface is able to configure how many such tasks can run in parallel. Finally, we implemented the proposed architecture to support faults and to be scalable.

Keywords

Queuing systems Batch processing Real-time processing Big data processing Smart cities 

Notes

Acknowledgment

The research presented in this paper is supported by projects: DataWay: Real-time Data Processing Platform for Smart Cities: Making sense of Big Data - PN-II-RU-TE-2014-4-2731; CyberWater grant of the Romanian National Authority for Scientific Research, UEFISCDI, project 47/2012; clueFarm: Information system based on cloud services accessible through mobile devices, to increase product quality and business development farms - PN-II-PT-PCCA-2013-4-0870.

We would like to thank the reviewers for their time and expertise, constructive comments and valuable insight.

References

  1. 1.
    Snyder, B., Bosnanac, D., Davies, R.: ActiveMQ in Action. Manning (2011)Google Scholar
  2. 2.
    Kreps, J., Narkhede, N., Rao, J., et al.: Kafka: a distributed messaging system for log processing. In: Proceedings of the NetDB, pp. 1–7 (2011)Google Scholar
  3. 3.
    Lampkin, V., Leong, W.T., Olivera, L., Rawat, S., Subrahmanyam, N., Xiang, R., Kallas, G., Krishna, N., Fassmann, S., Keen, M., et al.: Building Smarter Planet Solutions with mqtt and IBM Websphere MQ Telemetry. IBM Redbooks (2012)Google Scholar
  4. 4.
    Krafzig, D., Banke, K., Slama, D.: Enterprise SOA: Service-Oriented Architecture Best Practices. Prentice Hall Professional, Upper Saddle River (2005)Google Scholar
  5. 5.
    Hohpe, G., Woolf, B.: Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions. Addison-Wesley Professional, Upper Saddle River (2004)Google Scholar
  6. 6.
    Brydon, S.P., Singh, I.: Web services message broker architecture. US Patent 7, pp. 702–724, 20 April 2010Google Scholar
  7. 7.
    Fiosina, J., Fiosins, M.: Resampling based modelling of individual routing preferences in a distributed traffic network. Int. J. Artif. Intell. 12(1), 79–103 (2014)Google Scholar
  8. 8.
    Demirkan, H., Delen, D.: Leveraging the capabilities of service-oriented decision support systems: putting analytics and big data in cloud. Decis. Support Syst. 55(1), 412–421 (2013)CrossRefGoogle Scholar
  9. 9.
    Videla, A., Williams, J.: Rabbitmq in action: distributed messaging for everyone. Rabbit MQ in action (2012)Google Scholar
  10. 10.
    Dossot, D.: RabbitMQ Essentials. Packt Publishing Ltd (2014)Google Scholar
  11. 11.
    Krishnan, S., Gonzalez, J.L.U.: Google compute engine. In: Building Your Next Big Thing with Google Cloud Platform, pp. 53–81. Springer (2015)Google Scholar
  12. 12.
    Vasile, M.A., Pop, F., Tutueanu, R.I., Cristea, V., Kolodziej, J.: Resource-aware hybrid scheduling algorithm in heterogeneous distributed computing. Future Gener. Comput. Syst. 51(C), 61–71 (2015)CrossRefGoogle Scholar
  13. 13.
    Makpaisit, P., Marurngsith, W.: Griffon-gpu programming apis for scientific and general purpose computing (extended version). Int. J. Artif. Intell. 8(S12), 223–238 (2012)Google Scholar
  14. 14.
    Costa, Â., Novais, P.: Mobile sensor systems on outpatients. Int. J. Artif. Intell. 8(S12), 252–268 (2012)Google Scholar
  15. 15.
    Neumeyer, L., Robbins, B., Nair, A., Kesari, A.: S4: distributed stream computing platform. In: Proceedings of the 2010 IEEE International Conference on Data Mining Workshops. ICDMW 2010, pp. 170–177. IEEE, Washington, DC (2010)Google Scholar
  16. 16.
    Toshniwal, A., Taneja, S., Shukla, A., Ramasamy, K., Patel, J.M., Kulkarni, S., Jackson, J., Gade, K., Fu, M., Donham, J., Bhagat, N., Mittal, S., Ryaboy, D.: Storm@twitter. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data. SIGMOD 2014, USA, pp. 147–156. ACM (2014)Google Scholar
  17. 17.
    Zaharia, M., Das, T., Li, H., Shenker, S., Stoica, I.: Discretized streams: an efficient and fault-tolerant model for stream processing on large clusters. In: Proceedings of the 4th USENIX conference on Hot Topics in Cloud Ccomputing, p. 10. USENIX Association (2012)Google Scholar
  18. 18.
    Sun, Q., Yu, W., Kochurov, N., Hao, Q., Hu, F.: A multi-agent-based intelligent sensor and actuator network design for smart house and home automation. J. Sens. Actuator Netw. 2(3), 557–588 (2013)CrossRefGoogle Scholar
  19. 19.
    Oliveira, T.J.M., Costa, Â., Neves, J., Novais, P.: A comprehensive clinical guideline model and a reasoning mechanism for aal systems. Int. J. Artif. Intell. 11(A13), 57–73 (2013)Google Scholar
  20. 20.
    Benazzouz, Y., Chikhaoui, B., Abdulrazak, B.: An argumentation based approach for dynamic service composition in ambient intelligence environments. Int. J. Artif. Intell. 4(S10), 137–152 (2010)Google Scholar
  21. 21.
    Gomez, C., Paradells, J.: Wireless home automation networks: a survey of architectures and technologies. IEEE Comm. Mag. 48(6), 92–101 (2010)CrossRefGoogle Scholar
  22. 22.
    Shelby, Z., Bormann, C.: 6LoWPAN: The Wireless Embedded Internet. John Wiley & Sons, New York (2011)Google Scholar
  23. 23.
    Cao, H., Leung, V., Chow, C., Chan, H.: Enabling technologies for wireless body area networks: a survey and outlook. IEEE Comm. Mag. 47(12), 84–93 (2009)CrossRefGoogle Scholar
  24. 24.
    Hargreaves, T., Hauxwell-Baldwin, R., Coleman, M., Wilson, C., Stankovic, L., Stankovic, V., Murray, D., Liao, J., Kane, T., Firth, S., et al.: Smart homes, control and energy management: how do smart home technologies influence control over energy use and domestic life? European Council for an Energy Efficient Economy (ECEEE) 2015 Summer Study Proceedings, pp. 1022–1032 (2015)Google Scholar
  25. 25.
    Johansen, N.T.: Z-wave protocol overview (zensys), document no. sds 10243 (2006)Google Scholar
  26. 26.
    Gill, K., Yang, S.H., Yao, F., Lu, X.: A zigbee-based home automation system. IEEE Trans. Consum. Electron. 55(2), 422–430 (2009)CrossRefGoogle Scholar
  27. 27.
    Alliance, Z.: ZigBee Home Automation Public Application Profile (2007)Google Scholar
  28. 28.
    Ghaffarian Hoseini, A.H., Dahlan, N.D., Berardi, U., Ghaffarian Hoseini, A., Makaremi, N.: The essence of future smart houses: From embedding ict to adapting to sustainability principles. Renewable and Sustainable Energy Reviews 24, 593–607 (2013)CrossRefGoogle Scholar
  29. 29.
    Bessis, N., Sotiriadis, S., Pop, F., Cristea, V.: Optimizing the energy efficiency of message exchanging for service distribution in interoperable infrastructures. In: 2012 4th International Conference on Intelligent Networking and Collaborative Systems (INCoS), pp. 105–112. IEEE (2012)Google Scholar
  30. 30.
    Dragoicea, M., Patrascu, M., Serea, G.A.: Real time agent based simulation for smart city emergency protocols. In: 2014 18th International Conference on System Theory, Control and Computing (ICSTCC), pp. 187–192. IEEE (2014)Google Scholar
  31. 31.
    Patrascu, M., Dragoicea, M., Ion, A.: Emergent intelligence in agents: a scalable architecture for smart cities. In: 2014 18th International Conference on System Theory, Control and Computing (ICSTCC), pp. 181–186. IEEE (2014)Google Scholar
  32. 32.
    Skön, J.P., Johansson, M., Raatikainen, M., Haverinen-Shaughnessy, U., Pasanen, P., Leiviskä, K., Kolehmainen, M.: Analysing events and anomalies in indoor air quality using self-organizing maps. Int. J. of Artif. Intell. 9(A12), 79–89 (2012)Google Scholar
  33. 33.
    Ellis, C., Hazas, M., Scott, J.: Matchstick: A room-to-room thermal model for predicting indoor temperature from wireless sensor data. In: Proceedings of the 12th International Conference on Information processing in Sensor Networks, pp. 31–42. ACM (2013)Google Scholar
  34. 34.
    Yamazaki, T., Kamimura, K., Kurosu, S., Yamakawa, Y.: Air-conditioning PID control system with adjustable reset to offset thermal loads upsets. In: Yurkevich, V.D., (Ed.) Advances in PID Control, InTech.INTECH (2011)Google Scholar
  35. 35.
    Lachapelle, A.C., Love, J.A.: Simulink\({\textregistered }\) model of single co2 sensor location impact on CO2 levels in recirculating multiple-zone systems. In: Proceedings of eSim 2012: The Canadian Conference on Building Simulation, ESIM.CA, pp. 189–201 (2012)Google Scholar
  36. 36.
    Kuch, J.: Rabbitmq hits one million messages per second on google compute engine, point of view (2014). https://blog.pivotal.io/pivotal/products/rabbitmq-hits-one-million-messages-per-second-on-google-compute-engine

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Florin Stancu
    • 1
  • Dan Popa
    • 1
  • Loredana-Marsilia Groza
    • 1
  • Florin Pop
    • 1
  1. 1.Computer Science Department, Faculty of Automatic Control and ComputetsUniversity Politehnica of BucharestBucharestRomania

Personalised recommendations