Personal and Ubiquitous Computing

, Volume 21, Issue 3, pp 457–473 | Cite as

Designing parallel data processing for enabling large-scale sensor applications

  • Milan Kabáč
  • Charles Consel
  • Nic Volanschi
Original Article


Masses of sensors are being deployed at the scale of cities to manage parking spaces, transportation infrastructures to monitor traffic, and campuses of buildings to reduce energy consumption. These large-scale infrastructures become a reality for citizens via applications that orchestrate sensors to deliver high-value, innovative services. These applications critically rely on the processing of large amounts of data to analyze situations, inform users, and control devices. This paper proposes a design-driven approach to developing orchestrating applications for masses of sensors that integrates parallel processing of large amounts of data. Specifically, an application design exposes declarations that are used to generate a programming framework based on the MapReduce programming model. We have developed a prototype of our approach, using Apache Hadoop. We applied it to a case study and obtained significant speedups by parallelizing computations over twelve nodes. In doing so, we demonstrate that our design-driven approach allows to abstract over implementation details, while exposing architectural properties used to generate high-performance code for processing large datasets. Furthermore, we show that this high-performance support enables new, personalized services in a smart city. Finally, we discuss the expressiveness of our design language, identify some limitations, and present language extensions.


Data processing Programming frameworks Sensors MapReduce Orchestration 


  1. 1.
    Garcia-Ayllon S, Miralles JL (2015) New strategies to improve governance in territorial management: evolving from smart cities to smart territories. Procedia Eng 118:3–11CrossRefGoogle Scholar
  2. 2.
    Mizuno Y, Odake N (2015) Current status of smart systems and case studies of privacy protection platform for smart city in Japan. In: 2015 Portland international conference on management of engineering and technology (PICMET), pp 612–624Google Scholar
  3. 3.
    Naphade M, Banavar G, Harrison C, Paraszczak J, Morris R (2011) Smarter cities and their innovation challenges. Computer 44(6):32–39CrossRefGoogle Scholar
  4. 4.
    Libelium (2016) Smart City project in Santander to monitor Parking Free Slots [Online].
  5. 5.
    Worldsensing SL (2016) Worldsensing and SIGFOX announce the world’s largest Intelligent Parking deployment with Micronet, the SIGFOX Network Operator for Russia [Online].
  6. 6.
    Lee K-H, Lee Y-J, Choi H, Chung YD, Moon B (2012) Parallel data processing with MapReduce: a survey. SIGMOD Rec 40(4):11–20CrossRefGoogle Scholar
  7. 7.
    Patsakis C, Venanzi R, Bellavista P, Solanas A, Bouroche M (2014) Personalized medical services using smart cities’ infrastructures. In: Proceeding of the 2014 IEEE international symposium on medical measurements and applications (MeMeA). IEEE, pp 665–669Google Scholar
  8. 8.
    Seeliger R, Krauss C, Wilson A, Zwicklbauer M, Arbanowski S (2015) Towards personalized smart city guide services in future internet environments. In: Proceedings of the 24th international conference on World Wide Web, ser. WWW ’15 companion. ACM, New York, pp 563–568 [Online]. doi: 10.1145/2740908.2743905
  9. 9.
    The Apache Software Foundation (2016) Apache Pig [Online].
  10. 10.
    The Apache Software Foundation (2016) Apache Hive [Online].
  11. 11.
    Chambers C, Raniwala A, Perry F, Adams S, Henry RR, Bradshaw R, Weizenbaum N (2010) FlumeJava: easy, efficient data-parallel pipelines. In: Proceedings of the 31st ACM SIGPLAN conference on programming language design and implementation (PLDI ’10). ACM, pp 363–375Google Scholar
  12. 12.
    Kabáč M, Consel C (2015) Orchestrating masses of sensors: a design-driven development approach. In: Proceedings of the 2015 ACM SIGPLAN international conference on generative programming: concepts and experiences (GPCE 2015). ACM, pp 117–120Google Scholar
  13. 13.
    Kabáč M, Consel C (2016) Designing parallel data processing for large-scale sensor orchestration. In: 13th IEEE international conference on ubiquitous intelligence and computing (UIC 2016), Toulouse, France, best Paper Award [Online].
  14. 14.
    Lämmel R (2008) Google’s MapReduce programming model—revisited. Sci Comput Program 70(1):1–30MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Dean J, Ghemawat S (2008) MapReduce: simplified data processing on large clusters. Commun ACM 51(1):107–113CrossRefGoogle Scholar
  16. 16.
    Taylor RN, Medvidovic N, Dashofy EM (2009) Software architecture: foundations, theory, and practice. Wiley, New YorkCrossRefGoogle Scholar
  17. 17.
    Fayad M, Schmidt DC (1997) Object-oriented application frameworks. Commun ACM 40(10):32–38CrossRefGoogle Scholar
  18. 18.
    Tilak S, Abu-Ghazaleh NB, Heinzelman W (2002) A taxonomy of wireless micro-sensor network models. SIGMOBILE Mob Comput Commun Rev 6(2):28–36CrossRefGoogle Scholar
  19. 19.
    The Apache Software Foundation (2016) Hadoop Wiki PoweredBy [Online].
  20. 20.
    White T (2012) Hadoop: the definitive guide. O’Reilly Media Inc, SebastopolGoogle Scholar
  21. 21.
    The Apache Software Foundation (2016) Projects directory [Online].
  22. 22.
    The Apache Software Foundation (2016) Apache Spark [Online].
  23. 23.
    Zaharia M, Chowdhury M, Franklin MJ, Shenker S, Stoica I (2010) Spark: cluster computing with working sets. In: Proceedings of the 2nd USENIX conference on hot topics in cloud computing (HotCloud’10). USENIX Association, pp 10–10Google Scholar
  24. 24.
    Zaharia M, Chowdhury M, Das T, Dave A, Ma J, McCauley M, Franklin MJ, Shenker S, Stoica I (2012) Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Proceedings of the 9th USENIX conference on networked systems design and implementation (NSDI’12). USENIX Association, p 2Google Scholar
  25. 25.
    The Apache Software Foundation (2016) Apache Storm [Online].
  26. 26.
    Cugola G, Margara A (2012) Processing flows of information: from data stream to complex event processing. ACM Comput Surv 44(3):15:1–15:62CrossRefGoogle Scholar
  27. 27.
    Hewlett-Packard (2016) HP Helion eucalyptus [Online].
  28. 28.
    DataStax (2016) DataStax enterprise [Online].
  29. 29.
    Mackowski D, Bai Y, Ouyang Y (2015) Parking space management via dynamic performance-based pricing. Transp Res Procedia 7:170–191CrossRefGoogle Scholar
  30. 30.
    Pfrommer J, Warrington J, Schildbach G, Morari M (2014) Dynamic vehicle redistribution and online price incentives in shared mobility systems. IEEE Trans Intell Transp Syst 15(4):1567–1578CrossRefGoogle Scholar
  31. 31.
    Kabáč M, Consel C, Volanschi N (2016) Leveraging declarations over the lifecycle of large-scale sensor applications. In: 13th IEEE international conference on ubiquitous intelligence and computing (UIC 2016). Toulouse, France [Online].
  32. 32.
    Patel P, Pathak A, Cassou D, Issarny V (2013) Enabling high-level application development in the internet of things. In: S-CUBE’13: 4th international conference on sensor systems and softwareGoogle Scholar
  33. 33.
    Serral E, Valderas P, Pelechano V (2010) Towards the model driven development of context-aware pervasive systems. Pervasive Mob Comput 6(2):254–280CrossRefGoogle Scholar
  34. 34.
    Dey AK, Abowd GD, Salber D (2001) A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Hum Comput Interact 16(2):97–166CrossRefGoogle Scholar
  35. 35.
    Ranganathan A, Chetan S, Al-Muhtadi J, Campbell RH, Mickunas MD (2005) Olympus: a high-level programming model for pervasive computing environments. In: Proceedings of the third IEEE international conference on pervasive computing and communications (PERCOM ’05). IEEE Computer Society, pp 7–16Google Scholar
  36. 36.
    Cassou D, Balland E, Consel C, Lawall J (2011) Leveraging software architectures to guide and verify the development of sense/compute/control applications. In: Proceedings of the 33rd international conference on software engineering (ICSE ’11). ACM, pp 431–440Google Scholar
  37. 37.
    Bertran B, Bruneau J, Cassou D, Loriant N, Balland E, Consel C (2014) DiaSuite: a tool suite to develop sense/compute/control applications. Sci Comput Program 79:39–51CrossRefGoogle Scholar
  38. 38.
    Gupta V, Tovar E, Pinho LM, Kim J, Lakshmanan K, Rajkumar R (2011) sMapReduce: a programming pattern for wireless sensor networks. In: Proceedings of the 2nd workshop on software engineering for sensor network applications (SESENA ’11). ACM, pp 37–42Google Scholar
  39. 39.
    Mottola L, Picco GP (2011) Programming wireless sensor networks: fundamental concepts and state of the art. ACM Comput Surv 43(3):19:1–19:51CrossRefGoogle Scholar
  40. 40.
    Pike R, Dorward S, Griesemer R, Quinlan S (2005) Interpreting the data: parallel analysis with Sawzall. Sci Program 13(4):277–298Google Scholar

Copyright information

© Springer-Verlag London 2017

Authors and Affiliations

  1. 1.Inria BordeauxBordeauxFrance
  2. 2.Bordeaux Institute of TechnologyBordeauxFrance

Personalised recommendations