Setting up Clusters of Computing Units to Process Several Data Streams Efficiently

  • Daniel Millot
  • Christian ParrotEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8385)


Let us consider an upper bounded number of data streams to be processed by a Divisible Load application. The total workload is unknown and the available speeds for communicating and computing may be poorly a priori estimated. This paper presents a resource selection method that aims at maximizing the throughput of this processing. From a set of processing units linked by a network, this method consists in forming an optimal set of master-workers clusters. Results of simulations are presented to assess the efficiency of this method experimentally. Before focusing on the proposed resource selection method, the paper comes back on the adaptive scheduling method on which it relies.


Adaptive scheduling Parallel processing Master-worker model Load balancing Heterogeneous context Dynamic context 


  1. 1.
    Lee, C., Hamdi, M.: Parallel image processing application in a network of workstation. Parallel Comput. 21, 137–160 (1995)CrossRefzbMATHGoogle Scholar
  2. 2.
    Altılar, D.T., Paker, Y.: An optimal scheduling algorithm for stream based parallel video processing. In: Yazıcı, A., Şener, C. (eds.) ISCIS 2003. LNCS, vol. 2869, pp. 731–738. Springer, Heidelberg (2003) Google Scholar
  3. 3.
    Robertazzi, T.G.: Ten reasons to use divisible load theory. IEEE Comput. 36(5), 63–68 (2003)CrossRefGoogle Scholar
  4. 4.
    Altilar, D., Paker, Y.: An optimal scheduling algorithm for parallel video processing. In: Proceedings of the International Conference on Multimedia Computing and Systems. IEEE Computing Society Press (1998)Google Scholar
  5. 5.
    Dong, L., Bharadwaj, V., Ko, C.C.: Efficient movie retrieval strategies for movie-on-demand multimedia services on distributed networks. Multimedia Tools Appl. 20(2), 99–133 (2003)CrossRefGoogle Scholar
  6. 6.
    Beaumont, O., Casanova, H., Legrand, A., Robert, Y., Yang, Y.: Scheduling divisible loads on star and tree networks: results and open problems. IEEE Trans. Parallel Distrib. Syst. 16(3), 207–218 (2005)CrossRefGoogle Scholar
  7. 7.
    Drozdowski, M., Wolniewicz, P.: Optimizing divisible load scheduling on heterogeneous stars with limited memory. Eur. J. Oper. Res. 172(2), 545–559 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  8. 8.
    Rosenberg, A.L., Chiang, R.C.: Toward understanding heterogeneity in computing. In: Proceeding of the 24th International Parallel and Distributed Processing Symposium (IPDPS’10), vol. 1, pp. 1–10. IEEE Computing Society Press, April 2010Google Scholar
  9. 9.
    Beaumont, O., Marchal, L., Robert, Y.: Scheduling divisible loads with return messages on heterogeneous master-worker platforms. In: Bader, D.A., Parashar, M., Sridhar, V., Prasanna, V.K. (eds.) HiPC 2005. LNCS, vol. 3769, pp. 498–507. Springer, Heidelberg (2005)Google Scholar
  10. 10.
    Saif, T., Parashar, M.: Understanding the behavior and performance of non-blocking communications in MPI. In: Danelutto, M., Vanneschi, M., Laforenza, D. (eds.) Euro-Par 2004. LNCS, vol. 3149, pp. 173–182. Springer, Heidelberg (2004) Google Scholar
  11. 11.
    Bharadwaj, V., Ghose, D., Mani, V., Robertazzi, T.: Scheduling divisible loads in parallel and distributed systems. IEEE Computing Society Press, Los Almitos (1996)Google Scholar
  12. 12.
    Drozdowski, M.: Selected problems of scheduling tasks in multiprocessor computing systems. Ph.D. thesis, Instytut Informatyki Politechnika Poznanska, Poznan (1997)Google Scholar
  13. 13.
    Bharadwaj, V., Ghose, D., Mani, V.: Multi-installment load distribution in tree networks with delays. IEEE Trans. Aerosp. Electron. Syst. 31(2), 555–567 (1995)CrossRefGoogle Scholar
  14. 14.
    Yang, Y., Casanova, H.: Extensions to the multi-installment algorithm: affine costs and output data transfers. Technical Report CS2003-0754, Dept. of Computer Science and Engineering, University of California, San Diego (2003)Google Scholar
  15. 15.
    Millot, D., Parrot, C.: Scheduling on unspecified heterogeneous distributed resources. In: Proceedings of the 25th International Symposium on Parallel and Distributed Processing Workshops (IPDPSW’11), vol. 1, pp. 45–56. IEEE Computing Society Press, May 2011Google Scholar
  16. 16.
    Millot, D., Parrot, C.: Fundamental results on the AS4DR scheduler. Technical Report RR-11005-INF, TELECOM sudParis, Évry, France (2011)Google Scholar
  17. 17.
    Millot, D., Parrot, C.: Some tests of adaptivity for the AS4DR scheduler. In: Proceedings of the 41th International Conference on Parallel Processing (ICPP’12), pp. 323–331. IEEE Computing Society Press, September 2012Google Scholar
  18. 18.
    Pisinger, D.: An exact algorithm for large multiple knapsack problems. Eur. J. Oper. Res. 114(3), 528–541 (1999)CrossRefzbMATHGoogle Scholar
  19. 19.
    Casanova, H., Legrand, A., Quinson, M.: SimGrid: a generic framework for large-scale distributed experiments. In: Proceedings of the 10th International Conference on Computer Modeling and Simulation (ICCMS’10), pp. 126–131. IEEE Computing Society Press, March 2008Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  1. 1.Telecom SudParisInstitut Mines-TelecomÉvryFrance

Personalised recommendations