Scheduling Algorithms for Distributed Cosmic Ray Detection Using Apache Mesos

  • Germán SchnyderEmail author
  • Sergio Nesmachnow
  • Gonzalo Tancredi
  • Andrei Tchernykh
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 697)


This article presents two scheduling algorithms applied to the processing of astronomical images to detect cosmic rays on distributed memory high performance computing systems. We extend our previous article that proposed a parallel approach to improve processing times on image analysis using the Image Reduction and Analysis Facility IRAF software and the Docker project over Apache Mesos. By default, Mesos introduces a simple list scheduling algorithm where the first available task is assigned to the first available processor. On this paper we propose two alternatives for reordering the tasks allocation in order to improve the computational efficiency. The main results show that it is possible to reduce the makespan getting a speedup = 4.31 by adjusting how jobs are assigned and using Uniform processors.


Image processing Distributed memory Containers Mesos Scheduling 


  1. 1.
    Tancredi, G., Cromwell, G., Deustua, S., Gonzalez, G., Nesmachnow, S., Schnyder, G.: Geophysics using Hubble Space Telescope. Hubble Space Telescope Cycle 24 approved proposal (2016)Google Scholar
  2. 2.
    NOAO: IRAF Project Home Page, July 2016.
  3. 3.
    Schnyder, G., Nesmachnow, S.: Improving the performance of cosmic ray detection using Apache Mesos. In: International Supercomputing Conference in México (2016)Google Scholar
  4. 4.
    The Apache Software Foundation: Mesos, July 2016.
  5. 5.
    Mesosphere Inc.: Marathon: a cluster-wide init and control system for services in cgroups or Docker containers, July 2016.
  6. 6.
    The Apache Software Foundation: Apache ZooKeeper, July 2016.
  7. 7.
    Golpayegani, N., Halem, M.: Cloud computing for satellite data processing on high end compute clusters. In: International Conference on Cloud Computing (2009)Google Scholar
  8. 8.
    Ali, M., Kumar, J.: Implementation of image processing system using handover technique with map reduce based on big data in the cloud environment. Int. Arab J. Inf. Technol. 13(2), 326–331 (2016)Google Scholar
  9. 9.
    Adam, T.L., Chandy, K.M., Dickson, J.R.: A comparison of list schedules for parallel processing systems. Commun. ACM 17(12), 685–690 (1974)CrossRefzbMATHGoogle Scholar
  10. 10.
    Coffman, E.G., Sethi, R.: Algorithms minimizing mean flow time: schedule-length properties. Acta Informatica 6(1), 1–14 (1976)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Graham, R.L.: Bounds on multiprocessing timing anomalies. SIAM J. Appl. Math. 17(2), 416–429 (1969)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Kovács, A.: Tighter approximation bounds for LPT scheduling in two special cases. J. Discret. Algorithms 7(3), 327–340 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Oyetunji, E.O.: Some common performance measures in scheduling problems: review article. Res. J. Appl. Sci. Eng. Technol. 1(2), 6–9 (2009)Google Scholar
  14. 14.
    Wiley, K., Connolly, A., Gardner, J., Krughoff, S., Balazinska, M., Howe, B., Kwon, Y., Bu, Y.: Astronomy in the cloud: using MapReduce for image co-addition. Publ. Astron. Soc. Pac. 123(901), 366–380 (2011)CrossRefGoogle Scholar
  15. 15.
    Singh, N., Browne, L.M., Butler, R.: Parallel astronomical data processing with Python: recipes for multicore machines. Astron. Comput. 2, 1–10 (2013)CrossRefGoogle Scholar
  16. 16.
    Graham, R., Lawler, E., Lenstra, J., Kan, A.: Optimization, approximation in deterministic sequencing, scheduling: a survey. Ann. Discret. Math. 5, 287–326 (1979)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Eshaghian, M.: Heterogeneous Computing. Artech House, Norwood (1996)Google Scholar
  18. 18.
    Horowitz, E., Sahni, S.: Exact and approximate algorithms for scheduling nonidentical processors. J. ACM 23(2), 317–327 (1976)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Nesmachnow, S.: Parallel multiobjective evolutionary algorithms for batch scheduling in heterogeneous computing and grid systems. Comput. Optim. Appl. 55(2), 515–544 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Leung, J., Kelly, L., Anderson, J.: Handbook of Scheduling: Algorithms, Models, and Performance Analysis. CRC Press Inc., Boca Raton (2004)Google Scholar
  21. 21.
    Cirne, W., Brasileiro, F., Sauvé, J., Andrade, N., Paranhos, D., Santos-Neto, E.: Grid computing for bag of tasks applications. In: Proceedings of 3rd IFIP Conference on E-Commerce, E-Business and E-Government (2003)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Germán Schnyder
    • 1
    Email author
  • Sergio Nesmachnow
    • 1
  • Gonzalo Tancredi
    • 1
  • Andrei Tchernykh
    • 2
  1. 1.Universidad de la RepúblicaMontevideoUruguay
  2. 2.CICESE Research CenterEnsenadaMexico

Personalised recommendations