Skip to main content
SpringerLink
Log in

Modeling Instability for Large Scale Processing Tasks Within HEP Distributed Computing Environments

  • Conference paper
  • First Online:
Advances on P2P, Parallel, Grid, Cloud and Internet Computing (3PGCIC 2017)

Part of the book series: Lecture Notes on Data Engineering and Communications Technologies ((LNDECT,volume 13))

  • 1341 Accesses

Abstract

The scale of large processing tasks being run within the scientific Grid and Cloud environments have introduced a need for stability guarantees from geographically spanning resources, to ensure that failures are detected and handled preemptively. Performance inefficiencies within stacked service environments are a challenge to detect, where failures stem from a multitude of causes, often requiring expert intervention. Online reporting and classification of performance fluctuations can aid experts, central services, and users to target service areas where optimizations can be introduced. This paper describes an approach for modeling performance states for production tasks running within the ALICE Grid. We first provide an overview for the ALICE data and software workflow, focusing on the production job computational profile. Data center event state is then developed, based on data center job, computing, storage, and user behavior. With across site analysis, we then train groups to classify service domain states. Our approach is able to detect periods of service instability and the affected service domains. This can guide users, central, and data center experts to take action in advance of service failure effects.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    WLCG: http://wlcg-public.web.cern.ch/.

  2. 2.

    ALICE Collaboration: Our Mission - http://aliceinfo.cern.ch/node/3935.

  3. 3.

    Geant4 website: http://geant4.cern.ch/.

  4. 4.

    ROOT Data Analysis Framework: https://root.cern.ch/.

References

  1. Aerni, S., Farinas, H., Muralidhar, G.: Digital drivers in the age of massive datasets. XRDS 21(4), 60–66 (2015)

    Article  Google Scholar 

  2. Bal, H., Casanova, H., Dongarra, J., Matsuoka, S.: Application-level tools. In: Kesselman, I.F.C. (ed.) The Grid 2, The Morgan Kaufmann Series in Computer Architecture and Design, 2 edn., pp. 463–489. Morgan Kaufmann, Burlington (2004)

    Google Scholar 

  3. Betev, L., Gheata, A., Gheata, M., Grigoras, C., Hristov, P.: Performance optimisations for distributed analysis in ALICE. J. Phys. Conf. Ser. 523, 012014 (2014)

    Article  Google Scholar 

  4. Bird, I., Buncic, P., Carminati, F., Cattaneo, M., Clarke, P., Fisk, I., Girone, M., Harvey, J., Kersevan, B., Mato, P., Mount, R., Panzer-Steindel, B.: Update of the computing models of the WLCG and the LHC experiments. Technical report CERN-LHCC-2014-014. LCG-TDR-002, CERN, Geneva, April 2014

    Google Scholar 

  5. Buncic, P., Krzewicki, M., Vande Vyvre, P.: Technical design report for the upgrade of the online-offline computing system. Technical report CERN-LHCC-2015-006. ALICE-TDR-019, CERN, Geneva, April 2015

    Google Scholar 

  6. Cardwell, N., Cheng, Y., Gunn, C.S., Yeganeh, S.H., Jacobson, V.: BBR: congestion-based congestion control. Commun. ACM 60(2), 58–66 (2017)

    Article  Google Scholar 

  7. Carminati, F.: A reflection on Software Engineering in HEP. J. Phys: Conf. Ser. 396(5), 052018 (2012)

    Google Scholar 

  8. Carminati, F., Schutz, Y.: Conception, realisation et exploitation du traitement de données de l’expérience ALICE pour la simulation, la reconstruction et l’analyse. Ph.D. thesis, Nantes U. December 2012. Presented 22 February 2013

    Google Scholar 

  9. Date, S.: Should you upload or ship big data to the cloud? Commun. ACM 59(7), 44–51 (2016)

    Article  Google Scholar 

  10. Edwards, C.: Using patient data for personalized cancer treatments. Commun. ACM 57(4), 13–15 (2014)

    Article  Google Scholar 

  11. Flix, J., Accion, A., Acin, V., Acosta, C., Casals, J., Caubet, M., Cruz, R., Delfino, M., Lopez, F., Pacheco, A., Yzquierdo Perez-Calero, A., Planas, E., Porto, M., Rodriguez, B., Sedov, A.: Getting prepared for the LHC Run2: the PIC Tier-1 case. J. Phys. Conf. Ser. 664(5), 052014 (2015)

    Article  Google Scholar 

  12. Foster, I., Kesselman, C.: Chapter 4 - Concepts and architecture. In: Kesselman, C., Foster, I. (eds.) The Grid 2. The Morgan Kaufmann Series in Computer Architecture and Design, 2 edn., pp. 37–63. Morgan Kaufmann, Burlington (2004)

    Google Scholar 

  13. Kennedy, K.: Chapter 25 - languages, compilers, and run-time systems. In: Ian Foster Carl Kesselman, editor, The Grid 2. The Morgan Kaufmann Series in Computer Architecture and Design, 2 edn., pp. 491–512. Morgan Kaufmann, Burlington (2004)

    Google Scholar 

  14. Legrand, I., Newman, H., Voicu, R., Cirstoiu, C., Grigoras, C., Dobre, C., Muraru, A., Costan, A., Dediu, M., Stratan, C.: MonALISA: an agent based, dynamic service system to monitor, control and optimize distributed systems. Comput. Phys. Commun. 180(12), 2472–2498 (2009)

    Article  MATH  Google Scholar 

  15. Legrand, I.C., Newman, H.B.: The MONARC toolset for simulating large network-distributed processing systems. In: Proceedings of the 32nd Conference on Winter Simulation, WSC 2000, pp. 1794–1801. Society for Computer Simulation International, San Diego (2000)

    Google Scholar 

  16. Limoncelli, T.A.: The small batches principle. Commun. ACM 59(7), 52–57 (2016)

    Article  Google Scholar 

  17. Mattmann, C.A.: Computing: a vision for data science. Nature 493, 473–475 (2013)

    Article  Google Scholar 

  18. O’Driscoll, A., Daugelaite, J., Sleator, R.D.: ‘Big data’, Hadoop and cloud computing in genomics. J. Biomed. Inform. 46(5), 774–781 (2013)

    Article  MATH  Google Scholar 

  19. Rademakers, F.: Evolution of parallel computing in high energy physics, pp. 177–199. Springer, Heidelberg (2012)

    Google Scholar 

  20. Reed, D.A., Dongarra, J.: Exascale computing and Big Data. Commun. ACM 58(7), 56–68 (2015)

    Article  Google Scholar 

  21. Robertson, L.: Computing services for LHC: from clusters to grids, pp. 69–89. Springer, Heidelberg (2012)

    Google Scholar 

  22. Zimmermann, M.: The ALICE analysis train system. J. Phys. Conf. Ser. 608, 012019. 5 p (2015)

    Google Scholar 

Download references

Acknowledgments

The authors thank the ALICE collaboration and in particular Latchezar Betev for the valuable comments and suggestions. We also would like to thank Iossif Legrand and Federico Carminati for consultations in regards to monitoring data analytics, performance evaluation and statistical methods respectively.

Author information

Authors and Affiliations

  1. University of Houston, 4800 Calhoun Rd, Houston, TX, 77004, USA

    Olga Datskova & Weidong Shi

Authors
  1. Olga Datskova
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Weidong Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Olga Datskova .

Editor information

Editors and Affiliations

  1. Technical University of Catalonia, Barcelona, Spain

    Fatos Xhafa

  2. Open University of Catalonia, Barcelona, Spain

    Santi Caballé

  3. Fukuoka Institute of Technology (FIT), Fukuoka, Japan

    Leonard Barolli

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Datskova, O., Shi, W. (2018). Modeling Instability for Large Scale Processing Tasks Within HEP Distributed Computing Environments. In: Xhafa, F., Caballé, S., Barolli, L. (eds) Advances on P2P, Parallel, Grid, Cloud and Internet Computing. 3PGCIC 2017. Lecture Notes on Data Engineering and Communications Technologies, vol 13. Springer, Cham. https://doi.org/10.1007/978-3-319-69835-9_32

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-69835-9_32

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-69834-2

  • Online ISBN: 978-3-319-69835-9

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation