Skip to main content
SpringerLink
Log in

Optimising NGAS for the MWA Archive

  • ORIGINAL ARTICLE
  • Published:
Experimental Astronomy Aims and scope Submit manuscript

Abstract

The Murchison Widefield Array (MWA) is a next-generation radio telescope, generating visibility data products continuously at about 400 MB/s. Efficiently managing and archiving this data is a challenge. The MWA Archive consists of dataflows and storage sub-systems distributed across three tiers. At its core is the open source software—the Next-Generation Archive System (NGAS)—that was initially developed in ESO. However, to meet the MWA data challenge, we have tailored and optimised NGAS to achieve high-throughput data ingestion, efficient data flow management, multi-tiered data storage and processing-aware data staging.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Lonsdale, C.J., Cappallo, R.J., Morales, M.F., Briggs, F.H., Benkevitch, L., Bowman, J.D., Bunton, J.D., Burns, S., Corey, B.E., deSouza, L., et al.: Proc. IEEE 97(8), 1497 (2009)

    Article  ADS  Google Scholar 

  2. Tingay, S.J., Goeke, R., Bowman, J.D., Emrich, D., Ord, S.M., Mitchell, D.A., Morales, M.F., Booler, T., Crosse, B., Wayth, R.B., Lonsdale, C.J., Tremblay, S., Pallot, D., Colegate, T., Wicenec, A., Kudryavtseva, N., Arcus, W., Barnes, D., Bernardi, G., Briggs, F., Burns, S., Bunton, J.D., Cappallo, R.J., Corey, B.E., Deshpande, A., Desouza, L., Gaensler, B.M., Greenhill, L.J., Hall, P.J., Hazelton, B.J., Herne, D., Hewitt, J.N., Johnston-Hollitt, M., Kaplan, D.L., Kasper, J.C., Kincaid, B.B., Koenig, R., Kratzenberg, E., Lynch, M.J., Mckinley, B., Mcwhirter, S.R., Morgan, E., Oberoi, D., Pathikulangara, J., Prabu, T., Remillard, R.A., Rogers, A.E.E., Roshi, A., Salah, J.E., Sault, R.J., Udaya-Shankar, N., Schlagenhaufer, F., Srivani, K.S., Stevens, J., Subrahmanyan, R., Waterson, M., Webster, R.L., Whitney, A.R., Williams, A., Williams, C.L., Wyithe, J.S.B.: Publ. Astron. Soc. Aust. 30, e007 (2013). doi:10.1017/pasa.2012.007

    ADS  Google Scholar 

  3. Wayth, R.B., Greenhill, L.J., Briggs, F.H.: Publ. Astron. Soc. Pac. 121(882), 857 (2009)

    Article  ADS  Google Scholar 

  4. Ord, S., Mitchell, D., Wayth, R., Greenhill, L., Bernardi, G., Gleadow, S., Edgar, R., Clark, M., Allen, G., Arcus, W., et al.: Publ. Astron. Soc. Pac. 122(897), 1353 (2010)

    Article  ADS  Google Scholar 

  5. HPCWire: http://www.hpcwire.com/hpcwire/2013-05-01/sgi_providing_data_management_infrastructure_at_pawsey_supercomputing_centre.html (hpcwire.com) (2013)

  6. Tody, D., Plante, R.: http://www.ivoa.net/documents/SIA/20091116/REC-SIA-1.0.html (International Virtual Observatory Alliance) (2009)

  7. Dowler, P., Rixon, G., Tody, D.: http://http://www.ivoa.net/documents/TAP/ (International Virtual Observatory Alliance) (2010)

  8. Wicenec, A., Knudstrup, J.: The Messenger 129, 27 (2007)

    ADS  Google Scholar 

  9. Wicenec, A., Farrow, S., Gaudet, S., Hill, N., Meuss, H., Stirling, A.: Astron. Data Anal. Softw. Syst. (ADASS) XIII 314, 93 (2004)

    ADS  Google Scholar 

  10. Manning, A., Wicenec, A., Checcucci, A., Gonzalez Villalba, J.A.: Astronomical data analysis software and systems XXI. In: Ballester, P., Egret, D., Lorente, N.P.F. (eds.) Astronomical Society of the Pacific Conference Series, vol. 461, p 681 (2012)

  11. Oldfield, R.A., Ward, L., Riesen, R., Maccabe, A.B., Widener, P., Kordenbrock, T.: In: 2006 IEEE International Conference on Cluster Computing, pp 1–11 (2006)

  12. Harris, C.: In: Proceedings of the ATIP/A* CRC Workshop on Accelerator Technologies for High-Performance Computing: Does Asia Lead the Way? p. 20. A* STAR Computational Resource Centre (2012)

  13. Givelberg, E., Szalay, A., Kanov, K., Burns, R.: In: Proceedings of the First International Workshop on Network-aware Data Management, NDM ’11, pp. 57–64. ACM, New York (2011). doi:10.1145/2110217.2110226

  14. Raicu, I., Zhao, Y., Foster, I.T., Szalay, A.: In: Proceedings of the 2008 International Workshop on Data-aware Distributed Computing - DADC ’08, pp. 9–18. ACM Press, New York (2008) doi:10.1145/1383519.1383521

  15. Bharathi, S., Chervenak, A.: In: Proceedings of the Second International Workshop on Data-aware Distributed Computing, p. 5. ACM (2009)

  16. SKA: http://www.skatelescope.org/press/first-announcement-of-opportunity-for-time-allocation-on-ska-precursor-the-mwa-2/ (SKAtelescope.org) (2013)

  17. Bowman, J.D., Cairns, I., Kaplan, D.L., Murphy, T., Oberoi, D., Staveley-Smith, L., Arcus, W., Barnes, D.G., Bernardi, G., Briggs, F.H., et al.: Publ. Astron. Soc. Aust. 10, 28 (2013)

    Google Scholar 

Download references

Acknowledgments

The authors would like to thank iVEC for providing the PB Store, Fornax and Pawsey supercomputing facilities. The Pawsey centre is funded from Australian federal and Western Australian state grants. ICRAR is a joint venture between Curtin University, The University of Western Australia and received grants from the Western Australian Government. The authors would like to acknowledge the excellent work of the original main developer of NGAS, Jens Knudstrup from ESO, without which this work would have been impossible. The authors would also like to thank two anonymous reviewers for providing valuable feedback to improve the paper.

Author information

Authors and Affiliations

  1. International Centre for Radio Astronomy Research, University of Western Australia, 35 Stirling Hwy, Crawley, WA, 6009, Australia

    C. Wu, A. Wicenec & A. Checcucci

  2. International Centre for Radio Astronomy Research, Curtin University, Kent St, Bentley, WA, 6102, Australia

    D. Pallot

Authors
  1. C. Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. Wicenec
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. D. Pallot
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. A. Checcucci
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to C. Wu.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wu, C., Wicenec, A., Pallot, D. et al. Optimising NGAS for the MWA Archive. Exp Astron 36, 679–694 (2013). https://doi.org/10.1007/s10686-013-9354-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10686-013-9354-1

Keywords

Search

Navigation