Abstract
The availability of magnetic observatory data has evolved rapidly with the transition of observatories from analogue photographic magnetograms to digital electronic recordings, and the advent of the internet for instant global access to information of every sort. Metadata (information about the data) is undergoing its own transformation in order to accompany the rapid and extensive dissemination of these data. This chapter describes the types of data historically and currently produced by geomagnetic observatories and introduces new data types such as one-second and quasi-absolute data recently discussed at the 11th IAGA Scientific Assembly in Sopron, Hungary. We review the availability of these data types from the World Data Centres, INTERMAGNET and other sources. Finally, we discuss developments in metadata describing the current efforts in the geomagnetism community to gather, store and distribute this information about the data to better assist scientific discovery.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abston CC, Papitashvili NE, Papitashvili VO (1985) Combined international catalog of geomagnetic data, Report UAG-92, U.S. National Geophysical Data Center
Akasofu S-I (1981) Relationships between the AE and Dst indices during geomagnetic storms. J Geophys Res 86:4820–4822
Baillie O, Clarke E, Flower S, Reay S, Turbitt C (2009) Reporting quasi-definitive observatory data in near real time. 11th IAGA Scientific Assembly, Abstract Book 94Â pp
Bartels J, Heck NH, Johnston HF (1939) The three-hour-range index measuring geomagnetic activity. J Geophys Res 44:411–454
Boteler DH (2006) Comment on time conventions in the recordings of 1859. Adv Space Res 38:301–303
Brooke C (1847) Description of apparatus for automatic registration of magnetometers and other meteorological instruments by photography. Phil Trans R Soc London 137:69–77
Buneman P (2005) What the Web Has Done for Scientific Data, and What it Hasn’t. In Advances in Web-Age Information Management: 6th International Conference, Web Age Information Management 2005, Hangzhou, China, October 2005 Proceedings, pp 1–7
Chulliat A (2009c) Evaluation of IGRF-11 candidate models, secular variation. From IGRF-11: Progress reports, candidate models, evaluations and test models http://www.ngdc.noaa.gov/IAGA/vmod/EVALS/Eval_Chulliat.pdf
Chulliat A, Peltier A, Truong F, Fouassier D (2009b) Proposal for a new observatory data product: quasi-definitive data. 11th IAGA Scientific Assembly, Abstract Book 94Â pp
Chulliat A, Savary J, Telali A, Lalanne X (2009a) Acquisition of 1-Second Data in IPGP Magnetic Observatories. Proceedings of the XIIIth IAGA Workshop on Geomagnetic Observatory Instruments, Data Acquisition, and Processing: U.S. Geological Survey Open-File Report 2009–1226, pp 54–59
Chulliat A, Thébault E, Hulot G (2010) Core field acceleration pulse as a common cause of the 2003 and 2007 geomagnetic jerks. Geophys Res Lett 37:L07301. doi:10.1029/2009GL042019
Clarke E, Flower S, Humphries T, McIntosh R, McTaggart F, McIntyre B, Owenson N, Henderson K, Mann E, MacKenzie K, Piper S, Wilson L, Gillanders R (2009) The digitization of observatory magnetograms. 11th IAGA Scientific Assembly, Program Abstract 43Â pp
Curto JJ, Sanclement E, Torta JM (1996) Automatic measurement of magnetic records on photographic paper. Comput Geosci 22:359–368
Davis TN, Sugiura M (1966) Auroral electrojet activity index AE and its universal time variations. J Geophys Re 71:785
Dawson E, Reay S, Macmillan S, Flower S, Shanahan T (2009) Quality Control Procedures At The World Data For Geomagnetism (Edinburgh). 11th IAGA Scientific Assembly, Program Abstract 44Â pp
Feldstein YI, Grafe A, Gromova LI, Popov VA (1997) Auroral electrojets during geomagnetic storms. J Geophys Res 102:14223–14236. doi:10.1029/97JA00577
Fischman D, Denig WF, Herzog D (2009) A Proposed Metadata Implementation for Magnetic Observatories. Proceedings of the XIIIth IAGA Workshop on Geomagnetic Observatory Instruments, Data Acquisition, and Processing: U.S. Geological Survey Open-File Report 2009–1226, pp 82–85
Finlay CC, Maus S, Beggan C, Hamoudi M, Lesur V, Lowes FJ, Olsen N, Thébault E (2010) Evaluation of candidate geomagnetic field models for IGRF-11. Earth Planets Space (in press)
Fouassier D, Chulliat A (2009) Extending backwards to 1883 the French magnetic hourly data series. Proceedings of the XIIIth IAGA Workshop on Geomagnetic Observatory Instruments, Data Acquisition, and Processing: U.S. Geological Survey Open-File Report 2009-1226, pp 86–94
Friis-Christensen E, Lühr H, Hulot G (2006) Swarm: A constellation to study the Earth’s magnetic field. Earth Planets Space 58:351–358
Hejda P, Herzog D, Linthe HJ, Mandea M, Schott J-J, Svalgaard L (2009) On the derivation of hourly mean values with incomplete data. 11th IAGA Scientific Assembly, Abstract Book 94Â pp
Herzog DC, Lupica CW (1992) Magnetic observatory data on CD-ROM. Eos Trans Am Geophys Union 73:236
Herzog DC (2009) The effects of missing data on mean hourly values. Proceedings of the XIIIth IAGA Workshop on Geomagnetic Observatory Instruments, Data Acquisition, and Processing: U.S. Geological Survey Open-File Report 2009-1226, pp 116–126
IAGA (2009) IAGA News 46, http://www.iugg.org/IAGA/iaga_pages/pubs_prods/Newsletters/IAGA_News_46.pdf
ICSU Panel on World Data Centers (1996) Guide to the World Data Center System. http://www.ngdc.noaa.gov/wdc/guide/wdcguide.html
ICSU (2008) Ad hoc Strategic Committee on Information and Data. Final Report to the ICSU Committee on Scientific Planning and Review
Iyemori T (1990) Storm-time magnetospheric currents inferred from mid-latitude geomagnetic field variations. J Geomagnetics Geoelectrics 42:1249–1265
Iyemori T, Nose M, McCreadie H, Odagi Y, Takeda M, Kamei T, Yagi M (2005) Digitization of Old Analogue Geomagnetic Data. Proceedings of PV-2005: Ensuring Long-term Preservation and Adding Value to Scientific and Technical data, Royal Society, Edinburgh, UK. http://www.ukoln.ac.uk/events/pv-2005/pv-2005-final-poster-papers/014-poster.doc
Jankowski J, Sucksdorff C (1996) Guide for magnetic measurements and observatory practice. International Association of Geomagnetism and Aeronomy, Warsaw, Poland
Kerridge DJ (2001) INTERMAGNET: Worldwide near-real-time geomagnetic observatory data, Proc. ESA Space Weather Workshop, ESTEC, Noordwijk, The Netherlands, http://www.esa-spaceweather.net/spweather/workshops/SPW_W3/PROCEEDINGS_W3/ESTEC_Intermagnet.pdf
King T, Thieman J, Roberts DA (2010) SPASE 2.0: a standard data model for space physics. Earth Sci Infor. doi:10.1007/s12145-010-0053-4
Korte M, Mandea M, Olsen N (2007) Worldwide Observatory Hourly Means from 1995 to 2003: Investigation of Their Quality. In: Reda J (ed) XII IAGA Workshop on Geomagnetic Observatory Instruments, Data Acquisition and Processing Belsk, 19–24 June 2006, Publications of Institute of Geophysics Polish Academy of Sciences, C-99 (398)
Korepanov V, Marusenkov A, Rasson J (2009) 1-second INTERMAGNET Standard Magnetometer—1-year operation. 11th IAGA Scientific Assembly, Abstract Book 94 pp
Love JJ (2009) Missing data and the accuracy of magnetic-observatory hour means. Ann Geophys 27:3601–3610
Mabie J, Denig W, McLean S (2009) Geomagnetic Data Management at the NGDC. 11th IAGA Scientific Assembly, Program Abstract 114 pp
Macmillan S (2007) Observatories: an overview. In: Gubbins D, Herrero-Bervera E (eds) Encyclopedia of Geomagnetism and Paleomagnetism. Springer, Netherlands, pp 708–711
Marsal S, Curto JJ (2009) A new approach to the hourly mean computation problem when dealing with missing data. Earth Planet Space 61:945–956
Mandea M (2002) 60, 59, 58, … How many minutes for a reliable hourly mean? Proceedings of the Xth IAGA Workshop on Geomagnetic Instruments, Data Acquisition, and Processing, Hermanus Magnetic Observatory, Hermanus, South Africa, pp 112–120
Mandea M, Papitashvili V (2009) Worldwide Geomagnetic Data Collection and Management. Eos Trans Am Geophys Union 90:409–410
Matsushita S, Campbell WH (eds) (1967) Physics of geomagnetic phenomena (2 vols). Academic, New York-London
Mayaud PN (1967) Calcul preliminaire d’indices Km, Kn et Ks ou Am, An, et As, mesures de l’activité magnétique a l’échelle mondiale et dans les hémispheres Nord et Sud. Ann Géophys 23:585–617
Mayaud PN (1972) The aa index: a 100-year series characterizing the geomagnetic activity. J Geophys Res 77:6870–6874
Mayaud PN (1980) Derivation, Meaning, and Use of Geomagnetic Indices, AGU Geophys Monogr 22
Menvielle M, Iyemori T, Marchaudon A, Nose M (2011) Geomagnetic indices. In: Mandea M, Korte M (eds) Geomagnetic Observations and Models, IAGA Special Sopron Book Series 5, DOI 10.1007/978-90-481-9858-0_7
Menvielle M, Papitashvili N, Hakkinen L, and Sucksdorff C (1995) Computer production of K indices: review and comparison of methods. Geophys J Int 123:866–886
Merka J, Narock TW, Szabo A (2008) Navigating through SPASE to heliospheric and magnetospheric data. Earth Sci Inform 1:35–42
Minamoto Y (2009) Ongoing Geomagnetic field 1-second value measurement by JMA. Proceedings of the XIIIth IAGA Workshop on Geomagnetic Observatory Instruments, Data Acquisition, and Processing: U.S. Geological Survey Open-File Report 2009-1226, pp 190–193
Minster JH, Capitaine N, Clark DM, Mokrane M (2009) Initial Progress in Developing the New ICSU World Data System. American Geophysical Union, Fall Meeting 2009, abstract #IN23A-1059
Narod B (2009) Private communication, Further information can be obtained from Narod Geophysics Ltd., #310, 2475 York Ave., Vancouver, B.C., Canada, V6K-1C9. Tel/fax: 604-732-9083
Nevanlinna H, Häkkinen L (2010) Results of Russian geomagnetic observatories in the 19th century: magnetic activity, 1841–1862. Ann Geophys 28:917–926
Nagarajan N (2008) Retrieval and Distribution of Digital Archives of Photographic Records of Magnetic Variations at Ngri, India. Abstract for XIIIth IAGA Workshop on Geomagnetic Observatory Instruments, Data Acquisition, and Processing http://mines.conference-services.net/resources/328/1322/pdf/IAGA2008_0017.pdf
Newitt LR (2009) The Effects of Missing Data on the Computation of Hourly Mean Values and Ranges. Proceedings of the XIIIth IAGA Workshop on Geomagnetic Observatory Instruments, Data Acquisition, and Processing: U.S. Geological Survey Open-File Report 2009-1226, pp 194–201
Peltier A, Chulliat A (2010) On the feasibility of promptly producing quasi-definitive magnetic observatory data. Earth Planet Space 62(2):e5–e8, doi:10.5047/eps.2010.02.002
Peng F, Shen X, Tang K, Zhang J, Huang Q, Xu Y, Yue B, Yang D (2007) Data-Sharing Work of the World Data Center for Geophysics. Beijing Data Sci J 6:404–407
Rasson JL (2007) Observatories, INTERMAGNET. In: Gubbins D, Herrero-Bervera E (eds) Encyclopedia of geomagnetism and paleomagnetism. Springer, New York, NY, pp 715–717
Reay SJ, Allen W, Baillie O, Bowe J, Clarke E, Lesur V, Macmillan S (2005) Space weather effects on drilling accuracy in the North Sea. Ann Geophys 23:3081–3088
Reay S, Dawson E, Flower S, Macmillan S, Herzog D (2009) Towards a metadata standard for geomagnetic observatory data. 11th IAGA Scientific Assembly, Abstract Book 94Â pp
Reda J, Fouassier D, Isac A, Linthe H-J, Matzka J, Turbitt CW (2011) Improvements in geomagnetic observatory data quality. In: Mandea M, Korte M (eds) Geomagnetic Observations and Models, IAGA Special Sopron Book Series 5, DOI 10.1007/978-90-481-9858-0_7
St-Louis BJ, Sauter EA, Trigg DF, Coles RL (2008) INTERMAGNET Technical Reference Manual, version 4.4 http://www.intermagnet.org/publications/im_manual.pdf
Schmidt, A (1905) Ergebnisse der magnetischen Beobactungen in Potsdam, Veröffentl. des Preuss. Meteol Inst Berlin
Schott J-J, Linthe HJ (2007) The hourly mean computation problem revisited. Proceedings of the XIIth IAGA Workshop on Geomagnetic Instruments, Data Acquisition, and Processing, Institute of Geophysics, Polish Acad Sci Warszawa 135–143
Shanahan TJG, Turbitt CW (2009) Evaluating the Noise for a Commonly Used Fluxgate Magnetometer—for 1-second Data. Proceedings of the XIIIth IAGA Workshop on Geomagnetic Observatory Instruments, Data Acquisition, and Processing: U.S. Geological Survey Open-File Report 2009–1226, pp 239–245
Stauning P, Troshichev O, Janzhura A (2009) The polar cap (PC) index. Present (unified) and past index calculations 11th IAGA Scientific Assembly, Abstract Book 99Â pp
Sugiura M (1964) Hourly values of equatorial Dst for IGY. Annals of the International Geophysical Year, vol 35. Pergamon Press, Oxford, pp 945–948
Svalgaard L, Cliver EW (2007) Interhourly variability index of geomagnetic activity and its use in deriving the long-term variation of solar wind speed. J Geophys Res 112:A10111. doi:10.1029/2007JA012437
Trigg DF, Serson PH, Camfield PA (1971) A solid-state electrical recording magnetometer. Department of Energy, Mines, and Resources 41 Ottawa, Canada.
Troshichev OA, Andrezen VG (1985) The relationship between interplanetary quantities and magnetic activity in the southern polar cap. Planet Space Sci 33:415–419
Tsurutani BT, Gonzalez WD, Lakhina GS, Alex S (2003) The extreme magnetic storm of 1–2 September 1859. J Geophys Res 108(A7):1268. doi:10.1029/2002JA009504
Turner GM, Rasson JL, Reeves CV (2007) Observation and measurement techniques. In: Schubert G (ed) Treatise on geophysics, vol 5. Elsevier, Amsterdam, pp 93–146
Wayne L (2005a) Metadata in action. In: Proceedings of the 2005 GISPlanet Conference Proceedings http://www.fgdc.gov/metadata/documents/MetadataInAction.doc (Accessed on 28th July 2009)
Wayne L (2005b) Institutionalize Metadata Before It Institution-alizes You, Federal Geographic Data Committee http://www.fgdc.gov/metadata/documents/InstitutionalizeMeta_Nov2005.doc (Accessed on 28th July 2009)
Wienert KA (1970) Notes of geomagnetic observatory and survey practice. UNESCO Earth Sciences 5
Worthington EW, Sauter EA, Love JJ (2009b) Analysis of USGS One-Second Data. In: Proceedings of the XIIIth IAGA Workshop on Geomagnetic Observatory Instruments, Data Acquisition, and Processing: U.S. Geological Survey Open-File Report 2009–1226, pp 262–266
Worthington EW, White T, Sauter EA, Stewart DC (2009a) Evaluation of 1-Hz data from USGS geomagnetic observatories. 11th IAGA Scientific Assembly, Abstract Book 94Â pp
Acknowledgments
The authors wish to thank David M. Clark, Dr. Hans-Joachim Linthe, Dr. Jürgen Matzka and Dr. Susan Macmillan for their valued contributions. We would like to thank Ellen Clarke for her critical review and many helpful additions to the manuscript. We would also like to thank Prof. Mioara Mandea and Dr. Monika Korte for the opportunity to prepare this chapter. This paper is published with the permission of the Director of BGS (NERC).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendices
Appendix 1: Data File Formats
With the advent of digital data records it was soon recognised that a common data format for storing and presenting geomagnetic observatory was required. Without data standards it would be impossible for users of worldwide observatory data to make use of the vast collection of data available from many different institutions and countries.
Over time different data formats have been established with different motivations behind each approach. The most prevalent standard for one-minute data is currently IAGA-2002 whereas WDC Exchange Format is still often used for hourly mean data.
7.1.1 World Data Centre Exchange Format
In the 1980s a common file format for dissemination of geomagnetic data was developed. This WDC Exchange Format [36] was a continuation from the format for punched card and was designed to make maximum use of the limited RAM and disk capacity of computers at that time. It is not a convenient format for the user and there is no space for any metadata. For hourly means each value is decomposed into a tabular base value (shared by all hourly values for a given day), and a tabular value (Fig. 7.9). For each day the tabular base and the 24 values are presented with a daily mean value at the end of the row. For minute means each value in an hour is displayed along one horizontal row with an hourly mean value given at the end.
7.1.2 INTERMAGNET GIN Dissemination Formats
INTERMAGNET defined a number of format standards for the dissemination of data from their GINs [37]. There are standards for one-minute (IMFV1.22), hourly (IHFV1.01) and daily (IDFV1.01) data. For example, for one-minute means, data are organised on a day-file basis. One file contains 24 one-hour blocks, each containing 60 minutes worth of values. Blocks are padded with 9’s if incomplete. Information is encoded in ASCII and there is only very limited metadata included e.g., noting if data are reported, adjusted or definitive.
7.1.3 IAGA-2002 Format
The IAGA ASCII Exchange Format [38] was adopted in 2001. It is used as a data exchange format for geomagnetic data (samples and means) from observatories and variometer stations at cadences from milliseconds up to and including monthly means. This ability to accommodate data of different cadence is a major strength of this file format. It is also flexible and can account for files containing different durations of data (e.g. a month file of one-minute data) with its well-defined file naming scheme. The file names themselves provide critical metadata and within each file there are twelve mandatory file header records for further metadata. These include the type of data (definitive, provisional, variation), if the data are instantaneous or a mean, and how the values are centred. It also has an optional, variable length comment field for more descriptive metadata associated with the dataset. The data records are presented in a fixed four-column ASCII format which is convenient for the user to read and manipulate.
Appendix 2: Internet Links
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
- 30.
- 31.
- 32.
- 33.
- 34.
- 35.
- 36.
- 37.
- 38.
Glossary
- ANZLIC
-
—Australia New Zealand Land Information Council
- BGS
-
—British Geological Survey
- CDF
-
—Common Data Format
- CETP
-
—Centre d’étude des Environnements Terrestre et Planétaires
- CODATA
-
—Committee on Data for Science and Technology
- CSIC
-
—Consejo Superior de Investigaciones CientÃficas (Spanish Research Council)
- DACGSM
-
—Data Analysis Center for Geomagnetism and Space Magnetism
- DCP
-
—Data Collection Platform
- DMI
-
—Danish Meteorological Institute
- ERSI
-
—Environmental Systems Research Institute
- FAGS
-
—Federation of Astronomical and Geophysical data analysis Service
- FGDC
-
—Federal Geographic Data Committee
- FTP
-
—File Transfer Protocol
- GEODAS
-
—Geophysical Data System
- GIN
-
—Geomagnetic Information Nodes
- GFZ
-
—GeoForschungsZentrum
- HMV
-
—Hourly Mean Value
- IAGA
-
—International Association of Geomagnetism and Aeronomy
- ICSU
-
—International Council for Science
- IGRF
-
—International Geomagnetic Reference Field
- IGY
-
—International Geophysical Year
- IIG
-
—Indian Institute of Geomagnetism
- IPY
-
—International Polar Year
- IMO
-
—INTERMAGNET Observatory
- INTERMAGNET-
-
International Real-time Magnetic Observatory Network
- ISGI
-
—International Service of Geomagnetic Indices
- ISO
-
—International Standards Organisation
- IUGONET
-
—Inter-university Upper atmosphere Global Observation NETwork
- LATMOS
-
-Laboratoire Atmosphères, Milieux, Observations Spatiales
- LPM
-
—Low Power Magnetometer
- LPMNET
-
—Low Power Magnetometer NETwork
- NERC
-
—Natural Environment Research Council
- NGDC
-
—National Geophysical Data Center
- RAM
-
—Russian Arctic and Antarctic Magnetometer network
- SEP
-
—Solid-Earth Physics
- sfe
-
—Solar Flare Effects
- SPASE
-
—Space Physics Archive Search and Extract
- SPIDR
-
—Space Physics Interactive Data Resource
- ssc
-
—Sudden Storm Commencements
- STP
-
—Solar-Terrestrial Physics
- USGS
-
—United States Geological Survey
- VO
-
—Virtual Observatory
- WDC
-
—World Data Centre
- WDS
-
—World Data System
- WMM
-
—World Magnetic Model
- WWW
-
—World Wide Web
- XML
-
—eXtensible Markup Language
- XSLT
-
—eXtensible Stylesheet Language Transformations
Rights and permissions
Copyright information
© 2011 Springer Science+Business Media B.V.
About this chapter
Cite this chapter
Reay, S.J. et al. (2011). Magnetic Observatory Data and Metadata: Types and Availability. In: Mandea, M., Korte, M. (eds) Geomagnetic Observations and Models. IAGA Special Sopron Book Series, vol 5. Springer, Dordrecht. https://doi.org/10.1007/978-90-481-9858-0_7
Download citation
DOI: https://doi.org/10.1007/978-90-481-9858-0_7
Published:
Publisher Name: Springer, Dordrecht
Print ISBN: 978-90-481-9857-3
Online ISBN: 978-90-481-9858-0
eBook Packages: Earth and Environmental ScienceEarth and Environmental Science (R0)