The Greenwich Photo-heliographic Results (1874 – 1976): Procedures for Checking and Correcting the Sunspot Digital Datasets

Abstract

Attention is drawn to the existence of errors in the original digital dataset containing sunspot data extracted from certain sections of the printed Greenwich Photo-heliographic Results (GPR) 1874 – 1976. Calculating the polar coordinates from the heliographic coordinates and comparing them with the recorded polar coordinates reveals that there are both isolated and systematic errors in the original sunspot digital dataset, particularly during the early years (1874 – 1914). It should be noted that most of these errors are present in the compiled sunspot digital dataset and not in the original printed copies of the Greenwich Photo-heliographic Results. Surprisingly, many of the errors in the digitised positions of sunspot groups are apparently in the measured polar coordinates, not the derived heliographic coordinates. The mathematical equations that are used to convert between heliographic and polar coordinate systems are formulated and then used to calculate revised (digitised) polar coordinates for sunspot groups, on the assumption that the heliographic coordinates of every sunspot group are correct. The additional complication of requiring accurate solar ephemerides in order to solve the mathematical equations is discussed in detail. It is shown that the isolated and systematic errors, which are prevalent in the sunspot digital dataset during the early years, disappear if revised polar coordinates are used instead. A comprehensive procedure for checking the original sunspot digital dataset is formulated in an Appendix.

Appendix: Formal Procedure for Detecting and Correcting Errors in the GPR

1.1 A.1 Aim of the Appendix

A comprehensive and detailed procedure for checking the original sunspot digital dataset (which is introduced and defined in Sections 3 and 5 of Paper 1) is formulated in this appendix. Although the various checks apply specifically to the original sunspot digital dataset (RGO–SDD), the methodology could easily be adapted to apply similar checks to the more recent sunspot and faculae digital dataset (RGO–S&FDD). A preliminary procedure for checking the sunspot and faculae digital dataset (RGO–S&FDD), developed at the NOAA National Geophysical Data Center (NGDC), has already been used to identify a number of typographical errors in the RGO printed observations, bulletins and annals (RGO–POBA). This preliminary procedure is defined and discussed in Section 2 of Paper 3. Section 3 of this third paper includes a table presenting the typographical errors in the RGO–POBA for the interval 1874 – 1917, which have been identified in a preliminary investigation.

1.2 A.2 ‘Unit Testing’ to Validate that the GPR are Free from Errors

The present approach to validating that the RGO–SDD is free from errors is to construct a series of simple, self-contained tests that this dataset must pass in order to be classified as “error free”. This approach has been adopted from the discipline of software engineering, where it is typically applied to functional code. For software, a ‘unit’ describes the smallest testable part of an application. Ideally, tests should be independent.

In applying the unit-testing methodology to this dataset (RGO–SDD), particular care has been taken to ensure good coverage. By considering the data from different abstractions (logical, physical), exploiting redundancy (polar coordinates and heliographic coordinates) and the duplication of information in different datasets (RGO–SDD, RGO–S&FDD, RGO–POBA), coverage is maximised. It is important to gain good coverage, since no acceptance tests for the dataset as a whole are available.

For the RGO–SDD dataset, a list of unit tests is provided in the tables below. The procedure for developing a unit test is as follows. Identify an assertion that must hold for every element or group of elements in the dataset (Column: Assertion). Identify a method to test if this assertion holds for a given element or group of elements (Column: Test). Identify a correction or remedy to perform on elements or groups that fail the test (Column: Failure). Record any relevant additional information (Column: Notes).

Tests that are based on different conceptual views of the database are collected together into groups. These abstractions include: Group I: File Format (line endings must be consistent); Group II: Data Format (all lines must be the same length); Group III: Published Errata (check that all published errata have been implemented); Group IV: Observational Method (observation times must be consistent with the known method of observation); Group V: Observational Data (recorded values must be within physically acceptable bounds); Group VI: Physical Properties (sunspot observations must be in accordance with physical understanding of the Sun); Group VII: Redundant Data (redundant values must be self-consistent, as discussed in Section 6 of Paper 1). Group VIII: Completeness Checks (confirm that the results of any manual checks performed by individual scientists have been incorporated). This last test should provide a sampling of the different types of error and successful completion of this test should then confirm that the previous tests provide good coverage.

By iteratively testing and correcting the given dataset it is possible to arrive at a final dataset that is error free – where an ‘error’ is defined as a failure to pass one or more tests. This unit testing stresses simple ‘atomic tests’, each of which has one of two possible outcomes – success or failure. An intermediate dataset is created between each successive test. These intermediate datasets can be recorded as ‘versions’ to provide an exact list of changes introduced and, if necessary, allow for additional subsequent changes to the dataset in an arbitrary manner.

1.3 A.3 Test Group I: File Format

Table 1 File format.

This test may be defined as a pre-processing test for the consistency of the dataset. This test can be applied with little or no knowledge of the physical properties that are represented by the data.

1.4 A.4 Test Group II: Data Format

Table 2 Data format.

Interpolated values are indicated in the sunspot digital dataset (RGO–SDD) by a ‘special’ time format after the decimal point in the decimal part of a day (.__0). This time format is not specified in the metadata, which currently accompanies the sunspot data. In any future revision of the following test, the complication of interpolated values will be addressed more rigorously but until then such values should be excluded from the dataset, as recommended in Section 6 of Paper 1.

1.5 A.5 Test Group III: Published Errata

Table 3 Published errata.

Errata have been published by the RGO, as noted in Section 5 of Paper 1. These errata have been included in the sunspot digital dataset (RGO–SDD) but have not yet been fully incorporated in the sunspot and faculae digital dataset (RGO–S&FDD). It is also necessary to signify (with a special character) those values that have been adjusted or included because of these published errata. Then, if subsequent reference is made to the books with respect to later tests, it is known whether to examine the original records or the corresponding errata.

1.6 A.6 Test Group IV: Observational Method

Table 4 Observational method.

This set of tests exploits a priori knowledge of the data collection method. This knowledge includes explicit and implicit aims of the observers as well as physical restrictions on the method of observation.

1.7 A.7 Test Group V: Observational Data

Table 5 Observational data.

This set of tests exploits a priori knowledge of the data collection method. These tests are intended to validate recorded values.

1.8 A.8 Test Group VI: Physical Properties

Table 6 Physical properties.

This set of tests exploits a priori knowledge of the physical properties of the Sun.

1.9 A.9 Test Group VII: Redundant Data

Table 7 Redundant data.

This set of tests searches for numerical inconsistencies using “redundant” information in the original sunspot digital dataset (RGO–SDD), as outlined in Section 6 of Paper 1. Some of these tests use ephemeris values, which poses problems already documented (see Section 2.2).

1.10 A.10 Test Group VIII: Completeness Check

Table 8 Completeness check.

This test is intended to check that all known errors have been corrected. For example, it is important to include the results of any manual checks that may have been performed by individual scientists.