Assessment of the FAIRness of the Virtual Atomic and Molecular Data Centre following the Research Data Alliance evaluation framework

Abstract

In this paper, we present the result of the analysis made on the Virtual Atomic and Molecular Data Centre infrastructure, following the FAIR Data Maturity Model framework defined by the Research Data Alliance. After recalling the technical architecture of the VAMDC e-infrastructure (Sect. 1.1), we will introduce the RDA FAIR evaluation framework (Sect. 2) and define the methodology we adopt to perform our analysis (Sect. 2.1). In Sect. 2.2, we will present the result of this analysis (the fine-grained granularity analysis from which Sect. 2.2 is discussed in “Appendix A”). After having identified some lines of work aimed at improving the FAIRness of VAMDC (Sect. 3), we will conclude with some ideas for further developments of this work (Sect. 4).

Graphical Abstract

Appendix A: Details of the evaluation

• RDA-F1-01 M—Essential

  ${\mathcal {D}}$ :

  Metadata are identified by a persistent identifier;

  ${\mathcal {A}}$ :

  VAMDC federates heterogeneous databases, and each one of those has its own PID strategy and procedure for metadata. Moreover, these databases are dynamic. To address the issue of persistently identifying metadata associated with VAMDC data extractions, we implemented the RDA Dynamic Data Citation recommendation [9]: The Query Store achieves the persistent identification of both metadata and data extracted from the VAMDC infrastructure;

  ${\mathcal {L}}$ :

  = 4

• RDA-F1-01D—Essential

  ${\mathcal {D}}$ :

  Data are identified by a persistent identifier;

  ${\mathcal {A}}$ :

  The analysis is the same as in the case of the indicator RDA-F1-01 M;

  ${\mathcal {L}}$ :

  = 4

• RDA-F1-02 M—Essential

  ${\mathcal {D}}$ :

  Metadata are identified by a globally unique identifier;

  ${\mathcal {A}}$ :

  The Query Store assigns to any extracted digital object (i.e., the data together with its relevant metadata) a UUID¹⁷ and, upon user request, a DOI. These identifiers are unique by construction;

  ${\mathcal {L}}$ :

  = 4

• RDA-F1-02D—Essential

  ${\mathcal {D}}$ :

  Data are identified by a globally unique identifier;

  ${\mathcal {A}}$ :

  The analysis is the same as in the case of RDA-F1-02 M;

  ${\mathcal {L}}$ :

  = 4

• RDA-F2-01 M—Essential

  ${\mathcal {D}}$ :

  Rich metadata are provided to allow discovery;

  ${\mathcal {A}}$ :

  As we discussed in Sect. 2.1, VAMDC provides rich metadata to describe its data. We identified three different sets of metadata within the infrastructure, each set being associated with a particular infrastructure component:

  1.:

  The metadata associated with the served queries characterize the queries (e.g., the query execution time, the query text string, the version of nodes having generated the data, the version of the standard used to format the output data, etc.). These metadata are stored on the Query Store;

  2.:

  The metadata associated with the available resources (e.g., list of supported requestable, returnable and restrictables,¹⁸ node capabilities, supported access protocols, names of node maintainers, contacts, etc.). These metadata constitute the data stored into the Registry;

  3.:

  The metadata about the species and processes available for each node. These metadata constitute the data stored into the Species database;

  These three sets of metadata help users in discovering which database may contain interesting information.

  ${\mathcal {L}}$ :

  = 4

  ${\mathcal {I}}$ :

  A better integration between the Species database information and those from the Registry is encouraged. Users should have a single access point giving all the information on a given node, without having to use two different services. Today the information is fragmented.

• RDA-F3-01 M—Essential

  ${\mathcal {D}}$ :

  Metadata include the identifier for the data;

  ${\mathcal {A}}$ :

  1.:

  In the case of the Query Store, the PID associated with the query resolves to a Digital Object containing both the metadata and the underlying data. These are accessible for both human and machines;

  2.:

  In the case of the Registry, the underlying data associated with the metadata are the resources (e.g., nodes and processors). These are clearly identified by their IVOID (cf. Table 1) and their service endpoint;

  3.:

  In the case of the Species database, the underlying data associated with the metadata are the nodes and the processes supported by the nodes. In the web–human interface, these are clearly identified (by a direct link to the autonomous version of the database, not the VAMDC IVOID or TAP endpoint), whereas in the Excel machine-oriented interface,¹⁹ the IVOID of resources is provided. The link with the processes is a simple text string (only human-readable information in the web interface) and is missing in the machine-actionable interface;

  ${\mathcal {L}}$ :

  = 4

  ${\mathcal {I}}$ :

  In the case of the Species database, improve the machine actionability of the information, by putting, for example, links to the VAMDC IVOID of ressources in the human–web interface. Also pointers to processes should be machine-actionable.

• RDA-F4-01 M—Essential

  ${\mathcal {D}}$ :

  Metadata are offered in such a way that they can be harvested and indexed;

  ${\mathcal {A}}$ :

  The above-described three sets of metadata are offered in a way which may be indexed. VAMDC itself indexes those metadata and provides discipline-oriented (atomic and molecular data community) services and portal. However, it does not seem that the metadata from VAMDC are harvested and/or indexed outside VAMDC (e.g., Google or other indexing services). From the technical point of view, there are no particular barriers to achieve a better integration between indexing services and VAMDC, following an approach similar to the one described in [26];

  ${\mathcal {L}}$ :

  = 3

  ${\mathcal {I}}$ :

  VAMDC experts may define strategies to improve metadata harvesting and indexing from outside their data services. Two different strategies may be sketched: prepare metadata in formats that may be directly ingested by indexing services [27], or register VAMDC endpoints and standards in public catalogues of services (e.g., EOSC Portal,²⁰ FAIRsharing [28], etc.) so that harvesting services may discover and interact with VAMDC resources.

• RDA-A1-01 M—Important

  ${\mathcal {D}}$ :

  Metadata contain information to enable the user to get access to the data;

  ${\mathcal {A}}$ :

  The data in VAMDC are completely free and open. In the case of the Query Store, the digital object may be accessed directly by resolving a given PID. In all the other cases, data are accessible following classic https links. Since the access to the data is straightforward (in the sense that no authentication or particular procedure is required to use the services), VAMDC does not require specific metadata to enable users to get access to the data.

  ${\mathcal {L}}$ :

  = 0

  ${\mathcal {I}}$ :

  It could be useful for users to systematically provide information about the conditions of usage for VAMDC services, personal data and privacy policy and licenses on data. Those conditions are already defined for the Portal and for the Query Store. These should be extended to the Species database and to the Registry cf. indicator RDA-R1.1-01 M.

• RDA-A1-02 M—Essential

  ${\mathcal {D}}$ :

  Metadata can be accessed manually (i.e., with human intervention);

  ${\mathcal {A}}$ :

  The core central services of VAMDC (cf. Sect. 1.1) have a web interface designed for humans. All the metadata are accessible through those interfaces;

  ${\mathcal {L}}$ :

  = 4

• RDA-A1-02D—Essential

  ${\mathcal {D}}$ :

  Data can be accessed manually (i.e., with human intervention);

  ${\mathcal {A}}$ :

  The VAMDC portal [4] has a web interface designed for humans. All the data are accessible through those interfaces;

  ${\mathcal {L}}$ :

  = 4

• RDA-A1-03 M—Essential

  ${\mathcal {D}}$ :

  Metadata identifier resolves to a metadata record;

  ${\mathcal {A}}$ :

  1.:

  In the case of the Query Store, the PID resolves directly to a digital object where the metadata and the underlying data are packed together. Resolving a Query Store PID gives access both to query metadata and query produced data;

  2.:

  In the case of the registries, a resolution service is implemented which, starting from the resources IVOID (cf. Table 1) resolves to the underlying Registry node resource record;

  3.:

  In the case of the Species database, a resolution service is implemented which, starting from a given species identifier resolves to the underlying metadata;

  ${\mathcal {L}}$ :

  = 4

  ${\mathcal {I}}$ :

  The endpoints of machine-oriented resolution services for the Species database and the Registry should be advertised and indexed in some ad hoc service, so everybody may discover these endpoints.

• RDA-A1-03D—Essential

  ${\mathcal {D}}$ :

  Data identifier resolves to a digital object;

  ${\mathcal {A}}$ :

  In the case of the Query Store, the PID resolves directly to a digital object where the metadata and underlying data are packed together. Resolving a Query Store PID gives access both to query metadata and to the extracted data;

  ${\mathcal {L}}$ :

  = 4

• RDA-A1-04 M—Essential

  ${\mathcal {D}}$ :

  Metadata are accessed through standardized protocol;

  ${\mathcal {A}}$ :

  All the communications between the VAMDC web services are built over the http protocol (and its secured version https), which is free and standard. All the data and metadata are accessed using http. If we go deeply into details, the messages encapsulated into this http envelope also follow standard and open protocols: The Registry relies over the IVOA recommendation,²¹ and the data nodes implements the TAP protocol for data extraction²²and accepts queries formatted into a standard query language²³;

  ${\mathcal {L}}$ :

  = 4

• RDA-A1-04D—Essential

  ${\mathcal {D}}$ :

  Data are accessible through standardized protocol;

  ${\mathcal {A}}$ :

  The analysis is the same as in the case of RDA-A1-04 M;

  ${\mathcal {L}}$ :

  = 4

• RDA-A1-05D—Important

  ${\mathcal {D}}$ :

  Data can be accessed automatically (i.e., by a computer program);

  ${\mathcal {A}}$ :

  The architecture of VAMDC infrastructure relies on web services designed for machine-to-machine interaction. The web interfaces and tools designed for humans are an overlay to those services. Data extraction and handling are fully machine-actionable;

  ${\mathcal {L}}$ :

  = 4

• RDA-A1.1-01 M—Essential

  ${\mathcal {D}}$ :

  Metadata are accessible through a free access protocol;

  ${\mathcal {A}}$ :

  The analysis is the same as in the case of RDA-A1-04 M;

  ${\mathcal {L}}$ :

  = 4

• RDA-A1.1-01D—Important

  ${\mathcal {D}}$ :

  Data are accessible through a free access protocol;

  ${\mathcal {A}}$ :

  The analysis is the same as in the case of RDA-A1-04 M;

  ${\mathcal {L}}$ :

  = 4

• RDA-A1.2-01D—Useful

  ${\mathcal {D}}$ :

  Data are accessible through an access protocol that supports authentication and authorization;

  ${\mathcal {A}}$ :

  As previously mentioned, all the VAMDC data and services are open and free. However, as discussed in [20], the protocols may support authentication and authorization;

  ${\mathcal {L}}$ :

  = 2

• RDA-A2-01 M—Essential

  ${\mathcal {D}}$ :

  Metadata are guaranteed to remain available after data are no longer available;

  ${\mathcal {A}}$ :

  The Species database and the Registry are services describing the real-time state of the infrastructure. The legacy service for VAMDC data is the Query Store. Its metadata are kept permanently, even after the data deletion;

  ${\mathcal {L}}$ :

  = 4

• RDA-I1-01 M—Important

  ${\mathcal {D}}$ :

  Metadata use knowledge representation expressed in standardized format;

  ${\mathcal {A}}$ :

  1.:

  The DOI associated with a query in the Query Store resolves to a Digital Object whose metadata follow the DataCite metadata format.²⁴ The metadata, obtained while resolving the PID on the Query Store side, are formatted in JSON text, but do not follow any specific standard;

  2.:

  The metadata in the Registry follow the IVOA standards already cited, and the information provided is built over standard dictionaries²⁵;

  3.:

  The metadata in the Species database do not use specific standards to express knowledge;

  ${\mathcal {L}}$ :

  = 3

  ${\mathcal {I}}$ :

  1.:

  The format of the metadata returned by the Query Store while resolving a PID should be documented (in both human-readable and machine-actionable way) and published both through the VAMDC standards and in metadata registries, such as FAIRsharing²⁶[28];

  2.:

  In the Registry, when a specific term is used (for example, a given value for a restrictable or returnable) there should be a pointer, under the form of a PID, resolving to the definition of the term itself. A less desirable solution should be to point to the VAMDC dictionary repository from the Registry. Also, the version of standards should contain a PID resolving to the standards definition. Up to now, the links between the access protocol, the dictionaries and the other components are explained in human-readable documentations but not explicitly expressed in machine interfaces;

  3.:

  A dictionary specific to the knowledge representation of the Species database should be introduced and published (both in human-readable and machine-actionable way), both on the VAMDC standard page and in metadata registries such as FAIRsharing.

• RDA-I1-01D—Important

  ${\mathcal {D}}$ :

  Data use knowledge representation expressed in standardized format;

  ${\mathcal {A}}$ :

  The data extracted from VAMDC are formatted using the community-specific XSAMS standard. The XSAMS standard is documented in a human-readable form and in a machine-actionable form via an XSD schema;

  ${\mathcal {L}}$ :

  = 4

  ${\mathcal {I}}$ :

  Three aspects may improve the interoperability of the XSAMS standard:

  1.:

  Register the XSAMS standard as a standard format into registries of types such as FAIRsharing;

  2.:

  The VAMDC format contains enumerations of terms which are defined neither in the scheme, nor in an external dictionary: for example, CategoryType in Methods object, dataDescription values in DataSets object;

  3.:

  For codes of processes, PID resolving to the definition of the codes should be introduced. In particular for IAEA DCN codes,²⁷ the relevant metadata associated with each code is resolvable at as a JSON object (e.g., https://amdis.iaea.org/databases/processes/HCX/).

• RDA-I1-02 M—Important

  ${\mathcal {D}}$ :

  Metadata use machine-understandable knowledge representation;

  ${\mathcal {A}}$ :

  1.:

  The DOI associated with a query in the Query Store resolves to a Digital Object whose metadata follow the DataCite metadata format, which is machine-actionable. The metadata obtained while resolving the PID on the Query Store side are formatted in JSON text, but do not follow any specific standard. Metadata are machine-readable but not machine-understandable;

  2.:

  Metadata in registry are built to be machine-understandable;

  3.:

  Metadata in the Species database are formatted in JSON text, but do not follow any specific standard. Metadata are machine-readable but not machine-understandable;

  ${\mathcal {L}}$ :

  = 3

  ${\mathcal {I}}$ :

  Improve the machine understandability of the information from the Query Store and from the Species database.

• RDA-I1-02D—Important

  ${\mathcal {D}}$ :

  Data use machine-understandable knowledge representation;

  ${\mathcal {A}}$ :

  The XSAMS standard (cf. Sect. 1.1) is, as its name indicates, an XML schema which is designed to be machine-understandable. VAMDC provides several tools and converter to automatically interact with XSAMS data;

  ${\mathcal {L}}$ :

  = 4

• RDA-I2-01 M—Important

  ${\mathcal {D}}$ :

  Metadata use FAIR-compliant vocabularies;

  ${\mathcal {A}}$ :

  1.:

  The DOIs related to query store resolve to Digital Objects which use DataCite metadata (cf. Tables 1, 2), and as a consequence dictionaries that are compliant with the Dublin Core extended dictionary.²⁸ The PIDs of the Query Store use no FAIR dictionaries;

  2.:

  The registries use elements from VAMDC dictionaries to define capacities of nodes (e.g., restrictables and returnables). However, only the label of the term is used. There is no pointer to a PID resolving to the term definition. Moreover, the VAMDC dictionaries are not fully FAIR since they are not designed for machine but only for human fruition (e.g., there is just a web site and no API to access the terms and their definition);

  3.:

  The species DB does not use dictionaries to define the nature of metadata;

  ${\mathcal {L}}$ :

  = 2

  ${\mathcal {I}}$ :

  Make the VAMDC dictionaries more FAIR by improving machine actionability and term identification. Introduce dictionaries for the Species database.

• RDA-I2-01D—Useful

  ${\mathcal {D}}$ :

  Data use FAIR-compliant vocabularies;

  ${\mathcal {A}}$ :

  The VAMDC format contains enumeration of terms which are defined neither in the scheme, nor in an external dictionary, e.g., CategoryType in Methods object or dataDescription values in DataSet object. The codes for processes (including IAEA DCN codes) are simple text strings;

  ${\mathcal {L}}$ :

  = 2

  ${\mathcal {I}}$ :

  Improvements identified at item ${\mathcal {I}}$-2 and ${\mathcal {I}}$-3 of indicator RDA-I1-01D apply also here.

• RDA-I3-01 M—Important

  ${\mathcal {D}}$ :

  Metadata include references to other metadata;

  ${\mathcal {A}}$ :

  1.:

  The Query Store provides information about the node generating the query and the version of the standard. These references are only human-readable text string;

  2.:

  In the Registry, the records of data nodes contain some references to other metadata (services endpoints, curators, versions of standard) in a machine-understandable way, but not necessary machine-actionable: Those references cannot be resolved, as, for example, in the registry entry for Basecol²⁹ [29] one can find these two fields:

  

  and

  

  We see that we have no direct link to the definition of the cited standard, nor to the repository for the cited software. The two text strings enclosed by the XML tags may be understood only by expert users;

  3.:

  The Species database contains references to the nodes providing a given species. By now this points directly to the historical web sites of databases as they existed (and still exist) before the building of VAMDC and not to the VAMDC data node versions (i.e., to the corresponding data node resources, as defined in the Registry);

  ${\mathcal {L}}$ :

  = 2

  ${\mathcal {I}}$ :

  1.:

  The Query Store should provide references pointing to the node record in the registries and to definition of standards;

  2.:

  In the Registry, the metadata should point to definition of standards or to the software repositories in a machine-actionable way;

  3.:

  The Species database should have a link to the Registry record of each node.

• RDA-I3-01D—Useful

  ${\mathcal {D}}$ :

  Data include references to other data;

  ${\mathcal {A}}$ :

  In the XSAMS schema, the Source object may be used to reference other data and/or papers playing the role of source for the current data;

  ${\mathcal {L}}$ :

  = 4

  ${\mathcal {I}}$ :

  By now it does not exist a mechanism to reference connected data (e.g., similar data, previous version of the same data, etc.) directly inside XSAMS. VAMDC should investigate if such a need exists in its community.

• RDA-I3-02 M—Useful

  ${\mathcal {D}}$ :

  Metadata include references to other data;

  ${\mathcal {A}}$ :

  The DOIs assigned by the Query Store are versioned. The filiation of data with previous version are a typical example of “metadata include references to other data.” In all the other cases this indicator is not applicable;

  ${\mathcal {L}}$ :

  = 0

• RDA-I3-02D—Useful

  ${\mathcal {D}}$ :

  Data include qualified references to other data;

  ${\mathcal {A}}$ :

  The analysis is the same as for indicator RDA-I3-01D. The references are qualified in a machine-readable way (but not necessarily-actionable since the DOI is not mandatory in the XSAMS Source object) as “sources for the data”;

  ${\mathcal {L}}$ :

  = 4

• RDA-I3-03 M—Important

  ${\mathcal {D}}$ :

  Metadata include qualified references to other metadata;

  ${\mathcal {A}}$ :

  The references described while evaluating RDA-I3-01 M are all qualified in a human-oriented way and in a machine-readable way;

  ${\mathcal {L}}$ :

  = 4

  ${\mathcal {I}}$ :

  Metadata references to other metadata should be machine-actionable, cf. the analysis of RDA-R1-01 M.

• RDA-I3-04 M—Important

  ${\mathcal {D}}$ :

  Metadata include qualified references to other data;

  ${\mathcal {A}}$ :

  The analysis is the same as for indicator RDA-I3-02 M;

  ${\mathcal {L}}$ :

  = 0

• RDA-R1-01 M—Essential

  ${\mathcal {D}}$ :

  Plurality of accurate and relevant attributes are provided to allow reuse;

  ${\mathcal {A}}$ :

  As explained while evaluating the previous indicators, VAMDC uses a wide number of metadata to describe its data. The quality of those metadata is high;

  ${\mathcal {L}}$ :

  = 4

  ${\mathcal {I}}$ :

  Use standards and norm for all metadata to get them machine-actionable.

• RDA-R1.1-01 M—Essential

  ${\mathcal {D}}$ :

  Metadata include information about the license under which the data can be reused;

  ${\mathcal {A}}$ :

  There are no metadata describing the license on data. Data in VAMDC are free, but the consortium never discussed about a license to apply on VAMDC extracted data. As of today, there are no metadata field defined to specify the adopted license;

  ${\mathcal {L}}$ :

  = 1

  ${\mathcal {I}}$ :

  Define a license for all the data/metadata in VAMDC (cf. Sect. 3.1).

• RDA-R1.1-02 M—Important

  ${\mathcal {D}}$ :

  Metadata refer to a standard reuse license;

  ${\mathcal {A}}$ :

  Since there is no license (and no metadata field to provide license information), we cannot point to a standard reuse license;

  ${\mathcal {L}}$ :

  = 1

  ${\mathcal {I}}$ :

  The adopted license for data/metadata should be a widely adopted and accepted one (e.g., one from the Creative Commons ecosystem³⁰).

• RDA-R1.1-03 M—Important

  ${\mathcal {D}}$ :

  Metadata refer to a machine-understandable reuse license;

  ${\mathcal {A}}$ :

  The analysis is the same as for indicator RDA-R1.1-02 M;

  ${\mathcal {L}}$ :

  = 1

  ${\mathcal {I}}$ :

  The adopted license should be machine-actionable.

• RDA-R1.2-01 M—Important

  ${\mathcal {D}}$ :

  Metadata include provenance information according to community-specific standards;

  ${\mathcal {A}}$ :

  We recall that provenance metadata in the XSAMS is well defined using the Source element;

  1.:

  Provenance of data in the Query Store is clearly stated, since all the metadata of a landing page are about provenance. This information does not follow any particular standard but are human-readable;

  2.:

  Provenance of data in the registries is well defined. The information follows IVOA standard;

  3.:

  Provenance in the Species database is well defined. (We know from which node the information comes.) However, we miss the time stamp information about the databases that provide the species information.

  ${\mathcal {L}}$ :

  = 3

  ${\mathcal {I}}$ :

  Adopt standards for provenance in the Query Store and in the Species database.

• RDA-R1.2-02 M—Useful

  ${\mathcal {D}}$ :

  Metadata include provenance information according to a cross-community language;

  ${\mathcal {A}}$ :

  Cross-community standard from provenance (W3C³¹ or IVOA-prov³²) are not used (cf. Sect. 3.2 for a discussion);

  ${\mathcal {L}}$ :

  = 1

• RDA-R1.3-01 M—Essential

  ${\mathcal {D}}$ :

  Metadata comply with a community standard;

  ${\mathcal {A}}$ :

  1.:

  The DOIs assigned by the Query Store use metadata compliant with the DataCite metadata standard;

  2.:

  The metadata constituting the registries follows the IVOA standard for registries;

  3.:

  In the Species database, we use Inchi and InchiKey chemical standards for species identifications. The other information do not follow any particular standard;

  ${\mathcal {L}}$ :

  = 3

• RDA-R1.3-01D—Essential

  ${\mathcal {D}}$ :

  Data comply with a community standard;

  ${\mathcal {A}}$ :

  Data extracted from VAMDC are formatted using the XSAMS standard;

  ${\mathcal {L}}$ :

  = 4

• RDA-R1.3-02 M—Essential

  ${\mathcal {D}}$ :

  Metadata are expressed in compliance with a machine-understandable community standard;

  ${\mathcal {A}}$ :

  1.:

  The DOIs assigned by the Query Store use metadata compliant with the DataCite standard, which are machine-actionable;

  2.:

  Metadata constituting the Registry follow the IVOA standard for registries, which are machine-understandable and machine-actionable;

  3.:

  The Species database metadata do not follow any particular standard;

  ${\mathcal {L}}$ :

  = 3

  ${\mathcal {I}}$ :

  Use machine-actionable standards for the information returned by the Query Store and by the Species database.

• RDA-R1.3-02D—Important

  ${\mathcal {D}}$ :

  Data are expressed in compliance with a machine-understandable community standard;

  ${\mathcal {A}}$ :

  Data extracted from VAMDC are formatted using the XSAMS standard, which is fully machine-understandable;

  ${\mathcal {L}}$ :

  = 4

  ${\mathcal {I}}$ :

  Register XSAMS into registries of types [30], such as FAIRsharing [28].