Citizen science, accessibility, art & science, critical thinking, policy and engagement: thoughts and lessons learned from the REINFORCE experience

Abstract

REINFORCE (Research Infrastructures FOR Citizens in Europe) is a Research & Innovation Project, supported by the European Union’s Horizon 2020 SwafS, ‘Science with and for Society’ work programme (GA872859). The project, which ran from December, 2019, to November, 2022, engaged the public in a variety of innovative ways. Four citizen-science demonstrator projects were developed on the world-leading Zooniverse platform, each focussing on a different area of frontier physics: gravitational waves; neutrino astronomy; particle physics; and muography. A range of art and science events were launched and undertaken. A data-sonification tool—sonoUno—was developed in order to improve the accessibility of the data used in the four demonstrator projects. A course on critical thinking and a history of the Second Scientific Revolution was provided on YouTube and in podcast form, while a senior-citizen-science course was designed, co-developed and implemented. These initiatives were supported by a detailed engagement plan, a dedicated communications and dissemination strategy, and a constantly evolving assessment and evaluation approach. The experiences garnered during the project, in conjunction with consultations with project participants, volunteers and stakeholders, were built into the form of a policy roadmap explaining how to integrate citizen science into research infrastructures in Europe. The roadmap identifies a series of policy objectives and related policy gaps, associated challenges and lays out a series of recommendations. This article describes the results of the REINFORCE project and draws together the experiences of each of the involved twelve partner organisations.

1 Introduction

Large research infrastructures working in the field of frontier physics continue to open new observational windows on to the universe and to explore the structure of matter in ever greater detail. As these discoveries become more and more sophisticated, so the expertise required in order to understand them also increases. As a consequence, the public are often only able to access and interpret the results of research through the prism of outreach activities, and are thus not able to contribute directly to the development of new scientific knowledge. This distance contributes to a gap between science on one hand and society on the other; a gap that can be problematic in the context of public-funded scientific research.

REINFORCE has aimed to co-design, along with citizen scientists themselves, an approach that breaks down the barriers to access that contribute to the widening of this gap and to build communities of practice around four demonstrator projects on the Zooniverse platform. Volunteers in these communities have contributed directly to the production of scientific knowledge and have worked closely with different research teams to classify and understand data subjects in both quantitative and qualitative ways.

The REINFORCE project pursued a set of key aims: engage people to contribute to online frontier science; create a community of citizens that actively participates in scientific endeavours; introduce responsible research and innovation into the frontier citizen science landscape; assess the impact of frontier citizen science in science and society; create a policy roadmap for the implementation of citizen-science projects in large research infrastructures (LRI); and explore the potential of frontier citizen science for inclusion and diversity.

To reach these goals, a series of initiatives were developed and implemented, ranging from Zooniverse demonstrator projects, to art and science initiatives, via courses on critical thinking and science for senior citizens, to the development of dedicated data-sonification tools and courses. All of these initiatives were encompassed within clearly-defined engagement and dissemination strategies and a thorough and wide-ranging evaluative approach.

The REINFORCE Collaboration is formed of twelve different partners — Centre national de la recherche scientifique (CNRS), CONICET, Ellinogermaniki Agogi, European Gravitational Observatory (EGO), Institute of Accelerating Systems and Applications (IASA), Open University, Oxford University, The Lisbon Council for Economic Competitiveness and Social Renewal, Trust-IT Services, Université catholique de Louvain and Università di Pisa — from seven different countries across two continents — Austria, Belgium, France, Greece, Italy, the United Kingdom and Argentina. The European Gravitational Observatory took on the role of Project Coordinator.

The project was funded within the EU Science with and for Society (SwafS) work programme of the Research & Innovation Project thematic area. Dedicated to the integration of society in science and innovation issues, policies and activities, it promotes the integration of the interests and values of citizens in these areas with the aim of increasing the quality, relevance, social acceptability and sustainability of research. REINFORCE also focussed on supporting two specific United Nations Sustainable Development Goals (4. Quality Education and 5. Gender Equality).¹

By developing the work of the project through participatory design, involving all members of the project, from partners to volunteers, to stakeholders in other fields, the aim was to build a committed, diverse community of practice. As the project evolved, the experiences garnered across its life cycle, fed into the development of a policy roadmap, designed to signpost issues and provide recommendations for best practice in terms of the integration of citizen science in LRI across Europe.

This article looks first at the demonstrator projects on Zooniverse themselves. For each of them, a description of the project is provided, as well as an examination of the successes; what worked well and why. Interaction with volunteers and the demonstrator project community are also covered, as well as the impacts of the engagement strategy and activity implemented for each of them. The report on each of the projects concludes with a look at the lessons learned, at the areas that might be approached differently and the outcomes and future prospects for the long-term exploitation of the project and its resources. Following the demonstrator projects, the article explores the effectiveness of approaches to increase accessibility to scientific research within the project and the ways in which the effectiveness of these were understood and how they ultimately fed into the development of the policy roadmap, which stands as a keystone resource at the end of the project.

2 Demonstrator projects on Zooniverse

At the core of REINFORCE were four demonstrator projects developed on the Zooniverse online platform for people-powered research.² On Zooniverse, volunteers contribute to research projects and participate in the study and classification of data. The ethos behind the platform is that anyone can be a researcher; that working together can accelerate research and enable research that otherwise would not be possible, as well as potentially leading to new discoveries.

Each of the four REINFORCE demonstrator projects centred on a different area of frontier physics. GWitchHunters³ introduces volunteers to the data produced by the Virgo gravitational-wave detector to characterise its noise sources and improve its sensitivity.⁴ New Particle Search at CERN⁵ allows volunteers to work through different stages classifying displaced vertices; identifying the signatures of known particles; searching for Higgs boson decays; and looking for long-lived particle decays, in data provided by the ATLAS detector at the Large Hadron Collider.⁶ In Deep Sea Explorers,⁷ volunteers classify bioluminescence signals that constitute noise in the KM3NeT neutrino detectors⁸ and identify bioacoustic signals recorded by hydrophones located at the detector sites, in order to provide information useful for the study of the surrounding marine environment. In Cosmic Muon Images,⁹ volunteers identify particle tracks as they passed through the different stages of a muon detector.

Memoranda of understanding between the REINFORCE collaboration and each of the individual collaborations providing data for the demonstrator projects — Virgo, KM3Net and ATLAS — were agreed in the early stages of the project, which were all initially developed only in English. Following the launch of each of them, however, it was agreed to also implement alternative language versions, in order to render them more accessible and to better facilitate their dissemination across target groups and to contribute to the engagement targets of the project. At time of writing, GWitchHunters is available in both English and Italian, with the Spanish version due to go online shortly, while New Particle Search at CERN is available in English, Greek and Spanish.

2.1 GWitchHunters

The GWitchHunters project [1] focuses on supporting research in the field of gravitational waves through the contribution of citizen scientists. Citizens are asked to contribute to the study and characterisation of the data recorded by the Advanced Virgo interferometer [2]. Virgo is the main gravitational-wave detector in Europe and, together with Advanced LIGO [3] and KAGRA [4], forms part of an international network of experiments for gravitational-wave detection. These detectors are extremely sensitive machines, and, as such, need to be shielded as much as possible from potential sources of noise, which can produce spurious signals in the detector data that can be much larger than those expected from waves produced by astrophysical sources. It is therefore of paramount importance to properly characterise noise artefacts and to mitigate and possibly remove them. In particular, rapid transient noise events, known as glitches, are particularly detrimental for the detectors, since they can affect the duty cycle and can mask or mimic real astrophysical signals.

The characterisation of glitches is not trivial, however, as they appear arbitrarily in time and can exhibit very different types of behaviour. When plotted on a time-frequency map representing the time evolution of their spectral energy content (spectrogram), glitches can show diverse morphologies, and it is possible to group them into families.

This approach is based on the premise that, at the origin of the various glitch families, there are also distinct noise sources; identifying all the times when some of these sources manifest is the first step in further pursuing their identification for the subsequent implementation of mitigation strategies. Given the high rate of these glitches, and their large number over entire data-taking periods, studying them one by one can be very time-consuming. Machine learning (ML) is therefore of great help in their rapid identification, as it has the potential to automatically classify glitches on the basis of their individual spectrogram. As has been shown in different works, e.g. Zevin et al. [5], Razzano and Cuoco [6], convolutional neural networks are very effective at the task of glitch classification and can form the basis for an automatic classification pipeline. Additionally, considering the complexity of their morphology, Artificial Intelligence algorithms are proven to provide precious support in these classification tasks by noticing details and similarities over very large datasets that the human eye would struggle to identify.

Fig. 1

Identifying a glitch family in GWitchHunters

GWitchHunters volunteers provide classification information that can be used to train automatic classification pipelines. The project has aimed to expand the contribution of citizen scientists to tasks beyond a simple classification, in order to provide more complex and diverse data and to allow them to dig deeper in the investigation of the causes of the noise in the detector. In particular, volunteers choose from a variety of workflows with different degrees of difficulty. The first workflow comprises a simple classification of the shapes that appear in the spectrograms of the strain channel, the main channel in which gravitational-wave signals are detected, choosing from a limited number of glitch families (Fig. 1). In the second workflow, volunteers identify the glitches drawing one or more rectangles around the spectrogram regions exhibiting some excess of energy, and then perform their classification as in the previous workflow. In the third and more complex workflow, volunteers are asked to compare the shapes of the glitches appearing in strain channel with those of the so-called auxiliary channels, i.e. data coming from sensors that constantly monitor the detector parts and their physical environment (Fig. 2). The project has been officially launched on the Zooniverse platform the 16th of November, 2021, and also provides a set of mobile workflows, i.e. tasks that can easily be performed on mobile devices.

Fig. 2

Looking for correlations between Virgo auxiliary channels and a glitch in the strain channel. The glitch morphology is superimposed on to each of the auxiliary channel spectrograms

During the lifetime of REINFORCE, the GWitchHunters project obtained considerable success, both in terms of classifications and feedback from volunteers. To date, more than 4,600 individual volunteers have participated in the project, undertaking more than 700,000 classifications of 41,000 data subjects. These numbers are in broad alignment with the expectations for the project that were built on the basis of another Zooniverse project available on Zooniverse: Gravity Spy.¹⁰ Volunteers have proven to be very enthusiastic; often going beyond the more basic tasks. In particular, they were also asked to look for glitches that were not in the initial set of available glitch families. This aspect is very important in terms of discovering unknown noise sources, an aspect that is difficult to solve using ML only. Another aspect that has been particularly successful, has been the availability of translations in languages other than English. In order to test this aspect, a complete translation of the project into Italian was prepared, and a dedicated press event was held, in order to properly present the project to the volunteers in Italy. The results have been very interesting, showing that this approach can be very useful in promoting access to the project beyond only people who speak English to at least some degree.

The GWitchHunters classification results, using data from the O3 data-taking run, have been input to a ML algorithm built on a convolutional neural network (CNN) [1]. The performance of the ML has been compared to that of citizens, and the results have shown that this approach, based on citizen science, is promising, not just in terms of preparing datasets for the training of ML algorithms, but also in terms of contributing actively to the research in general. New datasets are in preparation, also taking into consideration the upcoming observational campaign — O4 — and it is expected that the contribution of volunteers will become more and more important. There are also other plans to further optimise the way in which the data subjects are presented to volunteers, as well to improve the levels adaptation for mobile devices. It is also hoped that introduction of the Spanish-language version of the project will help to broaden the participant base further.

2.1.1 Sonification of gravitational-wave data

The spectrogram images are a valuable tool for representing gravitational-wave detector data, particularly for the general public, without signal analysis expertise. To make this kind of data representation even more inclusive, the GWitchHunters project has developed a dedicated sonification algorithm to convert the images into sounds. At the base of this algorithm there is the association of frequencies with musical notes, and the signal energy with the notes intensities, as we shall describe with the following example.

Let’s consider the first detection event of a gravitational-wave signal from the coalescence of a pair of black holes, GW150914 [7]. Its spectrogram is shown in the left-hand side part of Fig. 3. The vertical axis of the spectrograms, representing the frequency, has been divided into intervals determined by the frequencies of the notes of the C-major scale of occidental music, as shown in the image on the right-hand side part of the figure. This choice, although arbitrary, aligns with the white piano keys and helps convey the meaning of different frequencies.

The energy in each band corresponds to the intensity of the note, and its evolution can be represented as a sequence of notes, resulting in the conversion of the spectrogram into a musical score and, in turn, into a melody. No need for musical knowledge is required to enjoy the latter and to let people able to distinguish different spectrogram morphologies as well as different melodies.

Additional data transformations can be implemented. For example, the gravitational-wave signal from the coalescence of a binary black hole system is characterised by a duration of a few tenths of seconds and frequencies up to a few hundred Hz. These times are too fast to appreciate the details of the signal evolution but, once converted to a musical score, the execution can be slowed down at will.

Similarly, the frequencies of this signal are a bit too low to be felt as enjoyable by the human ear. However, they can easily be shifted to a higher pitch by transposing the musical score by one octave (that is, doubling the frequencies) to match the human ear sensitivity range better.

The multisensorial representation of gravitational-wave data, in the form of images and sounds, fosters inclusiveness and increases the reach of the project by making the data enjoyable to a more vast public. Most importantly, it facilitates the engagement of individuals with visual impairments in both gravitational-wave research and the exploration of the cosmos.

Fig. 3

Alternative spectrogram representations of the GW150914 gravitational-wave signal as observed by the LIGO Hanford detector. a Usual time–frequency representation of the energy of the signal, as in the original article [7]. b Alternative representation where the energy corresponding to the different frequency bands is represented by the series of curve. The higher the curve, the greater the intensity of the corresponding musical note

2.2 New Particle Search at CERN

The New Particle Search at CERN¹¹ demonstrator project engages citizens in the state-of-the-art particle research performed at the Large Hadron Collider (LHC) of CERN, in the quest for the understanding of the ultimate structure of matter. The demonstrator is based on data collected by the ATLAS experiment¹² which are produced by high-energy proton-proton collisions at the LHC.

Volunteers perform a visual inspection of data samples consisting of events, namely the registered products of proton-proton collisions. In this way, they contribute to the search for yet undiscovered hypothetical particles predicted by theories Beyond the Standard Model (BSM) [8]. Their search could lead to a discovery, which would be a direct proof of new physics and would highlight a path for future research.

To enable the volunteers in their work, a three-stage architecture was adopted in the demonstrator project. The first two stages are based on selected samples of simulated data and are used to train volunteers, but also to allow for a quantitative assessment of their performance and a comparison with specially developed automated algorithms. The third stage of the demonstrator is a discovery stage, employing datasets of real events from the ATLAS Open Data Set Release [9]. The third stage provides two research paths: (a) study of Higgs boson decays to two photons, one of which could be converted to an electron-positron pair by interaction with detector material; and (b) search for yet undiscovered neutral long-lived particles, predicted by certain theories of the BSM. The volunteer research involves the identification of specific signatures, which are produced from the decay of these new long-lived particles. These decay products could originate from displaced vertices (DVs), namely vertices — formed by two or more tracks — that are displaced with respect to the main collision point of the two protons.

The demonstrator project provides visual analysis tools that allow the citizens not only to classify static images — in order to recognise the DVs — but also to interact with the event displays, select specific tracks and calculate kinematical quantities characteristic of the sought-after particles.

In Stage 1, which is hosted entirely on Zooniverse, volunteers are trained to recognise DVs in a high-purity sample of simulated data, corresponding to the various scenarios of new particles with displaced vertices. The volunteers only inspect stationary images of the traces that charged particles leave in the inner part of the ATLAS detector. They look for tracks that intersect at a point other than the main interaction point and need to inspect both views of the inner detector in order to be able to properly recognise track intersection. In the two different projections of the inner detector, which are depicted in Fig. 4, the tracks are given different colours, so that a user may identify the same track in both views. The volunteers are then asked to spot the DVs in both views, and the answer is internally assessed by Zooniverse, based on the truth information which is also provided to the platform.

Fig. 4

The first stage of New Particle Search at CERN on Zooniverse

The quality of the volunteers’ work is assessed and compared against a specially developed automated algorithm, which detects the presence of displaced vertices in the events. The algorithm extrapolates the tracks and looks for their intersection, using only detector information that is also available to the volunteers. The overall goal was to compare the DV-identification efficiency of the volunteers to that achieved by the algorithm. The efficiency is defined as the fraction of long-lived particle decays that were successfully identified by the volunteers. The results, from 180,000 classifications provided by the users, were analysed in multiple ways. The most interesting results were obtained by considering the user consensus as an identification criterion. The identification efficiency of the user consensus is found to be about 93% on average, depending on the projection of the ATLAS detector provided in the image (89% in the transverse detector view, 96% in the longitudinal detector view). This efficiency is very close to the respective identification efficiency obtained by the dedicated automated algorithm, which was 94%.

For Stages 2 and 3, the users are directed from Zooniverse to the HYPATIA [10] event display. Instead of simply examining static images and locating DV they interact with the event display in order to perform in-depth analysis of the events. In the context of the demonstrator project, significant additions were made to the HYPATIA platform, in order to incorporate the functionality necessary to enable the volunteers to perform their analysis in each stage of the project. In addition, a new event format was developed to include additional information, such as the display of DV.

In Stage 2 volunteers are asked to identify certain particle types that are useful for the next discovery stage. They look for signatures of muons, electrons, photons and converted photons. The HYPATIA particle information table, shown in Fig. 5, displays momentum, charge and direction information for each track or cluster of the event. The volunteer can identify tracks and clusters, using the button that corresponds to the type of track or cluster to which they have determined it belongs (electron, muon, photon or converted photon). When the volunteer clicks on the Next button to display another event, the selections made are stored in a back-end database for later processing. Since the sample consists of simulated events, the particle generation identification is already known and thus can be compared to the user classification, in order to determine the validity of the choice made. The volunteers made a total of 37,000 classifications in this way, with 80.4% of them being correct.

Fig. 5

The second stage of new particle search at CERN on HYPATIA

The volunteer output was also compared to an ML algorithm, which was developed based on Boosted Decision Tree classifiers, and which uses exactly the same information that is available to the volunteers, in order that this comparison be fair and impartial. The comparison shows that the identification ability of the volunteer cannot be better than a dedicated ML algorithm, but, in the cases of the electron and photon, the user identification efficiency is not very far from it.

In the Higgs boson stage, the volunteer can indicate that a cluster belongs to either a photon or a converted photon that could originate from Higgs boson decays. HYPATIA automatically calculates the invariant mass when the volunteer selects a pair of photons. In addition, the volunteer can rate the event (from one to five stars) based on the instructions given on the platform. In this study, users selected 94% of all photon-pair masses in the 106–160 GeV mass range. Distinguishing the Higgs boson signal, which lies on top of the background, requires a much larger dataset than that available to the volunteers and a sophisticated statistical analysis of the data. The volunteers rated 1156 events as being worthy of five stars, with most events receiving low star-ratings as they did not contain unusual Higgs candidate decays. Of those 1156 five-star-rated events, only a few contain extra leptons, but the invariant mass of the photons (or converted photons) is outside the mass range of the Higgs boson mass, therefore those events cannot be attributed to complex Higgs boson production and decay mechanisms.

In the Long-Lived-Particle (LLP) Hunting part of the demonstrator project, volunteers search for long-lived particles predicted by some BSM models, through the identification of secondary vertices of particle decays. Following the vertex identification, they were asked to look for muons originating from the displaced vertex and to examine certain kinematic variables, which could be characteristic of a candidate long-lived particle. Volunteers were also asked to rate each event from one to five stars, according to how similar they believed it to be to one of the sought-after decays. A total of 81,894 classifications were performed on a sample consisting of 2440 events. Events which satisfy all given requirements have a much higher probability of originating from a new long-lived particle. Since these particles are as yet undiscovered, the relevant candidates are extremely rare. The volunteers only viewed 27 of them (volunteers viewed each one multiple times) and correctly marked with five stars 23 of those events. Two of the five-star events were good candidates for new BSM long-lived particles (hypothetical super symmetric particles). The events were scrutinised by our team and further information (which was not available to the volunteers) was inspected. It turns out that, most probably, both events are due to the expected background. A number of volunteers did isolate 24 events with more than one muon in the DV and posted their findings in the project Talk forum, where they were discussed by the wider community. After further investigation by the research team, it was revealed that these events were either due to the interactions of a known particle with the detector or faults in the reconstruction of particle tracks.

2.2.1 Interaction with users, engagement and impacts

From the launch of New Particle Search at CERN as an official Zooniverse project on the 19th of October, 2021, until the end of the REINFORCE project, on the 30th of November, 2022, the project research team received and replied to 2,852 Talk messages (seven messages per day on average) posted by volunteers on the project forums available on Zooniverse.¹³ Volunteers communicated technical questions regarding the stages of the project (especially during its first few months), questions on physics related (or not) to the tasks provided to them, their results and observations, as well as interesting suggestions and features they would like to see added. The members of the scientific research group were delighted to respond to volunteer posts on a daily basis, but also motivated discussion between volunteers, while assigning the role of expert to the most talented among them. Furthermore, in collaboration with the Ellinogermaniki Agogi (EA) team, various activities during the course of the project were prepared, which served to ensure the motivation of volunteers remained high. These included:

• Awarding hard-copy certificates to interested volunteers who had completed over 200 classifications with over 50% efficiency, acknowledging their citizen-science contribution;

• Meeting with volunteers in person at the REINFORCE Summer School, 2022, in Marathon, Greece.¹⁴ During this summer school, volunteers were presented with the ongoing research and the latest achievements in the field of high-energy physics, and had the opportunity to work together with our scientific team;

• Four challenges (Winter Challenge 2022,¹⁵ Easter Challenge 2022,¹⁶ Challenge for Greek teachers,¹⁷ EPS competition¹⁸) during which the performance of each volunteer was closely monitored. At the end of each challenge, the most successful volunteers were awarded prizes, with the results announced on the Zooniverse forums;

• Online meetings with volunteers, in which their questions and ideas were discussed, as well as virtual visits to LHC experiments (ATLAS and ALICE). These activities maintained the engagement of citizens in New Particle Search at CERN and volunteers ended up contributing an impressive 179,887 classifications (441 classifications per day on average) to Stage 1 of the project alone. According to the data gathered by the Zooniverse platform, volunteers spent about five minutes per classification, on average.

In conjunction with the work undertaken by the REINFORCE engagement team, the New Particle Search at CERN team also developed a set of feedback metrics and mechanisms, which supported volunteers during their early encounters with the projects. During the final months of the REINFORCE project, all of the New Particle Search at CERN materials were translated into Greek, while the REINFORCE partner CONICET also provided a Spanish translation. All three language versions are now available on the project’s Zooniverse page. Furthermore, data on the particle identification from the second stage was provided to the CONICET team in order to develop a way to represent the different particle types with distinct sounds. Sound files from a small number of events were generated as part of this effort and tested during different dissemination events. It was determined that the second stage of the demonstrator project is the only part of it that can be sonified, as the other two are too visually complex and don’t lend themselves to auditory representation.

In addition to motivating volunteers, the scientific community was also motivated towards citizen science through presentations of the project at six international scientific conferences, with contributions to the respective conference proceedings.

2.2.2 Outcomes and the future

The final goal of the discovery stages is to have thousands of citizen scientists rate each event with 1–5 stars, according to how similar they believe them to be to one of the new long-lived particle decays. In this way, the volunteers are directly involved in potential discoveries, without requiring any prior knowledge of high-level physics or computing skills. Furthermore, they contribute to a frontier research science topic, performed at the largest particle physics facility in the world.

Finally, a large number of dedicated volunteers left very informative and useful comments in the Talk forum during the classifications of the two discovery stages. Overall, New Particle Search at CERN should be considered a success, as it managed to get a large number of volunteers involved in complicated analysis of high-energy-physics signatures collected by the state-of-the art ATLAS detector.

The REINFORCE project ended at the end of 2022, but the demonstrator project remains open and available on the Zooniverse platform and volunteers continue to classify events. When a new ATLAS Open Data release is available, together with other resources, new, fresh data will be added.

2.3 Deep Sea Explorers

The Deep Sea Explorers¹⁹ demonstrator project takes place in the context of the KM3NeT experiment,²⁰ a neutrino telescope currently being deployed in the Mediterranean Sea. While the main focus of the KM3NeT Collaboration is on the search for neutrinos, Deep Sea Explorers volunteers were asked to study events caused by sea mammals and benthic fauna.

KM3NeT is equipped with light sensors to detect Cherenkov light produced by a neutrino interaction in the sea, as well as hydrophones that are used to calibrate the instrument as it moves with the sea currents. In addition to being able to detect elementary particles crossing the detector, KM3NeT is also sensitive to its environment. Volunteers were therefore asked to classify the various signatures of light and acoustic noise produced in this environment. The aim of these classifications was to help to better understand, on one hand, KM3NeT data and the response of the instrument, and, on the other, the presence of life in the deep sea in which the detector is deployed.

In the bioluminescence workflow, volunteers were asked to classify data subjects and determine the number of peaks visible within them. In the bioacoustics workflow, they were asked to study different visual and audio representations of data subjects recorded by the hydrophones around the detectors and to determine whether they signalled the presence of sperm whales, short-finned pilot whales, were just pure noise or were likely something else.

As the KM3NeT Collaboration is an experiment that is still in the deployment phase, it was decided to split the acquisition of the classifications performed by citizen scientists into two separate phases. In addition to giving the scientist team more time to acquire additional raw data to be classified by the participants, this also created a motivating feature for the citizens. The first phase of the demonstrator ended in February, 2022, after the launch of the project earlier that year. The first classifications were analysed and the results, presented in the next section, were presented to the Deep Sea Explorers participants before the start of Phase 2.

A much larger set of events, a factor 10 and 20 for the bioluminescence and bioacoustics workflows, respectively, was created for Phase 2 and uploaded onto the Zooniverse platform a month later. Both workflows, despite not being fully completed, made it possible to acquire sufficient data to carry out a comparative study.

2.3.1 Successes: what worked and why

Deep Sea Explorers proved to be a real success. It has been possible to demonstrate that the help of citizen scientists improves the output of a classifier when labelled datasets are available, i.e. in the case of the bioacoustics workflow. In the case of both workflows, sufficient trained classifiers were obtained to now be used on the data collected in KM3NeT over its entire lifetime.

After the first phase of the project, it became possible to clearly identify some events as belonging to one of the four categories made available to volunteers on Zooniverse. As an example, the two events shown in Fig. 6 have been classified as being one peak events with a 90% and 96% certainty (i.e., this is how often they were classified in this category by the participants). On the other hand, some events have not been classified unanimously by the participants, such as that displayed in Fig. 7.

Fig. 6

Bioluminescence events from Phase 1 that were identified as being one peak events by the citizen scientists

Fig. 7

Example of an event that was classified as a short-finned pilot whale (28%), sperm whale (35%), or pure noise (24%)

The goal of the project was to obtain a set of very well classified data subjects. Despite some events already fulfilling the requirements to reach this objective, more events were needed for Phase 2. In this second phase, the aim was to provide new subjects so that the number of events that are classified with a high probability in only one category could be increased. Phase 2 therefore, had many more events than Phase 1, in order to significantly increase the data available for the final statistics.

At the end of Phase 2, the events of the bioacoustics workflow were used to train a convolutional neural network (CNN), previously developed using data from hydrophones located at the surface. While the events used in the demonstrator, which were originally classified with less than 70% accuracy by the already existing neural network, were used to train the new CNN and led to a precision in the classification of sperm whale and short finned pilot whale of 90% and 100%, respectively.

We concluded that the data collected from the volunteer classifications provided through the demonstrator project can help us to enhance the accuracy of classification of KM3NeT events in comparison to the original neural network.

2.3.2 Areas that might be approached differently

The involvement of citizen scientists in active research projects taking place within the KM3NeT collaboration was a real success with concrete scientific results as proof. However, the personpower required for such a project was misevaluated by the Deep Sea Explorer team. As the volunteers were discovering new research fields, moving between neutrino physics and astronomy to marine biology, detailed explanations and continuous support from the science team was needed. The Talk forum, which forms part of every Zooniverse project, allows for an easy and direct interaction between the science team and the citizen scientists, but, as with any forum, an active and constant engagement in the discussion takes time.

2.3.3 Lessons learned

Deep Sea Explorers results demonstrated to the science team, as well as to a considerable fraction of the particle physics community, the substantial gains that citizen science can bring to event classification. As other projects are added to Zooniverse, an important characteristic of them might be to keep in mind the time and investment required to properly nurture, grow and support a project of this kind. The Deep Sea Explorers science team will be developing more projects of this nature and will aim to maintain a support team that is at least twice the size of the team used for this project.

2.3.4 Interaction with the demonstrator project community

Besides successfully achieving the classification goal, the Deep Sea Explorer volunteer community caused a considered reflection within the research team on how scientific wording can be perceived. The original name of the demonstrator project was due to be Deep Sea Hunters, as researchers in the KM3NeT Collaboration often called themselves Neutrino Hunters. During beta testing, however, the Zooniverse community pointed out that the word hunter could have a negative connotation when associated to marine fauna and could discourage people to take part in the project. The demonstrator was duly renamed as a result of this advice and became the Deep Sea Explorers project that ultimately went to official launch.

2.3.5 The future: next steps; short-, mid- and long-term exploitation of the project and its resources

This project has led some members to ask for the help of citizen scientists in relation also to other projects, such as, for example, in the classification of sub-threshold events recorded by the IceCube Neutrino Observatory, another neutrino telescope located at the South Pole. While no bioluminescence activity is expected in the ice, the detector itself creates patterns in the data that are not well modelled and for which the help of citizen scientists in their classification would be useful. This would constitute a new project on the Zooniverse platform that will be created using the know-how acquired during the development of Deep Sea Explorers.

The science team also created fruitful ground for interdisciplinary projects and collaboration with bio-informaticians and biologists. This collaboration will continue after the end of REINFORCE and throughout the lifetime of KM3NeT. Several grant applications to hire PhD students and postdocs in this framework have been submitted. The help of citizen scientists to disentangle neutrinos from the signal sent by living species at the bottom of the sea will be one of the methodologies used in these projects.

2.4 Cosmic Muon Images

The Cosmic Muon Images²¹ demonstrator project is based on muon tomography [11], an imaging technique to study the internal density distribution of massive objects. Muon tomography uses muons produced in cosmic ray atmospheric showers as a probe for inferring the density anisotropies of geological, archaeological and civil engineering objects like volcanoes, pyramids and tumuli.

The principle behind this technique is similar to that behind a medical X-ray; a particle detector is placed downstream of an object with respect to the incoming particle flux and takes measurements for an extended period of time. The angular profile of the reconstructed muon tracks is then compared to the theoretical expectation, taking into account the external geometric characteristics of the object as well as some basic assumptions on its composition. Combining all this information provides an image of the internal density distribution for the object under study.

The method was first used by Alvarez to search for voids inside the pyramid of Chephren at the Giza plateau, back in the 1960s [12]. It was proposed for volcanology in the mid-90 s [13] and reached its modern form in the mid-2000s [14]. Since then, the domain of study has rapidly extended into new fields, such as: the prediction of natural hazards, which includes monitoring for landslides, lahars and other similar mass displacements; the absorption/retention and distribution of rain water in underground repositories and aquifers; the investigation of pyramids for unknown voids, as well as other sites of archaeological interest, such as tumuli; the monitoring of industrial facilities like blast furnaces or nuclear waste disposal containers; and, last but not least, homeland security applications focused on the detection of nuclear contraband within transportation containers [15].

Since muon tomography brings together particle physics, cosmic ray physics and a diverse conglomerate of scientific fields it serves as a concrete example of how particle physics can leave the constraints of the laboratory, become mobile and address issues that affect our everyday lives in more immediate and palpable ways.

The Cosmic Muon Images project focuses on the analysis of data from an archaeological muon tomography expedition that took place during the summer of 2018 in Greece at the ancient tumulus of Apollonia in Khalkidhiki, which hosts an ancient Macedonian tomb [16]. In the context of REINFORCE, all data acquired were shaped into a Structured Query Language (SQL) database, which acted as a repository for plot creation and which were then uploaded to the Zooniverse platform, in order to allow citizen scientists to inspect and analyse [17].

The goal of the analysis was to identify track-like topologies that had been missed by the algorithm used by the research team. The extreme horizontality and the closeness of the detector to the studied object made the signal-to-noise ratio for this experiment a factor that could potentially be ameliorated through the input of the volunteers.

The track-selection algorithm generally used for volcanology data is based on a series of empirical selection criteria that mostly favour low-multiplicity events, which triggered a low number of detector channels, i.e., fewer than 100 out of the 2,304 possible combinations. This provides a strict sample of muon tracks for the tomography and rejects events with complicated topologies as background, even though a track (or more) might be hidden among these busy topologies.

These events were separated into two samples and were then provided to the volunteers in the form of detector-representation plots, with the possible particle strikes depicted on the detector surfaces. The task at hand was the identification of tracks crossing the detector and extra particle strikes on the three detector planes.

The first sample comprised simpler events, for which our algorithm could provide a candidate track, while the second contained events for which no reconstructed tracks were retrieved. The first sample was classified by the volunteers within the Introductory workflow, which also served as a stepping stone to becoming familiar with the event categorisation procedure. The second, more complicated sample, was incorporated in the FreesStyle workflow [18].

Both workflows ask volunteers to draw a pair of lines, representing a track, passing through the detector channels that showed a charge and mark up extra particle strikes, by pairing detector channels that did not participate in the formation of the line. The difference being that for the Introductory workflow, a pair of lines is already present (representing a particle track) and the volunteers need to decide if these are valid, if they need to be redrawn properly or if they are just an artifact of our algorithm and the event should be categorised as background. The logic for the FreeStyle workflow is the same, with the topologies being more complicated and the volunteers having to point out if it is clearly a background event or if there are tracks that might be identified. For both workflows there is an extra step after the basic task of identifying the tracks, which is the highlighting of extra particle strikes.

2.4.1 Successes: what has worked and why

Towards the end of the project the volunteers had analysed 9,099 events for the Introductory workflow, having performed 48,577 classifications. Of these, 5,765 events were identified as signal and 3,344 events as pure background. This was an interesting finding, as the algorithm used by the research team would not be expected to reach such a high percentage (30%) of false positives. This is the first useful result provided by the inspection of the events, and it needs to be cross-checked with appropriate simulations to see if and why it holds true.

Trying out different ML algorithms from the scikit-learn²² library, the Gradient Boosted Trees algorithm was identified as optimal for signal-to-noise discrimination. The unblinding of the data showed that this method could potentially provide a better muon sample for muon tomography imaging, with 10% less noise than the initial sample. The impact of this selection on the muon tomography result remains to be studied [19].

2.4.2 Improvements; what has not worked and why

The FreeStyle workflow did not yield a similar result. The first cause for this was the complexity of the events combined with the small number of classified data that did not provide the ML algorithm with enough clear-cut information on which to perform. The overall participation of the volunteers for this workflow was comparable to the Introductory workflow, with 46,866 classifications of a dataset of 8,529 subjects, so the failure to train likely relates to the task itself and its origin is more objective than subjective [19].

The second issue originated from the research team side and the design of the workflow. The fact that there were many degrees of freedom through which the volunteers could navigate the classifications led to diverse realisations as to what might be background or not. Some completionists would try to draw all possible tracks they could see, no matter the validity of the physical interpretation, while others would follow a lazier approach and reject events as background that would otherwise prove to have a track after careful examination. The complicated topology of a subject would only make things worse, leading to a weak consensus. The heuristic approach that the research team aspired to was eventually not suitable in the framework of the demonstrator project.

2.4.3 Lessons learned

A key takeaway from the Cosmic Muon Images project is that citizen science projects can provide help even for tasks that at a first glance might seem simple or mundane. That being said, workflows need to be designed with care and attention and an extended period for tests and preliminary analyses of the classifications, in order to catch early on any potential misunderstandings and confusion that might arise when non-experts try to grasp and implement concepts that experts perceive as given and trivial. In retrospect, a gradual involvement of audiences with decreasing levels of scientific literacy would be beneficial. Maybe going from university students to other audiences that can function under guidance, such as amateur astronomers, geologists etc., would be beneficial for the formation of the project, before it reaches its final form and becomes available to the general public.

This proposed strategy springs from the fact that a workflow needs to remain unchanged throughout its lifetime, so that all data are analysed by the volunteers in a uniform way. The Zooniverse staff, being experienced with this aspect, have similar procedures in terms of beta testing by volunteers, but the focus is more the optimisation of the tools rather than the scientific end product and its usefulness. This gradual opening to audiences of different scientific expertise should help with the latter aspect.

Concerning the FreeStyle workflow dataset, it is possible in retrospect to say that it would have been more successful if an unsupervised ML algorithm had initially been run over it, in order to have caught some basic patterns. It would then have been possible to identify which of them made physical sense, in terms of tracks and extra particles. In this manner, it would have been easier to guide and better constrain volunteer classifications. With this method we could have better limited the data sample to only those events that could have a physical interpretation, rather than leaving this task to the volunteers. This strategy could have improved the consensus and would have decreased the number of classifications needed per subject.

The first step to remedy these drawbacks would be to re-design the FreeStyle workflow, providing better guidance and a more constrained path for the classification procedure. It would also be useful to remove events with extreme multiplicities, as, following discussion with the volunteers, it is clear that these can be a factor of confusion.

2.4.4 Interaction with the demonstrator project community

The interaction with the volunteers has proven to be very useful. Since the start of the project, it has been possible to identify two kinds of participant: those interested in the actual categorisation process; and those wanting to discuss the instructions, guides and the other demonstrator materials. Most commonly, people have asked if a specific categorisation that they have done is right. Answering these questions led gradually to the establishment of a database of examples with proper categorisations. The answers from the research team clarified why something would work and how certain choices would have made more sense than others, taking into account the detector operation and the processes involved in the production of the events.

Some volunteers were more inclined to address appearance and form issues relating to the demonstrator project. The research team was provided with a full list of spelling and grammatical mistakes that had been missed during the writing and correcting of the project materials, while others offered ideas on how the event depiction of the workflows could have a better, more intuitive representation. This feedback was incorporated into the field guide section of the project, as implementing changes directly on the workflows after their launch is discouraged.

Interaction with the volunteers also showed that many of them choose to go directly to the workflows and start performing classifications right away. They would come back to the research team for help only when they realised that their intuition alone, without going through our explanations, led to inconsistencies. As the project progressed and more answers for issues accumulated, a gradual decrease of this kind of question was observed.

At the start of the project, the research team aspired to create a small team of volunteers willing to delve deeper into muon tomography, but it quickly became clear that even the most dedicated individuals only wanted to help through the Zooniverse platform and nothing more. Another aspect that we found, was that uploading data in small bunches of 5,000 subjects per upload created some frustration among the volunteers. Most of them wanted the entire dataset to have been uploaded in one go, so that they would be able to have a clear time horizon for the completion of the project, rather than watch the progress bar reach the 100% mark, signifying that the dataset had been completely classified, only for it to go down again. It is difficult to say to what extent this affected the churn of the project, but it is clear that the practice was not received well and should not be repeated. Overall, feedback proved very useful and the communication with the volunteers serves now as a guide, with which to shape future developments of the project.

2.4.5 The future

The demonstrator has reached almost 170,000 classifications over a total of 13,626 subjects for the Introductory workflow and 12,867 subjects for the FreeStyle workflow, adding, respectively, 4,257 and 4,338 classified subjects to the numbers mentioned previously. It is now time to automate the work of volunteers for the Introductory workflow by means of a suitable neural network architecture. In the case that the current number of subjects is not enough for this task, the research team will need to explain the new course of its work and enrich the workflow with more data.

The FreeStyle workflow is, at present, suspended and the data analysis should focus more on the different tendencies of the volunteers towards the classification procedure. It will be necessary to identify those that are pointed in the right direction and to steer the citizen scientists towards them. The research team will continue to adopt a heuristic approach, but this time there will be a focus on narrowing down the outcomes to a degree that would make them useful, as is the case with the Introductory workflow. This should eventually result in a new workflow, with different datasets, tutorials etc.

Finally, an entirely new workflow is being considered. The idea being that this new workflow would provide volunteers with tomography plots for analysis and would focus on pattern recognition for the fast deciphering of objects and formations. Based on the experience gained so far, it is clear to the research team that this would be a much more complex task than the tasks provided so far. It represents an opportunity for the muon tomography community to provide data, offer perspectives and expertise, but also to involve many expert participants to support and evolve the project. The muon tomography community has limited experience with citizen science and a discussion on how to integrate the approach should prove beneficial for the domain as a whole.

The demonstrator project is now also an organic component of the outreach activities carried out by the research team and functions as a useful tool in terms of introducing audiences to the experimental prerequisites of muon tomography, while, at the same time, benefiting data analysis techniques through the categorisation of more data subjects.

3 Sonification activities

The use of sonification in astronomy has existed for years. Researchers have long highlighted the need for multi-modal approaches in teaching and learning environments, presenting principles for inclusive material, based on user-centred and universal design.

Recently [20], following an international sound workshop, held in August, 2021, created a repository of existing software, which collects the results of 98 projects that have been developed since 1962, many of them now discontinued, with a lack of documentation or without evidence of applications in science. Almost 80% of these sonification projects were carried out between 2011 and 2021. In most cases, the sonification mapping of the dataset was defined by its creator and shared as a final product or even with some musicalisation, not clearly devoted to the study of the data, the identification of features or even to research.

Taking into account the increasing examples of sonification in astrophysics, sonoUno is one of the sonification software that translates data from two or more column tables into sound. It is an open-source application, based on a modular design, which allows users to open different datasets, and explore them through visual and auditory display, the last permitting them to adjust visual and sound settings to enhance their perception. The project has been user centred from the beginning, and has been designed following different steps during which the user has always guided improvements, from the analysis of accessibility using the ISO 9241-171:2008 standard [21, 22], to focus group sessions, with people with and without visual disabilities testing the different sonoUno versions [23].

The software emerged with an early first release, at the early stages of the REINFORCE project, and, because of this, it was possible to add multiple new functionalities and developments, including the sonification of diverse sets of data from the demonstrator projects.

3.1 The team working on sonification

The sonification team required specialists and, to this end, a group with people acting on specific modules of the software, working in a cooperative way, was created. This allowed to improve both the desktop version of the software and the web interface, using new resources. The team is multidisciplinary, including astronomers, engineers, software designers, educators, disability specialists, neurologists, and sociologists, both disabled and not.

3.2 User-centred design

From the beginning, sonoUno was designed with the focus on the final user, first with a theoretical framework based on user-testing analysis of other platforms [22] and the ISO 9241-171:2008 standard; then with a focus group using the sonoUno prototype [23]. One of the conclusions of the focus group sessions was that sonoUno allows people with different sensory styles to explore and analyse scientific data presented in tabular format, through a synchronised visual and sound representation.

Following the first release, each update of the software was tested with different users. They tested the software with their own data sets and sent feedback back to the development team by email or through the completion of a form. In general, the users are contacted once a year, and even when not all participants answer, feedback from at least five people is obtained. This methodology of user testing gives visibility to sonoUno even, in some cases driving new use cases, such as: Sensing the Dynamic Universe,²³Taller de sonorización dictado por el IES José de Frugoni Pérez en España^24,25 and The Sounds of BEARS.²⁶

3.3 sonoUno development

Two versions of the software have been developed: a browser-based version, accessible via the web, and a desktop version.

3.3.1 Desktop-based version

The first approach to sonification was developed for a desktop computer; it used the graphic user interface design obtained from the previous software and user-centred analysis. Then, with the first prototype based on MIDI sonification, a focus group session was conducted to test the software’s usability and efficiency. From that analysis, it becomes apparent that user-centred software allows visually impaired people to explore the data, not only as a user, but also as part of the development and improvement community [23]. Some bugs were also detected and fixed in future updates; one of the notable modifications was the change of the sound library to one that allowed for more resolution in the sound.

The current version of sonoUno now contains sonification and visualisation of one-dimensional data sets; math functions; sound-setting configuration and plot-setting functionalities; a bash script that uses data provided by the user and stores the sonification and plot files; and three additional scripts to sonify specific data sets related to the demonstrator projects.

3.3.2 Web-based version

The web-based version of the software was developed in response to user recommendations and in order to ensure wide and global access to the software

The browser version was developed using HTML, CSS and JavaScript and uses the ARIA protocol to ensure communication with screen readers. For the sound synthesis itself, the tone.js²⁷ library was selected. The result was a versatile application for use in the sonification of any data set using a tabular format and that can be used via a web browser on a computer, tablet, or mobile device.

Fig. 8

A Blip type glitch, with neither grid nor axis edges. The blue vertical bar indicates the sonification position

3.4 Sonifying the demonstrator-project data

Different approaches were adopted in order to sonify the datasets of the different demonstrator projects. These are detailed below.

Fig. 9

Example of a Deep Sea Explorers bioacoustics workflow data subject. A click (millisecond-range soundwave) from a sperm whale

Fig. 10

A data subject from the Deep Sea Explorers Bioacoustics workflow, sound-tracked with the code for sonoUno image sound enhancement

• For GWitchHunters and Deep Sea Explorers data, image sonification was used. The openCV²⁸ library for image manipulation was used in conjunction with the sonoUno sound library. This consisted of converting a spectrogram image (Fig. 8,²⁹ Figs. 9, 10) to grayscale and translating to sound the normalised total grey value of each column of data. For the sound parameters, the brightness value corresponded to the highest tone, and the darkest value corresponded to the lowest tone (silence). The other parameters, such as instrument and frequency limits of sound, are fixed at the beginning of the code, but it is intended to add a way to indicate them via command line in the future;

• For New Particle Search at CERN, a different approach was necessary in order to be able to classify different particles using only sound. Upon detecting activity in any of the detector layers (calorimeter or inner detector) a scanning process was initiated within a narrow cone covering the active region. This scanning commenced at the centre of the detector and extended radially outward in three-dimensional space. Exploiting the distinctive interactions of different particles in each detector layer, various sound patterns and audio volumes were utilised to provide comprehensive information about the activity within the specific detector region.

  Each particle is visually represented in two views (an example is shown in (Fig. 11): an electron with a track in the inner detector that points to a cluster in the calorimeter; a converted photon with two very close tracks in the inner detector that point to a cluster in the calorimeter; a muon with a long track that traverses all of the detector layers and which could, although this is not necessarily the case, be pointing to a cluster; a photon with a cluster in the calorimeter, but with no track in the inner detector; or an unknown particle with only a track in the inner detector or another representation that is not covered by any of those described above.

  The audio representation of the above cases progressed simultaneously in both views and was based on the existence (a continuous sound of two seconds) or absence (a silence of two seconds) of the track in the inner detector, the existence (a characteristic sound of one second) or absence (a silence of one second) of the cluster in the calorimeter (the sound volume representing the energy), and the existence of a track beyond the inner detector (in this case, the initial continuous sound length changed from two seconds to four seconds)³⁰;

• In the case of Cosmic Muon Images, the representation with sound was based on the possibility of correlating the deposit of energy through the three layers of the detector (Fig. 12³¹). The sonification was based on the one-dimensional plot, with the top and bottom representations containing 32 channels and the middle one only 16 channels. In this case, 16 piano notes are used, one for each channel in the middle plot, and for the other plots, one note for two close channels. In the sonification process, the note heard corresponds to a deposit of energy in the plot; when more than one channel presents a deposit of energy, a combination of notes is provided.

Fig. 11

A comparison of an event displayed in HYPATIA with its representation in sonoUno

Fig. 12

Views of the same event representing the existence of a muon: 3D view (left), 2D view(right), note the channels mentioned in the text

3.5 Tactile models

Within REINFORCE, the team working on the sonification aspects of the project, also produced a toolbox of tactile models; physical representations of data subjects used in each of the demonstrator projects. These proved useful resources in workshops held during the project.

3.6 Challenges: difficulties and issues faced; how they were overcome

One of the most significant challenges of the work of the sonification team in REINFORCE was understanding how best sonify the individual data subjects, keeping in mind the different data formats and ways in which the visual representations varied. This required extensive collaboration with the individual demonstrator project research teams and numerous iterations in order to be able to arrive at a point that was satisfactory for all involved.

It proved necessary to develop the software several times during this, including introducing specific piano notes to describe the particle tracks, and tick marks, to indicate the beginning and end of a sonification.

3.7 What worked particularly well

Working with the different demonstrator project research teams proved to be a huge benefit to the development of sonoUno overall. The exchange with the different teams and different ways of investigating and exploring the data, made it possible to produce a versatile sonification library applicable to multiple scenarios. In this sense, the work done in REINFORCE particularly emphasises the importance of trans-disciplinary collaborations.

3.7.1 Course on sonification

Previous studies have shown that auditory performance improves when combined with visual stimuli and vice versa [24]. Taking this into consideration, the sonification team used two sensory pathways, sound and vision, with the intention of performing a test of multi-sensory training and with the aim of achieving the familiarisation of participants with this type of technique in signal detection. In order to schedule training that made it possible to presenting visual and auditory signals, allowing for the identification of patterns and the saving of this user interaction, the PsychoPy software was used [25]. This software was designed for the creation of experiments in behavioural sciences (such as psychology, neuroscience, and linguistics, among others) and allows precise spatial control and synchronisation with different stimuli.

Over two sessions a complete introduction to multi-modal perception (visual, auditory and tactile) as well as specific training in sonification, was performed. The course was designed so that the complexity of the actions required on the part of the participants grew over the two sessions. Each session was made up of three blocks, one for each type of data: (1) three types of glitches; (2) two LHC events; and (3) four muography events (two with the presence of a muon and two without). The first part of the training session was devoted to understanding the data and knowing how to classify it, presenting examples of each type. In the second part of the activity, different events were presented for each type of, randomly selected, data.

Following the activity, participants were asked to complete a survey about the web version of sonoUno and the training course. Responses in direct relation to the web interface were overwhelmingly positive, with 77% of participants feeling that the software could be used to improve their own work, research, professional and educational activities, while responses in relation to the complexity of the sonifications of the demonstrator project data proved what were expecting; i.e., that the GWitchHunters glitch sonifications and those relating to the Cosmic Muon Images project were relatively easy to interpret, while those relating to the New Particle Search at CERN demonstrator project were less straightforward, owing to their complexity and depth.

3.8 Plans for the future

In the framework of the REINFORCE project, progress was made towards the achievement of two objectives. The software was made more versatile in terms of the handling of sound effects, going beyond a single time series; while work also progressed in the provision of sound as a service. This novel idea of embedding the service in a web browser was investigated using a Web Server Gateway Interface (WSGI) application that acts as a sonoUno server using WebAssembly³² and would maximise code reuse between the actual sonoUno server, used by researchers for computationally intensive or batch jobs, local browsers for the computers of citizen scientists and users of Zooniverse. A prototype, using Flask,³³ looks very promising, and is being explored.

4 Citizen-engagement strategy

In REINFORCE, citizen science acts as the vehicle that aims to bridge the gap between large research infrastructures in physics and society, providing the framework and tools for effective and sustainable interaction between them. On one hand, citizens are trained in frontier science, they are in constant connection with researchers through dedicated communities of practice, they provide their feedback, they voice their concerns, and they actively contribute to the exploration of the boundaries of knowledge. On the other hand, researchers can receive help and support to refine their instruments and advance their research. This interplay leads to several questions, such as: how do we design a successful citizen science project that balances social inclusion and scientific efficiency [26]? What motivates citizens overall in an online citizen science project and how can we sustain this motivation over time [27, 28]? Who are the potential citizen scientists [29] and are they actually able to develop new knowledge at the frontiers of physics [30].

To address these questions, a citizen-engagement strategy was designed and implemented throughout the duration of the project implementation [31], with prime focus on balancing inclusion and scientific efficiency. Large-scale events and inclusive design are very important in terms of raising awareness of large numbers of citizens, offering everyone the opportunity to contribute to a citizen-science project [32]. According to Spiers et al. [29], a typical Zooniverse project has a classification curve that displays a peak of activity after launch, when then rapidly declines, a fact indicating that the majority of users contribute for a short time before moving to another project that captures their attention. Meanwhile, the advanced requirements in terms of content knowledge for the REINFORCE demonstrator projects and the large number of images for classification, combined with the retirement limits set, as well as the need for careful classification of every image in order to achieve a scientifically sound result, posed a series of requirements for the recruitment, continuous engagement and training of participants, which was estimated to be better facilitated by forming a core group of citizens working on the demonstrator projects.

The strategy formulated was based on: an in-depth review of literature of existing citizen-science activities and public-engagement frameworks, addressing citizens’ documented needs, motivations, characteristics, and interest in participating in citizen-science projects in general; a dedicated citizen survey to support the collection of data, together with a series of vision-building workshops and focus groups with small groups of citizens to identify key motivations of specific target groups (teachers, elderly citizens); a task analysis of the four REINFORCE demonstrator projects to provide a careful consideration of the kinds of projects offered, the scientific goals to be achieved, how to coordinate contributions, and arrange tasks within each demonstrator project.

To this end, the engagement team aimed to recruit volunteer citizens to participate in the project demonstrators, and also sought to minimise the effects of losing momentum following the initial peak in participation, and to even increase and maintain a higher participation over a longer period of time. More specifically, the strategy described the progressive approach for engaging citizens in the demonstrator projects, while, at the same time, describing the tasks and the expected contributions from citizens and the roles of the scientists in the process. A five-step approach is laid out, which involves enrichment of the demonstrator projects with dedicated educational resources and training materials and encompasses activities aiming to inform citizens, to involve them, to facilitate collaboration, to receive consultation and to empower them to become ambassadors of the projects in their communities.

Fig. 13

Details of the REINFORCE citizen-engagement strategy

Figure 13 details the engagement activities, citizens’ expected contributions and the researcher’s role in each step of the engagement strategy. The engagement strategy was translated into a series of engagement activities that were implemented throughout the duration of the project, both before the launch of the demonstrators and throughout their piloting period.

5 Participatory-engagement activities

The REINFORCE engagement activities focused both on broad-reaching activities that increased the project’s visibility and supported the continuous influx of new interested participants; and on dedicated activities that engaged citizens to enhance their contribution to the building of a core community of practice around a project that provides meaningful scientific output. It was estimated that reaching out to at least 100,000 citizens would be an achievable and realistic performance indicator in terms of achieving the project science and inclusion objectives. The ambitious goal of reaching out to this number of citizens was hindered by the emergence of the Covid-19 pandemic and the consequent long periods of quarantine in Europe. This situation posed a great challenge, since traditional forms of impactful engagement activities (such as public visits to large research infrastructures) could not be implemented for most of the lifetime of the project. To this end, the project engagement activities were shifted to online and hybrid formats (for example moving from face-to-face visits to virtual visits), while new types of activities were tested for the first time with great success. Different audiences require different engagement methodologies. To this end, the REINFORCE consortium worked with varying target groups, including: teachers, students, multi-modally-impaired citizens and senior citizens. Two particular cases are expanded upon in detail here: using art as a vehicle to engage students in complex topics that go beyond the standard curriculum (Sect. 6.1); and science for senior citizens (Sect. 7). Throughout the duration of REINFORCE, a total of 96 events, online and face-to-face, were organised, all following the project’s engagement framework as described in Fig. 13. 47% of the events addressed the general public and specific citizen groups; 25% were focused on reaching out to academics and researchers; 17% to teachers through dedicated workshops; and 11% to students through dedicated implementation activities. Aggregating the citizens reached through events, as well as through the online awareness activities for the project (such as Zooniverse newsletters), more than 500,000 citizens were informed, while 30,000 citizens participated in events aiming to inform them and provide training. 21,400 citizens collaborated with the project team and contributed classifications. The project team received consultation from 1,400 citizens, while it is estimated that more than 1,000 citizens were empowered to become project ambassadors.

Fig. 14

Engagement activities in REINFORCE. Top left: Informing citizens through face-to-face visits at the Virgo detector; Top middle/right: Involving citizens through hands-on activities; Bottom left: Inviting citizens to collaborate in dedicated challenges; Bottom middle: Receiving consultancy from citizens through dedicated meetings; Bottom right: Empowering citizens to become REINFORCE ambassadors through the intensive International Training Course

5.1 Impact of engagement activities and community building

To assess the impact of different types of events in terms of citizen participation and contribution, citizen engagement was systematically monitored throughout the project implementation using the Zooniverse portal and Google analytics. Quantitative measures were applied to track the quantity of citizen contributions and type of engagement over time. This made it possible to build an understanding of the dynamics of engagement in each of the REINFORCE demonstrator projects. The overall results are presented in Table 1.

Table 1 Metrics and results for citizen collaboration with the REINFORCE demonstrator projects

According to analysis, citizen scientists devoted double the effort (calculated based on the time allocated by the user logs) invested by the research team. This is evident from the time dedicated by volunteers compared to the total live-time of the demonstrator projects. This is a unique outcome of the work performed in the framework of the project, as it demonstrates that the contribution of citizen scientists could really be quite significant. Figure 15 displays citizen classifications over time during the piloting period of the project, which began on the 19th of October, 2021, and ended on the 25th of October, 2022.

Fig. 15

Daily Classifications versus time overall (black) and per demonstrator project (blue: GWitchHunters, orange: Deep Sea Explorers, red: New Particle Search at CERN, green: Cosmic Muon Images) during the piloting period. The peaks that correlate with the organisation of specific events have been labelled accordingly

By organising similar events for the different demonstrator projects, the project team had the chance to acquire useful data in terms of the user-friendliness of the approaches and the interfaces used and the complexity of the tasks assigned to citizens. It was observed that the period with the most engagement activities corresponds to a higher classification rate, as well as a considerably higher number of participants. The engagement of the citizens over time and the overall demonstrator-project appeal were periodically assessed, allowing the engagement team to providing the individual demonstrator project research teams with triggers as to when to act and ensure their project remained interesting and motivating for the volunteers.

Awareness campaigns and dissemination activities, such as the events and newsletters flanking the official launches of the demonstrator projects on Zooniverse, appear to have had a high impact in terms of daily classifications, with a low number of classifications per participant and a high drop-out rate. According to the data, online events with massive participation, such as dedicated challenges and competitions, provided 41% of the total classifications of the demonstrators, with the majority of these provided by a core team of dedicated participants. As examples, the REINFORCE Winter Challenge³⁴ acted as a social trigger that engaged 720 citizens to perform a total of 80,000 classifications, while the EPS Citizen Science Competition³⁵ engaged 374 citizens, who registered and performed a total of 201,000 classifications. The sustained participation of citizens in these events displays a high dependence on these challenges, as well as the related prizes.

It is possible to argue that the engagement activities managed to balance inclusion and scientific efficiency by reaching out to a large audience through dedicated large-scale events, while forming a core community of participants who contributed the bulk of the project classifications. This community-building was made possible by the commitment and accessibility of the researchers operating the demonstrator projects. The Zooniverse Talk forum was the place where day-to-day interactions between researchers and volunteers took place. For the research teams, this included answering questions and discussing the work of the demonstrator project, as well as more general topics, announcing events, recognising citizen participation, empowering citizens and announcing project results. Beyond forum interactions, community building was further facilitated through meetings (both face-to-face and online) which helped citizens and researchers to also put a face to the person on the other side of their monitor; the person they had been collaborating with for a long time.

5.2 Highlights of individual demonstrator-project characteristics

The effectiveness of the engagement activities heavily depended upon the respective characteristics of each demonstrator project. Table 2 provides an overview of individual demonstrator-project metrics for the piloting period.

Table 2 Overview of the engagement metrics for the REINFORCE demonstrator projects. Data correspond to the period starting on the 19th of October, 2021, and ending on the 25th of October, 2022

The differences between the four demonstrator projects stem from the convolution of different contributing factors, including the nature and difficulty of the tasks (from pattern recognition of noise features to track reconstruction), the respective characteristics of the interfaces, the live-time of each demonstrator, the community support, the knowledge background required, the popularity of the respective science topic, the support from other dissemination channels, etc. As an example, the easy-to-use game-based interface of the GWitchHunters demonstrator, as well as its launch in a mobile-friendly version, have contributed significantly to the success of the demonstrator project. A major contributing factor was also the support of a large research infrastructure, which increased its dissemination activities in relation to the project a lot. With a median classification time of seven seconds, the effort per classification can also be argued to be lower than that of the other demonstrators. These facts contribute to the appealing nature of the topic of this demonstrator. According to Spiers et al. [29], “All Zooniverse projects display unequal volunteer classification contribution”. This characteristic is also present in the four REINFORCE demonstrator projects. In order to identify the inequality in terms of citizen contributions, the Gini coefficient for each project was calculated. The Gini coefficients presented in Table 2 [0.84$-$0.89] indicate an unequal distribution of classifications among the participants of each demonstrator project, with a maximum variation of the order of 5% between demonstrators. Based on the results provided by Spiers et al. [29], the Gini coefficients reported for the REINFORCE demonstrators are within the range of coefficients reported for the bulk of Zooniverse astronomy-oriented projects. A further comparison of REINFORCE with other existing relevant Zooniverse projects is currently a research in progress.

As an epilogue, for one to reverse or delay the rapid fall of interest in the demonstrator and all online citizen-science projects in general, the engagement model tested in REINFORCE has great added value, a fact that is shown in the shapes of classification distributions over time and effectively in the classification output of the project. which ultimately resulted in more data for the demonstrator project research teams. Furthermore, organisation of such engagement activities goes hand in hand with community-building activities, which require the commitment of researchers and citizens and affects the overall sustainability of citizen-science projects over time.

6 Art & Science

As Root-Bernstein et al. [33] remind us, more than fifty years ago, the Nobel Prize winner van’t Hoff [34] proposed a correlation between creative activities outside of science and scientific imagination. Other studies have shown that a positive association exists between creative activities and success in science [35]. More generally, adopting art practice in elementary schools, has proven to enhance young children’s learning in science [36]. Indeed, by using art in science, children are stimulated to explore scientific facts and concepts in a multi-dimensional way, referring not only to inquiry and analysis, but also to imagination, emotions, and creativity [37, 38].

6.1 International Youth Art & Science competition

Starting from these assumptions, the REINFORCE project launched an International Youth Art & Science competition under the theme Humans as observer and listener of the cosmos. The focus of the contest was on concepts of fundamental physics, such as space, time and the nature and structure of the Universe. These same concepts underlay a cross-reflection between the artists and the scientists involved in the project.

The aim of the contest was to stimulate the participation of young people in the re-questioning of the fundamental concepts of physics that are mentioned above, by asking them to translate into artistic manifestations their understanding of these notions. To this end, entries in the REINFORCE International Youth Art & Science contest were grouped into two different age categories, from four-to-twelve years old and from twelve to eighteen years old. The individual calls for submission were designed accordingly, in order to capture the attention of potential participants and stimulate their interests in the underlying science. More precisely, for young children the project sought artwork representing humans observing space; listening to the sound of stars; discovering the nature of space and time; and the structure of the Universe. On the other hand, participants in the 12–18 range were asked to represent humanity observing space and more generally nature, through a multitude of senses, from vision to sound and touch; discovering the embedding of REINFORCE research in the Universe, including the environment; wondering about the nature of space, time and matter. Around twenty artworks were submitted by young artists aged five to seventeen. These were contributed from three different continents (Europe, Asia and America) and six different countries (Italy, Greece, Bahrain, Ecuador, Argentina and North Macedonia), with a prevalence of female artists (60%). All participants were awarded a certificate of participation and invited to attend a remote or in-person tour of the Virgo interferometer, located at the European Gravitational Observatory.

Although the number of entries was not high (mainly because the contest was launched during the summer recess period, when it becomes much more difficult to engage schools, teachers and students), the contest afforded the children the opportunity to explore and learn about fundamental physics, the Virgo experiment and the project itself. For the youngest artists, the contest was also the first time they were confronted with new concepts, such as the sound of the stars and the nature of space and time. The REINFORCE project, on the other hand, built on this initiative to involve young citizens in science and organised a visit to Virgo, as well as a Kid’s Lab on Messengers from Space,³⁶ during the European Researcher’s Night events of 2022.

6.2 Art & Science events organised within REINFORCE

The REINFORCE project has shown very clearly how the tools of citizen science can involve anyone in fundamental physics research, something that is so distant from people’s daily lives: from exploring the fundamental constituents of matter with particle accelerators to listening to the most violent cosmic phenomena through gravitational waves, to the study of neutrinos and cosmic rays. This is research that almost always generates applications that bring us back to the context in which our lives take place, through the development of innovative technologies or monitoring the natural or human-made environment of experiments.

However, there is another aspect that establishes an important link between fundamental physics research and society: its profound influence on thought, art and culture in general. The new visions of the world and the cosmos, often revolutionary and counter-intuitive, imagined and then verified with experiments by physicists, answer ancestral questions and humanity’s irrepressible aspiration to investigate the origin and reflect on its place in the cosmos. Because of this, they inspire new artistic and philosophical visions, generate stories that enter the common imagination at all levels, from cinema to comics, from science fiction to interactive and artistic installations.

Research infrastructures operating at the frontier of our knowledge can be among the leading actors in this process, developing collaborations with visual artists and musicians, stimulating the production of installations for the general public and organising public events, theatrical performances, conference-shows, in which scientists and artists can dialogue directly with the public, using different languages. These experiences have a dual significance: on the one hand, they make knowledge and ideas that are often not yet widely disseminated, accessible to the general public; on the other, they are a true laboratory for the development of new languages and syntheses between science and imagination, knowledge and art.

In this sense, the dimension of game and entertainment also plays a crucial role in communicating effectively and bringing the public closer to subjects with which they are not familiar. Recently, it has been emphasised in several contexts how an intertwining of education and basic research, with innovative languages, is increasingly urgent, not least to counter the negative and persistent effects of the pandemic on the education of the youngest [39].

Finally, these actions related to public communication and cultural imagery are complementary, supporting and reinforcing the motivation of citizens directly engaged in citizen-science initiatives. There is a virtuous circle between public communication and citizen-science actions, which evidently reinforce each other.

In this context and with this inspiring vision, EGO, in collaboration with partners, developed, within the framework of the REINFORCE project, a series of initiatives and events aimed at the general public and, in some cases, particularly at the youngest.

The events mainly explored topics related to gravitational and multi-messenger astronomy, i.e. listening to the cosmos via gravitational waves. The theme of cosmic sounds was especially explored and developed, both as a metaphor and narrative suggestion, and as an actual object of musical research, also through the sonification and elaboration of scientific data, in collaboration with musicians and sound designers.

Several public events were organised in this direction, also taking advantage of the intensive research on the sonification of data undertaken within the REINFORCE project. Below is a brief description of a selection of some of the art and science events and initiatives organised within REINFORCE.

• Black holes and gravitational waves in the cosmic concert - Concert/Installation - Online and in Rome (31st of December, 2020) Black holes and gravitational waves in a cosmic concert by Tomàs Saraceno, organised as part of the Festa di Roma, the New Year’s Eve celebration in Rome. The event involved a public concert and installation by Tomas Saraceno, along with a discussion involving the artist, the curators and scientists.³⁷

• E quindi uscimmo a riveder le stelle - EGO, Cascina (Italy) (1st of October, 2021) A special evening under the stars of Virgo to discover Dante’s cosmos and how we listen to and narrate the Universe today, in the unique setting of the European Gravitational Observatory. Accompanying Lina Bolzoni, historian of Italian literature at the Scuola Normale Superiore, Stavros Katsanevas, physicist and director of the European Gravitational Observatory, and Riccardo Pratesi, professor of mathematics and scholar of the Divine Comedy, were the canti cantati of the ensemble A Ricuccata and the irreverent humour of David Riondino.³⁸

• Lo Spazio e i Sensi - Pisa (Italy) (24th of September, 2021) Event held at La Nunziatina, Pisa. A night of science and music on the many ways we can explore and listen to the cosmos, as well as the world around us, in the name of diversity and inclusion. The guests were two exceptional protagonists of scientific research: Marica Branchesi, internationally renowned for her contribution to the birth of so-called multi-messenger astronomy, and Wanda Diaz-Merced, a blind astronomer of Puerto Rican origin, one of the global leaders in research into the sonification of astronomical signals. Stavros Katsanevas spoke with them, while the conversation between space, sounds and gravitational waves was accompanied by music from the Italian-Greek singer Marina Mulopulos.³⁹

• Il suono dell’Universo, Lajatico (Italy) (19th of July, 2022) The Sound of the Universe - Dialogue under the stars between gravitational waves and cosmic sounds. During the event, EGO director Stavros Katsanevas and Wanda Diaz Merced, world leader in the sonification of astronomical data, talked about data sonification, what it means to listen to the cosmos and how this is done, accompanied by the music of sound designer Massimo Magrini. Magrini’s contribution was developed within the framework of REINFORCE.⁴⁰

• Il suono dell’Universo, Genova (Italy) (30th of October, 2022 Within the context of the Genoa Festival of Science, The Sound of the Universe took place: a dialogue on what it means to listen to the cosmos with Wanda Diaz-Merced and Stavros Katsanevas, and the musical contribution of sound designer Massimo Magrini.⁴¹

• Athens Science Festival, Athens (Greece) (22nd and 23rd of October, 2022 Over 600 children, students and people of all ages discovered Virgo and the science of gravitational waves through many interactive activities organized by Ellinogermaniki Agogi. A room dedicated to gravitational waves was available, including an art and science installation developed by the Italian Institute of Nuclear Physics (INFN) and a model interferometer. Visitors were able to find many work-stations at which they could experiment with the GWitchHunters Zooniverse demonstrator project.⁴²

7 Senior-citizen science

Another initiative implemented in REINFORCE designed and developed with the intention of increasing accessibility to the work of the demonstrator project research teams was the Senior Citizen Science course. The course was co-designed with members of the Università della libera età (University of the free age) group, based in Cascina, in the vicinity of the European Gravitational Observatory (EGO). Sessions in the course were provided by members of the REINFORCE Collaboration based at the University of Pisa and at EGO.

Running a course for senior citizens in the middle of a global pandemic was inevitably more complicated than it might otherwise have been and required a certain logistic elasticity in terms of the calendar, which had to be shuffled and re-shuffled, as well as in terms of participation numbers, with the number of people available to participate from one session to the next being relatively fluid. Despite this, the implementation ultimately proved a success and even resulted in a subsequent edition for the 2022/23 academic year, i.e. well beyond the lifetime of the REINFORCE project.

7.1 The course syllabus

The first edition of the course took place during the 2021/22 academic year and was formed of nine sessions, plus a concluding visit to the Virgo interferometer, situated at EGO. The course took the following form:

• Classical particle physics was given by Giancarlo Cella, Researcher at the Italian National Institute for Nuclear Physics (INFN), and Professor of Astroparticle Physics at Pisa University, on the 6th of October, 2021, and covered the concept of space and time in Galileo, Descartes and Newton; particle trajectory, mass, speed, acceleration; and the Laws of Newton and the movement of the planets;

• Particles & waves in the XX century was delivered by Massimiliano Razzano, Professor of Experimental Physics at the University of Pisa, on the 8th of October, 2021, and covered: the scale and structure of the Universe; sub-nuclear structures; Newton, Maxwell, Faraday, Hertz; electromagnetic waves; Einstein and Relativity; Planck, Schrödinger and the quantum revolution; Rutherford; linear and circular accelerators and the Standard Model;

• Waves: concept and detection took place on the 10th of November, 2021, and was given by Stefano Rinaldi, PhD at the University of Pisa, and covered: the concepts of wave in Huygens, amplitude, frequency, interference; electromagnetic theory and waves, aether; their detection with the Michelson-Morley interferometer;

• The cosmology of the (in)visible universe took place on the 3^rd of December, 2021, and was delivered by Stavros Katsanevas, REINFORCE Coordinator and Professor of Physics (Exceptional Class) at the Université de Paris. The session ranged across a broad spectrum of themes, covering: Newton and Maxwell; Einstein’s Theory of Gravity; quantum physics — Bohr, Schrödinger, Dirac, Heisenberg, Feynman; and the Standard Model of subatomic particles; astroparticle physics; cosmic rays; Hubble; gravitational lensing; redshift; neutrinos; quarks, gluons, protons, neutrons; Higgs theory and supersymmetry; ‘shadow’ particles; matter and antimatter; nucleosynthesis; and gravitational waves.

• The fifth session, Citizen Science: from theory to practice, was delivered by Francesco Di Renzo, postdoctoral researcher at the University of Pisa, on the 9th of March, 2022, and focussed on the work of the GWitchHunters demonstrator project;

• The General relativity session was delivered by Valerio Boschi, Researcher at the INFN, on the 12th of January, 2022, and covered: spacetime, gravitational waves and their detection, LIGO and Virgo; sources of noise, analyses of waves and glitches;

• On the 29th of April, 2022, Gary Hemming, REINFORCE Technical Manager, of EGO, held a session under the heading Brainstorming and resolution of technical and theoretical problems, with the aim of evaluating directly with course participants, the ways in which the course had worked well up to that point, as well as to try to establish how and where it could be modified to ensure it reached its full potential;

• The sonification of gravitational waves session was delivered by Francesco Di Renzo on the 6th of May, 2022, and covered: signals and what they are; a particular example of a gravitational-wave signal: GW150914; signals as vehicles of information; the characteristics of a sound; pentagrams and time-frequency representations; listening to gravitational-wave signals; handling noise; and the voice of a glitch;

• The final formal session in the course, Art & Science was again delivered by Stavros Katasanevas and focussed on: space-time-matter; the senses and beyond: the two infinities; the notion of cosmos, order and violence, the singularities; from multi-messenger to multi-sensorial; and from Uranus to Gaia.

The course concluded with a final visit of the participants to the Virgo detector at EGO on the 18th of May, 2022 (Fig. 16). The tour included a view of the interferometer from the top of the North Arm Technical Building; a view of the Main Experimental Hall from the atrium of the Central Building of the interferometer; the Virgo Control Room; the Hall of the Main Building, which is home to a miniature model of a Michelson-Morley laser interferometer, an original prototype of the Superattenuator suspension system, used to reduce seismic isolation at the level of sensitive optical components in Virgo and a mock-up of one of the mirrors used in the detector; entering into the North Arm tunnel of the detector in order to see the interferometer vacuum tube at close quarters.

Fig. 16

Senior Citizen Science course participants visiting the Virgo gravitational-wave interferometer, based at the European Gravitational Observatory

7.2 What needed to be worked on?

An analysis of the evaluation session that was held with course participants, combined with feedback from session providers, brought to light some interesting outcomes. Participants were genuinely interested in the subject matter approached, although finding the right way to pitch it to them required some calibration. It became clear that a classical lecture-based environment was not the best approach in terms of ensuring the active participation of the group, which favoured a more relaxed, informal back-and-forth. This way of delivering sessions was subsequently adopted, with significant success.

It was also clear from the evaluation session that some members of the group were concerned that the level of the course had aimed too high. Again, this feedback proved useful in recalibrating the sessions slightly, in order to try and ensure that they remained as accessible as possible to as wider number of participants as possible.

7.3 What worked well?

Certainly, the more informal approach that was adopted as the course evolved proved a real success, building enthusiasm and motivation within the group and resulting in an enjoyable visit to EGO, followed by a request to continue the course into the subsequent academic year. One fascinating discovery from the feedback sessions, however, related to the session contents and their degree of accessibility. The The Cosmology of the Visible Universe session, delivered by Stavros Katsanevas, was the only session of the entire course that was delivered in English. Despite the fact that all of the group were mother-tongue Italian speakers, with a self-confessed limited knowledge of English, the feedback from the group showed that this session was the one that they had found most interesting and inspiring. Accompanied by a useful presentation, containing many images for reference, the group were clearly inspired by the session and felt that they had been able to put together the pieces, picking up enough information from Stavros, together with the slides he presented, to build a picture that was enthusiastically received.

Above all, the course proved the importance of creating an environment in which participants feel comfortable, can ask questions and not have to worry about getting things wrong.

8 Critical and Scientific Thinking course

The REINFORCE Course on Critical and Scientific Thinking took place over the course of five different days between the 24th of October and the 2^nd of November, 2022. On the first day, Nobel Prize laureate Professor Saul Perlmutter, professor at the University of California, Berkeley, Lawrence Berkeley National Laboratory, delivered a session to an audience connected remotely, via Zoom. The second and third days were delivered, also via Zoom, by Stefano Gattei, Professor of Science in Society, at the University of Trento, Italy. The fourth and fifth days were again delivered by Prof. Gattei, but were in-person at the European Gravitational Observatory, with participants following both on-site and from remote.

Following the delivery of the course, the sessions were divided into 18 separate sessions, which have been made available on the REINFORCE YouTube channel.⁴³ The individual sessions are also available in podcast format, via a dedicated playlist.⁴⁴

Fig. 17

The REINFORCE Critical and Scientific Thinking podcast playlist on YouTube

These sessions were broken down into the following themes, with the first three dedicated directly to the concepts at the core of critical thinking and the remainder providing a detailed history of the evolution of scientific thinking following the Second Scientific Revolution.

• 01 - A Missing Ingredient in Science Education – collective critical thinking and the ways in which crises of scientific knowledge are also crises of decision-making;

• 02 - The Four Rs – the ‘Can-Do’ aspects of science, such as Scientific Optimism, as well as the ideas of Confirmation Bias and Blind Analysis; and Group Thinking; should the Three Rs evolve into the Four Rs: Reading, Writing, Arithmetic and Scientific Reasoning;

• 03 - A Critical Thinking Q &A;

• 04 - What is this thing called science? – the so-called Second Scientific Revolution in a historical context, the key figures involved and its foundations;

• 05 - How to provide science with solid foundations...at the origins of critical thinking (Part 1) – how the Second Scientific Revolution intertwined with a revolution in the philosophy of science;

• 06 - How to provide science with solid foundations...at the origins of critical thinking (Part 2) – Wittgenstein, Logical Positivism and the life of the Vienna Circle;

• 07 - Verification versus Falsification – Popper and the publication of The Logic of Scientific Discovery; Logical Asymmetry and metaphysics;

• 08 - Science as Falsification – Popper’s criticisms of Logical Positivism; the theories of Einstein, Marx and Adler;

• 09 - The Test of Experience – Popper’s approaches to induction and deduction;

• 10 - Falsifiability as a standard of evaluation of scientific theories – Popper’s work and the ideas of conjecture and refutation;

• 11 - Dogmatism versus Infinite Regress versus Pyschologism - Fries’s Trilemma;

• 12 - The Game of Science – Rules and the Supreme Rule;

• 13 - The New Philosophy of Science - The New Philosophy of Science; Kuhn; Feyerabend; Toulmin, Hanson and Polanyi;

• 14 - Divergent versus Convergent Thinking in Science – convergent and divergent thinking, scientific revolutions and normal research, in the works of Thomas Kuhn;

• 15 - Normal Science as Puzzle Solving – Kuhn, Pre-Science and Normal Science, Paradigms and Puzzles, Ptolemy and Kepler;

• 16 - Normal Science versus Extraordinary Science – Puzzles and Anomalies and how crises in science cause scientific revolutions and new Paradigms;

• 17 - Incommensurability in Science – do scientific revolutions lead progress and how can this be measured?;

• 18 - Truth and Scientific Progress – the evolution of the Paradigm into the Disciplinary Matrix.

9 Raising awareness and sustainability

Across the lifespan of the project, REINFORCE adopted a wide spectrum of communication, outreach and dissemination activities, with a view to building up and nurturing a thriving community of engaged members and to ultimately pursue the primary objective of minimising the knowledge gap between large research infrastructures and society through citizen science.

The execution of a wide-ranging communication plan, inclusive of a series of joint actions, allowed to achieve an effective overall visibility of the project across the major stakeholders, summarised as follows:

• Potential citizen scientists;

• The scientific and research community;

• The citizen-science community;

• Policymakers;

• The academic/innovation community.

The overarching key goal of the REINFORCE dissemination activities was to meaningfully reach out to each of these groups, to become engaged and to take part in the exchange of good practices, to liaise with other initiatives in the field of citizen science and to promote the overall concept that the research community has to move beyond traditional outreach programmes and learn to design effective citizen-science activities that benefit their research and can ensure citizen involvement.

9.1 Social media

Social-media channels represented the centerpiece of the projects’ dissemination plan; the aim being to efficiently provide brief real-time updates and news, but also to be used as a tool to share information about citizen science on the whole, inform about the projects’ outcomes, to reach scientific and research community players and promote critical events’ activities and deliver real-time coverage of virtual events wherever the project was involved.

9.2 Webinar series

The webinar series proved to be one of the most effective and tailored dissemination activities and allowed the project to attract the interest of interested communities and participants. A set of five specific webinars was carried out and broadcast to support the most tangible results of each demonstrator project and to spark interest around the correlated Zooniverse theme: this brought a considerable stream of community members to Zooniverse and supported the development of each demonstrator project.

The webinar series covered the following themes:

• (on the 1^st of June, 2020) Bridging the gap between Science and Society through Citizen Science⁴⁵ was the opening webinar in the series and provided a general overview of the REINFORCE project, with interventions from members of the research teams of the individual demonstrator projects, as well as an explanation of the ways in which the engagement strategy would be implemented;

• (on the 16th of October, 2020) How to help scientists in the Gravitational Wave noise hunt⁴⁶ provided an introduction to Zooniverse and concentrated on the GWitchHunters demonstrator project and the work of the Virgo Collaboration;

• (on the 26th of February, 2021) Discovering the unexplored deep marine environment⁴⁷ focussed on the work of the Deep Sea Explorers demonstrator project, presenting the work of the KM3NeT collaboration along with interventions on the themes of bioluminescence, bioacoustics and marine ecology;

• (on the 11th of May, 2021) How to use cosmic rays in the study of geosciences and archaeology⁴⁸ provided an introduction to the Cosmic Muon Images demonstrator project, along with an overview of multiple different muographical applications;

• (on the 7th of July, 2021) Identifying undiscovered particles at the Large Hadron Collider⁴⁹ presented the New Particle Search at CERN demonstrator project and an introduction to the work of the ATLAS collaboration;

• (on the 24th of March, 2022) Opening Research Infrastructures – How citizens can play an active role in the advance of ground-breaking research?⁵⁰ was a high-level round-table and interactive workshop aimed at policymakers, education-policy experts, think-tankers and academics, and had the aim of contributing to the co-creation process for a policy roadmap on research infrastructures for citizen science in Europe;

• (on the 21^st of November, 2022) REINFORCE-ing citizen science in large research infrastructures – Delivery of a policy roadmap⁵¹ concluded the series of webinars, bringing together the members of the collaboration to present the policy roadmap and reflect on the legacy of the project and explore the future.

10 Evaluation activities

10.1 Overview of methodology

The evaluation framework of REINFORCE is based on the logic model developed by Kurz and Kubek [40] and presents the methodology regarding the evaluation of the project processes, outputs, outcomes and impact. The logic model provides a systematic overview of logical relationships between a project’s resources, activities and results and thus constitutes an excellent framework for measuring the outcomes and impacts of a project. The distinction between these levels is important in preserving the direction of the project and thus the impact orientation. In this paper the focus is on the outcomes on the side of the participants, as well as on the wider impacts.

To operationalise the evaluative dimensions, different evaluation instruments were developed: i) The Zooniverse feedback survey to measure participant satisfaction; ii) the pre- and post-questionnaires, for participants to understand the individual outcomes in terms of gained knowledge, scientific skills, and change of attitudes towards science; and iii) a self-assessment questionnaire, based on the work by Kieslinger et al. [41], for the research teams implementing the demonstrators to discuss processes & feasibility as well as outcomes & impact on the scientific, citizen science, and socio-ecological dimensions (Fig. 18).

Fig. 18

The citizen-science evaluation framework used within the REINFORCE project

10.2 Data analysis and results

10.2.1 Participant satisfaction

In order to evaluate the participant experience on Zooniverse, a short online survey was developed, to be completed by the participants. The survey consists of seven questions and basic demographic data (gender and age) and has been analysed (n=168).

On a scale from 1 (=strongly disagree) to 5 (=strongly agree), averages range from approximately 3 to above 4, indicating a general moderate-to-high level of user satisfaction. When compared to one another, it is item 3 (Fig. 19), I find the project tasks difficult, which has the lowest score. Considering the nature of the item, this is a satisfactory response. An average difficulty is ideal as according to motivation theory [42], tasks should be neither too difficult nor too easy, in order to motivate people to continue with them. Those with high motivation tend to choose moderately difficult tasks, feeling that they are challenging, but within reach [43].

Fig. 19

Average scores for participant satisfaction

10.2.2 Individual impact

The online pre and post survey is meant to detect new knowledge, new skills and changes of attitude, among participants in the four Zooniverse demonstrator projects, with questions related to knowledge on the role of research infrastructures, basic physics concepts, data recognition and analysis skills, awareness of science and scientific work. It also includes the assessment of change actions and behaviour: e.g. science career motivation; and living conditions: e.g. empowerment through contributing to citizen science. In addition, the questionnaire contains some basic demographic information, such as age, gender, highest educational level and a question regarding a background in science.

In the pre-post-test design as applied in our study, the pre-test aims to measure the baseline, in this case in three distinct areas: knowledge, skills and attitudes. The following table shows the average scores on a five-point Likert scale (n = 301) from the pre-test, i.e. when first logging into the Zooniverse project and post-test, completed after one month. As the paired sample t-test values reveal, three differences are significant (Fig. 20).

In a paired sample t-test, the T-value and df (degrees of freedom) are two important statistical values that help in the interpretation of the results of the test:

• The T-value, also known as the T-statistic, measures the size of the difference relative to the variation in the sample data. It is calculated by taking the difference between the mean of the paired samples and dividing it by the standard error of the differences. Generally, the larger the T-value, the more likely it is that the difference between the means is statistically significant. A higher absolute T-value indicates a larger difference between the two means. The sign of the T-value (±) indicates the direction of the difference. If it is positive, the first sample mean is higher; if it is negative, the second sample mean is higher.

• Degrees of Freedom (df): in statistics, degrees of freedom represent the number of values in the final calculation of a statistic that are free to vary. In the context of a paired sample t-test, the degrees of freedom represent the number of pairs of observations minus 1. For a paired sample t-test, df = n − 1, where n is the number of pairs of observations. Degrees of freedom are important because they determine the critical value of the t-distribution, which is used to determine statistical significance.

• Sig stands for significance: values below.05 mean the difference is statistically significant and cannot be explained by coincidence.

In summary, the T-value tells you the size and direction of the difference between the means of the paired samples, while the degrees of freedom indicate the number of independent pieces of information used to calculate the T-value and are important in determining the critical value for significance testing.

Fig. 20

Pre- and Post-questionnaire scores

There are significant changes in science-related skills, and in both reported and tested knowledge. In other words, participants improve their knowledge and their science-related skills over the course of their participation in the demonstrator projects. They join already with a very positive attitude towards science and this level does not change. The same applies to science-related self-efficacy.

10.2.3 Perceived outcomes and impacts by researcher teams

The self-assessment tool was meant to be conducted as a critical reflection exercise by the demonstrator project teams during the lifetime of the projects. The exercise was conducted twice: first, one month after the official launch of the demonstrator project on Zooniverse and, second, at a more advanced stage. This allowed the evaluation team to observe progress in reaching objectives and reflect together with the researcher teams on the outcomes and possible impact of their projects.

The self-reflection exercises were set up as four separate Zoom calls with each of the individual research teams. In these, the members of each component research team agreed, in a discussion, on joint values on the Likert scale that ranged from 1 (=strongly disagree) to 7 (=strongly agree). The evaluation team led the discussion via the online questionnaire and recorded the discussions and reflections, which added further details to the allocated scores.

The four reflection exercises with the demonstrators were meant to reflect on processes during the implementation of the citizen-science demonstrator projects, to see how processes were set up to be feasible in a scientific dimension, as well as for participating citizens in a wider socio-ecological dimension. Furthermore, the teams were asked to reflect on the impacts of their projects on these different dimensions, i.e. science, citizens and society. As the results show (Fig. 21), the demonstrator-project teams perceived the processes as well-designed and feasible and rather equally implemented in relation to citizens, science, and socio-ecological systems. The teams recognised already manifested outcomes and impacts, again referring mostly to participating citizens and the science side, and somewhat less to socio-ecological systems.

Fig. 21

Mean self-assessment results

10.2.4 Evaluation conclusions

To conclude, in general we can say that the demonstrator projects have successfully reached their objectives. According to the research teams in the self-assessment exercise, the science side has benefitted, as the algorithms used to categorise data have been improved thanks to the manual classifications of the citizen scientists.

While the citizen scientists have enjoyed their participation, as the results of the Zooniverse experience survey show, and benefited at the same time by gaining new scientific knowledge and scientific skills.

11 The policy roadmap

The policy roadmap [44] aims to promote the integration of citizen science within major research frameworks. The REINFORCE project seeks to bridge the divide between scientific research and the broader public by emphasizing citizen science. Various factors contribute to the existing status of citizen science in the academic realm. Over the past ten years, research bodies have diligently pursued strategies to promote the approach, but, REINFORCE stands out as a pioneering and comprehensive venture in promoting citizen science within major research infrastructures, specifically targeting astroparticle physics research.

Given REINFORCE’s unique stance compared to other initiatives, it is crucial for its participants to gather insights and reflect on their experiences. Their objective is to provide effective guidelines that benefit policymakers, researchers, citizen scientists, and the general public. The roadmap outlines the steps taken to execute these tasks and presents the findings of this policy initiative.

In detail, the roadmap incorporates results and evaluations from the REINFORCE demonstrator-project and monitoring feedback. Key challenges are seen in the governance models of extensive research systems, especially concerning the recognition and resources available to scientists for citizen-science endeavours. Fundamentally, the hurdles to integrating citizen science into these frameworks revolve around capacity, capability, and motivation.

The roadmap provides suggestions to overcoming these barriers, capitalising on the chance to firmly establish citizen science in expansive research environments. REINFORCE is uniquely positioned to serve as a mediator between scientific research and society. While the adoption of citizen science in major research settings is on the rise, it is yet to become the norm. Nevertheless, REINFORCE showcases that with a well-defined, strategic approach, there’s a promising path forward for the mainstream acceptance of citizen science in astroparticle physics research infrastructures.

11.1 The Wider Landscape of the European Open Science Cloud

The European Open Science Cloud (EOSC) is a major strategic priority of the European Commission, who have invested over €500 million in making the data from all major European research facilities Findable, Accessible, Interoperable and Reusable (FAIR). The aspiration of the EOSC Strategic Research and Innovation Agenda (SRIA [45]) is to deliver a working Web of FAIR data and services, providing European scientists and social scientists with the benefits of data-driven research, with Europe taking a global lead not just in scientific research but also in research data management.

This initiative is well-timed. The extraordinary recent growth in the volume and complexity of scientific data, across many scientific disciplines, has opened up new avenues of scientific discovery, from genomics to astronomical imaging to social sciences working with the firehose of data from social networks. However, the scale and complexity of the data avalanche poses significant computational and operational challenges. Machine learning/artificial intelligence can alleviate some of these problems, but unless the specific problems at hand are amenable to unsupervised ML techniques, the technologies still require labelled training sets.

Data mining by experts, or by volunteers, therefore still plays a central role in the science exploitation of open data. Furthermore, human classifiers easily exceed the capabilities of even the most sophisticated ML, except usually in speed and scale. ML cannot take a step back and un-ask the fundamental question being asked in response to oddities in the data. Serjeant, [46] gave a hypothetical example of an ML algorithm tasked with recognising and counting faces in images of commuters. In this thought experiment, one image contained a very prominent man in a clown costume. The machine vision would just count the face, but the human would spot the outlier in the data and respond to it. Human volunteers can also identify features in the data that are too rare for training ML, such as in the Snapshot Serengeti citizen-science project that measured biodiversity by examining camera-trap images. Over 100 thousand wildebeest were identified, more than sufficient for training ML, but the volunteers also found 17 examples of the extremely elusive zorilla [47].

Citizen science is therefore key to many science applications of FAIR data in EOSC. Fundamentally, citizen science exists as a tool for doing better research. For example, data-mining by citizen scientists can be likened to deploying a giant biological computer to the research problem. It is a tool, in the same sense that a detector beam-line or a spectrometer is a research-enabling tool. As with any other research facilities or capabilities, there are research objectives that are well-suited to the deployment of citizen science, and objectives that are not. Citizen science can also help with education and public engagement, but it exists for the purposes of conducting research and is optimised on that basis. If education or public engagement is the primary goal, there are often alternative tools or activities that may perform better than citizen science against non-research metrics.

There is a subtle logical fallacy in trying to make citizen science optimal for public engagement. It is not logically possible to optimise against two different things simultaneously. For example, there is no highest and hardest mountain to climb, because the highest (Everest) is not the same as the hardest (K2, arguably). One can optimise subject to a constraint, e.g. the best research subject to the constraint that it uses volunteer classifications, or the most educational benefit that happens to also use research, or the highest point that is also accessible by car. However, one always has to choose what is being optimised for, and in citizen science it is conducting scientific research. It is the highest point, or the nicest road, but it cannot be both. It has to be one or the other.

To encourage the growth of citizen science as a research tool, as opposed to viewing it as mechanism for education, the ESCAPE and EOSC-Future Horizon 2020 projects have been building exemplar projects, similar to those developed in REINFORCE, as well as several Jupyter notebooks and tutorials for creating citizen-science projects, managing them and folding in ML. These demonstrator projects, in brief, are:

• Galaxy Zoo: Clump Scout. This project sought volunteers to identify clumps of star formation in images of galaxies, in order to illuminate the physical causes of the rapid drop in comoving volume-averaged star formation rate since Cosmic Noon. The science goal requires a human assessment for what features are regarded as clumps. Over 14 000 volunteers provided over 1.7 million classifications.⁵²

• SuperWASP: Black Hole Hunters. This project used volunteers to identify the characteristic signature of a stellar-mass black hole gravitationally lensing a companion star. The data are inhomogeneous and require a human assessment of data quality. Over 5,000 volunteers provided over 2.1 million classifications.⁵³

• Radio Galaxy Zoo: LOFAR. This project used data from the Low Frequency Array (LOFAR) European radio telescope array. LOFAR is a science and technology pathfinder for the Square Kilometer Array. The classification task asked volunteers to identify regions of radio emission and to find the counterpart optical galaxy. Despite the complexity of the workflow, over 11 000 volunteers contributed nearly one million classifications.⁵⁴

• Galaxy Zoo: Cosmic Dawn. This was an incarnation of the longest-running project on the Zooniverse citizen science platform, Galaxy Zoo. The project asked volunteers to classify the morphologies of galaxies in images from the HyperSuprimeCam camera on the 8 m Subaru telescope in Hawaii, which had been produced by the Cosmic Dawn team in preparation for their future imaging by the Euclid space telescope as one of its deep fields. The Galaxy Zoo: Cosmic Dawn project was completed in June, 2023, and a general data release paper is currently in production.

• Knitting Leaflet Project. The ESCAPE project was focussed on the development of the astro/physics cell of EOSC, but there were still aspirations through EOSC-Future to develop EOSC citizen science demonstrators to wider subject areas. EOSC-Future therefore assisted in the development of a project to characterise historic knitting patterns in early Twentieth Century domestic magazines from the UK. The project will provide insights into how versions of femininity connected to fashion and consumption, and how they changed with time.⁵⁵

• African Indigenous Knowledge. EOSC-Future also assisted in the development of this citizen-science demonstrator project, aiming to capture, document and share African indigenous food system knowledge for promoting sustainable nutritious food production, processing and consumption, initially in Sierra Leone.

The SRIA contains many mentions of citizens as stakeholders in various contexts of the EOSC, as well as several exemplars of good EOSC practice involving citizen scientists. Perhaps partly as a result of our advocacy for citizen science as a research tool across all disciplines, the latest Horizon work programme includes explicit EOSC support of citizens and citizen scientists for societal challenges. However, one indicator of the barriers that still need to be overcome is that in the four key priority areas identified for EOSC in the SRIA, citizen science features in Priority 2: Bridging the education gap: coordinating and aligning curricula for students and researchers. It is true that all use of FAIR data in EOSC for research will involve educational materials, and the further that the data are from one’s discipline area, the more important these educational tools become. However, our policy aspiration is that future versions of the SRIA have citizen science expressed as one of the research tools and methodologies.

11.2 Development

The Policy Roadmap on Research Infrastructures for Citizen Science in Europe seeks to offer policy guidelines to enhance the role of research infrastructures in citizen science by consistently engaging the public.

This roadmapping process was divided into three primary phases:

• 1. Recognising policy voids that prevent the initiation and execution of citizen-science projects within research infrastructures;

• 2. Drafting potential future policy challenges and outlining plans related to advancing research infrastructures focused on citizen science;

• 3. Establishing clear policy guidance and actionable suggestions for all participating entities.

In a more detailed context, the roadmap addresses the following considerations:

• What are the principal policy voids and hurdles to standardise the adoption of citizen science within research infrastructures?

• Which tools and incentives are essential to mitigate these challenges?

• How might these challenges impact various policy areas and society at large?

• What overarching advice can be given to policymakers, researchers, and civil society groups to hasten the integration of citizen science in research settings?

• What measures need initiation to enact these recommendations?

A comprehensive methodology for the construction of the roadmap is depicted in the subsequent chart (Fig. 22). In it, blue boxes depict specific executed activities, purple symbols indicate input sources for the roadmap, and the golden pointers signify progression in the roadmap’s creation.

Fig. 22

Methodology for the development of the roadmap

From the analysis, it is evident that the evaluation of the project outcomes and acquired insights played a crucial role in identifying both the driving factors and obstacles, as well as the policy goals. Subsequently, aligning these policy objectives with current policy measures allowed for a thorough examination of discrepancies and external challenges that impede the integration of citizen science within research infrastructures. Based on these identified gaps, the research team formulated corresponding policy challenges, recommendations, and steps for implementation.

11.3 Main recommendations

• Prioritise STEM in education. In REINFORCE and through a collaboration with the FRONTIERS⁵⁶ network, a series of initiatives that brought the REINFORCE citizen-science projects into the realm of education was organised, including the production of educational resources and teacher guidelines, as well as the organisation of dedicated workshops and courses for teachers.

• New rules of attribution of scientific discoveries and merits. REINFORCE provides an example of an array of effective tools for, and approaches to, consultation. Within the REINFORCE demonstrator projects, several consultation activities were conducted with both scientists and citizen scientists, such as reflection workshops, to identify meaningful actions that might be used in the recognition of citizen-science effort.

• Foster design experience to ensure motivation, sustained engagement, and inclusivity. In REINFORCE, the participatory engagement activities informed more than 1,000,000 citizens in a variety of events organised by the project and through a systematic communication strategy aimed at targeted audiences, to involve around 30,000 citizens in the project activities, and to collaborate with 21,437 citizens in the demonstrator work.

• Incentives to open data on the side of research infrastructures. REINFORCE clearly worked in this regard, as all data were made available openly as part of a specific community on the Zenodo platform.⁵⁷

• Boost the European Open Science Cloud. Many members of the REINFORCE consortium have relationships with EOSC through its providers.

• Introduce citizen science into educational strategy. The REINFORCE consortium includes many education institutions that are introducing citizen science into their programmes. This is in line with the European Astroparticle Physics Strategy 2017–2026.⁵⁸

• Boost evaluation and monitoring of citizen science. The REINFORCE sustainability plan calls for the establishment of a think tank, which would ultimately carry out the long-term monitoring and consultation, as well as mentoring for, new citizen-science projects.

• Include educators in programme design. This has been shown to be an effective way of empowering citizens to become ambassadors for projects. Through dedicated training courses, organised by REINFORCE partner EA and which partners of the REINFORCE collaboration have co-organised, teachers and citizens were empowered through dedicated training activities.

• Community establishment, scale-up, sustain and engagement. With regard to this, the REINFORCE approach highlights the importance of task analysis and close collaboration with boundary organisations to help understand the educational perspective of any citizen-science initiative.

• Involve policymakers throughout the project life-cycle. In this regard, REINFORCE paved the way by adopting a strategic approach towards policy and organising workshops aimed at creating common ground in the conversation between scientists and policymakers through citizen science.

• Continue to pursue and encourage inclusion and diversity. In REINFORCE, CONICET and EGO have carried out a series of activities to extend the senses used in scientific inference, beyond the visual, and included in the overall effort and scientific community sense-disabled people (especially the visually disabled) and senior citizens, but have also profited from the different points of view of artists and artisans.

12 Thoughts and conclusions at the end of the REINFORCE project

Reflecting on the three years of REINFORCE it is difficult not to feel that the project can be regarded as a success. All four demonstrator projects were successfully officially launched on Zooniverse, despite one of them — New Particle Search at CERN — being of such complexity that it was necessary to move the more advanced stages of the project out on to the separate HYPATIA platform. All of the demonstrator projects have been successful at engaging volunteers in the scientific process with real-world scientific results. Targets set within the REINFORCE project were demanding and the citizen-engagement strategy (Sect. 4) and its subsequent implementation (Sect. 5) resulted in the successful delivery of more than 1 million classifications within the piloting period, contributed by a community of more than 20,000 committed individual citizens.

Collaboration between the demonstrator project and sonification teams has led to the availability of new sonification forms via the sonoUno library and the work of the sonification team has proven successful in engaging numerous volunteers in the further development of the software.

The senior citizen science programme, following initial disruptions because of the pandemic and subsequent to an evaluation activity undertaken with the participants, proved to be so successul that it led to a second edition, which took place, still under the auspices of REINFORCE, throughout the 2023/24 academic year, well beyond the lifetime of REINFORCE itself, while the art and science initiatives were very successful at bringing new participants into the world of the project and into the arena of the demonstrator projects.

The policy roadmap provides a series of useful recommendations and objectives to pursue in order to effectively embed citizen science in large research infrastructures. Implementation steps on the path to effective take-up are detailed, incorporating responsible research and innovation at the heart of the process. A broad range of recommendations have been provided, covering a number of different areas, including: open access to services; standards; collaboration; inclusion; science and society; sustainability; governance models; scaling-up; and policy-making processes.

Exploring the potential of frontier citizen science for inclusion and diversity was at the very heart of the project and much work has been done to this end, while the project also served to build a healthy international working collaboration. This in itself was not guaranteed, given that the Covid-19 pandemic broke just three months after REINFORCE started, with all of the related tumult and complication that it brought to an initiative that had been designed to heavily rely on in-person engagement activities. The fact that, despite this, the REINFORCE collaboration—built from twelve partner organisations, based in seven different countries, across two different continents, and involving, over the lifetime of the project, more than 70 people from twelve different countries—grew to be a respectful and friendly environment, in which all members were able to contribute, was also a very happy outcome.