1 Introduction

Digitisation of cultural heritage content, over the past few years, has created large digital collections that have the potential to open up cultural heritage and reach local and global audiences. However, users of cultural heritage can be diverse, and may include members of the general public, cultural heritage professionals, academics, historians, and industry workers, amongst others. Such audiences have different backgrounds and experiences, meaning cultural heritage objects can have multiple interpretations based on varied user types, including their cultural context and information needs [1]. Identifying user interests in different parts of an online collection, and investigating the related search behaviour, can help to improve system support in Interactive Information Retrieval where users are engaged in purposeful and directed searching [2,3,4,5].

Given their diverse backgrounds and information needs from cultural heritage content, users often have difficulties in locating objects of interest from very large and distributed collections [2]. Nevertheless, log analysis can inform engagement strategies by providing insights into users’ searching and navigational behaviours [6]. For example, a number of documented activities identified within server logs have enabled museums to identify potential user groups and categories [6,7,8]. In addition, some research studies suggest that access patterns identified through log analysis are useful in improving the clustering or categorisation of cultural heritage collections [9,10,11,12,13].

Evidence shows that online access to cultural heritage collections is increasing, but high numbers of users are looking at only one or two pages within a few seconds and then leaving [14]. Therefore, it is important to have a better understanding of the types of users, such as their motivations, tasks, engagement, and domain knowledge [14, 15]. Some research shows the relationships between online collections and their visitors [6, 7], but it is important for museums to create multiple forms of online experiences to reflect the kind of motivations, art background, context, and online behaviour of website users [7].

The findings of user behaviour studies can also change what an organisation is doing [16]. A further challenge to understanding user experience and information needs online comes from the variable quality of digital objects and collections themselves. Most of the time the metadata associated with cultural objects, such as images, is either sparse or inconsistent, and this makes keyword-based exploratory search difficult and therefore slows down the research or engagement process [17]. User modelling can describe the interaction process between users and cultural heritage applications and products [18]; however, despite a myriad of research reported over the past two decades or so, there is a lack of a richer and deeper understanding of digital users [19].

“Digital footprints and search pathways” was one of three projects supported by the Arts and Humanities Research Council’s (AHRC) Towards a National Collection programme (TaNC) as part of UKRI’s call for COVID-19 projects.Footnote 1 With the COVID-19 pandemic severely affecting every aspect of daily life, including the human need to connect with collections held in museums and galleries, there was a clear opportunity to investigate peoples’ engagement with cultural heritage sites in Scotland during lockdown. National Galleries of Scotland (NGS) and National Museums Scotland (NMS) offered two interesting case studies, since Scotland’s arts and culture sector was disproportionately affected by the pandemic in comparison with the rest of the UK.Footnote 2 The lessons learned from the findings of this project may therefore be used to improve online engagement as we move towards a new way of life where digital practices have a more prominent role throughout society.

Research reported in this paper is part of the AHRC project that aimed to investigate how people accessed cultural heritage information during the COVID-19 pandemic. Overall, the project examined user search behaviour and patterns via the log analysis of access data collected from NGS and NMS. Extensive detail of the log analysis, findings, and conclusions is available online.Footnote 3

This paper extends [20] by first reporting on some of the key findings from the log analysis on user access behaviour, with particular reference to the retention and return of users to specific content collections/items. These findings provide the foundations for our controlled experiments with two groups of users, and selected sets of digital content from NMS and NGS, to explore whether there are any differences in the way users search and the infrastructures (search pathways and interfaces) provided by cultural heritage organisations. More specifically, the user study aimed to offer an explanation on why people spend less time when they reach a specific collection or item after a search session, and why users do not return to the previously found collections and items. The research questions that shaped the design of the user study are as follows:

  1. 1.

    How does the metadata assigned by cultural heritage organisations meet or differ from the search needs of users?

  2. 2.

    How can the search strategies of users inform the search pathways provided by cultural heritage organisations?

The rest of the paper presents the methodology and protocols used in this research, along with key findings of the log analysis data and user study. We conclude by discussing how various user-defined metadata can be accommodated within the existing framework of the collection management standard (SpectrumFootnote 4) used throughout the cultural heritage sector in the UK.

2 Methodology

2.1 The log analysis

At the start of lockdown, due to the onset of the COVID-19 pandemic in the UK in March 2020, the online provisions of NGS and NMS varied widely. NGS’ collections included over 98,000 art works, of which about 80,000 had digital assets represented on the NGS website. Whereas the NMS collection comprises over 12.4 million objects and specimens, of which 783,319 items are accessible via the NMS website. Both worked with Google AnalyticsFootnote 5 to monitor and report on the traffic received by their respective websites. Due to the discrepancy in both the size and representation of each institutions’ online collection, their web traffic data was embedded in a separate longitudinal log analysis. Access patterns from the first 12 months of lockdown were split into blocks of four (April to June 2020; July to September 2020; October to December 2020; and January to March 2021) and compared with the same periods from the previous three years. Such a process meant that we were not only able to study the wider effects of COVID-19, but also the difference in access patterns as lockdown restrictions changed. Table 1 incorporates a list of the attributes considered during the log analysis, along with their reasons for inclusion.

Table 1 Attributes considered during the initial log analysis
Full size table

2.2 User study

2.2.1 Participants

In total, 10 people completed the virtual study between the months of January and March 2022. Table 2 includes the demographics of these participants, where a deliberate decision was made to recruit both experienced and first-time users of the NGS and NMS sites to understand whether there were any differences in the search behaviour between these two groups. Inexperienced users who had some knowledge of search were recruited from higher education institutions across Scotland, whilst more experienced users were contacted directly from the mailing lists of NMS. All participants had access to an information sheet during the recruitment process (see Online Resource 1) and provided informed consent before contributing to the study.

Table 2 Profiles of study participants
Full size table

2.2.2 Protocol

The user study was split into two separate tasks, which were completed virtually via the Zoom video conferencing system to adhere to social distancing measures imposed during the COVID-19 pandemic in UK. Both tasks were performed on Mural,Footnote 6 with task one consisting of an item categorization process where participants assigned search phrases to items from NGS and NMS before grouping them together to form “collections”, similar to [21]. The second task involved a scenario-based search observation process where participants performed live searches across the NGS and NMS websites to fulfil their information needs, similar to [22].

For task one, NGS and NMS selected one regularly accessed item and one less popular item from five of their collection departments. This was to ensure that consideration was also placed on harder to find items, which may have less impactful metadata. Digital flashcards were then developed for each of these items, which included the available metadata and an associated image; see Online Resource 2 for some examples. These flashcards were pooled into a Mural worksheet (see Online Resource 2 for a completed sheet), with the participant selecting the first item and assigning tags that would assist in its retrieval. They were then asked to describe their reasons for the tags they assigned, before placing the flashcard in an appropriate space in the worksheet, which may have included grouping similar items together to form “collections”. This process was repeated until the resource pool was empty, at which point the participant was given the opportunity to make amendments to the tags and/or groupings. Such a procedure enabled the participants to consider, outside of the infrastructures of NGS and NMS, the characteristics of collection items that are most important to them when searching. A comparison between these characteristics and the data management standards applied by NGS and NMS was made.

It was also important to consider the search strategies of users when fulfilling their information needs within the digital infrastructures of NGS and NMS, including the search terms employed. Therefore, task two involved a search observation process, with each participant being required to locate various items across each site. Based on Borlund’s evaluation framework [23] for interactive retrieval systems, four simulated search scenarios were created by NGS and NMS focusing on the following goals:

  1. 1.

    Researching a well-defined topical information need

  2. 2.

    Researching topics via data elements only, e.g. titles and locations

  3. 3.

    Researching an ill-defined topical information need

  4. 4.

    Researching a known item via data elements

These scenarios can be found in Online Resource 2 and were designed to ensure all search features across both sites were accessed. Additional consideration was also taken to ensure some of the scenarios focused on topics that would be of interest to NMS and NGS site users during the COVID-19 pandemic. For example, NMS scenario 2 involved researching information related to the Roman Empire, which would be of interest to students working from home. NMS scenario 3 centred on the exploration of medical artefacts, including those recently used to combat COVID-19. Finally, NGS scenario 4 focused on a drawing of a famous local landmark, Edinburgh Castle, which may have piqued the interest of worldwide users who were unable to physically visit NGS’ campus.

Participants completed one search task at a time and were permitted to utilise as many features and access as many pages as they deemed necessary to satisfy the information need. Whilst locating search items, each participant was encouraged to “think aloud” [24]: to talk through the rationale behind their actions as they were carried out. Help was not provided by the investigator unless explicitly requested, and no time limit was placed on the search tasks. On completion, a discussion took place about the features the participant liked on each site and the potential improvements that could be implemented.

2.2.3 Analysis

Both tasks were recorded and transcribed verbatim with participant consent for further analysis. The first task was primarily subjected to a deductive content analysis, using the Spectrum data management standard as the driving structural framework, since Spectrum is employed by both NGS and NMS. Content analysis is a term used to describe a number of text analysis strategies:

“It is a systematic coding and categorising approach used for exploring large amounts of textual information unobtrusively to determine trends and patterns of words used, their frequency, their relationships, and the structures and discourses of communication...The purpose of content analysis is to describe the characteristics of the document’s content by examining who says what, to whom, and with what effect”. [25]

With its added focus on the use of particular words, content analysis was an ideal method to determine the characteristics users find most interesting when searching for cultural heritage items and whether these align with Spectrum. An in-depth description of the steps involved in content analysis is described by Erlingsson and Brysiewicz [26]. During task two, participants employed a range of search strategies and therefore encountered a variety of barriers. Consequently, an inductive framework analysis [27] was considered to be the most appropriate method since it facilitates the natural comparison of participants’ views, which led to more concrete recommendations on how to improve the search pathways across NGS and NMS.

In addition, quantitative measures (such as time to completion, success rates, and number of pages accessed) were recorded for task two. Nevertheless, there were several factors that skewed the results for certain participants, including: poor internet connectivity that made it difficult to converse via Zoom; and a more limited proficiency of the English language. Since the n-size of the study was relatively small, we decided to omit these results and focus exclusively on the richer qualitative data.

3 General user access behaviour during the lockdown

The results of the log analysis reveal that, in general, the number of commenced sessions in lockdown across both organisations was low compared to previous years—see Fig. 1. This is not entirely surprising since there would have been a natural drop-off from users looking for information related to in-person visitations, e.g. opening hours and exhibitions on display. Nevertheless, there was an exception to this trend during the months of October to December 2020. In the lead up to the festive period, the number of sessions across NGS and NMS rose to the highest levels. This demonstrates that despite restrictions, the wider public’s interest in cultural heritage remained high during seasonal holidays—an aspect that organisations can capitalise on in the near future and during forthcoming crises. The phased reopening of both institutions in August 2020 also seemed to have an impact on the sessions commenced, with NGS experiencing a 20,000 rise (24.53%) and NMS a 50,000 rise (45.43%).

Fig. 1
figure 1

The number of sessions commenced across both sites. Less sessions occurred during lockdown across both organisations, except the period leading up to Christmas festivities (November–December), where sessions increased

Full size image
Fig. 2
figure 2

Total number of users vs. unique users across both sites during lockdown. Over 90% of users are unique users, meaning the data shows there are less returning/regular users

Full size image

The number of users accessing each site per month followed a similar trend to the sessions commenced. This was largely due to a severe lack of returning visitors during lockdown (see Fig. 2) where less than 8% of users frequented the sites more than once per month. Such a statistic matches the traffic received by NMS during previous years; however, NGS traditionally experienced a larger percentage of returning users, particularly in 2017 where up to 45% of visitors per month came back to the site. The publication rate of new content is likely to have an effect on returning users, whilst an exploration into the quality of content on offer is also required. Yet, this may also be caused by a mismatch of indexing/tagging of content done by the staff/curators of the cultural heritage institutions, and how content is searched by the end-users, i.e. use of terms/keywords. This led to our exploration of user search behaviour in the second part of the study.

Fig. 3
figure 3

Average number of page visited per session across both the NMS and NGS sites. NMS’ pages per visit followed the same trend as duration spent on the site—more pages were visited during the first half of lockdown. NGS visitors generally browsed less pages apart from a spike during August to September

Full size image
Fig. 4
figure 4

The total number of collection views received. Collection views were higher during lockdown for both NMS and NGS

Full size image
Fig. 5
figure 5

Average time spent on the sites per visit. NMS users spent more time on the website at the beginning of lockdown (April–October). NGS users increased their activity during the Christmas months (November and December)

Full size image

Figure 3 shows the average number of pages viewed by the users in the two sites. Many of the pages included in the NGS and NMS websites provide information regarding in-person visits (such as “What’s On’ or ‘Visit Us”), meaning they experienced a natural drop-off in interest when the physical institutions were closed. Nevertheless, collection pages offer a source of information for a variety of populations, not just those planning on visiting the museum or gallery. For example, academics and students may search collection databases to support their research or hobbyists can browse the collection to find items of interest. It was therefore important to analyse the impact of COVID-19 on the collection pages in comparison with the wider sites. As can be seen in Fig. 4, user views of collection pages and collection search databases were consistently higher for both organisations during the pandemic compared to the previous years, thereby demonstrating that cultural heritage continued to be an important aspect of life throughout the pandemic. NMS users also seemed to return to specific collection pages at a much higher rate than NGS (65% of the views were unique compared to 85%), although there was no explanation recorded for this gap.

Page views and user statistics only provide a shallow insight into engagement. It is therefore necessary to complement this data with information regarding the amount of time users spend on the sites and the number of pages accessed to provide a more complete view of the breadth and depth of interaction. In terms of average duration (Fig. 5), users spent more time on NMS’ site during the first six months of lockdown (up 40 s more than previous years) in comparison with the last six months analysed. NGS users spent a similar amount of time as the pre-lockdown rates, with a slight increase across the Christmas months (November and December). Unsurprisingly, the average number of pages per visit followed a similar trend to duration, with the exception of NGS experiencing a large spike in page access during August, which coincided with the easing of lockdown restrictions in Scotland and the reopening of the physical exhibition space.

4 Results: user study

As discussed in the previous section, users, on average, spent little time browsing pages across the NGS and NMS sites and were unlikely to return within the same month. Results from the user study will now be presented to highlight that search infrastructure barriers have the potential to contribute to limited engagement from end-users.

4.1 Task one: metadata tagging processes

We begin by presenting some of the tagging strategies carried out by the participants, and where relevant, highlight the differences compared to the Spectrum indexing standard currently utilised by NGS and NMS.

4.1.1 Wide ranging tags

Overall, the majority of the search tags created by the ten participants during task one could be retrofitted to meet the cataloguing fields proposed by Spectrum. Nevertheless, this process often consisted of assigning tags to wider encasing fields, such as description or physical description, where curators have some freedom in determining the characteristics that should be included. As such, there is a risk that potentially important information could be overlooked due to the structures of expertise and knowledge frameworks, or the lack of it, that inform the institutions’ indexing practices [28]. For example, in Ian Hamilton Finlay’s “Sea Pink” (see Online Resource 2), most of the less familiar participants focused on the colours pink and teal when assigning search tags due to their lack of knowledge about the object, yet such descriptors are not included in the metadata. NMS in particular offer no specific search functionalities related to colour, which participant 8 suggests would be helpful to distinguish between similar items from the same era: “I think something that might also be helpful to include within the websites is if you can kind of also add colours as ways to sort objects, especially within fashions and textiles. If there’s a lot of similar objects within the same era, then being able to identify them by colour might be helpful”.

4.1.2 Linking loosely coupled items

Some of the tags proposed by the participants could not be modelled under the existing Spectrum standards, with the majority of these aligning with the ability to link loosely coupled objects together. For example, in reflecting contemporary concerns with inclusion and equality, many of the participants honed-in on characteristics that related to disability (such as Mrs E.M. Wright being painted by an artist with no hands) and women’s rights (e.g. the suffragette banner), and therefore suggested that such topics could be grouped together under the same collection. This included highlighting female subjects or artists from older time periods, due to their previous exclusion from the field of art and culture. Currently, such information could be captured in Spectrum’s description field, yet this would not be sufficient to link inherently different items together, meaning an additional field would be necessary.

Similarly, some of the participants assigned search tags based on the presence of an animal or person, regardless of whether they were well known: participant 4: “There’s also people in this painting [Great Expectations] so I’ll put it here [next to portraits] and I’ll just put like a theme like people in general or something”; participant 1: “People do look for art that relates to animals in particular”. Creating new fields that enable users to search for people or animals in general (e.g. linking the Zoetrope with more obvious items such as Dolly the sheep) could help facilitate future research into areas such as class or the role of animals in human culture. Sub-categories may also be developed to support more specific research, as highlighted by participant 2: “This is going to be such an awkward one to do but it’s like famous or renowned. Yeah, it’s like famous faces. And I’m going to put in Stevenson, you can put in Dolly the Sheep, uh, where’s Van Gogh gone. I’m going to put him there and connect him to Burns”.

Participants also consistently assigned tags that group items from a particular domain. Some of these tags cited well indexed areas such as anatomy, Scottish History, space, and war; nevertheless, many were not, including animation, activism, taxidermy etc. Spectrum’s Object Category/Classification field can permit the retrieval of items from a particular subject, yet once again the nature of these subjects relies on the views of curators, which can differ from end-users.

4.1.3 Knowledge and expertise

Furthermore, there was a cultural difference observed in the way artefacts were being tagged. Those individuals with English as a second language particularly relied on Spectrum’s Object name category when tagging, which encapsulates more basic descriptions. Nevertheless, there were instances of local or culturally specific terms being embedded in this category, such as “claymore” in the highland sword, which had no meaning to these participants, who instead opted for simpler terms such as “sword”: participant 6: “Because I don’t know what [a] claymore [is], so I will just type sword”. This highlights the importance of providing synonyms to support search from a range of users, which may also include individuals who are not experts in a certain area. For example, the participants recognised that the tags they were implementing differed based on their own preferences and experiences: participant 10: “Yeah so it was easier because I have a background knowledge on Dolly I know what search terms would probably work for that one, whereas the other ones I don’t have any background knowledge on those”.

The variability of the available metadata also had an effect on the depth of the search terms assigned to an item. Some participants had great difficulty tagging items that had little description, whereas others were absorbed by more complex items and found themselves applying less relevant tags; participant 5: “Because there’s no information on it, it makes it hard to classify it and give it worth. I’d imagine walking past that and being like you’d want to know why it’s there and then when there’s no information on it you’re like there’s nothing there to tell me why it’s here and that someone made it”; participant 1: “I could list everything in that photo, waves, sea, boat, lighthouse but you know then I’m just listing everything in it rather than trying to generalise a theme...I don’t want thousands of themes. How do I encompass most of them?” There was also some evidence of participants breaking wider encompassing tags into smaller sub-tags.

4.1.4 Physical versus digital space

Finally, when attempting to group items, participant 8 consistently referred to the physical spaces of museums and how collections are formed: “Thinking about the actual physical space of where these objects would be and I think that is really important for a lot of people when it comes to sorting things especially if you’ve been to the physical spaces, they’re like oh this was probably in this room whereas this was in this room”. This was surprising since literature, e.g. Burke et al. [29], focuses on taking advantage of the different experiences offered by digital spaces and moving away from simply mirroring the layout of physical museums and galleries.

4.2 Task one: spectrum fields

This section provides a discussion on the Spectrum fields that were most effective in encapsulating the tags assigned by the participants. Note that Spectrum has a far wider catalogue and not all fields were referenced.

Dimensions: Participants consistently referred to the size of tangible items (i.e. physical, 3-D objects) when providing tags. This did not solely involve specific dimensions, particularly with the less experienced participants, where other more general descriptors were applied such as “miniature”. On the other hand, the more knowledgeable participants requested further information on the size of certain items, such as the claymore, which highlights the variability of the metadata being assigned to collection objects: participant 7: “Has it got the dimensions? No it doesn’t, um, because some of these were symbolic, you know, they were so big that they weren’t actually weapons but they’re classed as weapons”.

Location: The location tag in Spectrum calls for full location audit information, including current display locations. Some of the frequent visitors of NGS and NMS were interested in the exact rooms items were held, yet others cared more about whether they were on display to support their decision for an in-person visit: participant 1: “You want to group together things that were on display...but also if it wasn’t on display they [users] wouldn’t waste their time going to the museum to go see it if it wasn’t there because obviously, you know, during the pandemic everyone’s working from home. Some people may have moved away from the city and a lot of people who visit museums aren’t actually from the city...they might not want to visit or come to travel that far if that wasn’t there. And a lot of international people go to the museum”.

Materials: Materials were one of the most commonly tagged aspects for both experienced and inexperienced users, particularly when an unusual or defining substance was utilised by an artist: participant 4: “If I want like a more specific [tag] I would look at material, so here, like, it’s really different to have a wood material [for paintings]”. In addition, the participants would often fall back on the physical characteristic of items if they lacked knowledge on an artefact, participant 8: “I don’t think people would necessarily remember it’s a bridal set or anything like that, I think a keyword to be in here would be silver”.

Production dating: The Spectrum Production Dating field urges indexers to provide a specific date an item was made or a broader range if one is not available. This was evident in the participants’ own tags, where four different classes of date were mentioned: the exact date; the century; an era such as Victorian; and modern vs old art. Different indexing strategies could link vastly diverse items together, particularly via the latter method as highlighted by participant 2: “I would have guessed that [mummy portrait] would have been like, you know, maybe pre Victorian times but if that’s where that’s from then it’s ancient, that’s pretty amazing. So yes, to go with modern history there’s also your ancient history. Anything that’s over 1000 years would go into ancient history. Or like anything over 800 years. Yeah, I’d say 800 years cause then you get into like the Middle Ages, your dark ages and Middle Ages”.

Production place: Spectrum also places significance on the area an item is associated with, which may include multiple locations such as the place it was designed and the place it was manufactured. Both sets of participants also felt such information was important, and suggested emphasising Scottish and non-Scottish objects for tourists who may want to prioritise local artefacts: participant 9 “When I go to [anon] and they like present some Scottish local artists and some creation in a particular space. So I think some of the audience will be interested in Scottish artists. So I might put these kind of key words in it”.

School/style/culture and title: In terms of the style of an object and its title, many of the participants who had little experience in certain sub-domains of art and culture were hesitant to tag such fields unless they contained common knowledge such as Dolly the Sheep. Nevertheless, they recognised that users with more experience would deem these characteristics to be important, as also found in [28], where expert users searched for more characteristics than novice users: participant 8 “I mean I’m not an artist, I’m not, but is he classified as an impressionist or something? But I guess if people are looking for Van Gogh though they know about him”. Participant 10 “I don’t know a lot about guns so I wouldn’t know that [flintlock]. But I bet if someone knew something about guns and they were searching for it I’m sure they would know that term”.

4.3 Task two: search scenarios

In addition to evaluating the metadata tags utilised by NGS and NMS, it is also important to consider the overall user experience of individuals searching for information across the sites. Whilst completing the search tasks, the participants discussed aspects relating to the way they search, the search features (pathways) available, and the structure of the items returned.

4.3.1 Search procedures

Continuing on from the first task, the search terms employed by participants were generally basic, consisting of a few descriptive phrases such as “brooch, love”. Most, centred on terms that could be captured by Spectrum’s Object Name field, with colour, style, and materials also being used to narrow searches that returned a wide range of results. Barriers related to search terms primarily consisted of a lack of support for synonyms, misspellings, and grammatical constructs such as pluralisation: participant 2: “It would be dreadful if you type in something and it turns out you’ve missed your spelling slightly. Instead of archaeology I put archaeologists and got nothing”.

Two main search strategies were utilised by the participants depending on the topic being explored and their familiarity with the websites. First, if a topic was particularly broad, or the participant was new to the NGS or NMS websites, then they would prefer to use the site-wide search box: participant 4: “I feel like the advanced search is too narrow for this, like I don’t know where to put the COVID-19, like should this go into the collection or description, so I’m just gonna go with the normal search, COVID-19”. There was also evidence of participants falling back to the site-wide search bar if other features such as advanced search produced no relevant results: participant 7: “So when in doubt usually my last step, I think, is just going to the actual search bar up here and searching like art and culture”. Second, participants who were familiar with the websites tended to use more of the available search features, often beginning with advanced search when the object had a particularly distinguishable feature.

In general, participants tolerated between four and six pages of items being returned. If the results became too obscure, then they would narrow the search by adding further terms to the advanced search bar: participant 5: “We’re getting a bit obscure, well there’s a brooch but if I started to see like it was getting a bit abstract, like that plaid I’d be like, oh right, I may be going too far”.

4.3.2 Search features

The motivations behind utilising each of the available search features across NGS and NMS, as well as the advantages and barriers to using these features, are presented below.

Advanced search bars: As discussed earlier, the participants tended to use advanced search features when they were familiar with the websites and had a particular characteristic in mind that they wanted to search for, especially when narrowing results. Figure 6 highlights the differences in the advanced search features of NGS and NMS.

Users of NMS felt that the advanced search bar was missing crucial characteristics such as colour, whilst they were also unsure about what information to include in the categories that were provided. For example, all were hesitant to input a collection when searching for items as there was no easy way to find a list of collections made available by the museum: participant 5 “Knowing what the collections are called helps. But then again, I feel like that should just be something I can find out very easily rather than having to look for one example then work my way back up the chain”. In addition, the results were overly restrictive, in that inputting a wrong word or misspelling in one category would simply break the search.

The participants preferred the ability to select pre-determined search categories—like those offered by NGS—since this supports users who are less familiar with their item to find what they are looking for: participant 1: “I like that they both had an advanced search option. I like the fact that this one has the search option, where it kind of gives you things—if I wanted to search Van Gogh, you can see the artist and his artworks. It will give you, like, very specific things that might have been the actual search term to use”. Nevertheless, they felt that the free-text search bar was difficult to locate within the “More” menu item and should instead be embedded in the main Artworks page. Suggested improvements to NMS’ advanced search feature focused on guiding the user on what terms to use either via an autocomplete feature or similar drop-down menus to NGS: participant 2 “I think something that pops up with recommendations of tags that do exist...I think that would help”.

Fig. 6
figure 6

NGS and NMS advanced search bars

Full size image

Artists search NGS: The participants who utilised the “Artists” search feature from NGS appreciated the additional information that may be obtained—such as a link to the artist’s Wikipedia entry and biography—and felt that the pages were well structured overall. Yet, there were some instances where they attempted to find an unlisted artist using this feature and subsequently requested a more complete catalogue.

Collections at NGS : The “Collections” feature from NGS was misused by the participants who were unfamiliar with the site, as they felt that the page would offer a way to search for collection items (like the advanced search bar found in “Artworks”), as opposed to describing collections that are available in the gallery. This may suggest that a rethink of the headings may be necessary to support new users in accessing the features they are looking for but also encourage them to utilise a wider range of functionalities.

Glossary at NGS site: Surprisingly, NGS’ glossary was underutilised by the participants, especially those who were less familiar with art and culture. Nevertheless, when shown the feature, most suggested it could be extremely useful to identify potential search terms, with participant 4 advocating for a link to be embedded within the site-wide and advanced search features: “It’s difficult to find it. I feel like it should be near the search bar and then, like, under the search bar it should be written like ‘don’t know what terms to search, look at our glossary’ or something like that”.

Site-wide search bars: As discussed previously, the site-wide search bars were mostly utilised by new users or when participants were researching more open or new topics such as COVID-19. In addition, the experienced users of NMS used this feature in circumstances where an article would be more insightful than a collection page: participant 6: “The phrasing of that question, which was art is addressing the topic of climate change, that doesn’t make me think I’m looking for artwork for climate change because there’s probably lots of that but more maybe articles”. On the other hand, less experienced users expected a combination of articles and collection pages to be returned by the NMS site-wide search bar, which was not the case. In terms of the NGS bar, the participants appreciated the suggested terms drop-down menu that appears when typing but found it distracting when a suggestion permanently fills the search box once you have hovered over it.

Stories and resources NMS: This feature was mainly used by participants who were familiar with the NMS site. They suggested that “Stories and resources” offered an alternative way of gaining additional information on items via articles that are grouped together by themes and subjects: participant 7: “This is quite an interesting way to go because this includes lots more than just the actual artefacts, so I think the themes are quite good. I have found you’ve got to know to go there, and I think that that could be clearer. Romans’ life in the frontier, Romans, the Roman army. These are really, really good, these sorts of articles. I think that’s actually gonna tell me a bit more”. Improvements to the feature centred on the ability to restrict search results via subject, theme, and type as opposed to just one of those categories: participant 8 “I think if there was a way to sort of more narrow down, like if you could choose both the theme and subject because as you can see you can’t choose both. So having, like, explorer by type or subject or theme or a mixture of all of them I think would be a lot more helpful”.

4.3.3 Item descriptions

Three barriers relating to the descriptions of items were observed across both sites. First, participants found the collection search results to be difficult to navigate when the items were presented with the same, basic tags: participant 5 “It’s frustrating how they’re all called brooch. If they even had brooch brackets, something, a year, a period, a style anything because otherwise what you’ve got is brooch, brooch, brooch...even like a preview of what it could be [would be helpful]”. Some of the participants were also hesitant to conclude that their search tasks had been completed due to the omission of important metadata such as a date: participant 3 “I would be really missing a year. At the least, I like an approximate year because if it says Roman site at Newstead I don’t know whether there might be, like, an actual Roman site still now at Newstead and it’s been found like a week ago so it’s dated like 2021. I know I’m overthinking this but it’s clear for this object, but it might not be clear for other objects that are not so well known in history”. Finally, the lack of associated images hindered participants during the tasks where they had to use a picture of an object as a reference.

5 Conclusion

The goals of the project align with the wider research priorities of the United Nations Educational, Scientific and Cultural Organisation (UNESCO) that aim to increase access to culture and heritage online, support the resilience of artists during crisis situations, and bring governments together to improve on existing policies.Footnote 7 UNESCO estimated that at the height of the pandemic, 89% of all world heritage properties experienced some sort of closure and therefore lost millions in revenue each day. The findings of the project corroborate the findings of other researchers in that the cultural heritage institutions were often unprepared for a sudden shift towards digital provision due to patchiness in their online resources and a lack of expertise and budget to make changes,Footnote 8 which exacerbated financial concerns. This further highlights the need for improved digital provision in general, but especially in preparation for future crises.

Data from the log analysis provides interesting insights into user behaviour on the two digital cultural heritage information services. The analysis shows that, although user access patterns did not vary too much between the pre-pandemic and the pandemic lockdown period, the number of returning users was very low across both the services. Over 90% of the users were new, which raises the question of why they fail to come back to the digital collections of the cultural heritage institutions—in this case, NMS and NGS. In addition, Fig. 4 highlighted that, overall, engagement with collection items was higher during lockdown, but the number of pages searched for (Fig. 3), and time spent on the sites (Fig. 5), remained low. This strengthened the need for the user study, with the findings reinforcing our assumption that lower engagement levels may be attributed to the differences in the way users search for cultural heritage artefacts and the way they are indexed by institutions.

The population for the study was small, yet the results enabled a conclusion to be formed that the knowledge of stakeholder needs and preferences can help drive user-centred improvements to the digital infrastructures of cultural heritage institutions. All of the participants were highly educated and were either pursuing or had obtained a postgraduate degree. Professional and highly educated people form the majority of users of cultural heritage [29], and hence our selected user group reflects the education status of the bulk of users visiting the NGS and NMS sites. Nevertheless, this is a limitation of the study and future work should consider employing similar methodologies with more diverse groups of users to form a holistic understanding on the differences between the views of end-users and curatorial staff.

Overall, this research provides insights into the online search behaviour of NMS and NGS users that can inform future policies around digital presence and provisions for these institutions, and the sector as a whole. Existing collection management standards like Spectrum are not user-centred and often the metadata implemented by collection institutions to index objects are not designed for the diverse needs and contexts of users. This calls for more research—with diverse groups of both users and non-users, and selected collections/objects, to capture multiple perspectives of items. Such a process has the potential to ensure metadata is more user-centred and the search interface takes into consideration the needs of people with different backgrounds, motivations, ethnicities, and varied experience in cultural heritage. Research literature shows promising prospects for the use of AI (artificial intelligence) and ML (machine learning) to support more timely and wide-reaching metadata tagging [30]. However, this would require items to have a standard of existing data that neither NMS nor NGS currently have across their collections, which could be true for most cultural heritage institutions. Future investigations into this approach should start off small, focusing on collection items that have no licencing issues, good data standards, and which speak to diverse sets of users and their search motivations, before upscaling across entire collections.

Results from the first task in the user experiment highlight that indexing cultural heritage objects for a range of target users is an extremely difficult and time-consuming task, even with curators being guided by data management standards such as Spectrum. These standards encourage a variety of different characteristics to be considered when developing metadata, yet much of the content is left to the interpretations of curators, e.g. description or physical description. Rather, user- and context-specific guidelines could be beneficial in ensuring the aspects considered most important by consumers are indexed, whilst AI and ML techniques can expand on the resulting descriptions, thereby producing more relevant search results based on user profiles and access patterns.

Results from the second task indicate that a user-centred approach to designing cultural heritage websites would help to improve an individual’s experience when searching for information. Such a process requires institutions to form a concrete understanding of who their target users are before developing features and designs to suit their specific needs and interests. To elaborate, those participants who had less experience with art and culture, including the NGS and NMS sites, experienced different barriers than those who did, and used a narrower range of search features—primarily the site-wide and advanced search. Overall, the findings of this research corroborate the general argument that quantitative reporting, based on log analyses, lacks detail and nuance in terms of audience behaviour, and this often leads to a lack of richer and deeper understanding of digital users [19]. The user study reported in this paper, although conducted in a small scale, provides a methodology and strong argument for embedding the knowledge and perspectives of end-users within the indexing and representation of cultural heritage content. Our findings, which allude to the potential gaps between the institutional or curatorial views and the user-/community-views of cultural heritage content, echo the key arguments and the initial findings of some other AHRC funded projects in the UK under the TaNC (Towards a National Collection) theme (see, for example, the reports [31, 32]).