Is there an imbalance in the supply and demand for universal accessibility knowledge? Twenty years of UAIS papers viewed through the lens of WCAG

Abstract

Purpose: Some universal accessibility practitioners have voiced that they experience a mismatch in the research focus and the need for knowledge within specialized problem domains. This study thus set out to identify the balance of research into the main areas of accessibility, the impact of this research, and how the research profile varies over time and across geographical regions. Method: All UAIS papers indexed in Scopus were analysed using bibliometric methods. The WCAG taxonomy of accessibility was used for the analysis, namely perceivable, operable, and understandable. Results: The results confirm the expectation that research into visual impairment has received more attention than papers addressing operable and understandable. Although papers focussing on understandable made up the smallest group, papers in this group attracted more citations. Funded research attracted fewer citations than research without funding. The breakdown of research efforts appears consistent over time and across different geographical regions. Researchers in Europe and North America have been active throughout the last two decades, while Southeast Asia, Latin America, and Middle East became active in during the last five years. There is also seemingly a growing trend of out-of-scope papers. Conclusions: Based on the findings, several recommendations are proposed to the UAIS editorial board.

1 Introduction

To celebrate the twentieth anniversary of the Universal Access in the Information Society (UAIS) journal, this special issue paper takes a retrospective and introspective look at the research published in UAIS during its lifetime. This study was triggered by several anecdotal episodes experienced first-hand by the author in which practitioners working on universal accessibility commented on the imbalance and mismatch between the accessibility research presented at academic venues and the challenges faced in the practice fields. The commonly expressed view that research on visual impairment is overrepresented gave rise to the hypothesis that there may be a gap between the conducted accessibility research and practitioners’ need for new knowledge. Eyeballing accessibility conference programmes may seem to confirm this suspicion. However, to gather hard facts, this study set out to collect empirical evidence about the representation of various types of accessibility research.

Bibliometric methods have become an accepted methodology for assessing research efforts, quality, and impact. Kaye argues that it can be healthy to occasionally take an introspective look at a research field [1]. There are already quite a few bibliometric studies that have explored various aspects of human factors [2, 3], and human–computer interaction [4, 5]. A handful of recent studies have also applied bibliometric methods to analyse accessibility such as accessibility in transport [6], attitudes towards disability in education [7], and Web accessibility [8, 9].

Terminology and keywords are essential ingredients in bibliometric analyses as they provide insight into trends, patterns, and relationships (see for instance [5]). Researchers within the field of universal accessibility have a particular responsibility to use suitable terminology that does not discriminate, exclude, or demean certain, often vulnerable, user groups. Several guides give researchers valuable writing advice [10, 11]. The ACM endorsed writing by Hanson et al. [10] recommends that descriptions of participants with disabilities or ageing should be respectful. They argue that descriptions should be “explicit in describing the key characteristics relevant to the research questions”. Others argue for shifting the focus towards describing the relation between users and the environment [12]. The need to be sufficiently explicit in scientific studies collides with the notion of universal as used in areas of universal design where the goal is not to focus on specific disabilities to avoid stigma. This dichotomy between the universal and the specific can be illustrated by the legislature in Norway which at a high level defines that all ICT systems are to be universally designed (for all). In a sense, the legislature serves as a societal strategy. Yet, the practical implementation of the law is outlined in the regulations which refer to the Web Content Accessibility Guidelines (WCAG) [13], which is organized according to disability categories.

WCAG, which is probably one of the most widely and actively used documents for universal accessibility, utilizes terminology quite pragmatically. Its main sections define a taxonomy highly relevant to interactive computer systems and human–computer interaction models, i.e. perceivable, operable, and understandable. Perceivable deals with issues related to output issues, operable related to input issues, and understandable related to human cognition in context of the user interface. Although the fourth WCAG category, robust, is of high practical importance, it was not included in this study as it addresses issues related to the technical interoperability of different assistive technologies. Moreover, only 11 of the UAIS papers contained the keyword robust.

Categories similar to those in WCAG can also be found in the ETSI Accessibility guidelines for ICT products and services. Commonalities of these categories are that they are respectful and do not focus on the individual weaknesses of humans. Yet, in the detailed WCAG description there are references to classes of disabilities such as blindness and low vision, deafness and hearing loss, limited movement, speech disabilities, and photosensitivity. WCAG does not refer to specific disabilities in medical terms. It is thus of interest to gain insight into how universal accessibility researchers use technology. To what degree is the terminology used generally and specifically? To what degree is inclusive versus exclusive language used? Also, it is relevant to explore the use of respectful terms in context of an international community of non-native English-speaking researchers.

Although WCAG is specifically designed for the Web, the general principles of perceivable, operable, and understandable apply generally to all types of interfaces, even outside the digital domain. These categories are therefore also considered suitable to UAIS papers that do not address the Web. Examples include studies that address traditional assistive devices [14, 15], emerging assistive technologies [16, 17], self-service kiosks [18, 19], ensuring sufficient contrast regardless of where text is presented [20, 21], or accessibility tools [22].

Bibliometrics is possibly one of the few pragmatic approaches for exploring the impact of, or interest in research, through citation counts. Previous bibliometric studies on accessibility have ranked papers according to citation impact [7, 9]. Such information can be useful in identifying specific important works, or prolific authors. However, perhaps a relative comparison of various subfields is more useful. Is the interest in the published papers aligned with the quantity of published papers? Or are there discrepancies where some areas with few papers receive more attention? If we assume that it is easier to conduct scientific inquiry into low vision than reduced cognitive function, we may expect to find more papers on low vision. Yet, the demand for knowledge about reduced cognitive function may be larger. Such a result would support the claim that there are imbalances in the supply and demand for specific knowledge areas of accessibility.

There are recurring debates about whether research should be free, and curiosity driven (bottom-up), or governed and driven by practical and societal needs (top-down) [23, 24]. Funding mechanisms are sometimes used to guide the direction of research. Arguments in favour of top-down driven research include the ability to address established accessibility challenges in society and to consolidate complementing expertise in different institutions and countries. However, top-down driven research may be less capable of responding to rapidly emerging challenges, may be less innovative and ground-breaking, and less competitive. It is thus relevant to ask how the quality, novelty, and market availability of research that is controlled through funding calls, such as those of the European Union and the National research councils, compares to the quality, novelty, and market availability of curiosity-driven research. More specifically, what are the observable differences in impact between UAIS papers that document funded research versus those that report research without funding?

Universal accessibility in the digital domain is a global goal supported by the UN Convention on the rights of People with Disabilities. Still, the availability of resources, legislative push, and timeline for change vary in different geographical regions. It is therefore of interest to probe the global universal access research situation. Although some accessibility research results have global relevance, one should not underestimate the importance of research within local and regional contexts, for example, research connected to (sign) language and readability [22]. Bibliometric methods have been used to identify trends in different geographical regions. In the field of human–computer interaction, researchers have explored the situations in Southeast Asia [25], Brazil [26], India [27, 28], Australia [29], New Zealand [30], and the UK [31].

The following research questions (RQs) were therefore articulated:

• RQ1: What are the research intensities of different disability categories?

• RQ2: Has the research focus changed over time?

• RQ3: How do accessibility researchers use terminology?

• RQ4: What is the impact of different accessibility research areas?

• RQ5: Does funding affect accessibility research impact?

• RQ6: Does the accessibility research focus differ across geographical regions?

2 Related work

Ahmi and Mohamad [8] conducted a bibliometric study of student dissertations on Web accessibility published from 2013 to 2017. Their study was based on 39 dissertations obtained via Google Scholar, of which 19 were PhD-dissertations, 15 master dissertations, 3 bachelor dissertations, and 2 other types of dissertations. Descriptive statistics about where and at which institutions the works were conducted were provided.

A more recent and more comprehensive study of Web accessibility by the same authors [9] included 1103 publication records extracted from Scopus. The authors found that there has been a steady growth in Web accessibility research over time. They listed the most frequent terms including Web accessibility, accessibility, usability, WCAG, disability, and Internet. No specific terms related to perceivable, operable, and understandable (such as visual impairment) were reported. The authors also mapped the contributions of papers from different countries, where the USA was the most frequent contributor of Web accessibility papers, followed by the UK, Spain, Brazil, and Portugal. Japan was ranked as the 9th most frequent contributor of Web accessibility papers, China the 15th, and Taiwan the 24th most active contributor. European countries were listed individually, but collectively would have been ranked close to the USA if presented as one region. There were also a handful of papers from the African continent, that is, 5 from South Africa, 2 from each of Egypt and Ghana, and 1 from each of Algeria, Nigeria, and Tunisia. The authors presented a list of top cited papers on Web accessibility which were cited 55 times or more. Although UAIS is listed in Scopus, no UAIS papers were mentioned, indicating that UAIS papers were not included in this study as several UAIS papers have more than 55 citations in Scopus.

A bibliometric study of accessibility research within transportation [6] found that there has been a steady growth in publications over the past two decades. They showed that there is a citation lag of several years for a paper to attract citations. Their geographical analysis showed that the USA, Europe, and China were the most frequent contributors of papers. Nearly 8000 keywords were extracted and used to find six clusters addressing social exclusion, the built environment, travel behaviour, accessibility indicators, investment, and travel system performance evaluation.

Garcia-Fernandez et al. [7] used bibliometric methods to study attitudes towards disability in education using 925 Web of Knowledge (Social Science Index) records from the year of 2000 through to 2011. They found a noticeable growth in publications on the topic. Their study focussed on general topics and did not refer to specific disabilities. The authors also provided a list of the most prolific authors in terms of citations.

Bibliometric methods have been used to answer specific questions such as whether productive researchers receive higher pay [32] and effectiveness of incentive systems [33]. Sakamoto [25] used bibliometric methods to document the absence and presence of research from Southeast Asian researchers along a timeline. There has also been some interest in the assessment of the effects of funding investments on research [34]. A key question is to what extent the quality of research is affected by funding. Jung et al. [35] argued that the publishing of research in highly competitive journals correlates with the amount of research funding. They also observed that funding is more crucial in science and technology compared to social sciences due to the infrastructure and facility requirements. Amara et al. [36] collected empirical data showing that authors with low publishing activity are more likely of being affiliated with a top university and being the recipient of funding from national research councils compared to authors that do not publish. The difference between low performing scholars and highly active and cited scholars was connected to being full professor, having sufficient time to conduct research, as well as being affiliated with a top university and being recipients of funding. Jain et al. [37] reported positive effects of funding within chemical sciences, especially in terms of number of PhD degrees awarded. Chemical sciences are indeed highly depended on infrastructure and facilities. Lynch et al. found that commercially funded research was more likely to pass peer-review publication processes [38]. Svider et al. [39] correlated health researchers’ h-index and funding and explained that funding can predict a high h-index. However, in competitive systems a high h-index may be a prerequisite to get funding. Jacob and Lefgren [40] compared the productivity of health researchers with and without funding. They found that funding led to only one additional publication over a five-year period.

Co-authored papers have been connected to higher citation counts [41] than single-authored papers. A large diverse team of co-authors possess more collective competences than a single author. It follows that funded research consortia collaborating have a larger chance of producing quality research that generates citations than single researchers. Bu et al. [42] contended that research collaborations pose the potential of making more efficient use of limited resources including time, competences, funding, and equipment.

This study attempted to go deeper than the previous bibliometric studies on accessibility with the goal to obtain insight into the profile of accessibility research and how it has changed over time and across different geographical regions.

3 Method

To solicit answers to the research questions, bibliometric methods were chosen based on papers published in the journal Universal Access in the Information Society (UAIS). UAIS has the longest longevity of the handful of journals that specifically address universal accessibility of information technology. The 20-year history of the journal was deemed to hold promise as a source for exploring changes in archival research over time, for a field that emerged in the early 1990s [43]. Its international profile also made it suitable for exploring universal accessibility research from geographical perspectives.

3.1 Material

Scopus was used to extract UAIS publication records as this database has been useful in other bibliometric studies [9]. A total of 730 records were retrieved, from 2003 to the time of writing (2020) as a Comma Separated Value (CSV)-file with one record per row (see Online Resource 1). Web of Knowledge was also considered, but records only went back to 2010, the time Web of Knowledge started indexing UAIS papers. SpringerLink did not provide the same richness of data as Scopus, and the HCI bibliography [44] was discontinued in 2018.

Each record included author info, affiliations, title, abstract, keyword info, paper type, publication date, number of citations, as well as several other details such as Document Object Identifiers (DOIs). Scopus is believed to have a wider range of citation data compared to Web of Knowledge since it also includes citations to and from papers published in conference proceedings. Google Scholar is believed to have the widest citation range [45, 46], though Google Scholar does not allow records to be retrieved using automatic tools. The data were collected on 30 July 2020, and results for 2020 are therefore only partial. An additional Scopus search for funding was conducted on 19 December 2020. The number of UAIS articles had then grown to 768 records. Hence, the analysis related to funding was based on additional 38 records (see Online Resources 2 and 3).

To verify if the trends for UAIS papers generalize to the accessibility field, a similar comparison was made with what was considered the most related journals and conference indexed by Scopus. These publication channels included ASSETS—the International ACM SIGACCESS Conference on Computers and Accessibility (1342 papers) and the journals Disability and Rehabilitation: Assistive Technology (1254 papers), Technology and Disability (854 papers), Assistive Technology (735 papers), Journal of Assistive Technologies (277 papers), and Journal of disability and design for all (43 papers). Although influential conferences such as ICCHP (International Conference on Computers Helping People with Special Needs) and UAHCI (International Conference on Universal Access in Human–Computer Interaction) are indexed by Scopus, the isolation of these records was difficult as these all appear in the Springer Lecture Notes in Computer Science series. Statistics were obtained using the same keyword set as for the UAIS analysis. These Scopus searches were also conducted on 19 December 2020 (see Online Resources 4–9). Although other conferences and journals regularly publish papers related to accessibility, their main emphasis is typically general human computer interaction, not accessibility.

3.2 Classification

The WCAG taxonomy of functioning was used for organizing the publications, that is, papers that addressed issues related to perceivable, operable, and understandable. Perceivable deals with accessibility issues related to reduced sensory function such a low vision and hearing impairment. Operable refers to issues related to reduced motor function such as lack of limb, stiffness, or tremors. Understandable addresses issues related to reduced cognition.

In addition, a general accessibility category was defined to include other issues related to disability, inclusion, exclusion, and social participation that were not captured by the WCAG taxonomy. The final main category, outside (journal) scope, was defined as papers not belonging to either disability issues specifically or accessibility generally.

An Excel framework was configured to allow real-time search and iterative exploration of the keywords used in the title, abstract, and keyword sections. To classify a paper as addressing perceivable, it had to contain one of several specific keywords related to sensory issues such as blind, screen reader, and deaf. Similarly, papers were considered related to operable if they contained keywords such as input, mouse, and keyboard. Papers with keywords including cognitive, attention, and dyslexia were classified as dealing with issues related to understandable. Finally, papers with keywords including accessibility, inclusion, and exclusion were classified as dealing with general universal accessibility. A paper could belong to several categories; for instance, a paper could deal with both visual impairment (perceivable) and memory load (understandable). Such papers were therefore counted multiple times according to each of the identified categories. Table 1 in Results section lists the keywords used.

Table 1 Search terms used for the paper classification with frequency of occurrence. No occurrences were found for the lower set of terms

The keywords used in the first iteration were based on keywords found in key papers related to disability and accessibility. Several iterations of the process were conducted, and the generated classes of papers were inspected to assess the correctness of the classification. Papers with seemingly incorrect classification were analysed to identify additional keywords. The new keywords were included, and the process repeated.

3.3 Analysis

A fundamental assumption of this study was that publication counts mirror research activity. The JASP statistical software [47] was used for conducting statistical test to assess differences between groups. Shapiro–Wilks tests and Q–Q plots were used to assess if the observations were normally distributed. Nonparametric tests were used for data that were not normally distributed, namely Mann–Whitney test for groups of two, and Kruskal–Wallis tests were used for comparing groups of three or more.

When assessing citation counts for specific groups, two measures were explored, namely the median citation counts and the h-index. The median citation count gives an indication of the general level of which a paper in the given group is cited. This robust measure of centrality was used as citation observations contain large outliers where a few papers are cited much more (exponentially more) than other papers, and the citation distribution is heavily skewed due to the dominance of low citation counts.

The h-index is widely used to assess impact and gives an indication about the quantity and magnitude of highly cited papers [48, 49]. While the median is computed based on all the papers in the set, the h-index is only based on the most cited papers, i.e. an author with a h-index of H means that H papers by the author have been cited at least H times.

Geographical analyses were conducted by grouping papers from Europe (EU and EEU countries), Middle East (including Turkey, India, and Pakistan), Southeast Asia, Oceania (Australia and New Zealand), Africa, North America (USA and Canada), and Latin America. In cases of paper with authors from multiple countries, the country of the last author was used as a pragmatic simplification.

3.4 Ethics

The data used in this study are public and do not fall under the regulation of the General Data Protection Regulations. Mostly aggregated results are presented so as not to draw attention towards individual researchers and institutions. Also, the comparisons are not performed with the purpose of competition but rather with the goal of working towards narrowing gap as universal accessibility should benefit all. The raw data used for the analyses are available as supplementary files (see Online Resources 1–9).

4 Results

4.1 What are the research intensities of different disability categories?

Figure 1 shows the results of the publication counts for UAIS papers according to disability categories at four different aggregation levels. First, the results showed that 86.5% of the papers were within the scope of universal accessibility, while 14.5% of the papers (106 in total) were out of scope. Examples of papers considered outside the scope of universal accessibility include “Context awareness in healthcare: a systematic literature review”, “Incorporating digital games into anti-drug material: non-drug-addicted learners vs. drug-addicted learners”, “Investigating the effects of ubiquitous self-organized learning and learners as designers to improve students’ learning performance, academic motivation, and engagement in a cloud course”, “Distributed Web browsing: supporting frequent uses and opportunistic requirements”, to mention a few.

Fig. 1

Number of UAIS papers (2003–2020) according to disability categories at four levels of aggregation

Next, Fig. 1 shows the breakdown of papers that were within scope, divided into papers exclusively addressing issues related to specific disabilities (WCAG), papers exclusively addressing general accessibility issues not focussing on a specific disability (other accessibility), and those that addresses both (WCAG, other accessibility). Clearly, most of the papers included both general and specific issues, followed by general accessibility issues. Papers addressing specific disabilities comprised the smallest group. The out-of-scope category and the three inside-scope categories were all mutually exclusive.

Figure 1 also lists the number of papers addressing issues related to perceivable, operable, and understandable. Papers addressing perceivable constituted the largest group, followed by operable, while understandable constituted the smallest group. The perceivable group was nearly twice as large as the two other groups, while the operable group was only marginally larger than the understandable group. These three categories were not mutually exclusive as a paper could be counted in more than one category, i.e. addressing both perceivable and understandable, or any other combination. Overlaps were as follows: 38 papers (5.3%) addressed perceivable and understandable, 69 papers (9.4%) addressed understandable and operable, and 29 papers (4.0%) addressed operable and understandable.

The largest group, perceivable, was further subdivided into types of reduced sensory function. Papers addressing reduced vision comprised the largest group and nearly three times as large as the group of papers addressing hearing and touch. Papers addressing hearing and touch were of similar quantities. Only one paper addressed smell and none of the papers addressed taste. Again, the sensory groups were not mutually exclusive as some papers addressed multiple senses.

Comparisons with related publication channels are shown in Figs. 2, 3. Figure 2 reveals a distinct trend across the publication venues: the understandable category made up the largest category with close to 50% of the papers, followed by operable, with understandable as the smallest category. The differences were statistically significant (F(2, 18) = 14.4, p < 0.001, η² = 0.617). Tukey post hoc tests confirmed that perceivable was different to both operable (p = 0.002) and understandable (p < 0.001), while the difference between understandable and operable was not significantly different. Only the Journal of Assistive Technologies flagged more papers in the understandable category than the operable category. Technology and disability had the most balanced distribution of papers across the three categories followed by UAIS. ASSETS had the largest imbalance across the three categories.

Fig. 2

Percentage of papers in the main disability categories (in relation to the total) compared across publication channels

Fig. 3

Percentage of papers in the perceivable category addressing vision, hearing, and haptics compared across publication channels

Similarly, Fig. 3 reveals that papers in the perceivable category across all the publication channels were dominated by papers addressing visual issues (between 62.5 and 75.9%). ASSETS had the largest proportion of papers addressing visual issues, while Assistive Technologies had the least. Hearing was the least addressed issue with just 6.4% of the perceivable category papers in Journal of assistive technologies. UAIS had the largest portion (23.3%) of hearing-related papers in the perceivable category. The percentage of papers addressing haptics varied the most across the publication channels. Approximately every third paper in the perceivable category were related to haptics in ASSETS and the Journal of Disability and Design for All, while less than 10% of perceivable papers in Technology and Disability and Disability and Rehabilitation: Assistive Technology addressed haptics. Note that papers could be categorized as addressing several sensory issues simultaneously. The portions of sensory papers were statistically significant (F(2, 18) = 131.9, p < 0.001, η² = 0.938). Post hoc tests confirmed that the portion of vison papers was significantly different to the portion of papers addressing hearing (p < 0.001) and haptics (p < 0.001). There was no significant difference between the proportions of hearing-related papers and papers addressing haptics. Smell and taste were omitted in this comparison as only one paper related to smell (in ASSETS) and one paper related to taste (in Disability and rehabilitation: Assistive Technology) were identified.

4.2 Has the research focus changed over time?

Figure 4 shows the distribution of papers inside and out of scope over time. Percentages are normalized with respect to the total number of publications in each of the two groups, respectively. The plot reveals that most of the out-of-scope papers appeared between 2015 and 2020, while the inside-scope papers appeared throughout the lifetime of the journal with a constant yearly growth. A Mann–Whitney test confirmed that the out-of-scope papers appeared more recently than the inside-scope papers (U = 29,104.0, p = 0.048).

Fig. 4

UAIS papers inside scope and out of scope (normalized according to the total of each group)

Figure 5 shows the breakdown of papers addressing perceivable, operable, and understandable from the years 2003 to 2020. The approximate 2:1:1 proportion has remained nearly constant while the yearly quantity of papers has grown steadily over time. A Kruskal–Wallis test showed that there was no significant difference in time of publication for the three types of reduced function (χ²(2) = 0.918, p = 0.632). Similarly, no significant differences were found for the time of publication across the three perceivable categories (χ²(2) = 3.813, p = 0.149), that is, vision, hearing, and touch.

Fig. 5

Research effort on different disability categories over time

4.3 How do accessibility researchers use terminology?

Table 1 lists the terminology used for classifying the UAIS papers. The upper part of the table shows the terms found with the frequency of occurrence, while the bottom part lists terms that gave no hits among the UAIS papers.

The results show that terms related to specific disabilities in the operable, understandable, and perceivable columns tended to be general and colloquial including blind, vision impaired, deaf, input, cognitive, attention, etc. Other more specific terms (e.g., tunnel vision), terms often occurring in WCAG (such as timeout and reflow), and many medically specific terms were not found, including arthritis, muscular dystrophy, dyscalculia, prosopagnosia, photosensitive (epilepsy), cataract, and glaucoma.

It is also interesting to observe the presence of the terms handicap (25 papers) and able bodied (8 papers) that by many are considered inappropriate or outdated. The results revealed occurrences of both US and UK spelling of the word colour in the metainformation.

4.4 What is the impact of different accessibility research areas?

Figure 6 shows the median number of citations at the four levels of aggregation. According to the medians, a typical UAIS paper received 5 citations. Papers within scope received nearly twice as many citations (median = 5) than out of scope papers (median = 3), and a Mann–Whitney test revealed that the difference was statistically significant (U = 40,189.5, p < 0.001). Papers within scope that addressed understandable received most citations (median = 6) and perceivable the smallest number of citations (median = 4). However, a Kruskal–Wallis test did not show any significant differences in citations across the three categories of reduced functioning (χ²(2) = 0.144, p = 0.931).

Fig. 6

Median number of citations at different levels of aggregation. Error bars show 95% confidence intervals according to the median

Among the papers addressing issues related to perceivable, papers dealing with vision received most citations (median = 4.5), followed by touch (median = 4) and hearing (median = 3.5). A Kruskal–Wallis test did not flag any significant differences in citations across the three categories of reduced functioning (χ²(2) = 5.284, p = 0.071).

Figure 7 shows the h-index for the groups at the four levels of aggregation. The overall h-index of the UAIS journal was 41, and the most cited paper [50] had 388 citations. The h-index for papers within scope was nearly twice as high (h-index = 38) as for out of scope (h-index = 20). Of the papers within scope, papers addressing perceivable had the largest impact (h-index = 26), followed by understandable (h-index = 23) and operable (h-index = 19). Of the papers addressing perceivable, those tackling reduced vision had the highest impact (h-index = 23), followed by hearing (h-index = 13) and touch (h-index = 9).

Fig. 7

The h-index at different levels of aggregation

4.5 Does funding affect accessibility research impact?

Scopus data revealed that a total of 73 UAIS papers (9.5%) had received funding and 695 had no funding information (90.5%). Funding agencies included the Engineering and Physical Sciences Research Council (13 papers), European Commission (12 papers), European Regional Development Fund (11 papers), Seventh Framework Programme (11 papers), Ministry of Science and Technology, Taiwan (9 papers), Horizon 2020 Framework Programme (7 papers), Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (6 papers), National Science Foundation (papers 6), Conselho Nacional de Desenvolvimento Científico e Tecnológico (4 papers), and the European Social Fund (4 papers).

Papers documenting research without funding had a higher mean number of citations (M = 12.7, SD = 28.1), compared to papers documenting funded research (M = 8.4, SD = 12.5), and the corresponding median number of citations was 5 and 4, respectively. The set of funded papers had an h-index of 14 while the set of papers without funding had an h-index of 41. A Mann–Whitney U test confirmed that the difference in number of citations between papers reporting funded research and research without funding were significant (W = 22,260.5, p = 0.042). The first UAIS paper with registered funding [51] appeared in 2004. However, the data reveal that nearly a third (31.5%) of the papers with funding appeared in 2020.

4.6 Does the accessibility research focus differ across geographical regions?

The majority (60%) of the papers were contributed by European authors, followed by North America (15.6%) and Southeast Asia (11.5%). Middle East, Latin America, Oceania, and Africa each comprised less than 6% of the papers (see Fig. 8).

Fig. 8

Paper distribution according to geographical regions

Figure 9 shows the normalized thematic research profile of the seven geographical regions in terms of UAIS papers addressing perceivable, operable, and understandable, general accessibility, and non-accessibility contributions. A common trend is that the efforts related to perceivable were more intense than the efforts related to operable and understandable in all the regions, and operable issues were slightly more researched than understandable. Although following a similar pattern, Southeast Asia was the region with the most similar effort across these three themes. Middle East stood out as the region with the lowest ratio of papers related to understandable. Europe and North America, the most frequent contributors of papers, exhibited very similar ratios of papers addressing perceivable, operable, and understandable.

Fig. 9

Disability research focus in different geographical regions (normalized as percentage of total number of papers in each region)

The ratio of papers addressing general accessibility was similar for all the geographical regions (just under 40%), except for Oceania where about two thirds of the papers addressed general accessibility. However, this geographical region was represented by only 13 papers, which is too small to reliably map the research efforts.

Southeast Asia exhibited most papers out of scope (about 20%) closely followed by Latin America and Europe. North America, the second most frequent contributor of papers, also had a much lower ratio of papers out of scope (less than 10%).

Figure 10 shows the number of papers from different geographical regions over the lifetime of UAIS. Africa and Oceania were omitted due to the small number of papers. Europe was the most active contributor of papers throughout the lifetime of the journal with a constant growth throughout the two decades. North America has been a consistent contributor of papers at a relatively stable level.

Fig. 10

Regional breakdown of UAIS papers over time

Southeast Asia, Middle East, and Latin America have emerged with considerable publication activities during the last four years. The difference in time of publication for different regions was also statistically significant (χ²(6) = 110.441, p < 0.001), where the mean for Latin America is 2017 and the means for Africa, Middle East, and Southeast Asia were 2016, 2017, and 2016, while the means for Europe, North America, and Oceania were 2013, 2011, and 2010, respectively. Dunn’s post hoc tests confirm the significant differences between the established and emerging countries. In fact, since 2017 there have been more papers from Southeast Asia and Middle East than North America.

Note that there was a dip for 2020 as the data were collected during July 2020. The final tally for 2020 is thus likely to be nearly twice as large as the statistics shown if one assumes papers are published at a similar rate throughout 2020.

Figure 11 shows the median number of citations for different geographical regions. A significant difference was found across the groups (χ²(6) = 37.666, p < 0.001). Papers by North American authors generated more citations than papers by European authors even though European authors contributed nearly three times as many papers as North American authors. Dunn’s post hoc tests confirmed that papers from North America had assimilated more citations than all the other regions. Papers by Oceania came in second despite having contributed the 2nd fewest papers. Europe, the largest contributor of papers, came in the third place with a median of 5 citations per paper. Southeast Asia, the third most frequent contributor of papers, ranked second last with a median of two citations per paper. The difference between Southeast Asia and Europe was statistically significant.

Fig. 11

Median citation counts of the geographical regions

5 Discussion

5.1 What are the research intensities of different disability categories?

It is interesting to observe that 14.5% of the UAIS papers seemed to have addressed issues not directly related to universal accessibility. Inspecting some of the paper titles reveals that they addressed various general topics such as context awareness, learning, and health care. One possible cause for the high ratio of unrelated papers could be the many special issues or special Sects. (44 in total). It is possible that some guest editors may not have practised sufficiently strict scope checks. For example, a special issue such as “Distributed user interfaces: distributing interactions to facilitate universal access” could attract papers that both address accessibility and papers only addressing distributed user interfaces. Some of the special issues seem not to be related to universal accessibility, for example “Interactive and collaborative technological ecosystems for improving academic motivation and engagement”. It is unlikely that seemingly out-of-scope special issue calls will attract within scope papers.

Are these out-of-scope papers beneficial or disadvantageous? The citation results may provide one answer. Out-of-scope papers were cited significantly less than inside-scope papers. There are predominantly two ways UAIS papers get discovered and cited, either through database keyword searches or by systematically following the accessibility field. Clearly, researchers that belong to the latter group are likely to be accessibility researchers and hence will be more interested in inside-scope papers than out-of-scope papers. Out-of-scope papers are probably only cited when discovered through database searches. One could argue that practising a more thorough scope check of UAIS submissions may help shape the profile of the UAIS journal to become an even more distinct source of quality research on universal accessibility. The results also show that most out-of-scope papers emerged in recent years. Could it be that the editorial checks for scope have been relaxed recently?

The results confirmed the expectations that most of the papers addressed perceivable issues. One may speculate why most of the research was related to perceivable. Does the research effort invested in an issue correlate with its prevalence, i.e. that there are more users with reduced sensory function than users with reduced motor function or reduced cognitive function? Clearly, most individuals experience reduced sensory function with age, but age-related reduction in function also applies to motor functions and cognitive functions. Also, certain types of cognitive issues such as dyslexia are highly prevalent, i.e. nearly one in every ten persons. Therefore, unfilled needs for knowledge do not seem to be a convincing argument to explain the dominance of research addressing issues related to perceivable. Could it be that the dominance of research into perceivable issues is simply pragmatic or opportunistic? One may speculate that it is easier to approach problems related to vision as it is tangible and observable, while research into understandable is harder as the cognitive mechanisms are hidden from direct view, i.e. we are unable to easily observe the inner workings of cognitive processes. Such an explanation answers why there is more effort devoted to perceivable compared to understandable. But, what about the difference between operable and perceivable? Clearly, issues related to operable are highly observable and tangible. One possible explanation could be that input has not changed as much as output over the recent decades. Most input is still performed with keyboards (and switches) and pointing devices such as mice (and eye-trackers). The basic principles of keyboards, switches, and pointing devices have seemingly not changed much over time. Output paradigms on the other hand have changed regularly, and such paradigm shifts in visual interfaces have brought new accessibility challenges that have attracted the attention of accessibility researchers.

Another explanation for the low number of papers addressing operable issues could be that researchers specializing on these issues favour other publication channels than UAIS. Could it be that technical journals with higher citation impact than UAIS accept such papers and therefore become a more desirable choice? Or, are issues of operable more frequently associated with the physical world as opposed to the digital world? UAIS is clearly scoped by the digital domain as it concerns the “information society”, while researchers working on wheelchairs, orthoses, and other physical or physical–digital assistive technology may see specific journals focussing on operable issues as more relevant? UAIS could be defined as a highly multi-disciplinary journal, but could this multi-disciplinarity be viewed as too general for researchers with a narrow and specialized focus? The results of the comparisons with related publication channels presented in Sect. 4.1 suggest that these speculations may be rejected as the dominance of perceivable issues was observed in all the publication channels analysed, even within journals with an explicit assistive technology focus.

Among the works related to perceivable, vision is the largest group across all the publication channels including UAIS. Clearly, research into computer output, particularly for users with reduced vision, is a key area. The dominance of research related to vision could be attributed to the importance of visual communication in computer systems, while other senses such as audio, touch, smell, and taste are comparatively less used with computer output. Perhaps one exception is the key role of audio in multimedia, video, and communication applications. Still, audio was also ranked the second most frequently researched modality.

One could perhaps have expected to see an intensified research effort into operable issues during the last decade with recent paradigm shifts in input technologies, in particular input relying on touch-based smartphone pointing, gestures, eye tracking, speech, or emerging paradigms such as brain–computer interfaces. The results show that the profile of research intensity into different types of accessibility has not changed noticeably over the last 20 years, and there were no obvious correlations with recent technological advances in input technologies.

These results may also be connected to the recurring debate about whether research should be free, and curiosity driven (bottom-up), or governed and driven by practical and societal needs (top-down). From such a standpoint, one should not expect the profile of published research to fully align with the need for new knowledge. Comparisons of funded versus unfunded research in terms of citation impact are provided in Sect. 4.5. However, the number of UAIS papers documenting funded research is too low to warrant a reliable thematic comparison.

5.2 Has the research focus changed over time?

The observed growth in papers over time is as expected because a journal typically attracts more attention once it gains recognition. Growth in publications related to accessibility has also been reported in other studies [6, 7, 9]. This suggests that there are other factors at play, most notably an increased attention among researchers as governments have intensified their efforts through funding and/or legislature. However, the results did not reveal any noticeable changes in research focus over time. For instance, it was not possible to observe any connections between technological paradigm shifts, such as touch interaction and smartphones, and shifts in research focus.

5.3 How do accessibility researchers use terminology?

Publication records showed that accessibility researchers tended to use relatively colloquial and general umbrella terms such as visual impairment to describe user groups with various disabilities. The use of precise and even quite well-known medical terms was close to non-existent. On the one hand, a general and more colloquial term makes papers easier to read and disseminate more widely, which is useful within the multi-disciplinary field of accessibility. However, a drawback of the colloquial terms could be a lack of precision when describing users and their context, which may result in misleading generalizations. For instance, many papers refer to visual impairment and visually impaired users. This is indeed a very large group, and it is quite surprising not to find works addressing tunnel vision which manifests itself completely different to low vision caused by reduced visual acuity, as a person with tunnel vision may perceive perfect visual acuity in a small sector of the field of view (note that tunnel vision can also be considered a colloquial umbrella term). As pointed out by Hanson et al. [10], the term visual impairment does not give sufficient context in scientific writing even though it is a well-known and widely used term. Hanson et al. recommended that additional contextual details should be provided. Given the dominance of papers related to reduced vision in the journal, one may have expected a higher level of precision than what can be deciphered from the keywords. However, in defence of many UAIS papers using the term visual impairment, such papers often provide additional contextual information in the methodological descriptions such as whether the users use screen readers or use magnifiers.

There also seems to exist a methodological dilemma among accessibility researchers. On the one hand, the philosophy of universal design is not to focus on specific disabilities but rather focus on all citizens. A goal has been to distance oneself from the medical tradition of focussing on diseases and weaknesses of individuals. If subscribing to the philosophy of universal design, it is indeed a positive attribute that many accessibility researchers seemingly have diminished focus on the specifics of certain disabilities. On the other hand, one may argue that to achieve universal design for all one also needs to understand and study the phenomena surrounding special cases and specific disabilities. This may explain why some accessibility research focussed on specific disabilities.

It was somewhat surprising to find terms that are considered outdated or inappropriate in some of the papers, especially as there are several widely available general guides on inclusive language and how to write about disability specifically, such as Hanson et al.’s guide [10]. One may have expected that the review process would have picked up these issues giving the authors a chance to use more inclusive terminology. It may be more difficult for non-native English-speaking authors to distinguish between acceptable and unacceptable terms. In this regard, the reviewers and editorial staff are valuable resources for quality assurance.

5.4 What is the impact of different accessibility research areas?

There were no statistically significant differences in the number of citations for different types of functioning. Yet, looking at the practical differences the smallest group of papers addressing understandability had the largest mean number of citations. The largest group, namely papers addressing perceivable, had the fewest median number of citations. Hence, this could mean that research related to understandability gets more attention compared to both perceivable and operable, even though this group of papers was the smallest.

This anomaly was further explored by inspecting the h-index for these groups, as the papers addressing understandable had a higher h-index than papers addressing operable issues, even though there were more papers addressing operable issues and the papers addressing operable issues had a higher median number of citations. Hence, papers addressing understandable had a relatively higher impact than the papers addressing operable issues. Whether this is a result of exceptionally high-quality papers on issues related to understandable or whether there is a larger demand for, and interest in, papers addressing cognitive issues will merely remain a speculation.

This speculation may also be turned around. That is, do these results indicate a limited novelty in contribution of papers addressing perceivable and operable issues? Could it be that UAIS papers addressing perceivable and operable issues published over the last 20 years simply have reported variations on the same theme, or reported rediscoveries of ideas that are already known? Another explanation could be that the seminal works on these topics over the last 20 years have been published elsewhere and that UAIS have been unsuccessful in attracting the influential papers on perceivable and operable issues? However, the comparisons with related publication channels indicate that this imbalance is not unique to UAIS.

The significant difference in citations to papers inside scope and out of scope, where papers inside scope gained nearly twice as many citations, indicates that the papers published in UAIS were primarily used in other accessibility research. Perhaps more stringent scope checks on submissions may render UAIS even more relevant to the universal accessibility research community and consequently lead to increased citation impacts.

The citation statistics presented herein are probably moderate estimates due to the limited coverage of Scopus. For example, the most cited UAIS paper [50] was listed with 388 citations in Scopus, while Google Scholar lists this paper as having 734 citations. In this case, Google Scholar reports nearly twice as many citations.

5.5 Does funding affect accessibility research impact?

The observed lower citation impact of funded research support claims that top-down funded research is less innovative and ground-breaking than curiosity-driven bottom-up research. However, the results should be interpreted with caution. First, papers without funding information may indeed be the result of funded research that have not been successfully captured by Scopus. Second, it may not be the case that all the papers without funding information represent bottom-up or curiosity-driven research. It is indeed possible that some of the papers reporting research without funding are the result of top-down processes.

Next, the range of funding agencies is diverse and at national and international (EU) levels. One may also expect that the grant sizes, funding processes, and competitiveness vary considerably across different funding agencies and types of calls.

The total number of papers reporting funded research is small compared to the papers with no funding information. Moreover, most of these are published in recent years. These imbalances are likely to give unbalanced citations profiles as a larger set of papers will, from a statistical viewpoint, assimilate more citations than a smaller set. The funded papers published recently may not yet have assimilated citations due to citation lags [6].

It is relevant to ask how crucial funding is for accessibility research. Although it has been noted that funding is essential within science and technology [34], one could argue that accessibility research is not as dependent on costly facilities and laboratories. Accessibility studies often involve low-cost off-the-shelf equipment such as smartphones, tablets, and regular computers. The major bottleneck often seems to be the recruitment of participants from specific disability cohorts.

Although funding is considered a mechanism for guiding the direction of research according to some agenda, funding can also be framed to promote curiosity driven research [24]. A pragmatic position may be that both bottom-up accessibility research and top-down accessibility research are needed.

5.6 Does the accessibility research focus differ across geographical regions?

The geographical distribution of papers is as expected. Most of the contributions come from Europe and North America. Although UAIS is an international journal, it has a European publisher with two European editors-in-chief, while other related venues such as the ACM ASSETS conference are heavily US-centric in terms of published papers.

The low number of papers from African researchers agrees with Ahmi’s study of Web accessibility [9]. This result is unfortunate given Africa’s sizable population. It is likely explainable by socioeconomic factors [52].

The steady contribution of papers from North America may reflect its early focus on universal accessibility. There has also been a European focus on universal accessibility for several decades as revealed by the publication timeline. However, the increase over the years seems to suggest that the European focus on universal accessibility research has intensified over the last 20 years. This can probably be explained by the fact that the European category comprises many countries with different legislature and time schedules for regulating universal accessibility and that recent EU-level legislature has pushed most European countries to move towards regulation.

The recent emergence of research from Southeast Asia and the Middle East could indicate that there may be national events that have triggered an intensified effort on universal accessibility. Recent rises in Southeast Asian papers on human computer interaction have also been reported by Sakamoto [25]. However, researchers from Japan were known as early drivers of universal accessibility technology (see, for instance, Saito [53] and Ostroff [54]) although this is not obvious from the results presented herein. Eyeballing the papers in more detail reveals that there were only 7 papers authored by Japanese authors, of which one in 2003 and two in 2007. One explanation for the few contributions from Japan could be that Japanese researchers typically have published their research elsewhere. Another observation is that the UAIS editorial board only lists one scholar from Japan. Would more editorial board members from Japan help attract and engage more Japanese researchers with mature universal accessibility research portfolios?

Comparing Japan to other currently active countries, we find that authors from China contributed 12 papers (the first in 2011), South Korea contributed 8 papers (the first in 2016), and 37 papers were affiliated with Taiwan (the first in 2012). Clearly, Taiwanese authors contributed a majority of UAIS papers from Southeast Asia. It is also noteworthy that 12 of these papers were out of scope (nearly one third). The high portion of UAIS papers authored by Taiwanese authors does not match the findings for Web accessibility reported by Ami and Muhamad [9] who found both Japan and China as more frequent contributors with Taiwan in 24th place.

The dominant attention on perceivable issues was consistent across all the geographical regions. The most noticeable differences were observed for the ratios of papers in- and out of scope. Papers by North American authors were more likely to be inside scope among the active geographical regions. Perhaps North American academics are more conscious about strategically choosing publication channels that match the research contributions. Southeast Asia authors exhibited the highest ratio of out-of-scope papers. One may speculate as to why this is; could this trend be a side effect of an unhealthy publish-and-perish culture?

The results also show that papers from North America were the most highly cited. Is this an indication that the research conducted in North America has higher impact by being of better quality, especially when compared to European papers? European papers were greater in numbers and thereby statistically should have assimilated more citations. Another explanation could be that too many papers by European authors attracted too few citations due to low quality.

Papers by Southeast Asian authors have comparatively assimilated few citations although this region is the third most frequent contributor of papers to UAIS. A possible explanation could be that these papers have not generated enough interest in the research community. An alternative explanation, perhaps more convincing, could be that the papers from Southeast Asia were quite recent (from 2017 and onwards), and have therefore not yet had a chance to absorb citations due to citation lags [6]. In contrast, papers by North American authors published throughout the lifetime of the journal have had a chance to attract citations.

5.7 Limitations of this study

The results presented herein are based on an analysis of keywords, titles, abstracts, and author affiliations. Although the title, abstract, and keywords usually give a reasonable indication about the paper content, they do not necessarily always capture the gist of a paper accurately. Only the contents of a handful of papers were double checked. Consequently, there is a chance that some papers were misclassified, due to imprecise keywords and terminology in the titles and abstracts, leading to some false positives and false negatives. However, it is unlikely that misclassifications have significantly affected the results.

Most of the analyses presented herein are based on research published in the UAIS journal. Obviously, such results only give insight about the status of this journal specifically, and the results do not necessarily generalize to the field of accessibility. However, the comparisons of disability category proportions across related journals confirm that other publication channels exhibit similar trends as those observed for UAIS.

6 Conclusions

Bibliometric methods were used to analyse the papers that have been published in UAIS. The results support the claim that there is a gap between published research and knowledge needs. Most studies were published on visual impairment (perceivable), but these received the fewest citations, while understandable was covered by the smallest number of studies, yet with the highest number of citations. The distribution of research efforts appears to have remained stable over time and across different geographical regions. Based on the findings reported herein, the following recommendations are proposed to the UAIS editorial board:

• 1. Introduce more stringent scope checks for submitted papers and special issue proposals;

• 2. Initiate special issues dedicated to universal accessibility research in low-GDP-per-capita regions, to stimulate and support activity in these regions, and to help document the global accessibility situation (especially Africa);

• 3. Initiate special issues on the theme understandable to meet the knowledge needs;

• 4. Introduce a terminology policy, systematic checks, and a glossary to help authors from non-English speaking countries use inclusive, non-stigmatizing, and respectful language;

• 5. Invite more editorial board members from Japan to strengthen the quality of contributions from Southeast Asia.