1 Introduction and Motivation

Getting from ‘A to B’ is an important task in day-to-day life. It is a fundamental part of being integrated into wider society. For most people, this is something that can usually be taken for granted: the existing pedestrian paths are all accessible to them, and modern tools (e.g. Google Maps) operate seamlessly to plot an accurate route for them to travel to their desired destination. However, the same cannot be said for people with mobility impairments and other disabilities (e.g. people with visual impairments) that impact their movement within the built environment [8]. One group who are particularly disadvantaged in this regard are wheelchair users, who require a smoothly surfaced set of paths in order to safely drive a wheelchair. This is a substantial group of people with disabilities with a specific and well-defined set of accessibility needs, with there being 65 million people worldwide that are reliant on a wheelchair in their day-to-day lives [28]. This is more common in westernized countries: e.g. the Federal Statistical Office in Germany indicates that 1.5 million people use a wheelchair in Germany on a day-to-day basis [37], with a similar picture in the UK (1.2 million people [49]) and the USA (3.3 million people [48]).

In response, there has been a range of research aimed at documenting the built environment, so accessibility barriers can be identified (and thus be avoided or addressed). One aspect of this work focuses upon the accessibility of sidewalks, including the detection of surface quality for wheelchair users. These measurements have been made using both technical means such as depth cameras or acceleration sensors (e.g. [11, 13, 16, 17, 24]), or direct inspections, be they via crowdsourcing [45], or more formal inspections by experts (e.g. [46, 51]). These approaches either suffer from limited accuracy (i.e. the automated or crowdsourced approaches) or a paucity of coverage (in respect of the formal inspections by experts) [19].

Accordingly, there is an inevitable limitation in respect of a routing system for wheelchair users: it will always be operating on the basis of imperfect and incomplete information (and thereby make misdirection’s, see Fig. 1 for an example). An effective routing tool will be optimised to manage the risk that arises from this uncertainty. There have been nascent efforts to build routing systems for wheelchair users, including OpenRouteService [50], Routino [52], and Google has recently offered a separate prototype to serve wheelchair users [33]. There are also predecessors to these systems that were developed as academic prototypes (e.g. [4, 21, 26, 41]). Yet one limitation of these existing works is that they have not been subject to a real-world evaluation approach.

Fig. 1.
figure 1

A real example of a misdirection made by a wheelchair routing tool. On the left side of the street there is a narrow pavement with a horizontal slope (purple – diagonal strips). In addition, there is a street sign (yellow – horizontal strips) and a bin (red - squares) which hamper the use of this sidewalk. On the right side the narrow pavement has not only a horizontal slope but is also blocked by a bike (orange - full) and a car (pink – vertical strips). Even without the temporary barriers this sidewalk would not be usable for electric wheelchair drivers because of their narrow width. For manual wheelchair users it would only be possible with increased power and by travelling on the road itself (which itself presents substantial safety issues). (Color figure online)

Full size image

This paper makes two primary contributions. First, drawing upon both the literature concerning the experiences and needs of wheelchair users and separately the tool of cost-benefit analysis from economics, we develop a real-world evaluation approach for ‘A to B’ routing systems for wheelchair users. Second, we apply this framework to compare Google Maps – a pedestrian routing tool (and the most widely used mapping tool [30]) – with two different Wheelchair Routing systems in a real-world scenario. As with [42], our emphasis is upon the metrics and appropriate means for evaluation. We find that Google Maps is more effective than the existing Wheelchair Routing systems in identifying accessible routes for wheelchair users. By analysing how Google Maps is more effective, we are able to propose algorithmic and practical improvements to future ‘A to B’ systems for wheelchair users, as well as proposing new directions for research in respect of the documentation of accessibility barriers.

2 ‘A to B’: The Task of Wheelchair Routing

We frame the problem of wheelchair accessibility in respect of the ‘A to B’ problem, setting out the specific types of wheelchairs and wheelchair users, the effect of less than ideal driving conditions, and how this might in practice be mitigated. It must be emphasised that this is distinct to the at A or at B problem, which considers the accessibility of a location when someone arrives at it (e.g. does it have elevators to travel between floors?) and which is evaluated by a different set of tools (examples of which include Wheelmap [23] or Euan’s Guide [47]). The aim is to broadly frame and capture the problem in question, so that we can fully justify our proposed approach towards the identification and measurement of barriers.

2.1 Living with and Navigating Around Barriers in the Built Environment

In day-to-day life, an important task is getting from ‘A to B’, where A is one geographical location and B is another geographical location. This can be done using a broad range of transportation, including public transport or a private car, but in practice, there will be a portion of most trips that involve navigation of pedestrian routes of some kind (e.g. there are no roads leading to a location, a limited number of car parks and so forth). For a wheelchair user, those pedestrian routes may contain one or more obstacles: if they do, then they are either impassable, inconvenient or uncomfortable depending upon the person. The problem therefore is a matter of determining which route to take and how someone can know this in advance of traversing it, much as someone would use Google Maps. The distinction is that mapping software needs to take into account the barriers in determining which route to take and there is less of a possibility to ‘ask for directions’ (because most people are not wheelchair users and are therefore not familiar relevant with the barriers in the built environment). An example of a scenario involving a range of obstacles encountered after the failure of a routing system is provided in Fig. 1.

There are substantial practical effects upon the day-to-day lives of manual wheelchair users. The most notable recent work that directly considers ‘A to B’ routing systems is [12], which provides an extensive account of the way that Google Maps and other means are adapted by those with mobility impairments in order to assist with navigation (e.g. by performing ‘reccies’ to check routes in advance, or using Google Streetview to check a location): that work also proposed approaches for redesigning such systems. What is clear is that the uncertainty imposed upon people with mobility impairments – in respect of the accessibility of routes that are not already known to the person in question – has a substantial impact on their day-to-day lives, not only for the reasons given in [12], but also the wider socio-economic difficulties that often impact upon disabled people (e.g. a disproportionate number are on social security [22] and have low incomes, limiting the availability of alternatives such as taxis or more advanced mobility equipment).

2.2 Wheelchairs and Wheelchair Users

There are a multiplicity of underlying reasons why someone might be a wheelchair user. The common ground is that they are unable (or less able) to ambulate using their legs due to one or more impairments. The most usual case is manual wheelchairs driven using the arms of the wheelchair user: this group make up around 85% of all wheelchair users [35]. Manual wheelchairs generally comprise two large rear wheels (with grips known as ‘pushrims’) for propulsion, supported by two smaller caster wheels at the front of the chair [5]. Typically, these wheelchairs are highly customised to the specific anatomical requirements of the individual user [39].

In some cases, there are additional features, for instance a one-armed propulsion mechanism for someone who has had a stroke (and only has full use of one side of their body), or the use of a battery pack for someone who has limited strength and less ability to climb gradients. In others, a manual wheelchair is propelled by a caregiver or assistant, for those who may be unable to operate the chair themselves (e.g. due to a comorbid cognitive impairment): in this case there can be considerable physical demands placed on this operator [1, 2]. For persons who lack physical strength, coordination or control in respect of their upper arms, or those who simply lack physical stamina due to a health condition, an electric wheelchair or scooter would be used: as with manual wheelchairs, there can be a considerable degree of customisation involved in the design of these systems: they are bespoke to the user [18, 20, 39]. In general, an electric wheelchair or scooter is a four wheeled vehicle which is powered electrically by one or more motors, and normally controlled by some kind of joystick or steering wheel.

2.3 The Direct Effect of an Individual Barrier in the Built Environment

Despite this breadth in wheelchair users and wheelchairs, there is a point of considerable commonality. This starts with the ideal driving conditions for any kind of wheelchair which are the same in general, which in turn means that an accessibility barrier can be well-defined with reference to be a deviation from this ideal. To be specific, these ideal driving conditions are a sufficiently flat, smooth surface (albeit with sufficient grip), with sufficient space to maneuver their chair. Anything outside of that has consequences for wheelchair users: the precise effects are different depending on the individual and their abilities. They range from simple inconvenience, onto discomfort, gradual injury from repetitive interaction with the environment (or a more immediate injury were a feature to cause an accident), to impassibility. To provide a concrete understanding, we now describe the main types of barriers and the different effects that they can have.

  • Surface gradient. If a slope is too steep, physical fitness limits driving, it also involves undue effort and exertion for those who are able to navigate this [6]. In respect of gradients perpendicular to the direction of travel (i.e. with a notable camber), these can be uncomfortable, especially for (two-handed) manual wheelchair users, as they have to propel the wheelchair in an unbalanced manner [14].

  • Surface deformation. These typically take the form of an abrupt discontinuity in the surface, which raises the risk of a person driving an electric wheelchair getting stuck, or a manual wheelchair user falling out of the wheelchair. It is not just the risk in tripping that is detrimental, but also the additional concentration required in an effort to avoid these hazards (not all of which are readily visible).

  • Rough surface. As well as being a potential trip hazard and increasing the work required by a manual wheelchair user, it also increases harmful whole-body vibration, with a long term negative effect upon health [9, 44].

  • Lack of a dropped curb (or curb cut). These are impassable for existing electric wheelchairs but can be passed (uncomfortably) by ‘hopping’ in a manual wheelchair (presuming a suitably fit rider).

  • Stairs. For obvious reasons, these are generally impassable to a wheelchair user.

  • Narrow pavement or passageway. Wheelchairs are wider than most pedestrians and need a certain width in order to travel a given path. Accordingly, if a pavement is not wide enough (or is obstructed by a transient barrier such as a parked car or wheelie-bin), then a wheelchair driver may have to use the road (with the obvious risks) or the route is entirely inaccessible.

These features exclude barriers that go beyond wheelchair users: for example, whether there are sufficient tactile markings for visually impaired people [31], or if there is sufficient crowd management for people with relevant impairments (e.g. age related frailty or anxiety). By focussing upon a relatively fixed set of concrete barriers, our approach avoids subjectivity and enables a focus on the mechanical accessibility issues that apply specifically for wheelchair users.

3 Developing a Measurement Framework

3.1 The Task of a (Wheelchair) Routing System

It is necessary to briefly set out the task of a pedestrian routing system and provide some terminology. A routing systems task is to plot a route between geographical location A and geographical location B. This route will be a sequence of paths. An optimal route will be the one which allows the person to most efficiently traverse ‘A to B’. Typically, ‘most efficiently’ means the shortest distance, however it can be important to take into account the complexity of the route (because navigation can itself take additional time and effort, especially when a route is unfamiliar or more complex [10, 15]). In some cases, other factors might be optimised, for example the flatness of the route. We will say a route is effective if it is traversable by the person traversing it, i.e. it is not impassable. For someone who is not a wheelchair user, that means that the route exists and is not obstructed, for example by construction work. For someone who is a wheelchair user, this means that the route is not obstructed for them: in addition to that, it means that there are no accessibility barriers for them (i.e. those features as identified in Sect. 2.3 or a subset of them, depending on the person) impeding their route. The task of a wheelchair routing system is therefore to provide an optimal route from ‘A to B’ that (i) exists, (ii) is not obstructed and (iii) has no accessibility barriers for the wheelchair user in question.

3.2 What Errors Can Be Made by a Wheelchair Routing System?

Following on from the task of a routing system, it is important to be more specific as to what an error actually is and how it might arise. With respect to a given wheelchair user there are two primary types of errors that can be made by such a tool: A Type 1 error, where a longer route is proposed that presumably arises from a ‘virtual’ or ‘imaginary’ accessibility barrier which does not physically exist (i.e. a suboptimal route), and a Type 2 error, where a route is declared to be wheelchair accessible, but in fact has substantial obstacles within it. These errors have different consequences. In respect of a Type 1 error, the wheelchair user unnecessarily spends more time going to their destination than they otherwise would. In some cases, this may also mean additional time navigating, because longer routes on average are more likely to be complicated with an increased number of turns, in turn requiring further in-situ references to the route in question [43]. In respect of a Type 2 error, the result is the person has to do a ‘go around’, i.e. amend their journey in real time. A ‘go around’ has two implications, both of which substantially reduce efficiency: the first is that time is wasted travelling a route that is inaccessible; the second is that time has to be taken in order to identify or investigate an alternative route. Figure 2 depicts each type of error in turn.

Fig. 2.
figure 2

Examples of the types of errors that can be made by an ‘A to B’ wheelchair routing system. A Type I error (top), is where a longer route is proposed that presumably arises from a virtual or imaginary barrier. A Type II error (bottom) is where the routing algorithm sends a user down an inaccessible route, when there is an alternative accessible one.

Full size image

3.3 Cost-Benefit Analysis

Cost-benefit analysis is a widely-used comparative approach used to make decisions between options based on economic analysis. More specifically, it is defined as “the systematic and analytical process of comparing benefits and costs in evaluating the desirability of a project or programme” [32]. In our case, this is whether a wheelchair user should use a given routing algorithm. Cost-benefit analysis is an objective balancing exercise: it involves determining the costs and benefits of a given option. However, there are a variety of ways as to which this might be done: for example, there are different ways of measuring cost and benefit, including in respect of the units of analysis employed (e.g. time, financial cost). This means a bespoke approach must be designed for each individual scenario. This is what we do next.

3.4 Making the Subjective Objective: Classifying Individual Routes for Populations

One difficulty in developing any framework is the diverse nature of wheelchair users and their particular impairments in respect of accessing the built environment: this is captured in Sect. 2.2. Moreover, how this can impact upon a given individual is innately subjective. However, this does not mean we cannot adopt an objective approach towards evaluating wheelchair routing systems. This is because we are considering populations of individuals, not individuals per se, because the systems are supposed to be designed for wheelchair users at large. The remedy is to consider extrema, because it offers a strong indication in respect of the intermediate space which many wheelchair users fit within. For our purposes, this means the following broad categories of wheelchair users, with C1 and C3 capturing the extrema:

  • C1 (Electric/All users): A route which is fully accessible for an electric wheelchair or scooter user, which means it does not contain one or more errors identified in Sect. 2.3. Note: a route that is accessible for an electric wheelchair user will almost always be accessible for most manual wheelchair users.

  • C2: (Manual, Not Electric) A route that has one or more errors noted in C1, but is not inaccessible to a manual wheelchair user (i.e. there was no need for a ‘go around’ by them).

  • C3 (Not Accessible for all Wheelchair Users): A route which is impassable even to a highly skilled and able manual wheelchair user without any impairments asides in respect of using their legs to ambulate.

These categories can be readily considered pairwise to produce Type I and Type II errors, (i) Scenario 1 by comparing C1 vs C2 and C3 combined (hereon electric_wheelchair user accessibility), and separately (ii) Scenario 2 by comparing C1 and C2 vs C3 (hereon manual wheelchair user accessibility). This provides two different scenarios where a cost-benefit analysis can be performed on a set of routing algorithms.

3.5 Drawing This Together: How to Perform a Cost-Benefit Analysis

For our purposes, cost-benefit analysis takes two routing algorithms at a time, RA1 and RA2 and then compares them. To do so, one needs to prepare a sampleFootnote 1 of ‘A’s and ‘B’s: i.e. two different geographic locations and then use RA1 and RA2 in turn to compute routes. These routes can then be evaluated in turn, by computing (i) the length and complexity of the routes in question and (ii) the proportion of routes in which there is an accessibility barrier encountered, with reference to Scenario 1 and Scenario 2. A simple cost-benefit analysis would then involve computing – for each element of the sample – the additional distance and complexity between RA1 and RA2 by quantifying it as a numerical time, before deducting time for additional go-arounds.

For example, one could take distance at 6 km/h (the standard speed for moving in a wheelchair – thus 6000 m = 1 h), complexity at 5 s per turn (to take account of additional consultations of a smartphone) and a time penalty of 5 min for encountering an accessibility barrier. Now suppose RA1 was 24 km (so 4 h) over a sample of routes and RA2 was 30 km (so 5 h), but RA1 involved 30 more turns than RA2 and encountered 16 extra barriers, then RA1 would perform marginally worse than RA2, because the 30 extra turns (150 s or 2 and half extra minutes) and the 16 additional detours (80 min) add up to more time taken in total.

4 Methodology: Applying Our Framework in an Investigation of Wheelchair Routing Algorithms

As mentioned before, our method is a Cost-Benefit Analysis, following the framework set out in the foregoing section. Any analysis is based upon data. This work is no exception. We therefore collected a dataset for this investigation to which cost-benefit analysis could then be applied. There are three facets to this exercise: (i) the design of an expert assessment of routes to obtain a ground truth; (ii) the determination of how routes were to be generated using existing navigation tools and then (iii) how these routes were to be sampled (given the burdens of performing an expert investigation).

4.1 Making an Expert Assessment of a Route

Our goal was to produce an expert accessibility assessment of the routes generated by the mapping algorithms in order to act as an effective ground truth. Two manual wheelchair users with expertise in accessibility assessments independently traversed each of the routes in question to ensure that all the potential issues were captured. This ‘double annotation’ is in line with best practice when collecting accurate datasets [29]. The assessments took place in daytime to ensure sufficient visibility so that no features were occluded from the assessment. Whilst travelling the route, a detailed record was taken of all potential barriers (see the discussion in Sect. 2.3 above) within the built environment for wheelchair users of all abilities. If the route was impassable in a manual wheelchair, this was recorded and then a ‘go around’ was performed to travel as much of the proposed route as possible. To help ensure the necessary objectivity, there was a detailed checklist that had to be completed for each trip. Any discrepancies were resolved between the two annotators. The resulting exercise involved the collection of an extensive amount of data, with 160 km of routes being directly assessed by our two experts.

4.2 Navigation Tools

There are two existing systems for routing Wheelchair users from ‘A to B’, OpenRouteService (‘ORS’) [50] and Routino [52], which (i) presently operate in German cities and (ii) which also have an publicly available API that enables automated access to them. For each of these platforms, we generated 2715 routes. We adopt the approach towards generating origin-destination pairs as per Tannert et al. [38] This approach involves determining all possible routes ‘A’ and ‘B’ within a city, where in ‘A’ is a publicly available bathroom, and ‘B’ is a point of interest. Bathrooms are used because their locations are critical for wheelchair users in two respects: (i) the nature of many mobility impairments means an increased need to visit the bathroom (e.g. Spina Bifida is well known for having such an incidence [40]) and (ii) public accessible toilets are a rarity when compared to the facilities available for non-wheelchair users. Bremen was chosen because it is heavily populated, as well as having a variety of terrain (e.g. a medieval city centre) as to present a diverse range of challenges for a routing algorithm.

4.3 Sampling Approach

Our sampling approach has two goals. The first was to allow for an effective test of whether Type I or Type II errors (as defined in Sect. 3.2) impeded routes, so we could gain an insight into how (or more accurately why) a system offered the routes that it did. The second was to ensure that the overall sample was reasonably representative of how a wheelchair user might use the system, thereby enabling cost-benefit analysis. The challenge was to have a process that efficiently used expert assessments, given the degree of effort involved in collecting (and annotating) the data, whilst also meeting these goals. The way we achieved this was to have three different groups of routes, of which 15 examples were sampled (making 45 pairs for each comparison, totalling around 160 km of routes evaluated) These groups of routes were determined on the basis of the degree of overlap or alignment between RA1 and RA2 (where RA2 was always Google Maps): examples of each can be found in Fig. 3:

Fig. 3.
figure 3

These images provide examples of the generated routes in respect of each case. At the top is Case 1 (near 100% overlap/MAX), the middle is Case 2 (near 50% overlap/MID) and the bottom is Case 3 (near 0% overlap/MIN).

Full size image
  1. 1.

    MAX (overlap) the case where the route was closest to the one generated by Google Maps for fully ambulant pedestrians (i.e. starting from 100% overlap and descending until we have 15 routes). This is designed to help identify Type II errors, because it is highly probable that routes for wheelchair users that are equal to the routes of pedestrians could create a virtual barrier of Type 2.

  2. 2.

    MID (overlap) those in the middle (with ~50% overlap). This was added to provide a more balanced sample for cost-benefit analysis, given that cases 1 and 3 were extrema.

  3. 3.

    MIN (overlap) those routes with the least overlap (i.e. starting from 0% overlap and ascending until we had 15 routes). This is designed to help identify Type I errors, because if the routes for wheelchair users and pedestrians are almost different it is very possible that there could occur virtual barriers of Type 1.

To compute the degree of overlap between routes we used a fine grid to represent each path (with 10 m by 10 m cells) in order to identify which elements were overlapping and thus estimate the % overlap (which is different from the difference in distance) [34].

5 Results

5.1 The Routes Chosen by the Algorithms

We begin by summarizing the general (non-accessibility related) features of the routes we sampled. As one might expect (Fig. 4), the routes for the minimum overlap (whose average length is over 1.5 km) with Google Pedestrian are significantly longer that those with a maximum overlap (all of which are under 500 m), albeit there is substantial variance in lengths. Another key feature (Fig. 4) is that the routes generated by the wheelchair routing systems are on average both longer (in distance) and more complex (they have substantially more turns, indeed on average there are more than twice as many in each route). This is also to be expected: the algorithms of ORSFootnote 2. and RoutinoFootnote 3 both focus on (i) avoiding steps and (ii) providing paved routes whereby the ORS route has to be 100% paved and for the Routino route 90% must be paved.

Fig. 4.
figure 4

A summary for the routes generated by each system, in respect of each sample, with its paired pedestrian route in Google Maps. Each graph presents the mean values, with the error bars being one standard error. A proportion of the variance is explained by the fact that Google had up-to-date information on construction sites that necessitated substantial details, but neither wheelchair routing system was aware of them.

Full size image

5.2 Are the Routes Generated by a Wheelchair Routing System More Likely to Be Accessible?

The confusion matrices in Fig. 5 provide a comparison between Google and both of the mapping tools for each of the settings in question. The difference between the two approaches is mostly marginal, asides Routino’s relative failure in respect of electric wheelchair users, and ORS being better (by providing three more accessible routes compared to Google) for manual wheelchair users. In total 39 of the 45 ORS routes were accessible for manual wheelchair users and 29 for electric wheelchair users. Out of the 45 Routino routes, 34 were accessible for manual wheelchair users and only 21 for electric wheelchair users. For electric wheelchair users, both tools less often provide a more accessible route, meaning that the tools were (insofar as our sample was concerned) less accurate than using Google Maps, especially in respect of Routino. This appears to be for the reason that (i) the routes in question are longer and involve more turns and (ii) there is more likely to be a missing dropped curb within a given route (this only needs to happen once for the route to fail): see Table 1 which shows the increased number of average number of curbs and missing curbs (especially in respect of the longer routes with less overlap: i.e. Routinomin and ORSmin).

Fig. 5.
figure 5

Confusion Matrices comparing Google Maps Pedestrian Routing with the specialist Wheelchair Routing systems. The general picture is that under each condition, there is little or any practical difference between using a Wheelchair Routing system and simply using a general Pedestrian Routing system in terms of the accessibility (:Acc) or inaccessibility (:In) of the routes in question. The majority of the results are within the leading diagonal, i.e. neither system proposed a route that encountered an accessibility barrier.

Full size image
Table 1. This table provides a summary of the number of missing dropped curbs in respect of each routing algorithm. It can be seen that Google Maps is generally more successful in avoiding them.
Full size table

5.3 Our Cost-Benefit Analysis

In Table 2 we compute the additional time cost (denoted [Z] in the table) per additional accessible route identified. We make assumptions that generally favour the ‘A to B’ systems, including that it would take 5 min (on average) to perform a ‘go around’ in respect of a given accessibility barrier and that it would take only five seconds extra per turn to navigate the route in question. We also provide the success probability (denoted by [X] in Table 2 for Google, and [Y] for the respective routing system): namely the proportion of routes that were completely passable for each type of wheelchair user.

Table 2. This table details the underlying cost-benefit analysis of each Wheelchair Routing Algorithm compared to Google Maps. As the Increased Accessibility Probability for Electric Wheelchairs is negative to begin with (and the routing tools produce routes that are already on average longer), the table is only complete for Manual Wheelchair Users. The same modelling assumptions are used as in our example in Sect. 3.5.
Full size table

The best-case scenario is the Manual Wheelchair user with no other mobility impairments, as we have already found that the systems showed no improvement at all for Electric Wheelchair users (making the result of any cost benefit analysis be that Google Maps is more effective). For manual wheelchairs the ORS would need to be over six times more effective simply just to break-even (it takes over 30 min extra travelling time to avoid one accessibility barrier, so dividing by 5 min – our assumed time to perform a ‘go around’ gives us the figure in question), let alone to actually have a substantive impact on the wheelchair users chances of getting from ‘A to B’ in a timeous fashion. The picture is worse with Routino, which would need to be over 19 times more effective. We find that neither routing tool is effective compared to Google Maps under a cost-benefit analysis, even if profoundly more favorable assumptions were applied.

5.4 Virtual Barriers

As explained in our introduction, our concern was that these systems may be sending their users around the houses (i.e. on longer routes) to deal with imaginary accessibility barriers (i.e. making Type 1 errors). Table 3 makes it clear that this is so. As can be seen from Fig. 6 the most aggressive interventions appear to be made on the occasions where only the wheelchair routing system produces an accessible route (especially with respect to Routino): similarly, the increased distance of using a wheelchair routing system is markedly reduced on occasions where only Google Maps is accessible. This suggests where longer routes are being used, there is a real accessibility barrier in play, for Google Maps, we observed this was where a construction site was missed by the Wheelchair routing tool (in total, we identified four such occasions).

Table 3. This table enumerates the Types of Errors made in line with our Taxonomy in Sect. 3.2. of this document.
Full size table
Fig. 6.
figure 6

These two charts illustrate the additional travelling distance involved in using the tools in question, grouped by the same categories used in the confusion matrices.

Full size image

6 Discussion

6.1 Google (Normally) Knows Best

The result of our cost-benefit analysis is surprising: it was better to use Google Maps than a specialist routing tool to travel as a wheelchair user, even for manual wheelchair users. As Google Maps was able to identify transient barriers, such as construction sites (whereas the wheelchair mapping tools directed our user straight through them), it has an evident informational advantage over the alternative tools. This informational advantage partly explains why Google Maps is more effective.

Accordingly, rather than striking out alone for wheelchair users, it might be more appropriate to focus upon improving Google Maps and similar platforms, or at the least, integrate these additional sources of information into the analysis offered by a wheelchair routing tool. This is particularly so given that Google Maps is more developed: it provides a range of features both in the moment (where someone discovers a barrier) and offline (e.g. Street View for inspections), meaning that the foregoing analysis is an underestimate of the increased burden imposed by wheelchair-routing tools. Whilst Google Maps may have a disadvantage in that it is not Open Source (unlike the two tools we used), the reason for its advantage is the level of resource which arises from it not be purely focused upon Wheelchair users: there is no reason to expect that this advantage would not apply to similarly resourced Open Source tools.

6.2 Real Versus Synthetic Evaluations

There are a number of existing works that analyze the efficacy of a given routing algorithm on the basis of a synthetic evaluation. This includes two works which were the basis for the ORS system itself [26, 27]. Our investigation shows that in the real world, these synthetic analyses are not even sufficient to ensure that the routes offered are accessible, let alone the most effective ones. There are two specific flaws with these tools (i) they do not pick accessible routes often enough (which would have been identified if this was tested with a direct inspection) and (ii) they are not being optimized using metrics that are orientated from the perspective of the end user (i.e. they do not involve an effective cost-benefit analysis).

These are synthetic problems of different flavours, but ultimately the lesson is that some form of physical inspection process (i.e. obtaining ground truth) is required when evaluating any ‘A to B’ tool, as well as taking into account the real data that is likely to be available when navigating, rather than some idealized expectation of it. There is a further implication, namely that these tools cannot be used as proxies for the general accessibility of a city as proposed in [38], because (unlike what someone would reasonably expect) they simply do not sufficiently reflect the real world.

6.3 Reporting Routing Options

There is always a risk that accessibility data will be inaccurate: even for expert assessments, accessibility information is as only as good as the time of the assessment. In practice, simply due to resource constraints, most data is not even gathered by experts, making it inherently unreliable at the point of collection. The real concern is one of high-level strategy: Google Maps (and similar systems) may offer more direct (and more up to date) information about additional barriers (e.g. construction sites) and routes, but there is a need to merge that with reliable accessibility information.

How should these two information sources be combined? We suggest that a starting point should be the lived experiences of wheelchair users using ‘A to B’ tools, as sharply captured in [12]. These experiences are multifarious in nature, being a combination of the impairment that the person in question has, as well as their wider life circumstances and the particular city that they are based. Ultimately, this must be an exercise in exposing choices and the risk associated with them, so wheelchair users can decide in advance whether its worth (for example) the risk of being ‘X minutes late’ or having to go Y metres further, rather than having to do ‘reccies’ in advance or laboriously exploring routes using Google StreetView [12]. Given that Wheelchair users sometimes use existing networks, the information arising from this could also be provided, including perhaps the sharing of location and acceleration data to the mutual benefit of other wheelchair users.

Making meaningful choices would involve a system exposing what it knows about the accessibility of given segments, before presenting it as a range of choices. Taking into account that Google Maps is likely to suggest a shorter route, the choices would in effect amount to proposed ‘detours’, together with the information known about that detour (e.g. how much longer, what accessibility barrier it might be avoiding and the probabilities involved). Overall, this would be an exercise in exposing trade-offs that recognizes the imperfection inherent in any accessibility tool. Other advanced algorithms have been successful in a range of complex route-planning contexts (e.g. for ships at sea [3], pedestrians travelling at night [7] or personalized routing [36]), so adapting such approaches may be a partial remedy for wheelchair mapping as well.

6.4 Cost-Benefit Analysis and the Bigger Picture

In so far as we know, this paper is the first attempt at using cost-benefit analysis in analyzing individual assistive technologies. It is therefore important to recap the benefits of this approach. In this case, the further analysis allowed for a consideration that took into account a more holistic set of factors and considered the true utility of a system to someone with a disability. Whilst any cost-benefit analysis (like any model) has its limitations (it only captures what is modelled), the success in this example may mean that it could be appropriate for it to be considered more widely in respect of assistive technologies and thereby capture the wider benefits to individual lives. We consider this approach to be complementary to qualitative methods used to understand individual experiences and an efficient alternative where these studies might not be easily available (e.g. due to the well-known difficulty of easily accessing a pool of participants due to delays in ethical approval [25], or where a longer term consideration is required in advance of deployment), or at preliminary stages in design (thereby ensuring a better design to be tested and explored in a qualitative study or deployment).

6.5 Limitations

Our investigation was constrained to wheelchair users, meaning that its findings may not generalize to people with other impairments that limit their opportunity to access the built environment. This paper focusses on one city with a diverse landscape: future work may compare the effects identified within this paper with other cities. In particular, given that our city may have had limited accessibility, a more accessible city may well produce different results and be more (or less) effective a place for using ‘A to B’ tools (on the other hand, it is possible that the trade-off could in fact be worse). Similarly, where a city has been more (or less) thoroughly documented in respect of accessibility, this would also potentially impact on performance. For the comparison of the routes we chose Google Maps because it is the most widely used tool: future work may benefit with a comparison between other mapping tools (e.g. Apple Maps).

7 Conclusion and Future Work

We have explored the viability of ‘A to B’ mapping tools for wheelchair users, by developing a novel framework based on cost-benefit analysis in order to explore their real-world efficacy. In respect of existing tools aimed at wheelchair users the overall result is unfortunate: these tools are of little or no practical benefit when compared to Google Maps, with Routino also presenting routes that were less accessible overall. In arriving at this conclusion, we have also determined that the approaches used to evaluate ‘A to B’ tools are potentially misleading, and an economic approach based upon cost-benefit analysis from the perspective of wheelchair users would be more appropriate instead. Our analysis also suggests more appropriate means of presenting the data generated by these systems that reflects their real-world performance and the uncertainty inherent in that. It is hoped that this work would be the basis for ‘A to B’ mapping tools which are of genuine and long-lasting utility to Wheelchair users going forwards.