Introduction

In recent years, location data have been collected more than ever. Most moving objects, especially cars, are now equipped with GPS loggers in one way or another. While new cars have mostly GPS trackers, old cars' movements are recorded using GPS-enabled smartphones. These data have been used increasingly in mobility research in past years. Although it has provided many new opportunities for research, it has also introduced challenges [30].

Tracking data are collected by sampling the GPS location of a moving object at a specific sampling rate (GPS points per unit of time). These tracking data, combined with other data collected by the object's sensors, such as speed, are called Floating Car Data (FCD). FCD is a powerful source in smart-mobility management systems to analyze and predict traffic speed on road networks and measure traffic congestion. In addition, it is broadly used in research, for example, to find traffic patterns [21], to spot, explain, and predict accidents [20], and to infer travel mode [48].

GPS data are not accurate [5]. According to Plaudis et al. [35], a sequence of GPS points, called a GPS trajectory, is associated with two types of errors. First, measurement errors, i.e., the recorded location can deviate from the true location. This is due to noise from several sources when recording GPS data [19]. Second, sampling errors, which refer to lost information between the recorded points. Ignoring these errors in the data can lead to false analyses and misleading conclusions. Map-matching is an approach to minimize the errors in GPS data. It refers to matching the recorded points to a representation (usually in the form of a graph) of a road network. Figure 1 shows an example of map-matching.

Fig. 1
figure 1

GPS recorded points (red) and matched points (blue). The blue line shows the corresponding trajectory found by map-matching. basemap: ©OpenStreetMap ©CARTO

Full size image

Map-matching can either be offline or in real time [29]. Offline map-matching refers to matching the recorded coordinates after the trip is completed. Offline algorithms use all points of an entire trajectory to estimate the true positions on the road. These are primarily used when working with historical data. Real-time map-matching refers to matching the points during the trip as the points are being recorded. Obviously, real-time algorithms can use only previously recorded coordinates to estimate the true position and direction on the road at a certain point. These are used mainly by navigation services and in autonomous vehicles.

A comparison between different map-matching algorithms and generally evaluating the map-matching quality is not straightforward [38], since the “true” positions of the recorded points are unknown. However, almost all the matching algorithms implemented by available map-matching services demonstrate good performance when the sampling rate is high, i.e., time interval between measured GPS points of less than 30 s [4,5,6].

This paper demonstrates a practical guide to open-source map-matching. The section “Brief Overview of Available Approaches” reviews the historical evolution of map-matching algorithms and classifies their applications. The currently most prominent map-matching algorithm, which is based on hidden Markov models, is briefly explained in the section “Map-Matching Based on the HMM Algorithm”. Next, several available map-matching service providers and open-source routing engines are compared in the section “Available Map-Matching Services”. We chose the open-source routing engine Valhalla between available options to demonstrate the map-matching. The section “GPS Data Used” describes the GPS dataset used. The section “Building a Valhalla Routing Server” explains shortly how a Valhalla engine can be built. The map-matching process, API parameters and data flows are demonstrated in the section “Map-Matching Process”. The section “Results and Evaluation” discusses the results and evaluate the map-matching performed by Valhalla. Finally, the section “Conclusion” concludes the paper.

Brief Overview of Available Approaches

Some early algorithms [25, 39, 41] tried statistical estimation for map-matching. Their approach was to fit a curve to recorded points and find the road segment that matches the curve better. This approach worked only for some special cases, but it did not function well in most practical applications. The most naïve algorithm for map-matching, which matches the points to the nearest street segments, is discussed by Bernstein et al. [3], Kim [23], White et al. [45], and Taylor et al. [42]. This algorithm is sensitive to outliers and does not use the previously recorded points. However, it is fast and can provide a baseline for more advanced map-matching algorithms. Bernstein et al. [3] and White et al. [45] added a simple improvement to this algorithm by taking the heading information of the GPS data into account. If the GPS heading does not correspond to the road heading, then the road is discarded. Another improvement to this algorithm is suggested and discussed by Yang et al. [47] and Quddus et al. [37]. They use a Kalman filter to drop unreasonable GPS data points which leads to better map-matching results. A revolutionary map-matching algorithm is proposed by Newson and Krumm [34] which is the basis for many further approaches. This algorithm is also implemented in many open-source routing engines (such as Valhalla) as well as commercial map-matching services. This method, which is based on hidden Markov models (HMM), finds the most likely sequence of road segments that match the sequence of GPS points.

The HMM has been widely used for map-matching applications and optimized for different settings and environments (see, e.g., [12, 24, 28, 40, 46]. An essential aspect of map-matching using HMM is path-finding for estimating transition probabilities. HMM determines the shortest (least-cost) path between the two successive hidden states. This is a classic point2point shortest-path optimization problem (P2P), which has been extensively studied in the last decades (see, e.g., [1, 2, 8,9,10, 13, 22, 49]. There are exact and metaheuristics algorithms for solving the shortest-path problem. An example of an exact algorithm for finding the shortest path from a starting node S to a target node T in a weighted graph is the Dijkstra’s algorithm [10]. This algorithm finds the shortest path from S to T by exploring the shortest path from S to all other vertices in the graph.

A heuristic extension of Dijkstra’s algorithm is the A* search algorithm [16], which is implemented in Valhalla. Unlike Dijkstra, A* does not explore all routes from the source to the destination, and it only explores the promising routes. It minimizes the sum of the cost from the source to the current node and the current node to the destination. While the cost of the path from the source to the current node is known, the cost of the path from the current node to the destination is estimated using a heuristic function such as Euclidian or Manhattan distance. In recent years, heuristic approaches have gained significant attention for solving shortest-path problems [31]. For example, in Chen et al. [7], a heuristic method is proposed, which is an extension of Dijkstra’s algorithm, for finding the shortest path in traffic networks. The shortest-path problem is also relevant in other contexts rather than mobility. Hasan et al. [17] propose a heuristic genetic algorithm for finding the shortest path for Internet routing.

There are several literature reviews on map-matching attempting to classify existing methods. Quddus et al. [36] group the map-matching algorithms into geometric, topological, probabilistic, and advanced. Geometric map-matching is only based on geometric distance (e.g., point-to-curve), topological map-matching uses the contiguity of the roads, probabilistic methods approximate an error region using the uncertainty of position, and advanced map-matching is a combination of methods. In addition, this paper evaluates several available methods, and identify their constraints and limitations. Finally, it argues that the main problems are associated with the initial position identification. Wei et al. [44] review the map-matching algorithms and classify them as incremental max-weight, global max-weight, and global geometric. Max-weight methods integrate several factors (e.g., directions of recorded points and road, distance of the recorded point to road segments, and shortest path between recorded points) and select the candidate sequence with the highest score. Incremental max-weight only use previously recorded points for real-time applications and global max-weight use the entire trajectory for offline applications. For example, all algorithms using hidden Markov models (e.g., Newson and Krumm [34] fall under global max-weight. Global geometric methods refer to models which match the points only based on geometric measures. Hashemi and Karimi [18] provide an overview of map-matching methods mainly for navigational (real-time) applications and are therefore not discussed further. Kubicka et al. [27] divide map-matching algorithms based on their application. The intention for this division is that map-matching problems vary across applications, and different methods could be suitable for different applications. While real-time algorithms are suitable for navigation purposes, offline algorithms are used in surveying applications. A most recent review of map-matching algorithms is conducted by Chao et al. [6]. They argue that previous categorizations of map-matching algorithms are outdated and are not useful anymore. Geometric algorithms are not implemented and recent hidden Markov models map-matching algorithms can be used in both real-time and offline applications. Therefore, they provide a new categorization of map-matching models according to their algorithms and applications. Moreover, by evaluating several map-matching methods, they find that both too high and too low sampling rates can be problematic for map-matching models.

Map-Matching Based on the HMM Algorithm

Many map-matching service providers, such as Valhalla, Mapbox, and GraphHopper, use the HMM algorithm based on Newson and Krumm [34]. This algorithm, which is proposed by Microsoft, is also widely used by other companies such as Uber. In this approach, the problem is formulated as follows: given a sequence of (time-stamped) GPS points, each point has to match a road segment, what is the most likely sequence of road segments that match the sequence of GPS points? This formulation naturally fits the HMM, as the recorded GPS points (observations) refer to state measurements and the individual road segments refer to hidden states of the HMM algorithm. The most likely path is the sequence of states with the most likely transitions, where the transitions between road segments are ruled by connectivity in the road network.

The HMM algorithm works as follows: First, for each GPS point, the model finds the possible road segment matches, the so-called candidates. Figure 2 shows an example of a sequence of four GPS observations and their identified road segment candidates. These candidates are found within a certain search radius from the observation. Each observation has a set of candidates; for example, observation \({P}_{1}\) has two candidates, \({c}_{1}^{1}\) and \({c}_{1}^{2}\).

Fig. 2
figure 2

GPS points (observations) and their corresponding road candidates. ©OpenStreetMap ©CARTO

Full size image

Second, the measurement probability and the transition probabilities are calculated. The measurement probability is the likelihood of observing a particular measurement using only that measurement, assuming that GPS noise is Gaussian distributed with a zero mean. This way, the candidates being further away from the observation are less likely to be considered. The transition possibility of two successive candidates gives the likelihood that the vehicle drove between them. For example, some transitions which require complex maneuvers are less likely to be considered. Moreover, the algorithm favors the transitions, with a great-circle distanceFootnote 1 being about the same as the driving distance. Finally, considering both measurement and transition probabilities, the optimal path is identified. Figure 3 shows all the possible transitions between the road candidates for the GPS sequence of Fig. 2. The selected candidates and the optimal path are marked red.

Fig. 3
figure 3

Possible transitions between states and optimal path

Full size image

There are different algorithms to find the optimal path. A naive approach is a brute-force solution that explores all the possible transitions. This could be associated with a long computation time. A relatively faster approach is the Viterbi algorithm [11], which is used in the Newson and Krumm [34] approach. This algorithm also calculates the shortest path for all the consecutive pairs of nodes. Assuming a sequence has M measurements with an average of T states per measurement, the algorithm must calculate M*T*T shortest paths, corresponding to the time complexity of \(O({N}^{2}T)\). This calculation can also take much time, especially in dense urban areas with many possible transitions. Another method to reduce the number of shortest-path calculations is to use Dijkstra’s algorithm, which is implemented in Valhalla. It uses a greedy algorithm to extract only the most likely nodes at each state that could lead to a lower runtime than Viterbi.

Available Map-Matching Services

There are many map service providers which offer map-matching. Several examples and their API pricing are listed in Table 1. These services provide ease-of-use and fast implementation for applications, which require map-matching. For a few trips, map-matching is cheap, but costs for matching a large number of trajectories can become exorbitant. There are also limitations in each service. For example, in a single request, a maximum number of 100 locations is allowed while using Google Maps API or Mapbox API. This could be a problem for long trajectories which contain many location points or are recorded at a high sampling rate, since splitting trajectories increases the complexity of data engineering and reduces the quality the of map matching results. Further limitations in API usage, such as daily or minutely requests restrictions, are also present. For example, GraphHopper and Mapbox allow only 1500 requests per day and 300 requests per minute, respectively. These limitations could be acceptable for implementing these services in applications like smartphone apps. There are usually custom pricing plans for clients with scaling businesses. However, these limitations could be obstructive when working with historical data that require map-matching for a large number of trajectories at once.

Table 1 Map-matching API pricing of several map service providers
Full size table

Another way to conduct map-matching is to build a routing server capable of map-matching. Fortunately, there are several open-source routing engines that provide routing services, including map-matching. Two widely used open-source engines are OSRMFootnote 2 and Valhalla.Footnote 3 Some of the advantages and disadvantages of these two routing engines, as discussed by Kreiser [26], are listed in Table 2.

Table 2 Comparison between OSRM and Valhalla routing engines according to Kreiser [26]
Full size table

These engines have a high performance and are vastly used by large enterprises. Some of the above-mentioned service providers use these engines under the hood. OSRM and Valhalla are used (and also mainly developed and maintained) by Mapbox and Mapzen, respectively. Another advantage of both these engines is that they use OpenStreetMap data, making it easy for further analysis and visualization purposes. Here, we chose to build a Valhalla routing engine, since it needs significantly lower memory. A comprehensive study on map-matching using OSRM is performed by Vander Laan et al. [43]. They propose a scalable well-constructed enhancement framework for GPS data that could map-match millions of trajectories.

Other examples for map-matching frameworks are pgMapMatchFootnote 4 and Fast Map-Matching (FFM).Footnote 5 However, as the advantages of OSRM and Valhalla outweigh the other frameworks, they are not discussed here.

GPS Data Used

The GPS data consist of about 18 million trips with more than 5 billion GPS observations (locations with timestamp). These trips have at least one GPS location within a bounding box surrounding Frankfurt am Main in the year 2019. The data source is INRIX, a private company that provides location-wise data and analytics. The journeys are recorded using either a smartphone or embedded GPS devices. The dataset contains private as well as fleet trips, and it is distinguished between vehicle weight classes. Light, medium, and heavy vehicles range from 0 to 14,000 lb, 14,000 lb to 26,000 lb, and higher than 26,000 lb, respectively. An overview of the number of trips in each category can be seen in Table 3.

Table 3 Number of trips in each category regarding source, vehicle weight, and type
Full size table

The data are stored in compressed GZIP format on the AWS S3 storage and takes more than 300 GB. The waypoints are separated monthly. Each month has about 80 GZIP part files containing the waypoints. A 0.56% sample consisting of 100,000 trips is visualized in Fig. 4.

Fig. 4
figure 4

Waypoints of 100,000 trips visualized on the map. ©OpenStreetMap ©CARTO

Full size image

A description of the trips’ metadata can be seen in the Table 4.

Table 4 The descriptive statistics of trips’ mean and max speed, length, and sampling frequency of the whole dataset
Full size table

Building a Valhalla Routing Server

There are two ways to install Valhalla. The first one is to build it from the source. The second one is to run a Valhalla instance using Docker. The second option is less complex and is more efficient with regard to time and resources. The requirement is the latest docker and docker-compose running on an Ubuntu 20.04. It is recommended to run the Valhalla Docker image by GIS OPSFootnote 6, since it is more straightforward than the original Valhalla Docker image.

To build the Valhalla Docker with Germany’s tiles, the docker-compose.yml file should be properly configured. The tiles data are available on geofabrik,Footnote 7 which is an official member of OpenStreetMap. After integrating the geofabrik linkFootnote 8 in the YAML file, Germany’s tiles could be extracted directly from a PBF file while building Valhalla. An example of a YAML configuration file is shown in appendix.

Depending on the hardware, it can take several hours to build Valhalla completely with the Germany’s tiles. In our case, it took more than 4 h on a t2.2xlarge AWS EC2 instance with 32 GB RAM and 8 CPU cores. We store this server as an Amazon Machine Image (AMI) to make it reusable. Therefore, each time we need a map-matching server, we can simply start an EC2 instance with this AMI. We can even run multiple map-matching servers in parallel by starting several servers using this AMI.

Map-Matching Process

The first challenge with our dataset is assembling the journeys’ trajectories, since the waypoints are stored in many different files without any logical order. There are several different tools for big data analytics. Since our data are spatial and we need to perform spatial operations on the data, we chose the PostGISFootnote 9 spatial database. PostGIS is an extension to PostgreSQL, which adds support for geographic objects and allows to run spatial operations using SQL.

The first step is to import all the GZIP waypoint files from the S3 storage into a PostGIS server. Any required preprocessing step can be done on this server. For example, we drop all the GPS points outside Germany, since they are not of our interest for map-matching purposes. Figure 5 shows the same sample visualized in Fig. 4 after this preprocessing step.

Fig. 5
figure 5

Preprocessing step: only the waypoints in Germany are kept

Full size image

The next step is assembling the trajectories. We group the points by their TripID, order the points by their timestamp and store them in a LineString geometry type. Afterward, they are ready to be sent to the map-matching server. Figure 6 shows the data flow in our map-matching architecture. For each trip, a request is generated and sent to the map-matching server or, more precisely, to Meili using the Library API. Meili is a namespace within Valhalla that includes the map-matching code and is responsible for map-matching functionality.

Fig. 6
figure 6

Map-matching data pipeline

Full size image

We can control the performance and accuracy of Meili by adjusting the map-matching hyperparameters. Some of the parameters are listed in Table 5.Footnote 10 sigma_z is the standard deviation of the GPS measurement error and represents the GPS noise. It directly affects the emission probabilities. beta weighs the transition costs and affects the transition probability of two successive points. search_radius could affect the number of initial candidates that the HMM considers. The default values for these parameters in Valhalla are set regarding the recommendations from Newson and Krumm [34]. For instance, if the GPS data exhibit a lot of noise, we can increase the search_radius, beta, and sigma_z. This could increase the runtime as it considers more measurements and transition possibilities, but it could lead to better results depending on the data. There are also further parameters to set up the map-matching environment. For example, “transport mode” can be chosen between “auto”, “bicycle”, “pedestrian”, and “multimodal”.

Table 5 Examples of map-matching hyperparameters
Full size table

Integrating the timestamps of the coordinates in the API request is optional. Timestamps affect the calculated transition probabilities. The algorithm favors the transitions with durations (calculated by the shortest-path algorithm) being similar to the actual duration. This can enhance the map-matching performance.

Meili processes the request, performs the map-matching based on OSM tiles, and creates a response in JSON format. While parsing the response, we can split the result into three parts: matched points, trajectory, and narrative. Matched points are the corresponding locations of the recorded points on the road network. Trajectory is a sequence of corresponding road segments in a polyline format (convertible to LineString). It can be used to calculate the trip distance or to visualize the trip on the map. Narrative contains the maneuver points and their details (e.g., type of turn or verbal instruction by a navigation app) based on the actual path.

The parsed and processed responses are then directed to another PostGIS database responsible for holding the results. When the map-matching is finished for all the GPS points, we can export the results from the PostGIS database to the S3 storage or to send them to the machine learning data pipeline to do further analysis.

Results and Evaluation

We tested our map-matching environment with a sample of 1,216,476 trajectories. A description of the sample dataset can be seen in Table 6.

Table 6 The descriptive statistics of trips’ mean and max speed, length, and sampling frequency of the sample used
Full size table

In our Valhalla’s map-matching environment, the “transport mode” is set to “auto”. All other parameters are set to default. The first attempt succeeds in map-matching 78.24% of the journeys (n = 951,804). After exploring the unsuccessful cases, we could establish two systematic failures associated with two clusters of journeys: (1) journeys longer than 200 km and (2) journeys with too few GPS measurements. The reason why the journeys longer than 200 km could not be map-matched is unknown; however, a possible limitation might be implemented in Valhalla without being documented. After discovering this issue, the journeys with a trip length of more than 200 km are split into several parts, so that each part is less than 200 km. The map-matching process is conducted again, and the new success rate is 95.20% (n = 1,158,144).

To discover if any better success rate could be achieved, a grid search is done on a random sample of 5000 journeys. The selected hyperparameters, resulting in 556 (= 8*8*9) combinations, for the grid search are listed in Table 7.

Table 7 Range of hyperparameters chosen for the experiment
Full size table

The grid search results demonstrate that the success rate of map-matching is constant at 95.26% for all the hyperparameter combinations. This means a better result on this sample could not be achieved with the selected finite subset from the hyperparameters’ domain space. This could be due to the low number of GPS measurements and high noise in the “unsuccessful” journeys. However, we cannot make any claim about the success rate of the map-matching of our sample for the combinations of hyperparameters that fall out of the selected grid search.

The raw data before map-matching is visualized in Fig. 7, and the trajectories after map-matching are visualized in Fig. 8. It can be seen that before map-matching, the trajectories can deviate from roads, i.e., they are associated with errors. After map-matching, they perfectly lie on the road network.

Fig. 7
figure 7

Trajectories before map-matching

Full size image
Fig. 8
figure 8

Trajectories after map-matching

Full size image

We calculate the trip’s distance before and after map-matching to get an overview of the changes. A sample of 20,000 trips is chosen containing trips with a distance of more than 1000 m and a continuous LineString from the origin to the destination after map-matching. The trip distance is defined as the sum of the great-circle distances between the location points. The absolute and relative difference in trip distance after map-matching in comparison to the original trip distance can be seen in Fig. 9. The trip distance after map-matching is on average 1.4% longer than the original trip distance calculated from raw data. One reason for this increase is the fact that curves are mapped more realistic after map-matching (see Fig. 1).

Fig. 9
figure 9

Absolute (left) and relative difference (right) of trip distance after map-matching

Full size image

To provide an overview of map-matching runtime, the duration of map-matching of every journey is measured. Table 8 shows the average map-matching runtime for different groups of journeys based on the number of GPS measurements and trip length. We can see that the average runtime increases by increasing the points count and trip length. This is intuitive as the number of computational tasks increases.

Table 8 Average runtime of map-matching for each trip separated by different length-points count groups
Full size table

Map-matching runtime depends on the hardware. For example, in this case, the map-matching server is set up on an r5a.2xlarge EC2 instance. This instance has 64 GB RAM and 8 vCPUs and costs $0.548 per hour on-demand on a Linux machine at the time this paper is written. This indicates a relatively low cost for this map-matching approach.

Conclusion

In recent years, there has been a drastic increase in using GPS data in mobility research and smart-mobility applications. Moreover, much research has been done on dealing with inaccuracy in the GPS movement data. Against this background, map-matching has been recognized to be an essential preprocessing step for minimizing errors.

Using map-matching, we can find the road segments matched to the recorded points of a GPS trajectory, allowing further analysis based on road segments. There are several map service providers that offer map-matching. However, their APIs are usually restricted (e.g., number of locations per request) and costly, which make them not suitable for map-matching historical big GPS data. An alternative to these services is building a map-matching server using open-source routing engines such as Valhalla.

We built a cloud-based map-matching framework on AWS. Within this framework, PostGIS is used to perform spatial operations and to assemble trajectories, and Valhalla runs on the map-matching server. We tested Valhalla with a sample of about 1.2 million GPS trajectories and 95.20% of the journeys were successfully map-matched. Other journeys (the remaining 4.80%) contained too few GPS measurements to yield a reliable map-matching result. This proves that Valhalla can be considered a powerful tool for map-matching big GPS datasets regarding cost, runtime, and performance. A visualization of map-matched trips shows that their trajectories perfectly lie on the road network. The trajectory distance calculated after map-matching is on average 1.4% longer than the distance calculated using raw data before map-matching. Assuming that map-matching reveals the actual driven route, it leads to a significant error reduction while calculating trajectory distance. This could improve the accuracy of further analyses that use trip distance.

Map-matching using the suggested framework by Valhalla is fast enough for most applications, since it is supposed to be used for offline and not for real-time map-matching. However, it can still take significant time. Alternatives, such as OSRM, are relatively faster and could replace Valhalla when the overall runtime is an essential aspect. With the parallelization of Valhalla servers, the overall map-matching runtime could be reduced linearly (which could be associated with more infrastructure and cost). Overall, the proposed map-matching framework is efficient and scalable and can be used to conduct map-matching for terabytes of geospatial data.

Some of the limitations of our work are as follows: (1) We cannot make any claim about the accuracy of the results, since the true positions of raw GPS measurements are unknown. A possible approach to evaluate the accuracy of the results would be testing the framework with artificial GPS points and a deviation metric. (2) Because of the computation limits, only a small subset of hyperparameters are chosen for the grid search. The performance of the algorithm outside of this space is not studied.

Future work should use other open-source alternatives, such as OSRM. It should demonstrate how OSRM could be built and used for map-matching. Using the same dataset, its performance should be evaluated and a comparison between the open-source routing engines, i.e., OSRM and Valhalla, could help to decide which service to use depending on the application. In addition, using an artificial GPS dataset, the accuracy of both frameworks should be measured and compared.