Abstract
The overlaying of two map layers is a standard GIS procedure. As we saw in the previous chapter, it enables us to compute the intersection between two feature classes and cross-tabulate either the area or the pixel count of the intersecting features depending on whether raster or vector data are being used. Cross-tabulation can be used to evaluate different topics depending on the nature of the input data. In this chapter, cross-tabulation is used to assess land cover changes, the spatial agreement between maps and map accuracy. In Sect. 1, Land use/cover changes (LUCC) are quantified by comparing two LUC maps, computing different indices of change and creating a change matrix. In Sect. 2, we used various metrics to evaluate the spatial agreement between two maps. This procedure was applied to compare a LUC map with a reference map, a simulated LUC map with a reference map and a simulated LUCC map with a reference map of changes. Section 3 introduces the Kappa indices, which allow us to assess the agreement between two datasets, given the agreement expected by random coincidence. We used the indices to compare observed or simulated maps with a reference map. In Sect. 4 we evaluate the agreement between maps at a global level (the entire map) by focusing on a specific feature such as a smaller area or a particular category (stratum level). Finally, in Sect. 5, the cross-tabulation between a map and reference sample data is used to assess the thematic accuracy of the map by calculating various different accuracy indices. We present examples of analyses based on cross-tabulation for four different cases: To validate a series of maps with two or more time points, to validate a map against a reference map, to validate a simulation against a reference map and to validate simulated changes against a reference map of changes. In the example exercises, we use CORINE and SIOSE maps from the Asturias Central Area and Ariège Valley datasets and maps of the Marqués de Comillas region of south-eastern Mexico (MarquesLUC dataset). The cross-tabulation techniques proposed by Robert Gilmore Pontius Jr. are applied in Chapter “Pontius Jr. Methods Based on a Cross-Tabulation Matrix to Validate Land Use Cover Maps”.
You have full access to this open access chapter, Download chapter PDF
Similar content being viewed by others
Keywords
1 Change Statistics
Description
Land use/cover change (LUCC) can be quantified by comparing two maps or two classified images that represent land cover at two different dates.
Absolute change (AC) is the difference in the area covered by a category (category area) between two dates and is usually expressed in hectares or square kilometres.
where A1 and A2 are the category areas in question at dates 1 and 2, respectively.
AC can be divided by the number of years between the two dates to obtain the average annual change area over the study period.
Relative change (RC) is obtained by normalizing the absolute change value by the category area at date 1.
This formula expresses the proportion of the category area that changed over the study period.
Other indices of LUCC include rates of change. The most popular rate of change is the annual rate of deforestation proposed by the FAO (1995). This indicator is based on the compound interest law. It expresses the proportion of the category area that changes in one year.
An alternative equation, also based on the compound interest law, was proposed by Puyravaud (2003).
Both formulae give similar results except when LUCC is very high, in which case r is significantly higher than t (Puyravaud, 2003).
All the change indices presented above indicate net change, which results from the balance after gross losses have been subtracted from gross gains. For instance, a given forest category could show an absolute change of −2 ha, which could be erroneously interpreted as very little change, but in fact is the result of two opposing processes: the deforestation of 202 ha compensated by the reforestation of 200 ha. A more detailed analysis of change dynamics can be obtained by cross-tabulating the two maps at two different dates and drawing up a change matrix. The change matrix is a cross-tabulated table indicating the area covered by each change (or permanence) between a category at date 1 and another category at date 2. Many change indices can be obtained from this matrix (see, for example, Sect. 2).
Utility
Exercises | |
---|---|
1. To validate a series of maps with two or more time points |
Indices of change are widely used to assess LUCC. Normalized indices, such as rates of change, enable us to compare the rate of change between regions of different sizes.
QGIS Exercise
Available tools | |
---|---|
• Processing R provider Plugin Change_Statistics.rsx R script |
The indices of change proposed in this document are based on the area statistics for the two maps. These could be efficiently computed using a spreadsheet program. However, we suggest using a simple R script using the QGIS Processing R provider plugin. The script generates a table containing the absolute change (AC) area, the relative change (RC) area (both in hectares), the rates of change based on FAO and Puyravaud (2003) and the change matrix.
Exercise 1. To validate a series of maps with two or more time points
Aim
To assess LUCC in the Ariege study area using the CORINE Land Use maps dated 2000 and 2018.
Materials
CORINE Land Cover Map Val d’Ariège 2000
CORINE Land Cover Map Val d’Ariège 2018
Requisites
All maps must be in raster format and have the same resolution, extent and projection.
Execution
If necessary, install the Processing R provider plugin and download the R script Change_Statistics.rsx into the R scripts folder (processing/rscripts). For more information, see Chapter “About This Book”.
Step 1
Then, execute the script and fill in the required parameters (names and dates of the two maps and the output table) as shown in Fig. 1.
The script generates two tables in CSV format: a table showing the change indices (Table 1) and the change matrix (Table 2).
Results and Comments
The two land covers with the most significant absolute change are Categories 1 (built-up) and 2 (agriculture). During the period 2000–2018, the built-up area is increased by 1840 ha, and agriculture lost 1987 ha. The built-up area increased by over 50%. The rates of change resulting from the two equations are very similar. The two categories with the largest rates are built-up (Category 1) and water (Category 6) areas. Over the period 2000–2018, the area covered by these categories increased by around 2.5% a year. Categories 2 and 4 (agriculture and scrubs) present a negative net change rate, indicating that their areas have been shrinking. The change matrix gives us more information about the processes of change. One surprising change is the transition from 1 (built-up) to 6 (water). On closer observation, it was found that pits had been filled with water to create reservoirs.
2 Areal and Spatial Agreement Metrics
Description
Different authors have proposed a series of metrics that evaluate the areal and spatial agreement between two land use/cover maps or between any of their categories. These metrics are obtained from the cross-tabulation matrix and summarize in a single value the agreement between two maps.
The metrics are based either on the comparison of the proportion of total area occupied by a particular category on two maps or on the spatial coincidence of the pixels allocated to any given category on two maps. This review includes some of the most recently developed metrics.
Yang et al. (2017) proposed the overall spatial agreement (A0) and the individual spatial agreement (Ai) metrics. They are formulated as follows:
where \({X}_{i}\) refers to the number of pixels belonging to category \(i\) in map \(X\), \({Y}_{i}\) refers to the number of pixels belonging to category \(i\) in map \(Y\), \({XY}_{ii}\) refers to the number of pixels belonging to category \(i\) in both maps \(X\) and \(Y\), \(N\) is the number of categories into which the pixels are classified and \(M\) is the number of pixels into which the maps are divided.
The overall spatial agreement (A0) and the overall spatial inconsistency (OSI) metrics assess the spatial agreement between the categories in two maps. One metric can be obtained from the other. Whereas A0 shows the spatial agreement (0–100%), the OSI shows the spatial disagreement (0–100%). Added together, they come to 100.
Islam et al. (2019) proposed the overall areal inconsistency (OAI), the individual areal inconsistency (AIC) and the overall spatial inconsistency (OSI) metrics. They are formulated as follows:
where \({X}_{i}\) refers to the percentage of the total area represented by category \(i\) in map \(X\), \({Y}_{i}\) refers to the percentage of the total area represented by category \(i\) in map \(Y\), \(n\) is the total number of categories, \(N\) is the number of pixels and \({N}_{\left(i\ne j\right)}\) is the number of pixels assigned to one category in Map X and a different category in Map Y.
Overall areal inconsistency (OAI) shows the agreement between two maps in terms of category proportions and is expressed in values of between 0 and 100. Users can also assess the areal and spatial agreement/disagreement at a category level through the individual areal inconsistency (AIC) and individual spatial agreement (Ai) metrics. The values for the latter range from 0 to 100, and a value of 100 means perfect agreement.
AIC does not have a standard scale of values, as these depend on the proportion of the total area of the map allocated to the category. It is therefore very difficult to compare the values for this metric between classes, so limiting its usefulness.
Utility
Exercises | |
---|---|
1. To validate a map against reference data/map 2. To validate a simulation against a reference map 3. To validate simulated changes against a reference map of changes |
The areal and spatial agreement metrics assess the similarity between the two maps. They are obtained from the cross-tabulation matrix and therefore do not provide any additional information, in that the values they provide can also be obtained from the matrix. However, they are standard metrics that allow us to measure the agreement between two maps and summarize it in a single figure. In this sense, they are similar to the user’s and producer’s accuracy metrics and to Kappa indices. They are also complementary to quantity and allocation (dis)agreement metrics, as they can differentiate between spatial and quantity agreements.
These metrics can be used to assess how similar a land use/cover map is to another map used as a reference, i.e. the real situation on the ground. They can also be used to check the similarity between a simulation and the reference map for the same year.
QGIS Exercise
Available tools | |
---|---|
• Processing Toolbox R Areal and spatial agreement metrics Individual Areal Inconsistency.rsx Individual Spatial Agreement.rsx Overall Areal Inconsistency.rsx Overall Spatial Agreement.rsx Overall Spatial Inconsistency.rsx |
QGIS has no specific tool for calculating the metrics proposed by Yang et al. (2017) and Islam et al. (2019). However, these can be easily calculated using the cross-tabulation matrix via the formulae set out above. We have also developed various different tools with R to automatically calculate each metric with QGIS.
When using these R scripts, the categories in LUC rasters must be coded in consecutive numbers, from 1 to the maximum number of categories used in the map. Thus, in a raster with five categories, the categories must be coded as 1, 2, 3, 4 and 5.
Exercise 1. To validate a map against reference data/map
Aim
To validate the CORINE 2011 land use map, take the SIOSE 2011 land use map as a reference. We will be focusing particularly on how the “urban fabric” and “industrial and commercial areas” categories are mapped in CORINE 2011.
Materials
CORINE Land Use Map Asturias Central Area 2011
SIOSE Land Use Map Asturias Central Area 2011
Requisites
All maps must be rasters and have the same resolution, extent, projection and number of categories. LUC categories must be coded consecutively from 1 to the maximum number of categories considered.
Execution
If necessary, install the plugin Processing R provider and download the R scripts indicated above in the “Available Tools” table. Paste the R scripts into the R scripts folder. For more information, see Chapter “About This Book”.
Step 1
Our maps do not comply with one of the requisites of the tools we will be using, in that the categories in our LUC maps are coded from 0 (agricultural areas) to 12 (background). The first step is therefore to reclassify the maps to ensure that all the categories are coded consecutively from 1 to 13. This is done using the Reclassify by table tool (Figs. 2 and 3).
Step 2
Once the maps comply with the requirements of the tools, the different metrics can then be calculated. To test the overall agreement between the assessed and the reference maps, we will calculate the overall spatial agreement (A0), the overall areal inconsistency (OAI) and the overall spatial inconsistency (OSI). For their part, individual areal inconsistency (AIC) and individual spatial agreement (Ai) are used to assess agreement specifically for the “urban fabric” and “industrial and commercial areas” categories.
To calculate all these metrics, open the respective tool and select the maps you want to compare (Fig. 4): first the CORINE map and second the SIOSE map, which is used as a reference. In all cases, the background value of the maps (13) must also be indicated. Finally, specify the folder where the result from each tool will be stored.
For class-specific metrics indicate the codes of the classes you want to validate (Fig. 5). In this case, we will be calculating these metrics for two different classes: urban fabric, which is coded 3 after reclassification, and industrial and commercial areas, which is coded 4.
Results and Comments
After calculating all the different metrics, a numerical output is obtained for each one (Tables 3 and 4). This output is also stored in a CSV file in the selected folder.
There is a high overall spatial agreement (close to 90%) between the two maps and low areal inconsistency (around 3%). We can therefore consider the CORINE land cover map for 2011 as validated. The category proportions between CORINE and SIOSE are almost identical and the spatial agreement is very high. The disagreements between the two maps are due to their different degree of detail, which draws small features in SIOSE that are not detected at the scale used in CORINE.
At the class level, the picture is slightly different. For the two classes we assessed (urban fabric and industrial and commercial areas) spatial agreement between the two maps to be close to 70%. Although this is a high level of agreement, it is much lower than the overall figure. This could be due to the fact that these two classes are more sensitive than others to the scale difference between SIOSE and CORINE.
In order to interpret the AIC metric, we need to first understand the proportion of total area allocated to each class on the two maps. AIC is half of the difference between the two proportions (i.e. if the proportion allocated to one class is 3% on one map and 4% on the other, the difference is 1% and AIC is 0.5). In our case, the AIC value for urban fabric is less than 0.1, which means a high level of agreement between the two maps regarding the proportion of total area allocated to this category (around 3.9%). The proportion allocated to industrial and commercial areas is around 3% in both maps and the AIC value is slightly more than 0.1. This also indicates a high level of agreement, although less than for urban fabric.
Exercise 2. To validate a simulation against a reference map
Aim
To validate the simulation obtained by our land use/cover change modelling exercise. We will focus on the two categories we have modelled actively: “urban fabric” and “industrial and commercial areas”.
Materials
CORINE Land Use Map Asturias Central Area 2011
Simulation CORINE Asturias Central Area 2011
Requisites
All maps must be rasters and have the same resolution, extent, projection and number of categories. LUC categories must be coded consecutively from 1 to the maximum number of categories considered.
Execution
Step 1
The first step is to reclassify our maps to make them comply with the requisites of the tools we will be using. These tools require the categories to be consecutively coded from 1. This means that “agricultural areas” (coded 0) must be given a new code (Fig. 3). This is done using the Reclassify by table tool (see the previous exercise).
Step 2
Once the maps comply with the requirements of the tools, we can then calculate the different areal and spatial agreement metrics using the tools available in the R toolbox.
To evaluate the global agreement between the simulation and the reference map, we will calculate the overall spatial agreement (A0), the overall areal inconsistency (OAI) and the overall spatial inconsistency (OSI). To evaluate agreement for the categories that we actively modelled, we will calculate the individual areal inconsistency (AIC) and the individual spatial agreement (Ai).
To calculate the metrics, open the corresponding tools and indicate the following: the simulation to be evaluated, the reference map (CORINE 2011), the background value of the maps (13) and the folder where the results will be stored. For the class-specific metrics, you must also provide the codes of the classes you want to evaluate: in this case 3 (urban fabric) and 4 (industrial and commercial areas) (Fig. 6).
Results and Comments
Once you have finished the exercise, you will obtain an a CSV file for each metric. The results are summarized in Tables 5 and 6.
The results show almost perfect agreement between our simulation and the reference map. The maps share the same LUC in 99% of their area and the areal inconsistency is insignificant (0.26%). A similar pattern is observed in the actively simulated classes.
These results are misleading. There is perfect agreement between our simulation and the reference map in the persistence areas. However, it is not that high for those areas modelled as changes. Because there are relatively few changes in our study area, the disagreement between the two maps in areas where change is predicted has very little impact on the overall high levels of the agreement created by the correct simulation of permanence areas. To correctly validate the changes that we simulated, we should repeat this exercise, focusing exclusively on the areas that changed in the simulation and in the reference map, as compared to the initial map of the simulation (see next exercise).
Exercise 3. To validate simulated changes against a reference map of changes
Aim
To validate the changes simulated by our land use/cover change modelling exercise.
Materials
CORINE Land Use Changes Asturias Central Area 2005–2011
Simulated CORINE changes Asturias Central Area 2005–2011
Requisites
All maps must be rasters and have the same resolution, extent, projection and number of categories. LUC categories must be coded consecutively from 1 to the maximum number of categories considered.
Execution
Step 1
Our maps do not comply with the requirements for the tools. In the map of simulated changes, the categories are not consecutively coded from 1. In addition, the reference map of changes has many more categories than the map of simulated changes. Using the Reclassify by table tool we can adjust the number of categories on the two maps to the two categories that appear in both (urban fabric and industrial and commercial areas), plus a third category covering non-changing areas and changes that were not simulated. These categories will be assigned codes 1, 2 and 3, respectively. Figures 7 and 8 show the reclassification codes that must be inputted into the Reclassify by table tool.
Step 2
After reclassifying the maps, we will calculate the following metrics to validate the simulated changes: individual areal inconsistency (AIC) and individual spatial agreement (Ai). As we are only comparing two categories, the overall metrics provide the same information as the individual ones.
For each metric, we will open the corresponding tool, indicating the map of simulated changes to be validated (Land use map 1), the reference map of changes (Land use map 2), the background value of the maps (0), the category we are going to evaluate (urban fabric, 2, Fig. 9; industrial and commercial areas, 3, Fig. 10) and the folder where the results of the analysis will be stored. We use 999 as the background value in our maps because no specific value was assigned to the background. 0 means no change, another category that must be considered in this analysis.
Results and Comments
A CSV file will be created for each metric. The results are summarized in Table 7.
The same amount of changes took place in the reference map of changes as in our simulation. There is no disagreement on this point. However, unlike the previous exercise, the spatial agreement between the simulated and the reference changes was very low. The Ai value for the two categories that were actively simulated was quite similar (less than 25%).
These results mean that only a quarter of the simulated changes were allocated in the same places as the changes observed on the reference map. This result, by itself, is not sufficient to consider the simulation invalid. We need to gain a better picture of the location of the changes that were simulated and their pattern. Even if they were not allocated in exactly the same places as on the reference map, they may be allocated in the same general area and follow a similar pattern, indicating that the model has correctly simulated the processes of change. To assess these aspects, we can perform a visual inspection of the reference and simulated changes on the maps, cross-tabulate them at multiple resolutions (see Sect. 2 in Chapter “Basic and Multiple-Resolution Cross-Tabulation to Validate Land Use Cover Maps”) and calculate the spatial metrics (see Sect. 1 in Chapter “Spatial Metrics to Validate Land Use Cover Maps”).
3 Kappa Indices
Description
Kappa indices assess the agreement between two sources of spatial data, corrected by the agreement that is expected by chance. They are typically used to compare the agreement between two maps and to compare one map with reference information (e.g. a collection of validation points).
The first Kappa index (Cohen’s Kappa) dates from 1960 (Cohen, 1960) and has been widely used in LUC analysis. Many variants of this first original index have been proposed. They mainly apply to the comparison between two maps. Of these, the following are of particular interest:
-
Pontius Jr. (2000) split Cohen’s Kappa into three indices, called Kno, Kquantity and Klocation. These indices offer more information about the causes of the (dis)agreement between two compared maps, i.e. (dis)agreement in terms of the different allocation of the categories on the two maps and (dis)agreement in terms of the different proportions in which the categories appear on the two maps.
-
Hagen (2002), following the work done by Pontius Jr., split Cohen’s Kappa into two indices, called Khistogram and Klocation. These refer to the Kappa agreement in terms of the categories appearing in the same proportions (histograms) on the two maps and the Kappa agreement due to the categories appearing in the same location on the two maps.
-
Van Vliet et al. (2011) proposed the Kappa simulation, which was specifically designed for validating LUCC models. It assesses the agreement between the changes on two maps, as compared to a third map used as an initial point, corrected by the agreement expected by chance.
-
Hagen (2003) and Van Vliet et al. (2013) also incorporated fuzzy logic into the calculation of Kappa indices, creating fuzzy Kappa and fuzzy Kappa simulation. They took the degree of spatial and thematic mismatch into account when calculating Kappa. In other words, two maps may be said to show partial agreement if the validation pixel or point is close to the compared pixel. The same would apply if the pixels were allocated to different classes, but with similar meanings.
Utility
Exercises | |
---|---|
1. To validate a map against reference data/map 2. To validate a simulation against a reference map 3. To validate a simulation against a reference map at the category level |
Kappa indices enable us to test the similarity between two sources of spatial information. If we have one map and reference data, we can determine to what extent the map we want to validate agrees with the reference data.
The main advantage of Kappa indices is that they provide a standard measure. Kappa agreement always ranges between −1 and 1, where 1 means total agreement, −1 total disagreement and 0 random agreement. These are universal measures, which means that the performance of a LUC classification exercise or a LUCC modelling exercise can be compared with the performance typically achieved in these exercises.
There are many critics of the widespread use of Kappa metrics, especially in LUCC modelling validation. There is now a general consensus that these indices should not be the only validation measures used when evaluating modelling exercises and maps. More information about the limitations of Kappa indices and the criticisms levelled against them can be found in Pontius Jr. and Millones (2011) and Van Vliet et al. (2011).
QGIS Exercise
Available tools | |
---|---|
• Processing Toolbox GRASS Raster r.kappa • Semi-Automatic Classification Plugin Tab: Postprocessing Section: Accuracy |
QGIS does not include many tools for calculating Kappa indices. The Cohen’s Kappa index can be obtained through the associated GRASS module. The Semi-Automatic Classification plugin also calculates the Kappa index, globally and at the category level, when doing the cross-tabulation (see Chapter “Basic and Multiple-Resolution Cross-Tabulation to Validate Land Use Cover Maps”). The other variants of Kappa are not available through QGIS or any of its pattern software, like R. Those who would like to calculate these indices are referred to the Map Comparison Kit, which is also available for free.Footnote 1
Exercise 1. To validate a map against reference data/map
Aim
To test the validity of the CORINE 2011 land use map, take the SIOSE land use map as a reference. In this way, we can answer the following question: assuming that the SIOSE map shows the true situation, how true is the CORINE map?
Materials
SIOSE Land Use Map Asturias Central Area 2011
CORINE Land Use Map Asturias Central Area 2011
Requisites
The two maps must be rasters and have the same extent, spatial resolution, projection and legend. If they do not have the same legend, the user must reclassify the maps in such a way that they comply with this requirement.
Execution
Step 1
Open the r.kappa function and fill in the required parameters: raster to be validated (CORINE map) and reference raster (SIOSE map) (Fig. 11).
Results and Comments
Once the function has been executed, QGIS creates a new text file (.txt) in the specified folder. Users must manually access this folder to open the text file and see the results of the analysis. These include a cross-tabulation matrix of the maps, together with the Kappa value. For the two maps assessed, we obtained the following Kappa:
where 1 means total agreement, −1 total disagreement and 0 random agreement. A Kappa index value of 0.88 means that the two maps are very similar and therefore that our map has been validated. As a general rule, Kappas above 0.7–0.8 are considered good enough for validation. Kappas above 0.9 indicate very high agreement.
In our case, it is always important to bear in mind that SIOSE is made at a more detailed scale than CORINE. The two maps have different minimum mapping units and minimum mapping widths, which means that perfect agreement is impossible. The SIOSE map will always draw features that are not detected in CORINE because of its coarser scale. Kappa scores of almost 0.9, like this one, show almost perfect agreement between the two sources.
Users can also assess the agreement between CORINE and SIOSE at the category level so as to obtain more information about the similarities and dissimilarities between the two maps. To compute these metrics, they should refer to Exercise 3, using the Semi-Automatic Classification Plugin instead of r.kappa.
Exercise 2. To validate a simulation against a reference map
Aim
To validate the simulation obtained by our land use/cover change modelling exercise.
Materials
Simulation CORINE Asturias Central Area 2011
CORINE Land Use Map Asturias Central Area 2011
Requisites
The two maps being compared must be rasters and have identical resolution, extent, projection and legend. For proper validation, the reference map must refer to the same date for which the landscape was simulated.
Execution
Step 1
Open the r.kappa function and fill in the required parameters: raster to be validated (Simulation) and reference raster (CORINE 2011) (Fig. 12).
Results and Comments
QGIS will create a text file in the specified folder. This file contains the Kappa value for our simulation:
where 1 means total agreement. The Kappa value indicates that the two maps are almost the same. However, this does not mean that the changes we simulated are the same as the changes that took place on the reference map (CORINE 2011) as compared to the map used as the starting point for our modelling exercise (CORINE 2006).
In our simulation, most of the landscape remains unchanged. The high Kappa value indicates that we have correctly modelled the persistence of these unchanged areas. However, it is difficult to draw any meaningful conclusions about how closely the changes we simulated fit the changes observed between the CORINE 2011 and 2006 maps. These changes only affect very small parts of the maps and, therefore, do not have a meaningful impact on the Kappa index when evaluating the agreement between the entire area of the maps.
In order to gain a better picture as to how well the simulated changes fit the changes in the reference maps, other complementary metrics also described in this book can be used, such as the quantity and allocation disagreement or the figure of merit (see Sects. 3 and 4 in Chapter “Pontius Jr. Methods Based on a Cross-Tabulation Matrix to Validate Land Use Cover Maps”). The agreement between simulated and reference changes can also be assessed using Kappa simulation, although this metric is not currently implemented in any tool in QGIS or in its associated software, such as R.
Users can also evaluate the kappa agreement between the simulation and the reference map at the category level, for which purposes they should refer to the next exercise, Exercise 3.
Exercise 3. To validate a simulation against a reference map at the category level
Aim
To validate a simulation obtained by our land use/cover change modelling exercise at the general and category level, focusing on a specific category.
Materials
CORINE Land Cover Map Val d’Ariège 2018
Simulation LCM Val d’Ariège 2018
Requisites
The two maps to be compared must be rasters and have identical resolution, extent, projection and legend. For proper validation, the reference map must refer to the same date for which the landscape was simulated.
Execution
Step 1
The Kappa index can be calculated at the category level for all the categories in our map using the Semi-Automatic Classification Plugin. To this end, open the plugin and select the Accuracy (Postprocessing) option from the menu. Then choose the rasters to be assessed, i.e. the simulation and the reference map (Fig. 13). It is also important to indicate the code for no data or background. In our case, the code is 10.
Results and Comments
After executing the tool, we obtain a raster that cross-tabulates the compared maps and a CSV file with the cross-tabulation matrix, the overall, user’s and producer’s accuracy values and the Kappa indices of agreement, overall and per category. This information will also be displayed in the output window. For detailed information about how to interpret the matrices and the user’s and producer’s accuracy values, please refer to the Sect. 4 in Chapter “Pontius Jr. Methods Based on a Cross-Tabulation Matrix to Validate Land Use Cover Maps”.
The Kappa values for the two maps show high levels of agreement at both a general level and for all categories (Table 8). The Kappa values for the “Open spaces with little or no vegetation” and “Water surfaces” categories are 1, which means perfect agreement. In other words, there are no differences between the two maps for these classes. This makes sense because they were not simulated in our modelling exercise.
The class with the lowest Kappa value is “Built-up areas”. This indicates that many of the changes in this category have not been correctly simulated, which is to be expected given the dynamism of this category when compared with others such as forest or water surfaces. It is normally easier to simulate static land categories than changing ones. This explains why “Built-up” areas obtained a very low Kappa score compared to the overall score (Table 8).
Although these results offer some clues as to how well the changes in some categories were simulated, to obtain a more detailed understanding other methods and metrics should be used, such as the quantity and allocation disagreement and the figure of merit (see Sects. 3 and 4 in Chapter “Pontius Jr. Methods Based on a Cross-Tabulation Matrix to Validate Land Use Cover Maps”) or the Kappa simulation metrics. Whereas the Kappa metrics calculated here assess the agreement between persistent and changing areas in the compared and the reference maps, the other tools and methods focus on the specific areas that change between the initial and the final year of the simulation. This is a key element for understanding the success of our simulation, as it is easier to model persistence than change.
4 Agreement Between Maps at Overall and Stratum Level
Description
The aim is to assess the agreement between map pairs such as a reference map and a simulation map, at different levels: overall agreement for the whole map, agreement for a given stratum, a smaller area, formed by a particular territory, LUC category or transition or by sample areas according to a gradient such as distance to a road. The purpose of this validation method is encapsulated in the following question: Does a particular item or area of interest show the same prediction score as the whole map?
Utility
Exercises | |
---|---|
1. To validate simulated changes against a reference map of changes |
A given map (LUC map, simulation) can be evaluated more precisely at spatial level (specific territory), category level (Is the simulation closer to the real situation for built-up areas or for forests?) or specific transitions (Does the model work better for the transition from forest to agriculture or from forest to pasture?). In this context, the entire area of interest can be used as a guide for interpreting particular simulation scores.
QGIS Exercise
Available tools | |
---|---|
• Raster Raster calculator • Processing Toolbox GRASS Raster (r.*) r.kappa Raster analysis Reclassify by table Raster layer unique values report |
Agreement between maps at the overall and stratum levels is more a validation approach than a specific method. Accordingly, there are no specific tools available in QGIS to carry out this analysis, as the used tool will depend on what type of analysis will be carried out at the overall and stratum levels.
For general operations, we will make use of the QGIS Raster Calculator. a generic tool performing all kinds of raster calculations. To calculate Kappa indices at the global and stratum levels, we will make use of r.kappa. For more information about this tool, please refer to the previous section.
Exercise 1. To validate simulated changes against a reference map of changes
Aim
To find out if the agreement between an observed (reference map) and a simulated transition varies for several distance-based categories resulting from a driver (e.g. distance to roads).
Materials
CORINE Land Cover Map Val d’Ariège 2012
CORINE Land Cover Map Val d’Ariège 2018
Simulation LCM Val d’Ariège 2018
Distance to roads
Requisites
All maps must be in raster format with the same resolution, extent and spatial reference system (SRS).
Execution
Step 1
First, we have to obtain the observed and simulated transitions from agriculture and pasture land to built-up areas over the period 2012–2018. Using the raster calculator, we extract the observed (“CLC_2012@1” = 2 AND “CLC_2018@1” = 1) and the simulated (“CLC_2012@1” = 2 AND “CLC_predict_2018@1” = 1) transition from agriculture and pasture land (Category 2) to built-up areas (Category 1). The result is shown in Fig. 14 (observed change appears in cyan, simulated change in red).
Step 2
The Reclassify by table raster analysis tool is used to transform the map showing the continuous distance from roads into various different classes. Given the dense road network, we intentionally apply a progressive interval as shown in Fig. 15.
Figure 16 shows the general result and the result for a detailed area with the following classes: the road network itself (distance is zero), distance class 1 (less than 100 m), class 2 (100–300 m), class 3 (300–1000 m) and class 4 (more than 1000 m).
Step 3
The next step is to compute observed and predicted transitions from Category 2 to Category 1 as a function of the road distance classes. To this end, we use the Raster calculator again to calculate: i) the road distance class map multiplied by the observed transition map and ii), the road distance class map multiplied by the simulated transition map. The results can be seen in Fig. 17, in which the two maps show the transition from 2 to 1 as a function of road distance. The map on the left shows the observed transition and the map on the right shows the simulated transition, with a detailed area in both cases.
Step 4
Finally, we compare observed and simulated transitions as a function of distances classes (strata). We use the Raster layer unique values report raster analysis tool to calculate the number of pixels for each road distance category (observed and simulation) for the transition from category 2 to 1 as shown in Fig. 18 (left for observed, right for simulated transition).
The results are then converted into percentage as shown in Table 9.
Results and Comments
The result is that there are almost three times as many observed transitions as predicted transitions. However, the proportion of near-to-road transitions is approximately the same. In conclusion, the model underestimates the quantity of agriculture and pasture land that is transformed into built-up areas, although in the areas close to roads, it accurately predicted what happened in the Ariège Valley between 2012 and 2018.
5 Accuracy Assessment Statistics
Description
The thematic accuracy assessment statistics are a set of parameters that measure the degree of agreement between the LUC map and the reference data (for more details about reference data, see Chapter “Sample Data for Thematic Accuracy Assessment in QGIS”). Overall accuracy, user’s accuracy and producer’s accuracy are reported in many studies. Some additional accuracy measures such as the standard error of overall accuracy and the confidence intervals for the adjusted areas are also helpful.
All these parameters are mainly derived from the error or confusion matrix (see Chapter “Basic and Multiple-Resolution Cross-Tabulation to Validate Land Use Cover Maps”). This matrix is obtained from a cross-tabulation between the reference data and the thematic map. In the resulting table, the reference data are generally shown in the columns and the map data in the rows (Table 10).
In Table 10, nij refers to the sample count of spatial units in cell (i, j), ni+, n+j denote the sum of nij in each row and column, and n is the sample size; n+j is the number of spatial assessment units belonging to class j, according to the reference data, and ni+ is the number of spatial units belonging to class i according to the thematic map.
Expressing the error matrix in terms of area proportions instead of sample counts enables the calculation of unbiased area estimators. The area proportions (\({\widehat{p}}_{ij}\)) are defined as follows:
where Wi = (Map area of class i)/(Total area of the map).
Based on these area proportions, the overall estimated accuracy (\(\widehat{O}\)), user’s accuracy (\({\widehat{U}}_{i}\)) and producer’s accuracy (\({\widehat{P}}_{j}\)) are calculated with the following equations:
Errors of commission and omission are complementary concepts of the user’s and producer’s accuracy metrics, respectively (i.e. error = 1 − accuracy). An error of commission occurs when a feature is included in a thematic class to which it does not belong. In contrast, an error of omission occurs when a feature is excluded from the thematic class to which it belongs (Finegold et al. 2016).
Errors in the classification process can increase the uncertainty in area estimation. However, the pixel count multiplied by pixel size is often used as an estimator of the true area on the ground. This measurement is strongly affected by both omission and commission errors (Gallego, 2004). Olofsson et al. (2013) proposed an unbiased area estimate using an adjustment factor obtained from the error matrix:
\({\widehat{A}}_{j}\) is the unbiased area estimator or adjusted area. In this case, the area estimator obtained directly from the map (Atotal) is then adjusted by a factor obtained from the reference data. If there are more samples labelled as class j in the reference sample than in the map, then \({\widehat{A}}_{j}\) will be larger than the area obtained directly by pixel counting.
Utility
Exercises | |
---|---|
1. To validate a map against reference data/map |
The statistics obtained from the thematic accuracy assessment are not only descriptors of the map quality but also represent a fundamental input for calculating unbiased area estimators. Additionally, they provide the necessary elements to decide whether to increase the number of sampling sites in the reference data, if the precision obtained does not meet the initial mapping objectives.
QGIS Exercise
Available tools | |
---|---|
• MapAccurAssess Plugin |
In QGIS, several plugins, such as Semi-Automatic Classification, AcATaMa and MapAccurAssess, can be used to calculate the map accuracy statistics. All three plugins provide the overall accuracy, producer’s accuracy, user’s accuracy and the error matrix, although AcATaMa and MapAccurAssess also report some additional statistics about the adjusted areas and their levels of accuracy.
In this exercise, we use the MapAccurAssess plugin because it can use a shapefile directly with the reference data. The results provided by this plugin, based on Olofsson et al. (2013), include the error matrix and a table with the following statistics: the class area, the producer’s and user’s accuracy values, the adjusted areas and their confidence intervals. It also includes the overall accuracy and its respective standard error.
This plugin is a test version and has not yet been accepted in the official QGIS repositories.
Exercise 1. To validate a map against reference data/map
Aim
To validate a LUC map for the Marqués de Comillas study area by computing accuracy assessment statistics and the error matrix via cross-tabulation of the reference data and the thematic map.
Materials
Marqués de Comillas Land Use Cover Map 2019
Photointerpreted reference dataset—Marqués de Comillas 2019 (reference dataset resulting from the exercise in Sect. 2 in Chapter “Sample Sata for Thematic Accuracy Assessment in QGIS”)
Requisites
In order to compute the areas, the land cover map must be in raster format (GeoTiff) in any cartographic projection. The reference data must be contained in a shapefile with the same type of projection as the map. The shapefile attribute table must contain at least two columns, showing the value for the thematic class obtained from the land cover map and the value according to field ground-truthing or photointerpretation. Both columns must have the same data type (integer or text) to be comparable. Each row of the table corresponds to one reference site.
Execution
Step 1
Install the MapAccurAssess plugin. Should you need help, please see Chapter “About This Book” and the plugin’s documentation.
Step 2
If the plugin has been successfully installed, an icon should appear in the main graphics panel. To start the exercise, click on this icon. Alternatively, go to the Complements menu, select Accuracy Assessment and then Accuracy Assessment again.
Step 3
Select the shapefile with the reference samples (Photo-interpreted reference dataset—Marqués de Comillas 2019)Footnote 2 and indicate the column with the reference data and the column with the values for the thematic classes used in the map. After that, select the land cover map you want to assess (Marqués de Comillas Land Use Land Cover Map 2019). If the map is in vector format, indicate the column containing the thematic class values. Finally, select a folder where the results will be saved and click “Accept” (Fig. 19).
Results and Comments
The output of this plugin consists of two CSV tables. The first contains the error matrix (Table 11), and the second contains the map accuracy assessment statistics (Table 12). These statistics are as follows: user’s accuracy, producer’s accuracy, thematic class area (as retrieved from the map), the area adjusted by the error level (Area_adj), the confidence intervals for the adjusted area (CI_sup and CI_inf) and the overall accuracy (O).
According to the data from this exercise, the overall accuracy of the map is 0.91. In other words, there is a high probability (91%) that a randomly selected location on the map will be correctly classified. Note that the thematic class with the lowest accuracy is 130 (Wetland), with a user accuracy of 0.7 and a producer accuracy of 0.21. This class covers a small area (252 ha according to the map). We decided to keep this class to show that illogical situations can occur when there is only a small number of sampling sites, e.g. negative areas. However, we recommend merging class 130 with another class of similar characteristics and recomputing.
Notes
- 1.
- 2.
The photointerpreted reference dataset for Marqués de Comillas (RandomSample_Buffer.shp) was obtained from the exercise in Sect. 2 in Chapter “Sample Data for Thematic Accuracy Assessment in QGIS”. This layer has two columns, “class” and “refer_data”. The first contains the values for the thematic classes used in the map and the second contains the reference data, which were obtained from the photointerpretation of satellite images.
References
Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20:37–46
FAO (1995) Forest resources assessment 1990. In: Global synthesis. FAO, Rome
Finegold Y, Ortmann A, Lindquist E, d’Annunzio R and Sandker M (2016) Map accuracy assessment and area estimation: a practical guide. National forest monitoring assessment working paper no. 46/E. Food and Agriculture Organization of the United Nations. Rome
Gallego FJ (2004) Remote sensing and land cover area estimation. Int J Remote Sens 25:3019–3047. https://doi.org/10.1080/01431160310001619607
Hagen A (2002) Multi-method assessment of map similarity. In: 5th AGILE conference on geographic information science. Mallorca, pp 1–8
Hagen A (2003) Fuzzy set approach to assessing similarity of categorical maps. Int J Geogr Inf Sci 17:235–249. https://doi.org/10.1080/13658810210157822
Islam S, Zhang M, Yang H, Ma M (2019) Assessing inconsistency in global land cover products and synthesis of studies on land use and land cover dynamics during 2001 to 2017 in the southeastern region of Bangladesh. J Appl Remote Sens 13:1. https://doi.org/10.1117/1.JRS.13.048501
Olofsson P, Foody GM, Stehman SV, Woodcock CE (2013) Making better use of accuracy data in land change studies: estimating accuracy and area and quantifying uncertainty using stratified estimation. Remote Sens Environ 129:122–131. https://doi.org/10.1016/j.rse.2012.10.031
Pontius RG Jr (2000) Quantification error versus location error in comparison of categorical maps. Photogramm Eng Remote Sensing 66:1011
Pontius RG Jr, Millones M (2011) Death to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment. Int J Remote Sens 32:4407–4429. https://doi.org/10.1080/01431161.2011.552923
Puyravaud J-P (2003) Standardizing the calculation of the annual rate of deforestation. For Ecol Manage 177(1–3):593–596
Van Vliet J, Bregt AK, Hagen-Zanker A (2011) Revisiting Kappa to account for change in the accuracy assessment of land-use change models. Ecol Modell 222:1367–1375. https://doi.org/10.1016/j.ecolmodel.2011.01.017
Van Vliet J, Hagen-Zanker A, Hurkens J, Van Delden H (2013) A fuzzy set approach to assess the predictive accuracy of land use simulations. Ecol Modell 261–262:32–42. https://doi.org/10.1016/j.ecolmodel.2013.03.019
Yang Y, Xiao P, Feng X, Li H (2017) Accuracy assessment of seven global land cover datasets over China. ISPRS J Photogramm Remote Sens 125:156–173. https://doi.org/10.1016/j.isprsjprs.2017.01.016
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Copyright information
© 2022 The Author(s)
About this chapter
Cite this chapter
Mas, JF., García-Álvarez, D., Paegelow, M., Domínguez-Vera, R., Castillo-Santiago, M.Á. (2022). Metrics Based on a Cross-Tabulation Matrix to Validate Land Use Cover Maps. In: García-Álvarez, D., Camacho Olmedo, M.T., Paegelow, M., Mas, J.F. (eds) Land Use Cover Datasets and Validation Tools. Springer, Cham. https://doi.org/10.1007/978-3-030-90998-7_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-90998-7_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-90997-0
Online ISBN: 978-3-030-90998-7
eBook Packages: Earth and Environmental ScienceEarth and Environmental Science (R0)