Metrics Based on a Cross-Tabulation Matrix to Validate Land Use Cover Maps

Abstract

The overlaying of two map layers is a standard GIS procedure. As we saw in the previous chapter, it enables us to compute the intersection between two feature classes and cross-tabulate either the area or the pixel count of the intersecting features depending on whether raster or vector data are being used. Cross-tabulation can be used to evaluate different topics depending on the nature of the input data. In this chapter, cross-tabulation is used to assess land cover changes, the spatial agreement between maps and map accuracy. In Sect. 1, Land use/cover changes (LUCC) are quantified by comparing two LUC maps, computing different indices of change and creating a change matrix. In Sect. 2, we used various metrics to evaluate the spatial agreement between two maps. This procedure was applied to compare a LUC map with a reference map, a simulated LUC map with a reference map and a simulated LUCC map with a reference map of changes. Section 3 introduces the Kappa indices, which allow us to assess the agreement between two datasets, given the agreement expected by random coincidence. We used the indices to compare observed or simulated maps with a reference map. In Sect. 4 we evaluate the agreement between maps at a global level (the entire map) by focusing on a specific feature such as a smaller area or a particular category (stratum level). Finally, in Sect. 5, the cross-tabulation between a map and reference sample data is used to assess the thematic accuracy of the map by calculating various different accuracy indices. We present examples of analyses based on cross-tabulation for four different cases: To validate a series of maps with two or more time points, to validate a map against a reference map, to validate a simulation against a reference map and to validate simulated changes against a reference map of changes. In the example exercises, we use CORINE and SIOSE maps from the Asturias Central Area and Ariège Valley datasets and maps of the Marqués de Comillas region of south-eastern Mexico (MarquesLUC dataset). The cross-tabulation techniques proposed by Robert Gilmore Pontius Jr. are applied in Chapter “Pontius Jr. Methods Based on a Cross-Tabulation Matrix to Validate Land Use Cover Maps”.

1 Change Statistics

Description

Land use/cover change (LUCC) can be quantified by comparing two maps or two classified images that represent land cover at two different dates.

Absolute change (AC) is the difference in the area covered by a category (category area) between two dates and is usually expressed in hectares or square kilometres.

$${\text{AC}} = A_2 - A_1$$

where A₁ and A₂ are the category areas in question at dates 1 and 2, respectively.

AC can be divided by the number of years between the two dates to obtain the average annual change area over the study period.

Relative change (RC) is obtained by normalizing the absolute change value by the category area at date 1.

$${\text{RC}} = \left( {A_2 - \, A_1 } \right)/A_1$$

This formula expresses the proportion of the category area that changed over the study period.

Other indices of LUCC include rates of change. The most popular rate of change is the annual rate of deforestation proposed by the FAO (1995). This indicator is based on the compound interest law. It expresses the proportion of the category area that changes in one year.

$$t={\left(\frac{{A}_{2}}{{A}_{1}}\right)}^{1/\left({t}_{2}-{t}_{1}\right)}-1$$

An alternative equation, also based on the compound interest law, was proposed by Puyravaud (2003).

$$r=\frac{1}{\left({t}_{2}-{t}_{1}\right)}\mathrm{ln}\frac{{A}_{2}}{{A}_{1}}$$

Both formulae give similar results except when LUCC is very high, in which case r is significantly higher than t (Puyravaud, 2003).

All the change indices presented above indicate net change, which results from the balance after gross losses have been subtracted from gross gains. For instance, a given forest category could show an absolute change of −2 ha, which could be erroneously interpreted as very little change, but in fact is the result of two opposing processes: the deforestation of 202 ha compensated by the reforestation of 200 ha. A more detailed analysis of change dynamics can be obtained by cross-tabulating the two maps at two different dates and drawing up a change matrix. The change matrix is a cross-tabulated table indicating the area covered by each change (or permanence) between a category at date 1 and another category at date 2. Many change indices can be obtained from this matrix (see, for example, Sect. 2).

Utility

Exercises
1. To validate a series of maps with two or more time points

Indices of change are widely used to assess LUCC. Normalized indices, such as rates of change, enable us to compare the rate of change between regions of different sizes.

QGIS Exercise

Available tools
• Processing R provider Plugin      Change_Statistics.rsx R script

The indices of change proposed in this document are based on the area statistics for the two maps. These could be efficiently computed using a spreadsheet program. However, we suggest using a simple R script using the QGIS Processing R provider plugin. The script generates a table containing the absolute change (AC) area, the relative change (RC) area (both in hectares), the rates of change based on FAO and Puyravaud (2003) and the change matrix.

Exercise 1. To validate a series of maps with two or more time points

Aim

To assess LUCC in the Ariege study area using the CORINE Land Use maps dated 2000 and 2018.

Materials

CORINE Land Cover Map Val d’Ariège 2000

CORINE Land Cover Map Val d’Ariège 2018

Requisites

All maps must be in raster format and have the same resolution, extent and projection.

Execution

If necessary, install the Processing R provider plugin and download the R script Change_Statistics.rsx into the R scripts folder (processing/rscripts). For more information, see Chapter “About This Book”.

Step 1

Then, execute the script and fill in the required parameters (names and dates of the two maps and the output table) as shown in Fig. 1.

Fig. 1

Exercise 1. Step 1. Change Statistics R script

The script generates two tables in CSV format: a table showing the change indices (Table 1) and the change matrix (Table 2).

Table 1 Results from Exercise 1 displayed in the “output” window of the Change Statistics R script. Change indices

Table 2 Results from Exercise 1 displayed in the “output” window of the Change Statistics R script. Change matrix

Results and Comments

The two land covers with the most significant absolute change are Categories 1 (built-up) and 2 (agriculture). During the period 2000–2018, the built-up area is increased by 1840 ha, and agriculture lost 1987 ha. The built-up area increased by over 50%. The rates of change resulting from the two equations are very similar. The two categories with the largest rates are built-up (Category 1) and water (Category 6) areas. Over the period 2000–2018, the area covered by these categories increased by around 2.5% a year. Categories 2 and 4 (agriculture and scrubs) present a negative net change rate, indicating that their areas have been shrinking. The change matrix gives us more information about the processes of change. One surprising change is the transition from 1 (built-up) to 6 (water). On closer observation, it was found that pits had been filled with water to create reservoirs.

2 Areal and Spatial Agreement Metrics

Description

Different authors have proposed a series of metrics that evaluate the areal and spatial agreement between two land use/cover maps or between any of their categories. These metrics are obtained from the cross-tabulation matrix and summarize in a single value the agreement between two maps.

The metrics are based either on the comparison of the proportion of total area occupied by a particular category on two maps or on the spatial coincidence of the pixels allocated to any given category on two maps. This review includes some of the most recently developed metrics.

Yang et al. (2017) proposed the overall spatial agreement (A₀) and the individual spatial agreement (A_i) metrics. They are formulated as follows:

$$A_0 = \frac{{\sum_1^N XY_{ii} }}{M} \times 100$$

$$A_1 = \frac{{XY_{ii} }}{{\left( {X_i + Y_i } \right)/2}} \times 100$$

where ${X}_{i}$ refers to the number of pixels belonging to category $i$ in map $X$, ${Y}_{i}$ refers to the number of pixels belonging to category $i$ in map $Y$, ${XY}_{ii}$ refers to the number of pixels belonging to category $i$ in both maps $X$ and $Y$, $N$ is the number of categories into which the pixels are classified and $M$ is the number of pixels into which the maps are divided.

The overall spatial agreement (A₀) and the overall spatial inconsistency (OSI) metrics assess the spatial agreement between the categories in two maps. One metric can be obtained from the other. Whereas A₀ shows the spatial agreement (0–100%), the OSI shows the spatial disagreement (0–100%). Added together, they come to 100.

Islam et al. (2019) proposed the overall areal inconsistency (OAI), the individual areal inconsistency (AIC) and the overall spatial inconsistency (OSI) metrics. They are formulated as follows:

$$\mathrm{AIC}=\left|\left({X}_{i}-{Y}_{i}\right)\right|/2$$

$$\mathrm{OAI}={\sum }_{i}^{n}\mathrm{AIC}$$

$${\text{OSI}} = \frac{{N_{\left( {i \ne j} \right)} }}{N} \times 100$$

where ${X}_{i}$ refers to the percentage of the total area represented by category $i$ in map $X$, ${Y}_{i}$ refers to the percentage of the total area represented by category $i$ in map $Y$, $n$ is the total number of categories, $N$ is the number of pixels and ${N}_{\left(i\ne j\right)}$ is the number of pixels assigned to one category in Map X and a different category in Map Y.

Overall areal inconsistency (OAI) shows the agreement between two maps in terms of category proportions and is expressed in values of between 0 and 100. Users can also assess the areal and spatial agreement/disagreement at a category level through the individual areal inconsistency (AIC) and individual spatial agreement (A_i) metrics. The values for the latter range from 0 to 100, and a value of 100 means perfect agreement.

AIC does not have a standard scale of values, as these depend on the proportion of the total area of the map allocated to the category. It is therefore very difficult to compare the values for this metric between classes, so limiting its usefulness.

Utility

Exercises
1. To validate a map against reference data/map 2. To validate a simulation against a reference map 3. To validate simulated changes against a reference map of changes

The areal and spatial agreement metrics assess the similarity between the two maps. They are obtained from the cross-tabulation matrix and therefore do not provide any additional information, in that the values they provide can also be obtained from the matrix. However, they are standard metrics that allow us to measure the agreement between two maps and summarize it in a single figure. In this sense, they are similar to the user’s and producer’s accuracy metrics and to Kappa indices. They are also complementary to quantity and allocation (dis)agreement metrics, as they can differentiate between spatial and quantity agreements.

These metrics can be used to assess how similar a land use/cover map is to another map used as a reference, i.e. the real situation on the ground. They can also be used to check the similarity between a simulation and the reference map for the same year.

QGIS Exercise

Available tools
• Processing Toolbox      R         Areal and spatial agreement metrics           Individual Areal Inconsistency.rsx           Individual Spatial Agreement.rsx           Overall Areal Inconsistency.rsx           Overall Spatial Agreement.rsx           Overall Spatial Inconsistency.rsx

QGIS has no specific tool for calculating the metrics proposed by Yang et al. (2017) and Islam et al. (2019). However, these can be easily calculated using the cross-tabulation matrix via the formulae set out above. We have also developed various different tools with R to automatically calculate each metric with QGIS.

When using these R scripts, the categories in LUC rasters must be coded in consecutive numbers, from 1 to the maximum number of categories used in the map. Thus, in a raster with five categories, the categories must be coded as 1, 2, 3, 4 and 5.

Exercise 1. To validate a map against reference data/map

Aim

To validate the CORINE 2011 land use map, take the SIOSE 2011 land use map as a reference. We will be focusing particularly on how the “urban fabric” and “industrial and commercial areas” categories are mapped in CORINE 2011.

Materials

CORINE Land Use Map Asturias Central Area 2011

SIOSE Land Use Map Asturias Central Area 2011

Requisites

All maps must be rasters and have the same resolution, extent, projection and number of categories. LUC categories must be coded consecutively from 1 to the maximum number of categories considered.

Execution

If necessary, install the plugin Processing R provider and download the R scripts indicated above in the “Available Tools” table. Paste the R scripts into the R scripts folder. For more information, see Chapter “About This Book”.

Step 1

Our maps do not comply with one of the requisites of the tools we will be using, in that the categories in our LUC maps are coded from 0 (agricultural areas) to 12 (background). The first step is therefore to reclassify the maps to ensure that all the categories are coded consecutively from 1 to 13. This is done using the Reclassify by table tool (Figs. 2 and 3).

Fig. 2

Exercise 1. Step 1. Reclassify by Table

Fig. 3

Exercise 1. Step 1. Reclassification table of the Reclassify by Table tool

Step 2

Once the maps comply with the requirements of the tools, the different metrics can then be calculated. To test the overall agreement between the assessed and the reference maps, we will calculate the overall spatial agreement (A₀), the overall areal inconsistency (OAI) and the overall spatial inconsistency (OSI). For their part, individual areal inconsistency (AIC) and individual spatial agreement (A_i) are used to assess agreement specifically for the “urban fabric” and “industrial and commercial areas” categories.

To calculate all these metrics, open the respective tool and select the maps you want to compare (Fig. 4): first the CORINE map and second the SIOSE map, which is used as a reference. In all cases, the background value of the maps (13) must also be indicated. Finally, specify the folder where the result from each tool will be stored.

Fig. 4

Exercise 1. Step 2. Overall Spatial Agreement R script

For class-specific metrics indicate the codes of the classes you want to validate (Fig. 5). In this case, we will be calculating these metrics for two different classes: urban fabric, which is coded 3 after reclassification, and industrial and commercial areas, which is coded 4.

Fig. 5

Exercise 1. Step 2. Individual Spatial Agreement R script

Results and Comments

After calculating all the different metrics, a numerical output is obtained for each one (Tables 3 and 4). This output is also stored in a CSV file in the selected folder.

Table 3 Results from Exercise 1. Overall agreement indices

Table 4 Results from Exercise 1. Individual agreement indices

There is a high overall spatial agreement (close to 90%) between the two maps and low areal inconsistency (around 3%). We can therefore consider the CORINE land cover map for 2011 as validated. The category proportions between CORINE and SIOSE are almost identical and the spatial agreement is very high. The disagreements between the two maps are due to their different degree of detail, which draws small features in SIOSE that are not detected at the scale used in CORINE.

At the class level, the picture is slightly different. For the two classes we assessed (urban fabric and industrial and commercial areas) spatial agreement between the two maps to be close to 70%. Although this is a high level of agreement, it is much lower than the overall figure. This could be due to the fact that these two classes are more sensitive than others to the scale difference between SIOSE and CORINE.

In order to interpret the AIC metric, we need to first understand the proportion of total area allocated to each class on the two maps. AIC is half of the difference between the two proportions (i.e. if the proportion allocated to one class is 3% on one map and 4% on the other, the difference is 1% and AIC is 0.5). In our case, the AIC value for urban fabric is less than 0.1, which means a high level of agreement between the two maps regarding the proportion of total area allocated to this category (around 3.9%). The proportion allocated to industrial and commercial areas is around 3% in both maps and the AIC value is slightly more than 0.1. This also indicates a high level of agreement, although less than for urban fabric.

Exercise 2. To validate a simulation against a reference map

Aim

To validate the simulation obtained by our land use/cover change modelling exercise. We will focus on the two categories we have modelled actively: “urban fabric” and “industrial and commercial areas”.

Materials

CORINE Land Use Map Asturias Central Area 2011

Simulation CORINE Asturias Central Area 2011

Requisites

All maps must be rasters and have the same resolution, extent, projection and number of categories. LUC categories must be coded consecutively from 1 to the maximum number of categories considered.

Execution

Step 1

The first step is to reclassify our maps to make them comply with the requisites of the tools we will be using. These tools require the categories to be consecutively coded from 1. This means that “agricultural areas” (coded 0) must be given a new code (Fig. 3). This is done using the Reclassify by table tool (see the previous exercise).

Step 2

Once the maps comply with the requirements of the tools, we can then calculate the different areal and spatial agreement metrics using the tools available in the R toolbox.

To evaluate the global agreement between the simulation and the reference map, we will calculate the overall spatial agreement (A₀), the overall areal inconsistency (OAI) and the overall spatial inconsistency (OSI). To evaluate agreement for the categories that we actively modelled, we will calculate the individual areal inconsistency (AIC) and the individual spatial agreement (A_i).

To calculate the metrics, open the corresponding tools and indicate the following: the simulation to be evaluated, the reference map (CORINE 2011), the background value of the maps (13) and the folder where the results will be stored. For the class-specific metrics, you must also provide the codes of the classes you want to evaluate: in this case 3 (urban fabric) and 4 (industrial and commercial areas) (Fig. 6).

Fig. 6

Exercise 2. Step 2. Individual Areal Inconsistency R script

Results and Comments

Once you have finished the exercise, you will obtain an a  CSV file for each metric. The results are summarized in Tables 5 and 6.

Table 5 Results from Exercise 2. Overall agreement indices

Table 6 Results from Exercise 2. Individual agreement indices

The results show almost perfect agreement between our simulation and the reference map. The maps share the same LUC in 99% of their area and the areal inconsistency is insignificant (0.26%). A similar pattern is observed in the actively simulated classes.

These results are misleading. There is perfect agreement between our simulation and the reference map in the persistence areas. However, it is not that high for those areas modelled as changes. Because there are relatively few changes in our study area, the disagreement between the two maps in areas where change is predicted has very little impact on the overall high levels of the agreement created by the correct simulation of permanence areas. To correctly validate the changes that we simulated, we should repeat this exercise, focusing exclusively on the areas that changed in the simulation and in the reference map, as compared to the initial map of the simulation (see next exercise).

Exercise 3. To validate simulated changes against a reference map of changes

Aim

To validate the changes simulated by our land use/cover change modelling exercise.

Materials

CORINE Land Use Changes Asturias Central Area 2005–2011

Simulated CORINE changes Asturias Central Area 2005–2011

Requisites

All maps must be rasters and have the same resolution, extent, projection and number of categories. LUC categories must be coded consecutively from 1 to the maximum number of categories considered.

Execution

Step 1

Our maps do not comply with the requirements for the tools. In the map of simulated changes, the categories are not consecutively coded from 1. In addition, the reference map of changes has many more categories than the map of simulated changes. Using the Reclassify by table tool we can adjust the number of categories on the two maps to the two categories that appear in both (urban fabric and industrial and commercial areas), plus a third category covering non-changing areas and changes that were not simulated. These categories will be assigned codes 1, 2 and 3, respectively. Figures 7 and 8 show the reclassification codes that must be inputted into the Reclassify by table tool.

Fig. 7

Exercise 3. Step 1. Reclassification table of the Reclassify by Table tool (CORINE changes)

Fig. 8

Exercise 3. Step 1. Reclassification table of the Reclassify by Table tool (Simulated CORINE changes)

Step 2

After reclassifying the maps, we will calculate the following metrics to validate the simulated changes: individual areal inconsistency (AIC) and individual spatial agreement (A_i). As we are only comparing two categories, the overall metrics provide the same information as the individual ones.

For each metric, we will open the corresponding tool, indicating the map of simulated changes to be validated (Land use map 1), the reference map of changes (Land use map 2), the background value of the maps (0), the category we are going to evaluate (urban fabric, 2, Fig. 9; industrial and commercial areas, 3, Fig. 10) and the folder where the results of the analysis will be stored. We use 999 as the background value in our maps because no specific value was assigned to the background. 0 means no change, another category that must be considered in this analysis.

Fig. 9

Exercise 3. Step 2. Individual Spatial Agreemen R script (urban fabric)

Fig. 10

Exercise 3. Step 2. Individual Spatial Agreement R script (industrial and commercial areas)

Results and Comments

A  CSV file will be created for each metric. The results are summarized in Table 7.

Table 7 Results from Exercise 3.Individual agreement indices

The same amount of changes took place in the reference map of changes as in our simulation. There is no disagreement on this point. However, unlike the previous exercise, the spatial agreement between the simulated and the reference changes was very low. The A_i value for the two categories that were actively simulated was quite similar (less than 25%).

These results mean that only a quarter of the simulated changes were allocated in the same places as the changes observed on the reference map. This result, by itself, is not sufficient to consider the simulation invalid. We need to gain a better picture of the location of the changes that were simulated and their pattern. Even if they were not allocated in exactly the same places as on the reference map, they may be allocated in the same general area and follow a similar pattern, indicating that the model has correctly simulated the processes of change. To assess these aspects, we can perform a visual inspection of the reference and simulated changes on the maps, cross-tabulate them at multiple resolutions (see Sect. 2 in Chapter “Basic and Multiple-Resolution Cross-Tabulation to Validate Land Use Cover Maps”) and calculate the spatial metrics (see Sect. 1 in Chapter “Spatial Metrics to Validate Land Use Cover Maps”).

3 Kappa Indices

Description

Kappa indices assess the agreement between two sources of spatial data, corrected by the agreement that is expected by chance. They are typically used to compare the agreement between two maps and to compare one map with reference information (e.g. a collection of validation points).

The first Kappa index (Cohen’s Kappa) dates from 1960 (Cohen, 1960) and has been widely used in LUC analysis. Many variants of this first original index have been proposed. They mainly apply to the comparison between two maps. Of these, the following are of particular interest:

• Pontius Jr. (2000) split Cohen’s Kappa into three indices, called Kno, Kquantity and Klocation. These indices offer more information about the causes of the (dis)agreement between two compared maps, i.e. (dis)agreement in terms of the different allocation of the categories on the two maps and (dis)agreement in terms of the different proportions in which the categories appear on the two maps.

• Hagen (2002), following the work done by Pontius Jr., split Cohen’s Kappa into two indices, called Khistogram and Klocation. These refer to the Kappa agreement in terms of the categories appearing in the same proportions (histograms) on the two maps and the Kappa agreement due to the categories appearing in the same location on the two maps.

• Van Vliet et al. (2011) proposed the Kappa simulation, which was specifically designed for validating LUCC models. It assesses the agreement between the changes on two maps, as compared to a third map used as an initial point, corrected by the agreement expected by chance.

• Hagen (2003) and Van Vliet et al. (2013) also incorporated fuzzy logic into the calculation of Kappa indices, creating fuzzy Kappa and fuzzy Kappa simulation. They took the degree of spatial and thematic mismatch into account when calculating Kappa. In other words, two maps may be said to show partial agreement if the validation pixel or point is close to the compared pixel. The same would apply if the pixels were allocated to different classes, but with similar meanings.

Utility

Exercises
1. To validate a map against reference data/map 2. To validate a simulation against a reference map 3. To validate a simulation against a reference map at the category level

Kappa indices enable us to test the similarity between two sources of spatial information. If we have one map and reference data, we can determine to what extent the map we want to validate agrees with the reference data.

The main advantage of Kappa indices is that they provide a standard measure. Kappa agreement always ranges between −1 and 1, where 1 means total agreement, −1 total disagreement and 0 random agreement. These are universal measures, which means that the performance of a LUC classification exercise or a LUCC modelling exercise can be compared with the performance typically achieved in these exercises.

There are many critics of the widespread use of Kappa metrics, especially in LUCC modelling validation. There is now a general consensus that these indices should not be the only validation measures used when evaluating modelling exercises and maps. More information about the limitations of Kappa indices and the criticisms levelled against them can be found in Pontius Jr. and Millones (2011) and Van Vliet et al. (2011).

QGIS Exercise

Available tools
• Processing Toolbox GRASS Raster r.kappa • Semi-Automatic Classification Plugin Tab: Postprocessing Section: Accuracy

QGIS does not include many tools for calculating Kappa indices. The Cohen’s Kappa index can be obtained through the associated GRASS module. The Semi-Automatic Classification plugin also calculates the Kappa index, globally and at the category level, when doing the cross-tabulation (see Chapter “Basic and Multiple-Resolution Cross-Tabulation to Validate Land Use Cover Maps”). The other variants of Kappa are not available through QGIS or any of its pattern software, like R. Those who would like to calculate these indices are referred to the Map Comparison Kit, which is also available for free.¹

Exercise 1. To validate a map against reference data/map

Aim

To test the validity of the CORINE 2011 land use map, take the SIOSE land use map as a reference. In this way, we can answer the following question: assuming that the SIOSE map shows the true situation, how true is the CORINE map?

Materials

SIOSE Land Use Map Asturias Central Area 2011

CORINE Land Use Map Asturias Central Area 2011

Requisites

The two maps must be rasters and have the same extent, spatial resolution, projection and legend. If they do not have the same legend, the user must reclassify the maps in such a way that they comply with this requirement.

Execution

Step 1

Open the r.kappa function and fill in the required parameters: raster to be validated (CORINE map) and reference raster (SIOSE map) (Fig. 11).

Fig. 11

Exercise 1. Step 1. R.kappa

Results and Comments

Once the function has been executed, QGIS creates a new text file (.txt) in the specified folder. Users must manually access this folder to open the text file and see the results of the analysis. These include a cross-tabulation matrix of the maps, together with the Kappa value. For the two maps assessed, we obtained the following Kappa:

$${\text{Kappa}} = 0.88$$

where 1 means total agreement, −1 total disagreement and 0 random agreement. A Kappa index value of 0.88 means that the two maps are very similar and therefore that our map has been validated. As a general rule, Kappas above 0.7–0.8 are considered good enough for validation. Kappas above 0.9 indicate very high agreement.

In our case, it is always important to bear in mind that SIOSE is made at a more detailed scale than CORINE. The two maps have different minimum mapping units and minimum mapping widths, which means that perfect agreement is impossible. The SIOSE map will always draw features that are not detected in CORINE because of its coarser scale. Kappa scores of almost 0.9, like this one, show almost perfect agreement between the two sources.

Users can also assess the agreement between CORINE and SIOSE at the category level so as to obtain more information about the similarities and dissimilarities between the two maps. To compute these metrics, they should refer to Exercise 3, using the Semi-Automatic Classification Plugin instead of r.kappa.

Exercise 2. To validate a simulation against a reference map

Aim

To validate the simulation obtained by our land use/cover change modelling exercise.

Materials

Simulation CORINE Asturias Central Area 2011

CORINE Land Use Map Asturias Central Area 2011

Requisites

The two maps being compared must be rasters and have identical resolution, extent, projection and legend. For proper validation, the reference map must refer to the same date for which the landscape was simulated.

Execution

Step 1

Open the r.kappa function and fill in the required parameters: raster to be validated (Simulation) and reference raster (CORINE 2011) (Fig. 12).

Fig. 12

Exercise 2 Step 1. R.kappa

Results and Comments

QGIS will create a text file in the specified folder. This file contains the Kappa value for our simulation:

$${\text{Kappa}} = 0.99$$

where 1 means total agreement. The Kappa value indicates that the two maps are almost the same. However, this does not mean that the changes we simulated are the same as the changes that took place on the reference map (CORINE 2011) as compared to the map used as the starting point for our modelling exercise (CORINE 2006).

In our simulation, most of the landscape remains unchanged. The high Kappa value indicates that we have correctly modelled the persistence of these unchanged areas. However, it is difficult to draw any meaningful conclusions about how closely the changes we simulated fit the changes observed between the CORINE 2011 and 2006 maps. These changes only affect very small parts of the maps and, therefore, do not have a meaningful impact on the Kappa index when evaluating the agreement between the entire area of the maps.

In order to gain a better picture as to how well the simulated changes fit the changes in the reference maps, other complementary metrics also described in this book can be used, such as the quantity and allocation disagreement or the figure of merit (see Sects. 3 and 4 in Chapter “Pontius Jr. Methods Based on a Cross-Tabulation Matrix to Validate Land Use Cover Maps”). The agreement between simulated and reference changes can also be assessed using Kappa simulation, although this metric is not currently implemented in any tool in QGIS or in its associated software, such as R.

Users can also evaluate the kappa agreement between the simulation and the reference map at the category level, for which purposes they should refer to the next exercise, Exercise 3.

Exercise 3. To validate a simulation against a reference map at the category level

Aim

To validate a simulation obtained by our land use/cover change modelling exercise at the general and category level, focusing on a specific category.

Materials

CORINE Land Cover Map Val d’Ariège 2018

Simulation LCM Val d’Ariège 2018

Requisites

The two maps to be compared must be rasters and have identical resolution, extent, projection and legend. For proper validation, the reference map must refer to the same date for which the landscape was simulated.

Execution

Step 1

The Kappa index can be calculated at the category level for all the categories in our map using the Semi-Automatic Classification Plugin. To this end, open the plugin and select the Accuracy (Postprocessing) option from the menu. Then choose the rasters to be assessed, i.e. the simulation and the reference map (Fig. 13). It is also important to indicate the code for no data or background. In our case, the code is 10.

Fig. 13

Exercise 3 Step 1. Semi-Automatic Classification Plugin

Results and Comments

After executing the tool, we obtain a raster that cross-tabulates the compared maps and a CSV file with the cross-tabulation matrix, the overall, user’s and producer’s accuracy values and the Kappa indices of agreement, overall and per category. This information will also be displayed in the output window. For detailed information about how to interpret the matrices and the user’s and producer’s accuracy values, please refer to the Sect. 4 in Chapter “Pontius Jr. Methods Based on a Cross-Tabulation Matrix to Validate Land Use Cover Maps”.

The Kappa values for the two maps show high levels of agreement at both a general level and for all categories (Table 8). The Kappa values for the “Open spaces with little or no vegetation” and “Water surfaces” categories are 1, which means perfect agreement. In other words, there are no differences between the two maps for these classes. This makes sense because they were not simulated in our modelling exercise.

Table 8 Results from Exercise 3. Kappa indices: overall and per category

The class with the lowest Kappa value is “Built-up areas”. This indicates that many of the changes in this category have not been correctly simulated, which is to be expected given the dynamism of this category when compared with others such as forest or water surfaces. It is normally easier to simulate static land categories than changing ones. This explains why “Built-up” areas obtained a very low Kappa score compared to the overall score (Table 8).

Although these results offer some clues as to how well the changes in some categories were simulated, to obtain a more detailed understanding other methods and metrics should be used, such as the quantity and allocation disagreement and the figure of merit (see Sects. 3 and 4 in Chapter “Pontius Jr. Methods Based on a Cross-Tabulation Matrix to Validate Land Use Cover Maps”) or the Kappa simulation metrics. Whereas the Kappa metrics calculated here assess the agreement between persistent and changing areas in the compared and the reference maps, the other tools and methods focus on the specific areas that change between the initial and the final year of the simulation. This is a key element for understanding the success of our simulation, as it is easier to model persistence than change.

4 Agreement Between Maps at Overall and Stratum Level

Description

The aim is to assess the agreement between map pairs such as a reference map and a simulation map, at different levels: overall agreement for the whole map, agreement for a given stratum, a smaller area, formed by a particular territory, LUC category or transition or by sample areas according to a gradient such as distance to a road. The purpose of this validation method is encapsulated in the following question: Does a particular item or area of interest show the same prediction score as the whole map?

Utility

Exercises
1. To validate simulated changes against a reference map of changes

A given map (LUC map, simulation) can be evaluated more precisely at spatial level (specific territory), category level (Is the simulation closer to the real situation for built-up areas or for forests?) or specific transitions (Does the model work better for the transition from forest to agriculture or from forest to pasture?). In this context, the entire area of interest can be used as a guide for interpreting particular simulation scores.

QGIS Exercise

Available tools
• Raster      Raster calculator • Processing Toolbox      GRASS         Raster (r.*)           r.kappa      Raster analysis           Reclassify by table           Raster layer unique values report

Agreement between maps at the overall and stratum levels is more a validation approach than a specific method. Accordingly, there are no specific tools available in QGIS to carry out this analysis, as the used tool will depend on what type of analysis will be carried out at the overall and stratum levels.

For general operations, we will make use of the QGIS Raster Calculator. a generic tool performing all kinds of raster calculations. To calculate Kappa indices at the global and stratum levels, we will make use of r.kappa. For more information about this tool, please refer to the previous section.

Exercise 1. To validate simulated changes against a reference map of changes

Aim

To find out if the agreement between an observed (reference map) and a simulated transition varies for several distance-based categories resulting from a driver (e.g. distance to roads).

Materials

CORINE Land Cover Map Val d’Ariège 2012

CORINE Land Cover Map Val d’Ariège 2018

Simulation LCM Val d’Ariège 2018

Distance to roads

Requisites

All maps must be in raster format with the same resolution, extent and spatial reference system (SRS).

Execution

Step 1

First, we have to obtain the observed and simulated transitions from agriculture and pasture land to built-up areas over the period 2012–2018. Using the raster calculator, we extract the observed (“CLC_2012@1” = 2 AND “CLC_2018@1” = 1) and the simulated (“CLC_2012@1” = 2 AND “CLC_predict_2018@1” = 1) transition from agriculture and pasture land (Category 2) to built-up areas (Category 1). The result is shown in Fig. 14 (observed change appears in cyan, simulated change in red).

Fig. 14

Exercise 1. Step 1. Intermediate maps showing observed and simulated transitions from agriculture and pasture to built-up areas

Step 2

The Reclassify by table raster analysis tool is used to transform the map showing the continuous distance from roads into various different classes. Given the dense road network, we intentionally apply a progressive interval as shown in Fig. 15.

Fig. 15

Exercise 1. Step 2. Reclassify by Table

Figure 16 shows the general result and the result for a detailed area with the following classes: the road network itself (distance is zero), distance class 1 (less than 100 m), class 2 (100–300 m), class 3 (300–1000 m) and class 4 (more than 1000 m).

Fig. 16

Exercise 1. Step 2. Intermediate map showing the distance from roads reclassified by intervals

Step 3

The next step is to compute observed and predicted transitions from Category 2 to Category 1 as a function of the road distance classes. To this end, we use the Raster calculator again to calculate: i) the road distance class map multiplied by the observed transition map and ii), the road distance class map multiplied by the simulated transition map. The results can be seen in Fig. 17, in which the two maps show the transition from 2 to 1 as a function of road distance. The map on the left shows the observed transition and the map on the right shows the simulated transition, with a detailed area in both cases.

Fig. 17

Exercise 1. Step 3. Intermediate maps showing observed (left) and simulated (right) transition from 2 to 1 as a function of the road distance classes

Step 4

Finally, we compare observed and simulated transitions as a function of distances classes (strata). We use the Raster layer unique values report raster analysis tool to calculate the number of pixels for each road distance category (observed and simulation) for the transition from category 2 to 1 as shown in Fig. 18 (left for observed, right for simulated transition).

Fig. 18

Exercise 1. Step 4 presented in the “output” window. Number of cells and areas of observed (left) and simulated (right) transition from 2 to 1 as a function of the road distance classes

The results are then converted into percentage as shown in Table 9.

Table 9  Exercise 1 Step 4. Number and proportion of cells of observed and simulated transition from 2 to 1 as a function of the road distance classes

Results and Comments

The result is that there are almost three times as many observed transitions as predicted transitions. However, the proportion of near-to-road transitions is approximately the same. In conclusion, the model underestimates the quantity of agriculture and pasture land that is transformed into built-up areas, although in the areas close to roads, it accurately predicted what happened in the Ariège Valley between 2012 and 2018.

5 Accuracy Assessment Statistics

Description

The thematic accuracy assessment statistics are a set of parameters that measure the degree of agreement between the LUC map and the reference data (for more details about reference data, see Chapter “Sample Data for Thematic Accuracy Assessment in QGIS”). Overall accuracy, user’s accuracy and producer’s accuracy are reported in many studies. Some additional accuracy measures such as the standard error of overall accuracy and the confidence intervals for the adjusted areas are also helpful.

All these parameters are mainly derived from the error or confusion matrix (see Chapter “Basic and Multiple-Resolution Cross-Tabulation to Validate Land Use Cover Maps”). This matrix is obtained from a cross-tabulation between the reference data and the thematic map. In the resulting table, the reference data are generally shown in the columns and the map data in the rows (Table 10).

Table 10 Confusion matrix

In Table 10, n_ij refers to the sample count of spatial units in cell (i, j), n_i+, n_+j denote the sum of n_ij in each row and column, and n is the sample size; n_+j is the number of spatial assessment units belonging to class j, according to the reference data, and n_i+ is the number of spatial units belonging to class i according to the thematic map.

Expressing the error matrix in terms of area proportions instead of sample counts enables the calculation of unbiased area estimators. The area proportions (${\widehat{p}}_{ij}$) are defined as follows:

$${\widehat{p}}_{ij}={W}_{i}\frac{{n}_{ij}}{{n}_{i+}}$$

where W_i = (Map area of class i)/(Total area of the map).

Based on these area proportions, the overall estimated accuracy ($\widehat{O}$), user’s accuracy (${\widehat{U}}_{i}$) and producer’s accuracy (${\widehat{P}}_{j}$) are calculated with the following equations:

$$\widehat{O}={\sum }_{j=1}^{q}{\widehat{p}}_{jj}$$

$${\widehat{U}}_{i}=\frac{{\widehat{p}}_{ii}}{{\widehat{p}}_{i+}}$$

$${\widehat{P}}_{j}=\frac{{\widehat{p}}_{jj}}{{\widehat{p}}_{+j}}$$

Errors of commission and omission are complementary concepts of the user’s and producer’s accuracy metrics, respectively (i.e. error = 1 − accuracy). An error of commission occurs when a feature is included in a thematic class to which it does not belong. In contrast, an error of omission occurs when a feature is excluded from the thematic class to which it belongs (Finegold et al. 2016).

Errors in the classification process can increase the uncertainty in area estimation. However, the pixel count multiplied by pixel size is often used as an estimator of the true area on the ground. This measurement is strongly affected by both omission and commission errors (Gallego, 2004). Olofsson et al. (2013) proposed an unbiased area estimate using an adjustment factor obtained from the error matrix:

$${\widehat{A}}_{j}={A}_{total}\times {\widehat{p}}_{+j}$$

${\widehat{A}}_{j}$ is the unbiased area estimator or adjusted area. In this case, the area estimator obtained directly from the map (A_total) is then adjusted by a factor obtained from the reference data. If there are more samples labelled as class j in the reference sample than in the map, then ${\widehat{A}}_{j}$ will be larger than the area obtained directly by pixel counting.

Utility

Exercises
1. To validate a map against reference data/map

The statistics obtained from the thematic accuracy assessment are not only descriptors of the map quality but also represent a fundamental input for calculating unbiased area estimators. Additionally, they provide the necessary elements to decide whether to increase the number of sampling sites in the reference data, if the precision obtained does not meet the initial mapping objectives.

QGIS Exercise

Available tools
• MapAccurAssess Plugin

In QGIS, several plugins, such as Semi-Automatic Classification, AcATaMa and MapAccurAssess, can be used to calculate the map accuracy statistics. All three plugins provide the overall accuracy, producer’s accuracy, user’s accuracy and the error matrix, although AcATaMa and MapAccurAssess also report some additional statistics about the adjusted areas and their levels of accuracy.

In this exercise, we use the MapAccurAssess plugin because it can use a shapefile directly with the reference data. The results provided by this plugin, based on Olofsson et al. (2013), include the error matrix and a table with the following statistics: the class area, the producer’s and user’s accuracy values, the adjusted areas and their confidence intervals. It also includes the overall accuracy and its respective standard error.

This plugin is a test version and has not yet been accepted in the official QGIS repositories.

Exercise 1. To validate a map against reference data/map

Aim

To validate a LUC map for the Marqués de Comillas study area by computing accuracy assessment statistics and the error matrix via cross-tabulation of the reference data and the thematic map.

Materials

Marqués de Comillas Land Use Cover Map 2019

Photointerpreted reference dataset—Marqués de Comillas 2019 (reference dataset resulting from the exercise in Sect. 2 in Chapter “Sample Sata for Thematic Accuracy Assessment in QGIS”)

Requisites

In order to compute the areas, the land cover map must be in raster format (GeoTiff) in any cartographic projection. The reference data must be contained in a shapefile with the same type of projection as the map. The shapefile attribute table must contain at least two columns, showing the value for the thematic class obtained from the land cover map and the value according to field ground-truthing or photointerpretation. Both columns must have the same data type (integer or text) to be comparable. Each row of the table corresponds to one reference site.

Execution

Step 1

Install the MapAccurAssess plugin. Should you need help, please see Chapter “About This Book” and the plugin’s documentation.

Step 2

If the plugin has been successfully installed, an icon should appear in the main graphics panel. To start the exercise, click on this icon. Alternatively, go to the Complements menu, select Accuracy Assessment and then Accuracy Assessment again.

Step 3

Select the shapefile with the reference samples (Photo-interpreted reference dataset—Marqués de Comillas 2019)² and indicate the column with the reference data and the column with the values for the thematic classes used in the map. After that, select the land cover map you want to assess (Marqués de Comillas Land Use Land Cover Map 2019). If the map is in vector format, indicate the column containing the thematic class values. Finally, select a folder where the results will be saved and click “Accept” (Fig. 19).

Fig. 19

Exercise 1 Step 3. MapAccurAssess plugin

Results and Comments

The output of this plugin consists of two CSV tables. The first contains the error matrix (Table 11), and the second contains the map accuracy assessment statistics (Table 12). These statistics are as follows: user’s accuracy, producer’s accuracy, thematic class area (as retrieved from the map), the area adjusted by the error level (Area_adj), the confidence intervals for the adjusted area (CI_sup and CI_inf) and the overall accuracy (O).

Table 11 Result from Exercise 1. Error matrix

Table 12 Results from Exercise 1, Step 3 presented in the second “output” CSV file (accuracy indices)

According to the data from this exercise, the overall accuracy of the map is 0.91. In other words, there is a high probability (91%) that a randomly selected location on the map will be correctly classified. Note that the thematic class with the lowest accuracy is 130 (Wetland), with a user accuracy of 0.7 and a producer accuracy of 0.21. This class covers a small area (252 ha according to the map). We decided to keep this class to show that illogical situations can occur when there is only a small number of sampling sites, e.g. negative areas. However, we recommend merging class 130 with another class of similar characteristics and recomputing.