Appearance-based loop closure detection combining lines and learned points for low-textured environments

Hand-crafted point descriptors have been traditionally used for visual loop closure detection. However, in low-textured environments, it is usually difficult to find enough point features and, hence, the performance of such algorithms degrade. Under this context, this paper proposes a loop closure detection method that combines lines and learned points to work, particularly, in scenarios where hand-crafted points fail. To index previous images, we adopt separate incremental binary Bag-of-Words (BoW) schemes for points and lines. Moreover, we adopt a binarization procedure for features’ descriptors to benefit from the advantages of learned features into a binary BoW model. Furthermore, image candidates from each BoW instance are merged using a novel query-adaptive late fusion approach. Finally, a spatial verification stage, which integrates appearance and geometry perspectives, allows us to enhance the global performance of the method. Our approach is validated using several public datasets, outperforming other state-of-the-art solutions in most cases, especially in low-textured scenarios.


Introduction
Simultaneous Localization and Mapping (SLAM) is a fundamental task in robotics that allows an agent to build a map of an unknown environment while, at the same time, tracks its position within this map (Cadena et al. 2016). SLAM systems typically rely on Loop Closure Detection (LCD) methods to identify when the robot has returned to a previously visited place (Stewart et al. 2002). As it is well known, the accumulated drift can be greatly reduced using loop closure information. Additionally, LCD is also used for robot relocalization after a tracking failure (Williams et al. 2011). Cameras have been fundamentally used in the last decade, among other sensors that can be adopted for LCD, due to the richness of imagery as well as the low cost of these devices nowadays. These methods are globally referred to as appearance-based LCD (Angeli et al. 2008;Milford and Wyeth 2012;Cummins and Newman 2011;Galvez-López and Tardos 2012;Khan and Wollherr 2015;Garcia-Fidalgo and Ortiz 2018;Bampis et al. 2018).
The performance of appearance-based LCD methods is mainly determined by: (1) the image description technique adopted; and (2) the ability of the approach to efficiently index previous images (Garcia-Fidalgo and Ortiz 2015). Concerning image description, hand-crafted point features have extensively been used for LCD tasks. Among these solutions, binary descriptors (Calonder et al. 2010;Rublee et al. 2011) have progressively emerged as an alternative to real-valued descriptors (Lowe 2004;Bay et al. 2006) given their demonstrated advantages in computational terms. More recently, the aforementioned techniques have successfully been replaced by methods based on Convolutional Neural Networks (CNN), which have proven to be more robust to illumination and viewpoint changes than hand-crafted features (DeTone et al. 2018). CNN-based methods have typically been used as holistic descriptors. Other approaches (DeTone et al. 2018;Dusmanu et al. 2019;Revaud et al. 2019) have been recently shown to be able to detect and Fig. 1 Lines can support point features in low-textured scenarios: (left) query image, (right) loop-closing candidate image selected by our approach. Points and line segments are labelled as circles and lines, respectively. Matchings found are indicated by the same color in both images describe simultaneously interest points in an image with adequate performance; none of them, however, produce binary descriptors. On the other side, while hand-crafted point features have shown impressive results working on well-textured scenarios (Mur-Artal and Tardós 2017), their performance decreases when dealing with low-textured scenarios, where the number of feature points detected results to be rather low (Pumarola et al. 2017). Precisely because of their nature, low-textured, human-made environments usually exhibit structural regularities that can be described using line features. All this has taken us to the combination of both visual perspectives, lines and point features, in order to enhance loop closing performance and widen the range of environments that can be addressed (Company-Corcoles et al. 2020). By way of illustration, Fig. 1 shows the benefits of combining both kinds of features in a low-textured scenario.
In reference to image indexing, arguably the most used scheme is the combination of the Bag of Words model (BoW) [50], Nister and Stewenius (2006) with an inverted file. In this model, descriptors are quantized into visual words, according to the available visual vocabulary. Images are then described by a histogram of occurrences of each visual word in the image. Depending on how this visual vocabulary is generated, the available solutions can be classified as either off-line or on-line (Garcia-Fidalgo and Ortiz 2015). The off-line approach needs a training phase to build the visual vocabulary, which can be particularly time-consuming, and besides makes the resulting solution highly dependent on the diversity of the training set (Nister and Stewenius 2006). As an alternative, the on-line approach builds the visual vocabulary incrementally, as the robot navigates, which helps to solve the aforementioned issues. A fusion strategy is additionally required to obtain a consolidated list of loop closure candidates when different sorts of features are involved, maybe by means of several BoW instances, one for each kind of feature (Zuo et al. 2017;Gomez-Ojeda et al. 2019;Han et al. 2021).
Standard BoW schemes do not account for the spatial distribution of features in the image, which tends to reduce their accuracy under severe perceptual aliasing conditions (Galvez-López and Tardos 2012;Garcia-Fidalgo and Ortiz 2018). Due to this reason, a final spatial verification step is usually performed to check consistency at the geometric level for the resulting correspondences between the query and candidate images. In this regard, the most popular approach consists in using RANdom SAmple Consensus (RANSAC) (Fischler and Bolles 1981) to check whether the image features obey a specific motion model between images. Typically, this loop closing validation step degrades in performance when dealing with either non-rigid image transformations or a high number of outliers. To tackle this problem, some recent approaches check the consistency between images for local neighborhood structures around the correspondences found (Lowry and Andreasson 2018;Ma et al. 2019;Zheng and Doermann 2006;Bian et al. 2017).
Under this context, we introduce an appearance-based loop closure detection algorithm, named LInes and POints for Low-Textured scenarios' Loop Closure Detection (LiPo-LCD ++ ). In our approach, binarized CNN-based point descriptors and binary line descriptors are indexed using an incremental Bag-of-Binary-Words scheme and combined through a query-adaptive late fusion strategy in order to obtain an integrated list of loop closure candidates. After choosing an image as a final loop candidate, the loop closure is further validated performing a spatial verification procedure which naturally integrates point and line features. Our approach, as it is shown in the experimental results section, outperforms other solutions for generic environments, exhibiting a remarkable performance level for low-textured scenarios.
A preliminary version of LiPo-LCD ++ was introduced in Company- Corcoles et al. (2020). In this paper we extend this work with the following contributions: -A binarization procedure to transform a CNN-based realvalued local descriptor (DeTone et al. 2018) into a binary descriptor. This binarization procedure allows us to combine the advantages of CNN-based feature descriptors with the benefits of a binary visual vocabulary. -A query-adaptive late fusion approach to merge loop candidates retrieved using lines and points independently. This fusion strategy permits us to adapt our system dynamically to the environment by weighing automatically the candidates according to the images contents in terms of lines and point features, and, thus, increasing its performance for a wider range of scenarios. -A novel spatial verification method which combines appearance and geometric information of feature points and lines. This method leads to the detection of a larger number of inliers when dealing with lines and attains higher accuracy in general. Moreover, we propose a fusion of feature points and lines involving local neighborhood structures. This allows us to increase the efficiency of usual spatial verification stages for, particularly, low-textured environments.
Additionally, we report on an extensive set of experiments to validate the performance, adaptability and effectiveness of LiPo-LCD ++ . As it is shown in the experimental results section, our solution compares favourably with other stateof-the-art methods in the field.
The rest of the paper is organized as follows: Sect. 2 overviews the most important works in the field; Sect. 3 describes the proposed approach, while Sect. 4 discusses on a set of experimental results to evaluate LiPo-LCD ++ performance; finally, Sect. 5 concludes the paper and suggests some future research lines.

Related work
Appearance-based loop closure approaches can be categorized according to their image description method (Garcia-Fidalgo and Ortiz 2015). In this regard, some works (Sünderhauf and Protzel 2011;Milford and Wyeth 2012;Arroyo et al. 2014) have opted for global descriptors for image description. These descriptors are usually very fast to compute, though, typically, they are more sensitive to viewpoint and illumination changes than local features. Due to these reasons, point local descriptors, either real-valued (Cummins and Newman 2008;Angeli et al. 2008;Cummins and Newman 2011;Tsintotas et al. 2019) or binary (Galvez-López and Tardos 2012; Mur-Artal and Tardós 2014; Khan and Wollherr 2015;Garcia-Fidalgo and Ortiz 2018), have been widely used in the literature during last decades. More recently, approaches based on CNNs have emerged as an alternative, motivated by their demonstrated robustness to visual appearance changes (Chen et al. 2014;Sünderhauf et al. 2015;Chen et al. 2017;Kenshimov et al. 2017;Lopez-Antequera et al. 2017;Yue et al. 2019). These solutions typically employ CNNs to extract a global descriptor of the image, what makes more difficult the implementation of a spatial verification stage.
The BoW model [50], Nister and Stewenius (2006), used along with an inverted file, is the most used indexing scheme for appearance-based LCD (Garcia-Fidalgo and Ortiz 2015; Lowry et al. 2016). As discussed earlier, depending on how the visual vocabulary is generated, LCD approaches are classified as either off-line or on-line methods. Off-line methods (Cummins and Newman 2011;Galvez-López and Tardos 2012;Mur-Artal and Tardós 2014;Bampis et al. 2018) generate the vocabulary during a training phase. Conversely, on-line methods (Angeli et al. 2008;Labbé and Michaud 2013;Khan and Wollherr 2015;Garcia-Fidalgo and Ortiz 2018;Tsintotas et al. 2018;Tsintotas et al. 2019) build the visual vocabulary as images are received.
Sequence-based matching is a well-known technique that has proven to be useful in LCD approaches (Milford and Wyeth 2012). Recently, Garg and Milford (Garg and Milford 2021) introduced the concept of temporal matching in a hierarchical scheme. In their solution, a learning-based sequence descriptor is first employed to select a number of place candidates. A learnt single image descriptor is next used to decide the final match. Unlike these works, our approach employs two types of features and involves temporal information in the decision after selecting a set of candidates. Additionally, we do not rely on global descriptors.
Within the context of LCD, binarizing CNN features has been adopted by multiple solutions. For instance, Arroyo et al. (2016) applies min-max normalization to the vector resulting from the concatenation of several convolutional layers. The final descriptor is next obtained by randomly selecting a specific set of features and thresholding on each component. More recently, Garg and Milford (Garg and Milford 2020) describe a coarse quantization-based hashing scheme, which employs PCA for dimensionality reduction. In Neubert et al. (2019), the authors propose a sparsified binary adaptation of Locality-Sensitive Hashing (LSH) using random projections, while other works opt for a CNN-based approach to directly generate a binary descriptor (Lin et al. 2016;Lin et al. 2019). The main difference between our approach and these solutions is that they result into a global binary descriptor for the whole image. Furthermore, unlike our solution, some of the aforementioned works (Arroyo et al. 2016;Garg and Milford 2020;Neubert et al. 2019) require a dimensionality reduction stage.
Some SLAM approaches (Pumarola et al. 2017;Zhang et al. 2019) have benefited from the combination of points and lines for image description. However, during the LCD stage, they only rely on point features, discarding line information that can be useful for some environments, unlike our approach (Company-Corcoles et al. 2020), and others that adopt a dual scheme for LCD (Gomez-Ojeda et al. 2019;Zuo et al. 2017;Han et al. 2021). Our approach resembles the idea of combining lines and points described in these works, though it differs in multiple aspects, such as (1) the point features that are employed, (2) how the visual vocabularies are built, (3) how image scores are calculated from the visual vocabularies and later combined, and (4) the spatial verification process. All these points are revised in the following: -As for the generation of the vocabulary, all the abovementioned works rely on an off-line method which, as commented previously, makes them highly dependent on the diversity of the training set used. This dependence does not appear in our method thanks to the on-line vocabulary acquisition approach.
-Moreover, all of them make use of ORB (Rublee et al. 2011) Han et al. (2021), adopt the most widely used spatial verification approach, based on a RANSAC-based estimator of the fundamental matrix. We instead adopt a strategy based on analyzing the preservation of the topological relationships between neighbouring features. Unlike these approaches, we only use the appearance to validate loop closures. Furthermore, the proposed procedure can tolerate a high number of outliers and non-rigid camera motions. Figure 2 illustrates LiPo-LCD ++ . As can be observed, two sets of features, i.e. lines and points, are found in every new image that is processed. Each of these features is described by means of a binary descriptor, which is then used to: (1) obtain a list of the most similar images according to this visual cue; and (2) update the corresponding visual vocabulary. Next, the two lists are merged using a scored-voting procedure, giving rise to a combined list of loop-closing candidates. Subsequently, a temporal filter, based on the concept of dynamic islands (Garcia-Fidalgo and Ortiz 2018), groups those images close in time and prevent adjacent frames from competing among them as loop candidates. To validate the existence of a loop closure, the representative image of the best island is selected as the final loop candidate image and it is spatially assessed against the query image. We deal next with the details of every step.

Image description
As stated previously, LiPo-LCD ++ uses points and lines to describe images. The rationale behind this is that the combination of multiple and complementary descriptions improves the performance and robustness of LCD methods (Hausler and Milford 2020). By way of illustration, textureless scenes may be more distinctively described using lines than only points, while other environments that lack structure can be better described by means of point features. To this end, in our solution, the image I t at time t is described by φ(I t ) = {P t , L t }, being P t a set of local keypoint descriptors and L t a set of line-segment descriptors.

Point description
Given the robustness shown by CNN-based methods, we use the SuperPoint approach (DeTone et al. 2018) to obtain a set of keypoints from the image under consideration. SuperPoint is a learning-based feature detection and  description framework that is specially effective under illumination and viewpoint changes. Furthermore, compared to similar solutions such as R2D2 (Revaud et al. 2019) and D2-Net (Dusmanu et al. 2019), SuperPoint detects and describes features in an affordable time. In LiPo-LCD ++ , we use the default model provided by the authors, which was trained using a wide range of scenarios. This is to enhance the performance of our approach in a larger range of environments, in contrast to other CNN-based approaches, which are typically trained using a specific scenario. Figure 3 shows an example of how SuperPoint detects, in an indoor scenario, a higher number of features than other hand-crafted features like ORB (Rublee et al. 2011) or SIFT (Lowe 2004). Besides, we have observed that SuperPoint tends to find keypoints in a more uniform way throughout the image than other detectors. SuperPoint, similarly to other recent CNN-based approaches for keypoint detection, produces real-valued descriptors (DeTone et al. 2018;Dusmanu et al. 2019;Revaud et al. 2019). In this work, we are interested in binary descriptors to take advantage of their well-known computational benefits. To this end, our proposal inspires on other binary description methods (Calonder et al. 2010;Rublee et al. 2011), which perform pairwise comparisons between pixels within an image patch. In our case, we compare the values of different components of the real-valued descriptor. Formally, we arrange a D-dimensional real-valued descriptor d ∈ R D as a concatenation of 8-component M In this way, each 8-component vector leads to an 8-bit string for each pair, which can be stored using a single byte. Given two subvectors d x and d y from d, their corresponding 8-bit string β is hence defined as where τ is a comparison test given by: where ⊕ stands for the concatenation of binary descriptors. The binarization procedure is illustrated in Fig. 4. SuperPoint generates a 256-dimensional descriptor, i.e. D = 256, resulting into M = 32 subvectors d x , so that the set of N ≤ 32 2 ℘ j pairs leads to a binary descriptor d b of N × 8 bits. In this work, we assess three criteria for selecting these pairs, since the chosen pairs affect the performance of the LCD approach. For each case, we generate N = 32 and N = 64 pairs, which result into, respectively, 256-and 512-bit binary descriptors. We briefly review each selection method in the following.
The first proposal applies a dimensionality reduction procedure to the original real-valued descriptor and, then, combines all-against-all possible pairs from the resulting subvectors. To this end, we make use of Gaussian Random Projections (Dasgupta 2013) to transform the original space into a simpler K-dimensional space. The conversion is performed by means of a projection matrix R ∈ R D×k . Each entry is independently sampled from a standard normal distribution N(0,1). In our experiments, we project Super-Point descriptors onto 72 and 96 dimensions, which results into, respectively, 9 and 12 subvectors d x per descriptor. We choose all the possible pairs on each case, respectively 9 2 = 36 and 12 2 = 66, discarding four combinations in the first case -(0,1), (0,2), (6,8) and (7,8)-to reach the 32 pairs of the final binary descriptor, and discarding two combinations -(0,1) and (10,11)-to end with 64 pairs in the second case. There is no particular reason for discarding these pairs, except for the fact of involving the first and last components of the descriptor in both cases.
As a second proposal, we consider one of the approaches assessed by the BRIEF descriptor authors in Calonder et al. (2010), which choose randomly the N pairs. In the seminal work, this random criterion seems to provide better results than the other pair selection schemes considered. We thus generate two random patterns of 32 and 64 pairs from all the possible combinations of M subvectors.
As a third alternative, we propose a novel but simple method to produce the corresponding N pairs. To generate a 256-bit descriptor, each subvector d k is paired with its neighbours {d k−1 , d k+1 }. Similarly, the 512-bit descriptor is computed using the two nearest neighbours of d k , i.e.
For the 256-bit case, in practice, one has to consider pairs (d k , d k+1 ), k = 0 .. 30, together with pair (d 31 , d 0 ), 1 for a total of 32 pairs. For the 512-bit case, one has to consider the previous 32 pairs and pairs for a total of 64 pairs.
As is shown in Sect. 4.2, this last approach is the one leading to the best performance. To finish, we denote the final set of binarized descriptors as P t .

Line description
Lines are detected in the image using the Line Segment Detector (LSD) (Grompone von Gioi et al. 2010). LSD is a linear-time high-precision line segment detector with subpixel accuracy without parameter tuning. However, LSD tends to break long line segments into shorter segments or even duplicated segments. The existence of these repli-1 Considering the SuperPoint descriptor as a circular vector. cas often affects the matching procedure and increases the storage requirements. To minimize the incidence of these misbehaviours, we perform a line merging step after segment detection. Formally, each line segment k is represented by its start point s k and its end point e k in homogeneous coordinates. The direction vector l for segment k is thus given by: For segment merging, we consider every possible pair of lines (l i , l j ). Lines are merged if the minimum Euclidean distance for the four possible combinations of end-points is smaller than a threshold and the angle between line-segments θ i j , computed as is close to 0 or π radians. Similarly to LiPo-LCD (Company-Corcoles et al. 2020), the lines found are described using the binary form of the Line Band Descriptor (LBD) (Zhang and Koch 2013), conforming the final set of binary line descriptors L t .

Searching for loop closure candidates
Similarly to our earlier work (Company-Corcoles et al. 2020), we make use of a hierarchical tree structure to efficiently retrieve loop closure candidates, what allows us to manage an increasing number of visual words, i.e. online operation. To this end, we employ OBIndex2 (Garcia-Fidalgo and Ortiz 2018) as an incremental BoW scheme for binary descriptors. In combination with an inverted index, it allows for fast image retrieval.
More precisely, LiPo-LCD ++ maintains two instances of OBIndex2, one for point features and another one for line features. Each instance builds an incremental visual vocabulary jointly with an inverted index of images. For a given query image I t , a search is performed on each index to retrieve the most similar images with regard to each visual perspective. To this end, we search each visual feature descriptor within its corresponding dictionary to find the closest visual word. Next, using the inverted index, a list of images where the visual word has been observed is retrieved, and, for each one, a TF-IDF weight is added to the final score of the image. Candidate images are then sorted according to this final score. For a further explanation, the reader is referred to Garcia-Fidalgo and Ortiz (2018).
As a result, two lists are obtained: (1) the m most similar images using point features C t p = {I t p 0 , . . . , I t p m−1 } and (2) the n most similar images using line features C t l = {I t l 0 , . . . , I t l n−1 }. Each list is sorted by the associated scores, respectively s t p (I t , I t j ) and s t l (I t , I t j ), which measure the similarity between the query image I t and each candidate image I j . Since the range of these scores varies depending on the distribution of visual words in each vocabulary, we map them onto a common range [0, 1] by applying min-max normalization given by: where s t k I t , I t min and s t k I t , I t max correspond, respectively, to the minimum and the maximum scores of an image candidate list, being k ∈ {p, l}. To keep control of the size of the list of candidates for a specific feature C t k , k ∈ {p, l}, images whose normalized scores t k is lower than a predefined threshold are discarded. Finally, the respective visual vocabularies are updated accordingly to the set of point and line binary descriptors resulting from the current image.

Fusion of the lists of candidates
The next step is to integrate both lists C t p and C t l , each one providing loop candidates from a different visual perspective, into a joint list of similar images C t pl .

Overview
Techniques that combine multimodal information for image retrieval can be generically classified into early or late fusion approaches (Bhowmik et al. 2014). Early fusion refers to the combination of the information at the feature descriptor level, before being processed by a retrieval system. Conversely, late fusion works with the outputs of different retrieval systems.
Regarding the latter, a common approach is to aggregate multiple ranked lists, corresponding to different visual features, using a function that provides a global confidence measure. This function can be implemented in several ways: (1) using a ranked-voting procedure that considers the position of each candidate in the list for each retrieval system; or (2) using a score-based method that combines the candidates by weighing them according to the importance of each visual feature in the query image. In a previous work (Company-Corcoles et al. 2020), we adopted a ranked-voting scheme where each candidate was scored inversely proportional to their position in the list. Despite its good performance, this scheme did not take into account the relevance, or even the existence, of each visual feature in the query image, preventing the use of information that could certainly be used to fit the approach with dynamic adaptation to each case. To this end, in this work, we adopt a query-adaptive late fusion approach which exploits a score-based method to leverage the information coming from both sorts of visual features. Particularly, we propose a late fusion approach based on the work by Zheng et al. (2015), where the importance of a visual feature for a query image can be attributed to the shape of the curve of the resulting scores, sorted in descending order. To be more precise, they argue that this curve is L-shaped for a good feature, indicating that a small portion of the database images attain a high score, while a bad feature distributes the scores along more database images. LiPo-LCD ++ accounts for this behaviour essentially by means of the calculation of the Area Under the Curve (AUC) of the scores curve, to compute adaptively the importance of each feature for each query image. The details for this computation can be found next.

Computation of feature relevance
Following the aforementioned ideas, we propose to compute the importance of each visual feature, for a given query image, according to the AUCs of the sorted scores of lists C t k , k ∈ {p, l}. Formally, we denote f k as the curve of the corresponding normalized scoress t k , i.e. f k ( j) =s t k (I t , I t j ) for a specific image candidate j. As a first step, each curve f k is analysed to detect and remove flat tails, i.e. detecting when the curve stops decreasing. To this end, we compute a linear approximation, from lower to higher scores, for each pair of consecutive points of the curve. Then, the analysis is finished when the magnitude of the resulting slope is above a threshold, discarding the remaining candidates and leading to a shorter curve f k . Assuming that the final number of candidates is C k , its AUC, denoted by A k , is calculated by integrating f k using the composite trapezoidal rule using equally-sized partitions over the interval [0, C k − 1]: Once the AUC for points A p and lines A l has been calculated, the weights for point features w p and line features w l are given by: We use the inverse of A k to score higher smaller areas and so capture better the L-shape.

Combination of scores
Weights computed in the previous step are used to leverage scores of image candidates from the different visual perspectives, resulting into a global confidence measure. Among the different options considered to consolidate these scores in Zheng et al. (2015), we use the sum rule given its better tolerance to false positives. Therefore, we compute a joint similarity measure S t (I t , I t j ) between the query image I t and a candidate image I t j as: Finally, an integrated image list C t pl is obtained by sorting the resulting candidates according to their integrated scores S t (I t , I t j ). Figure 5 illustrates the above-described merging procedure with an example. Curves f p and f l for, respectively, feature points and feature lines are shown on the left, while the respective images are shown on the right. The images involved in the calculation of A p and A l correspond to the ends of the blue and green segments of the f k curves (for, respectively, lines and points). Removed image candidates for each case are shown as the ends of the red segments. Black captions below the images on the left denote correct loop closures (according to the ground truth), labelled with a coloured dot on the left curves, whereas captions for noncorrect loop closures are shown in red. As a result of the whole process, for this particular case, w p = 0.46 and w l = 0.54, indicating that the importance of each feature, for the example query image, is almost the same, as the f k curves already suggest. It can also be observed from this figure that discarded images are mostly incorrect loop closure candidates, while the selected frames to compute A k are almost the same in both cases. It can also be noticed that, in this example, points are more affected by perceptual aliasing than lines.

Dynamic islands computation for loop candidates filtering
After merging the resulting lists C t p and C t l into C t pl , we filter C t pl to prevent that images coming from the same area compete among them as final loop candidates. To this end, we make use of the concept of dynamic islands (Garcia-Fidalgo and Ortiz 2018). A dynamic island Υ m n groups images whose timestamps range from m to n. The size of an island is not fixed, but it depends on the similarities between neighbouring images and the camera velocity, to adapt to the specific image stream. For each query image I t , a set of islands Γ t is calculated processing images in the list C t pl in ascending order by using the following procedure. Every image I i ∈ C t pl that lies in the [m, n] interval is associated to an existing island Υ m n ; in such a case, the time interval of the island is updated to accommodate I t and some previous and posterior frames. I t is associated to a new island created around time t if I t does not fit in any existing island. Note that this procedure relies on a sorted list of images.
After processing all images in C t pl , a global score g is computed for each island as: which is an indicator of how similar an area of the environment is with regard to the query image, taking into account the two visual perspectives. We also rely on the idea of priority islands, defined as the islands in Γ t that overlap in time with the island selected at time t − 1, Υ * (t − 1). The rationale behind these islands is based on the fact that consecutive images should close loops with areas of the environment where previous images also closed a loop. Originally, the authors of Garcia-Fidalgo and Ortiz (2018) selected, as a final island, the priority island with the highest score g, if any. However, they did not consider the spatial arrangement of the features and the fact that perceptual aliasing might produce incorrect island associations, especially in human-made environments. For this reason, as an alternative, LiPo-LCD ++ retains an island for the next time instant if the final selected loop candidate satisfies the spatial verification procedure, as explained in Sect. 3.5. Then, when the best island Υ * (t) is determined, the image I c with the highest combined weighted score S t of Υ * (t) is selected as its representative and passed to the next spatial verification stage.

Spatial verification
One of the main drawbacks of BoW schemes is that they ignore the spatial arrangement of visual words in the image. Under severe perceptual aliasing conditions, this might lead to produce incorrect loop associations which can be detected by checking whether some minimal physical constraints on the camera motion are satisfied or not. Therefore, a spatial verification step usually takes place to validate the loop candidate. The most widely used approach in this regard assumes a rigid motion from the camera and verifies if the motion of image features is compatible with epipolar geometry computing the fundamental matrix within a robust estimation framework, e.g. RANSAC-based, and requiring a minimum number of inliers. However, RANSAC performance decreases, either when the number of outliers becomes large, or when the camera motion cannot be modeled by means of the fundamental matrix ( LPM is based on the fact that the true matches from a list of candidate matches should maintain the spatial neighborhood relationships among image features, i.e. should preserve the topological structures of the scene. In this work, we modify LPM by incorporating line matches into the procedure.

Feature matching
The first step is to find a set of consistent matches between the query and the candidate images. In our approach, we perform a separate matching procedure for points and lines. On the one hand, matchings between the point descriptor sets P c , for a candidate image I c , and P t , for the query image I t , are val-idated using the Nearest Neighbor Distance Ratio (NNDR) test (Lowe 2004). However, in our experiments, this approach has not led to adequate performance for lines when using a restrictive ratio, especially in low-textured environments. Instead, we propose, for lines, to combine appearance and geometrical information into the matching process. Firstly, segments are matched by their appearance. Next, geometric properties of lines are used to further filter the initial set of matches. The details can be found in the following.
Given L c and L t as the sets of line descriptors for, respectively, a candidate image I c and the query image I t , we first obtain an initial set of appearance-based line matches by applying the NNDR test, but with a more permissive threshold than for points. Next, we discard those matching pair segments whose lengths change significantly from one frame to the other. For this purpose, we calculate the ratio length μ i j for every candidate matching pair (l i , l j ), as Matching candidates with similar lengths, i.e. μ i j ≈ 1, are accepted. Subsequently, we asses whether the angle θ i j between the two line segments, calculated using Eq. 5, is approximately 0 or π . To avoid that camera rotations between images affect this step, θ i j is compensated using the global rotation between frames, as performed in Zhang and Koch (2013). Surviving line matches are considered for the local neighborhood evaluation.

Local Neighborhood Consistency Assessment
Once the set of putative matches for each feature are generated, we assess their consistency for their local neighborhood. For this purpose, we propose a modification of the LPM algorithm, which we will name as LP-LPM from now on, as it combines points and lines. This combination has shown to be able to enhance the matching performance in low-textured environments, where the performance of LPM usually drops. As it is explained in Ma et al. (2019), the distance between two feature points may vary under viewpoint changes, but local neighborhood structures should be similar. The generation of these local neighborhood structures unfortunately degrades when the number of detected features is low. Combining points and lines however allows for an increasing number of features over the image and leads to a better estimation of such structures.
For a start, point matches are incorporated into the LP-LPM scheme as usual, while line matches are introduced as point correspondences relating their particular endpoints s i and e i for, respectively, the start and the end of the line segment. First, for every line matching pair (l i , l j ), we deter-

Experimental results
In this section, we conduct a set of experiments on several publicly available datasets to evaluate the performance of LiPo-LCD ++ . We also compare our solution with a representative set of state-of-the-art approaches. All experiments have been run on an Intel Core i7-9750H (2.60 GHz) processor with 16 GB RAM. In addition, an Nvidia GeForce GTX 980 GPU has been used to execute SuperPoint, 2 using a Python implementation on TensorFlow.

Methodology
We have selected 8 public datasets, which correspond to different environmental conditions: CityCentre (Cummins and Newman 2008)  of these datasets. For each one, we use the ground truth provided by the original authors except for the KITTI sequences, for which we use the data referred by Arroyo et al. (2014), and the EuR5 and MLG datasets, for which we use the data available from Tsintotas et al. (2019). Precision-recall (PR) metrics are used to evaluate the global performance of every approach. Given that the aim of an LCD method is to be used in a real SLAM solution, false positives are considered critical. For this reason, we are interested in the maximum recall at 100% precision.
Regarding LiPo-LCD ++ parameters, we have used either default values, or values which experimentally have resulted in good performance. OBIndex2 and dynamic islands have been configured as explained in Garcia-Fidalgo and Ortiz (2018), as well as in our previous work (Company-Corcoles et al. 2020). The maximum number of features for Super-Point has been set to 1500. Line detection and description methods have been executed using the default parameters. The binarization of SuperPoint descriptors, apart from the desired size of the final descriptor, does not require any additional setup. Regarding the query-adaptive late fusion stage, the threshold for the segment slopes has been set to 0.025. We have also limited the maximum value of weights w p and w l to 0.8. Concerning the geometric spatial verification stage, 0.8 and 0.95 have been used as NNDR values for, respectively, points and lines. Line matches have been discarded if the ratio length μ i j is above 2.5, and in case θ i j deviates more than 30 • from 0 or π . LPM runs using the default parameters provided by the authors.

Effectiveness of the binary descriptors
In this section, we assess the effectiveness of the binarization procedure. We empirically evaluate the three configurations for selecting pairs introduced in Sect. 3.1.1, denoted by, respectively, All (A), Random (R), and Neighbors (N). For each configuration, we consider the two possible versions of 32 and 64 pairs, resulting into binary descriptors of 256 and 512 bits, respectively. Additionally, we also consider a less sophisticated method that quantizes a 256-dimensional SuperPoint descriptor by thresholding each real-value component, hence resulting into a 256-bit descriptor. The threshold for each component is chosen as the average value of that component in the processed dataset.
In the following, we denote this method as Q-256. The L6I and the EuR5 datasets have been chosen for this experiment. L6I is a full indoor scenario which can be considered as a low-textured environment with high perceptual aliasing. Conversely, EuR5 is also a low-textured indoor environment which presents scenes under severe illumination and scale changes. To measure the effectiveness of each configuration, we compute the maximum recall that can be achieved by the whole system maintaining precision at 100%. Results are shown in Table 2. The highest recall has been obtained for the N-512 configuration for both datasets. As it is shown later, the average time difference required per frame using the N-256 and the N-512 configurations is approximately 7% in the EuR5 dataset. Taking into account both performance perspectives, we decide to adopt the configuration leading to best PR.

LCD performance breakdown
In this section, we evaluate the net effect of the proposed contributions to the final LCD performance. More precisely, we compute what is the maximum recall obtained by LiPo-LCD ++ at 100% of precision, but using different alternatives on each of the steps of our pipeline. Results are shown in Table 3. For the Feature Extraction (FE) stage, we consider ORB and our binarized SuperPoint descriptor (b-SP) as point descriptors, along with the LBD descriptor for lines. Regarding the Late Fusion (LF) options, we consider: (1) our previous Borda Count-based system (Company-Corcoles et al. 2020) and (2) the Query-Adaptive Late Fusion (QALF) method proposed in this work. Finally, RANSAC and the proposed LP-LPM are the choices studied for the Spatial Verification (SV) stage. The L6I dataset is selected for this experiment because it only contains indoor scenes and therefore it can be regarded as the most low-textured dataset considered in this work. As it can be observed, every contribution improves a specific part of the pipeline, which results into a slightly increase of the final recall. Overall, the new pipeline exhibits enhanced performance with regard to our previous contributions: 11.16% over the recall reported in Garcia-Fidalgo and Ortiz (2018), where only points were

General performance
This section focuses on the global performance of LiPo-LCD ++ . We measure the accuracy of loop closure detection, and we report on the computational times as well as on the evolution of the vocabulary size. Figure 6 reports on the accuracy of loop closure detection of LiPo-LCD ++ for all datasets. As usual, we employ PR curves, which result from modifying the threshold on the total number of inliers required to accept a loop. As can be observed, LiPo-LCD ++ exhibits a stable behaviour for all datasets, achieving high recall values while keeping precision at 100%.

Computational times
Average computational times are summarized in Table 4. These values have been computed from the K00 dataset, since it is the largest one considered in this work. Feature Extraction (FE) includes line and point detection and description, as well as the proposed binarization procedure for points using the N-512 configuration. On average, point FE requires about 9.58 ms per image, from which only 1.96 ms corresponds to the binarization of point descriptors. These times are remarkably low in contrast to the 457 ms required by, for instance, D2-Net (Dusmanu et al. 2019) and the 542 ms of R2D2 (Revaud et al. 2019), involving only feature detection and the generation of real-valued descriptors. The Vocabulary Update (VU) and Search for Candidates (SC) steps are usually slower for points than lines, due to the higher number of descriptors that have to be handled. Times required to merge candidate lists and select dynamic islands are included in SC because they are almost negligible. Finally, the Spatial Verification (SV) step only makes sense when both points and lines are jointly computed. The average response time per image of the whole pipeline turns out to be 406.52 ms in a multi-thread implementation, where points and lines are each processed within a separate thread. The evolution of computational times as more images are processed is additionally illustrated in Fig. 7. We also use the K00 dataset for this assessment, for the same reason as above. As can be observed, times for VU and SC remain relatively stable over time (remember that our approach implements a dual incremental BoW scheme). SV times grow occasionally due to the amount of inliers required to validate these candidates within the LP-LPM scheme. If required for a specific application, the number of candidates can be reduced.

Evolution of the vocabulary size
As stated previously, LiPo-LCD ++ makes use of two vocabularies, one for point features and one for line features, that store visual words of, respectively, 512 and 256 dimensions. Figure 8 shows that (1) globally the number of visual words grows as more images have been processed, and (2) the amount of line features is considerably lower than the number of point features, both (1) and (2) as expected. (The Fig. 7 Evolution of the computational times for each part of the pipeline using the K00 dataset

More on the computational requirements
In this section, we evaluate the effect of two strategies to save computational resources on the performance of LiPo-LCD ++ , to contemplate the possibility of running it on computationally-restricted hardware (otherwise, one may be interested in achieving the highest performance in terms of recall). More precisely, among other alternatives (Zaffar et al. 2020), we first assess how reducing the map affects the performance of LiPo-LCD ++ . To this end, we consider a simple method, denoted by Map Size Reduction (MSR), which, using the N-512 configuration as a basis, processes one out of every two images of the input dataset. A second strategy focuses on limiting the vocabulary size. For this purpose, we reduce the number of point features extracted from each image using the relevance measure available for SuperPoints; we consider 1000, 750, 500 and 250 features per image. Regarding the vocabulary for line features, we do not care for its size since it is negligible in contrast with that of point features. The results in terms of vocabulary size and computational times are detailed in, respectively, Fig. 9a and b. The corresponding recall values at 100% of precision for the different datasets can be found in Table 5.
As can be observed in Fig. 9a, the vocabulary size required by MSR is similar to the one resulting when using 500 point descriptors. However, the computational times reported in Fig. 9b demonstrate that MSR leads to similar running times than using 1000 or 1500 point descriptors per image.
Regarding the reduction of the map size reported in Table 5, it can be observed that the recall is dependent on the frame rate/camera speed, e.g. for datasets with higher frame rate or low camera speed, better performance is exhibited. On the other side, the recall tends to decrease when using less point features per image. However, despite this reduction, results for K00, K06 and L6O keep similar for all configurations. Another relevant observation is that using 750 or 1000 descriptors per image achieves similar performace than using the default configuration, i.e. 1500 descriptors, in most cases. Regarding L6O, the number of features that are detected is no more than 700, this is the reason why the recall is the same for the 750, 1000 and 1500-point features cases. As for L6I, no more than 250 features can be found, reason by which it is missing in Table 5.

Comparison with other solutions
In this section, we compare the performance of LiPo-LCD ++ against other state-of-the-art solutions. The maximum recall achieved at 100% of precision for each approach is summarized in Table 6. The reported results come from As can be observed, the combination of points and lines achieves the highest recall for almost all datasets. This is specially evident for the case of L6I and Eur5, which are regarded as the most low-textured scenarios considered in this work, which actually is the focus of our approach. Furthermore, the incorporation of lines in an LCD method not only improves the performance in low-textured scenarios, but for these low-textured datasets, and provides a competitive performance for the remaining datasets. Furthermore, LiPo-LCD ++ obtains better recall values almost for every dataset compared to the solution by Yue et al. (2019), which describes the images using a non-binarized version of the SuperPoint descriptor. Despite LiPo-LCD ++ achieves better recall values than our previous solution in the MLG dataset, the performance attained is rather low with regard to other more successful approaches. We have observed that this can be the result of wrong line detections between consecutive frames, which are due to the bad quality of the images and the corresponding perceptual aliasing that occurs among those lines, leading to a subsequent decrease in the pipeline performance. Notice that our proposal also outperforms (Gomez-Ojeda et al. 2019;Han et al. 2021), which are the solutions most similar to LiPo-LCD ++ , given that they are the only ones that combine points and lines for loop closure detection. To finish, in Table 7, we compare the vocabulary size of LiPo-LCD ++ with other state-of-the-art solutions in terms of memory consumption. As expected, the requirements for LiPo-LCD ++ are higher than the ones for a method that only uses points (Garcia-Fidalgo and Ortiz 2018). This is obviously because of the two visual vocabularies, but also due to the fact that our binary point descriptor is 512dimensional, which is larger than the ORB descriptor used by Garcia-Fidalgo and Ortiz (2018). Notice also that limiting the number of point features to 1000 -250 we reduce the gap as for memory requirements with regard to Tsintotas et al. (2021) without compromising much our LCD performance (as reported in Table 5).

Conclusions and future work
This work has introduced LiPo-LCD ++ , an appearancebased LCD approach that combines point and line features to increase the performance in low-textured environments. To search for loop candidates, our approach is based on two instances of an incremental BoW scheme specifically devised for binary descriptors, one instance for each feature. Besides, we develop a binarization procedure to incorporate Super-Point into LiPo-LCD ++ as a feature point extractor, given the advantages of CNN-based methods recently reported. A query-adaptive late fusion approach is also adopted to merge lists of candidates obtained from the two visual perspectives. This fusion method is dynamically adapted to the current operating scenario, leveraging the final candidate scores according to the presence or absence of each kind of feature. Finally, we introduce a spatial verification method, which jointly validates the appearance and the geometric consistency of points and lines together. LiPo-LCD ++ compares favourably with several state-of-the-art approaches for different public datasets and hence different environmental conditions. Regarding future work, we plan to incorporate LiPo-LCD ++ into a SLAM framework and analyze its performance in low-textured scenarios. We also plan to explore the use of other geometric entities, e.g. planes, for loop closure detection. Balears). This publication reflects only the authors views and the European Union is not liable for any use that may be made of the information contained therein.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecomm ons.org/licenses/by/4.0/.