Abstract
Feature selection and its subsequent dimensionality reduction are significant problems in machine learning and it is at the core of several data science techniques. The ‘shape’ of data, or in other words its related topological properties, can provide crucial insights into the corresponding data types and sources and it enables the identification of general properties that facilitate its analysis and assessment. In this article, we discuss an information theoretic approach combined with data homological properties to assess dimensionality reduction, which can be applied to semantic feature selection.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
1 Introduction
Data science has become an extensively investigated discipline, where diverse methods are applied to identify and extract actionable information.
Mathematical areas including Network Theory, Dynamical Systems, Topology, as well as sophisticated algebraic approaches have been introduced, which have demonstrated their potential in addressing data complexity and their applications have led to the design of effective algorithms (Trovati et al. 2020; Shao et al. 2017; Ray and Trovati 2018; Xu et al. 2019; Ray et al. 2018; Trovati et al. 2022).
In this work, an approach based on the topological and information theoretic properties of networks is introduced, which aims to assess and analyse semantic features and and their reduction. The main motivation follows from the recent advance in Topological Data Analysis (TDA) which has been shown to identify invariant data properties that can be used to assess and analyse large quantities of unstructured data, otherwise difficult to investigate via standard approaches (Carlsson 2008).
We define a knowledge system as a set of concepts and mutually connecting semantic properties, which naturally define semantic networks. More specifically, they are defined as networks \(G=G(V,E)\), where \(V = \{ v_{i}\}_{i=1}^{n}\) is the node-set and \(E = \{ e(v_{i}, v_{j}) \}_{i\not = j=1}^{n}\) is the edge set; each edge can be represented as a multi-dimensional vector based on a predefined set of (semantic) relations. In other words, multiple relations typically exist between two concepts. Furthermore, each node is embedded into an multi-dimensional feature space. It is important to specify that features are assumed to be semantic, that is well defined relational concepts, which can be quantified based on their attributes. These properties define semantic features, which are the central part to this work. Loosely speaking, they refer to semantic properties, which are linked to (potentially multiple) edges, or embedded into the nodes. In particular, we will assume that each component of every edge can be evaluated within the interval [0, 1] via a weight \(w(e(v_{i}, v_{j})): E \mapsto [0,1]\) of an edge \(e(v_{i}, v_{j}) \in E\).
As discussed above, each edge contains specific information embedded in an \(n-\)dimensional feature space \(\mathcal {R}\subset [0,1]^{n}\). These features are typically captured by multiple edges between nodes, which can be flattened into a single edge. We will assume that such evaluation and assessment can be carried out independently from the approach introduced in this work, and as part of any suitable data pre-processing process. Furthermore, it is also assumed that an appropriate measure to capture the potentially multiple semantic links between pairs of nodes has been carried out.
As a consequence, each node \(v_i \in V\) can be represented as an element of \([0,1]^{n}\). Furthermore, every edge is associated with a map \(R_{k}: x \rightarrow y\) for the \(k-\)th feature in \(\mathcal {R}\) linking the nodes x and \(y \in V\). Finally, let
be the set of relations shared by the nodes x and y.
This article is structured as follows: in Sect. 2, existing research and methods are discussed. Sections 3 and 4 introduce the main background information and the main results, respectively. Finally, Sect. 5 concludes the article and prompts to new directions.
2 Related works
Much of the research in this area is focused on the extraction of information that may be utilised to make decisions from unstructured data (Chazal and Michel 2017). In fact, research and industry organisations, as well as general human activities, generate massive amount of data, which poses significant challenges, as well as significant opportunities. A major problem is comprehending and evaluating the importance and worth of the useful information that is included within this abundance of data.
With an emphasis on the structure and shape of data in terms of the relationships between its components, new theoretical methods based on topological techniques have been created (Carlsson 2008). The capacity to categorise things based on their shared characteristics, or in other words, those displaying invariant qualities, is one property of topology. The geometrical characteristics of the data under investigation are crucial in many clustering and classification techniques. In reality, a large portion of these methods concentrate on the idea of distance, which must be described in terms of the mathematical spaces in which the data is contained.
Persistent homology, which focuses on identifying the topological qualities that remain invariant (Edelsbrunner and Harer 2008), has been drawing increasing attention from the research community due to its applicability to AI, machine learning and data science in general. Simplicial complexes are the fundamental components in persistent topology. These are topological space triangulations consisting of coupled, non-overlapping polyhedra, as depicted in Fig. 1.
The graphical representation of simplicial complexes as described in Zhang et al. (2020)
Image pixellation is a simple but instructive example of a space triangulation in which a genuine image is covered in pixels to provide an accurate approximation.
The fact that a simplicial complex provides an ‘approximation’ of the object which is covered via a triangulation. Triangulations include, but are not limited to Voronoi diagrams, Delaunay triangulations, Vietoris complexes, and C̆ech complexes. In general, they are defined in terms of a certain distance or by ball intersections, whose centres are the data points. Additionally, the adjacency graphs produced by these triangulations can provide a number of details about their invariant topological characteristics, making them pertinent to data science (Carlsson 2008). Refer to Edelsbrunner and Harer (2008) for further information on simplicial complexes.
3 Observability
In this section, the concept of observability is introduced. The main motivation is that information is inter-twinned with its observability. In other words, information must be observed. We define the following types of observability which, as explained above, are embedded in a knowledge system, or in other words, a suitably defined semantic network.
-
Single object observability: O(x) measures the observability of x with respect to the other nodes. More specifically, we have
-
Absolutely observability: this is a single observability which covers the entire network
-
Local observability: similar to the above but covers only a subnetwork containing x
-
-
Mutual observability: this occurs between two (semantic) objects x, and y, and it is defined as O(x, y).
In the rest of this section, a formal definition of the relevant observabilities derived from the above instances, is introduced.
Definition 1
(Observabilities) Let \(G = G(V,E)\) be a (semantic) network (knowledge system) and let \(x,y \in V\) be two nodes, and \(e \in E\) be an edge. Assume that n(x) is the set of incidents edges with respect to x (or the degree of x), so that \(n(x \leftrightarrow y)\) is the set of (undirected) edges connecting x with y. Let \(w(e) \in (0, 1]\) be the weight of the edge e. We define the observability from x to y as
where
and
In this article, a simplified version of observability, the naïve observability of x, will be used, which is defined as
The difference between the naïve observability \(O_{n}(x)\) and those described above is that the former only consider the level of connectivity of x with its neighbours compared to all the other nodes in the network G. In other words, this type of observability is only affected by the immediate connections of x, as opposed to the overall topology of G and the corresponding connected components and paths. Despite this, it proves to provide a useful tool in the analysis of the results in the remaining of the article.
It is trivial to see that \(O(x \rightarrow y)\) is a premetric as in general it is not symmetric and the triangle inequality does not hold.
Let \(\underline{x} = x_{1}x_{2}\cdots x_{n}\) be a path joining a set of incident nodes. We can easily see that the observability of such a path is
In particular, using Eq. 2 this can be written as
3.1 Probability as a function of observability
Any probability measure defined over complex networks is closely related to their overall topology, which usually depends on the corresponding degree distribution.
Based on the above and on general network properties (Newman 2010), we have that
where p(x), p(x, y) and \(p(y\,\vert \,x)\) are the probability of x, the joint probability of x and y, and the conditional probability of y given x, respectively.
Using the notation introduced in Definition 1, let \(x_1\) and \(x_k\) be two nodes. Define the path \(\underline{x} = x_{1},\ldots , x_{k}\) and let the probability \(p(\underline{x})\) of reaching \(x_k\) from \(x_s\) along the is
Note that, if we define a path \(\underline{\tilde{x}} = x_{k},\ldots , x_{1}\), that is the path going in the opposite direction of \(\underline{x}\), we have that
and \(p(x) = \displaystyle {\frac{w(x)}{ \sum _{y \in V(\tilde{G})} w(y)}}\) for any node \(x \in V\).
The above shows that, in general, \(p(\underline{x}) \not = p(\underline{\tilde{x}})\). Again, we can easily see that \(p(\underline{x})\) can be written in terms of \(O(\underline{x})\) as
The above observations and remarks lead to the following result
Lemma 1
Let O(x, y) be the joint observability of y and x. Then
Proof
Note that
Since
then we have that
The result follows. \(\square\)
3.2 Mutual information
Mutual information plays an important role in assessing the feature space and in particular, in reducing its dimensionality (Shadvar 2012). It is defined as
Mutual information can provide useful insights into feature inter-dependencies. In particular, an optimal level of features is linked with a maximum value of I(x; y). Similarly to the above, we can write I(x; y) in terms of observabilities using Eq. 14, namely
Eq. 15 can be used to find a lower bound of I(x; y) with respect to the observabilities. Recall that
Hence
Since p(x, y) can also be written as \(p(x \,\vert \,y) p(y)\), or in other words as \(e^{-(O(y\rightarrow x) + O(y))}\), using a similar argument we have that
Recall from Definition 1, \(\displaystyle {O(y \rightarrow x) = O(x \rightarrow y)- \log {\frac{w(x)}{w(y)}}}\). This implies that, if no extra neighbouring nodes are added to either x or y, then if \(O(y \rightarrow x)\) increases, \(O(x \rightarrow y)\) will increase as well, and vice-versa. This leads to the following result.
Lemma 2
Let I(x; y) be the mutual information between x and y. If \(O(y \rightarrow x)\), or \(O(x \rightarrow y)\), are minimised then I(x; y) will increase towards its maximum value.
It is important to emphasise that Lemma 2 does not provide an exact evaluation of the behaviour of mutual information, but only an estimation. However, it provides a tool to identify which features should be removed likely to increase mutual information.
3.3 Entropy and its variations
From Eq. 17, we can easily calculate the conditional entropy as
If we consider such (semantic) networks as dynamical networks, which change with respect to a (discrete) ‘time’ parameter, Eq. 2 can be written to emphasise the iteration time k, as
Assume we remove one semantic feature from \(n(x_{1}, x_{2})\) and let the corresponding weight of such feature be \(\tilde{w}\). Equation 20 can be re-written as:
Since
From the above observations, for a path \(\underline{x} = x_{1},\ldots , x_{k}\), we have that
Let
From the above equations, we can derive that
using the notation in (24), and we can also derive that
If we use \(H_{k}(x_{i} \,\vert \,x_{j})\) for two adjacent nodes \(x_{i}\) and \(x_{j}\) at time iteration k, as a general entropy ‘indicator’, we can consider its overall contribution based on all the nodes in the network \(G_{k}(V,E)\). In other words,
This will allow to evaluate
Following a similar approach as above, we can also see that
Recall that we are removing one semantic relation across all the edges. Therefore, Eq. (29), can be arranged as follows
The above calculation lead to the following result
Lemma 3
The entropy variation \(\Delta _{k}(H(G)\) of the network G at the time iteration k, can be minimised by minimising
.
This lemma further demonstrates the role of observability in the dynamical and topological properties of the corresponding networks. Although it would be equivalent to maximise I(x; y) directly, this approach strengthen the motivation of using observability to achieve this, which can be used to further investigate and analyse the topological and stochastic properties of the corresponding network.
4 Feature dimensionality reduction
Lemma 2 provides a useful tool to assess a feature space based on the corresponding observabilities. This will be achieved by considering the (semantic) network, as defined above. This will be constructed as a simplicial complex as discussed in Sect. 4.1.
4.1 Simplicial complexes via observability
Observability, despite being a premetric, can naturally create a network structure, or more precisely a simplicial complex \(G = G(V,E)\). More specifically, we assume that a link (or edge) exists between any two nodes in V, if their observability is less than a given (numerical) threshold \(\theta > 0\), which will define a filtration of the simplicial complex (Chazal and Michel 2017).
More formally, we define the edge-set E of a simplicial complex G as
Note that (32) implies that G is an undirected network. Furthermore, as discussed in Sect. 1, the nodes of G are \(n-\)dimensional so that they can be embedded in an \(n-\) dimensional space to satisfy the homology construction as per Eq. 33.
We can define the usual simplicial homology
![](http://media.springernature.com/lw388/springer-static/image/art%3A10.1007%2Fs12530-023-09535-8/MediaObjects/12530_2023_9535_Equ33_HTML.png)
where \(\partial _{m}\circ \partial _{m+1} = 0\). The \(m-\)th homology group is defined as \(H_{m}(G):= \ker {(\partial _{m} )}/ {{\,\textrm{Im}\,}}{(\partial _{m} )}\)
The assessment of \(m-\)dimensional homology groups, will determine, loosely speaking, the number of \(m-\)dimensional holes in G, which can be associated with the corresponding Betti numbers \(b_{k}\). Recall that a contractible space has trivial reduced homology groups, Hatcher (2002). Based on Alexander and Bishop (1990), and the fact that G is typically a compact space, a non-contractible space is not necessarily a uniquely geodesic space. Therefore, G might not be a uniquely geodesic space and so there are some points (associated with nodes in the simplicial complex) which cannot be connected via shortest path (from an observability point of view). This implies that mutual information cannot be in general maximised for all nodes in V. Therefore, by iteratively removing the appropriate features associated to the nodes in the node-set V, the feature space would be contractible. The above facts lead to the following result.
Proposition 1
Let \(G = G(V,E)\) be a simplicial complex as defined in Sect. 4.1. Its overall mutual information can be maximised by removing those feature dimensions related to the nodes in V, that do not yield trivial reduced homology groups.
The above proposition can be written as the following algorithm
-
Step 1:
let \(G = G(V,E)\) be the simplicial complex as per Proposition 1.
-
Step 2:
let \(p:G_{n} \rightarrow G_{m}^{k}\) be the projection from the \(n-\)dimensional space G onto the \(m-\) dimensional space \(G_{m}^{k}\), by removing k features so that \(n = m+k\).
-
Step 3:
start from \(k=0\)
-
Step 4:
if \(b_{k} \not = 0\) for \(G_{m}^{k}\), then \(k=k-1\) and repeat this step
-
Step 5:
else, stop.
4.2 Evaluation
As part of this work, an evaluation of the above approach was carried out. The datasets introduced in Trovati et al. (2020) and Trovati et al. (2022) based on ConceptNet (Liu and Singh 2004) were used to create a selection of semantic networks (with approximately 150 nodes), based on different values of \(\theta\). Each node had 10 features.
Table 1 depicts the main results for different \(\theta\) which generates different edges and from 10 different reductions
The data used in this evaluation is limited due to the extensive manual labelling required, which has been created by the authors as part of ongoing research (Trovati et al. 2022). However, more datasets are being designed to provide more comprehensive data-driven evaluations for future research and implementations.
5 Conclusions
In this article, a method to identify the most likely feature dimensions to reduce based on on homological properties of observability is introduced. Despite the limited experimental evaluation, the potential of this approach is clear, both from theoretical and implementation perspectives. In fact, the use of the concept of observability suggests important research areas to further investigate where the dynamical and topological properties should be integrated to provide a more comprehensive understanding of the evolving nature of data. In particular, future steps include a deeper assessment of different network topology properties which can provide further insights.
Data Availability
The datasets generated during and/or analysed during the current study are not publicly available due to being under review but are available from the corresponding author on reasonable request
References
Alexander S, Bishop R (1990) The hadamard-cartan theorem in locally convex metric spaces. Enseign Math 36:309–320. https://doi.org/10.5169/seals-57911
Carlsson G (2008) Topology and data. Technical report
Chazal F, Michel B (2017) An introduction to topological data analysis: fundamental and practical aspects for data scientists. arXiv:1710.04019
Edelsbrunner H, Harer J (2008) Persistent Homology – a Survey. In: Surveys on Discrete and Computational Geometry, vol. 453, p. 257. Amer Mathematical Society
Hatcher A (2002) Algebraic Topology. Cambridge University Press, Cambridge
Liu H, Singh P (2004) Conceptnet – a practical commonsense reasoning tool-kit. BT Technol J 22(4):211–226. https://doi.org/10.1023/B:BTTJ.0000047600.45421.6d
Newman M (2010) Networks: an Introduction. Oxford University Press, Inc. http://scholar.google.de/scholar.bib?q=info:WDP-LHCuDj4J:scholar.google.com/ &output=citation &hl=de &as_sdt=0,5 &ct=citation &cd=0
Ray J, Trovati M (2018) A survey of topological data analysis (tda) methods implemented in python. In: Advances in Intelligent Networking and Collaborative Systems: The 9th International Conference on Intelligent Networking and Collaborative Systems (INCoS-2017), pp. 594–600. Springer
Ray J, Johnny O, Trovati M, Sotiriadis S, Bessis N (2018) The rise of big data science: a survey of techniques, methods and approaches in the field of natural language processing and network theory. Big Data Cognit Comput 2(3). https://doi.org/10.3390/bdcc2030022
Shadvar A (2012) Dimension reduction by mutual information feature extraction. CoRR arXiv:1207.3394
Shao Y, Trovati M, Shi Q, Angelopoulou O, Asimakopoulou E, Bessis N (2017) A hybrid spam detection method based on unstructured datasets. Soft Comput-A Fusion Found Methodol Appl 21(1):233–243
Trovati M, Zhang H, Ray J, Xu X (2020) An entropy-based approach to real-time information extraction for industry 4.0. IEEE Trans Ind Inf 16(9):6033–6041
Trovati M, Johnny O, Xu X, Polatidis N (2022) A new model for artificial intuition. In: Artificial Neural Networks and Machine Learning - ICANN 2022: 31st International Conference on Artificial Neural Networks, Bristol, UK, September 6-9, 2022, Proceedings, Part I, pp. 454–465. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-031-15919-0_38
Xu X, Hu N, Li T, Trovati M, Palmieri F, Kontonatsios G, Castiglione A (2019) Distributed temporal link prediction algorithm based on label propagation. Fut Gen Comput Syst 93(C), 627–636. https://doi.org/10.1016/j.future.2018.10.056
Zhang M, Kalies W, Kelso S, Tognoli E (2020) Topological portraits of multiscale coordination dynamics. J Neurosci Methods 339:108672. https://doi.org/10.1016/j.jneumeth.2020.108672
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
On behalf of all authors, the corresponding author states that there is no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Trovati, M., Farsimadan, E. Feature dimensionality reduction via homological properties of observability. Evolving Systems 15, 57–63 (2024). https://doi.org/10.1007/s12530-023-09535-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12530-023-09535-8