Hypergraph Modeling

Abstract

Hypergraph modeling is the fundamental task in hypergraph computation, which targets on establishing a high-quality hypergraph structure to accurately formulate the high-order correlation among data. In this section, we introduce different hypergraph modeling methods to show how to build hypergraphs using various pieces of information, such as features, attributes, and/or graphs. These methods are organized into two broad categories, depending on whether these correlations are explicit or implicit, to distinguish the similarities and differences. We then further discuss different hypergraph structure optimization and generation methods, such as adaptive hypergraph modeling, generative hypergraph modeling, and knowledge hypergraph generation.

4.1 Introduction

Although there are complex correlations among data in many applications, it is difficult to discover such complex correlations in many cases due to the limitations of observation technologies. Taking social networks as an example, the group information is a kind of high-order correlation that connects a number of people based on certain criteria. However, it is intractable to investigate all the groups when there are millions or even billions of vertices in a social network. Another typical example is the human brain network. Apparently, some functions of the brain are implemented by the communications among multiple brain regions rather than just two regions, which means that there exist high-order correlations among brain regions. Nevertheless, much manpower and material resources would be required to directly record such high-order correlations by neuroscience experiments. Therefore, it is necessary to study how to model such high-order correlations based on existing information in practical applications.

Hypergraph has shown its superiority on high-order correlation modeling. Hypergraph structure generation has attracted much attention and is still an open problem due to complex correlations among non-standard data. In this chapter, we systematically review the existing hypergraph modeling methods, including both the implicit hypergraph modeling strategy and the explicit hypergraph modeling strategy. The implicit hypergraph modeling strategy aims to generate the hypergraph structure using vertex representations based on their distances or similarities, in which the correlations are not directly provided. The explicit hypergraph modeling strategy targets at the data with explicit high-order correlation information, such as attributes and pairwise connections. For the implicit hypergraph modeling strategy, we mainly introduce the distance-based and representation-based hypergraph structure generation methods. For the explicit hypergraph modeling strategy, we focus on the attribute-based and the network-based hypergraph generation approaches. Figure 4.1 illustrates the hypergraph modeling methods.

Fig. 4.1

Different categories of hypergraph modeling methods

We further give four examples in computer vision, recommender system, computer-aided diagnosis, and brain network for hypergraph modeling in this chapter. In the last part, we discuss the topics of further research of hypergraph modeling, which have the potential of going beyond the limitations of current methods that are difficult to be adaptive to complex data. Part of the work introduced in this chapter has been published in [1,2,3,4].

4.2 Implicit Hypergraph Modeling

In implicit hypergraph modeling, the correlations among data are not directly provided. Under such circumstances, we need to explore different representations of the data to build the correlations. Two typical methods for implicit hypergraph modeling are distance-based methods and representation-based methods. In distance-based methods, we can explore the neighborhood information for each sample in some specific feature spaces, and the samples with high similarity/low distance can be connected by a corresponding hyperedge. In representation-based methods, the representation among different feature vectors for the samples is used to measure the neighborhood information, which can be used to generate hyperedges.

4.2.1 Distance-Based Hypergraph Generation

Distance-based hypergraph generation methods construct the hyperedges based on the distances in the feature space for all the vertices. In general, the construction of the hypergraph can be divided into two steps: the incidence matrix generation and the hyperedge weight generation. For the incidence matrix generation, the connectivity on the hypergraph, i.e., the hyperedge, is determined with the consideration of the neighborhood relationships, where the neighbors of the vertices in the feature space are connected by these hyperedges. For the hyperedge weight generation, the weights of these hyperedges are calculated based on the distance information.

The incidence matrix is generated based on the neighbors of the vertices. There are two major approaches to determine the neighbors [1], i.e., the nearest-neighbor-based hyperedge generation strategy (shown in Fig. 4.2) and the clustering-based hyperedge strategy (shown in Fig. 4.3). The nearest-neighbor-based hyperedge generation strategy searches the nearest vertices for the given vertex, i.e., the centroid, and connects these vertices by the hyperedges. The clustering-based hyperedge generation strategy groups the vertices with the features and constructs a hyperedge to connect all vertices fallen into the same cluster.

Fig. 4.2

An illustration of the nearest-neighbor-based hyperedge generation strategy. (a) shows the k-NN neighbors of the given vertex, and (b) shows the 𝜖-ball neighbors

Fig. 4.3

Illustration of the cluster-based hyperedge generation strategy. This figure is from [1]

The nearest-neighbor-based hyperedge generation strategy starts out with calculating the distances between all pairs of vertices in the feature space. Subsequently, two commonly used criteria [5] are applied to determine the neighbors of the given centroid, i.e., the k-NN neighbors [6] and the 𝜖-ball neighbors [2]. The given centroid and the selected neighbors are connected together by a hyperedge.

Here we denote $\mathbb {V}$ as the vertices set, $u\in \mathbb {V}$ as the given centroid, X(u) as the feature vector of u, d(x ₁, x ₂) = ||x ₁ − x ₂||₂ as the Euclidean distance between the vectors x ₁ and x ₂, $\mathbb {N}_k(u)$ as the k-NN neighbors set of u, and $\mathbb {N}_\epsilon (u)$ as the 𝜖-ball neighbors set of u. $\mathbb {N}_k(u)$ contains k vertices with the smallest distance to u, while $\mathbb {N}_\epsilon (u)$ contains the vertices with distance smaller than 𝜖, i.e.,

$$\displaystyle \begin{aligned} \mathbb{N}_\epsilon(u) = \{ v| d(X(u),X(v))\leq \epsilon \}. \end{aligned} $$

(4.1)

The vertex u and the neighbors $\mathbb {N}(u)$ (either $\mathbb {N}_k(u)$ or $\mathbb {N}_\epsilon (u)$) are grouped together to generate a hyperedge e(u):

$$\displaystyle \begin{aligned} e(u) = \mathbb{N}(u) \cup \{ u \}, \end{aligned} $$

(4.2)

and the hyperedge set $\mathbb {E}$ is formulated as

$$\displaystyle \begin{aligned} \mathbb{E} = \{ e(u) | u\in \mathbb{V} \}. \end{aligned} $$

(4.3)

The clustering-based hyperedge generation strategy starts out with grouping the vertices according to the corresponding features using the clustering algorithms, such like k-means. Subsequently, the vertices belonging to the same cluster are connected together using a hyperedge. Here we assume that the k-means algorithm clusters the vertex set $\mathbb {V}$ into K groups $\mathbb {V}_1, \ldots , \mathbb {V}_K$. Then, K hyperedges can be constructed using these clustering results:

$$\displaystyle \begin{aligned} \forall 1\leq k\leq K, e_k = \mathbb{V}_k = \{ v_{k_1}, v_{k_2}, \ldots \}, \end{aligned} $$

(4.4)

and the hyperedge set $\mathbb {E}$ is formulated as

$$\displaystyle \begin{aligned} \mathbb{E} = \{ e_k | \forall 1\leq k\leq K \}. \end{aligned} $$

(4.5)

Besides the similarity/distance in the feature space, other types of information, which can be used to measure the correlation in some specific space, such as the spatial information, can also be applied for hyperedge generation. For example, the spatial information of pixels in an image can be used to select a group of neighbor pixels for one centroid, which can be connected by a hyperedge, as shown in Fig. 4.4.

Fig. 4.4

An illustration of using spatial information of pixels to generate a hyperedge

Typically, an incidence matrix H is used to represent the structure of the hypergraph, i.e.,

$$\displaystyle \begin{aligned} {\mathbf{H}}_{ue} = \begin{cases} 1 & \text{ if } u\in e \\ 0 & \text{ otherwise} \end{cases}, \end{aligned} $$

(4.6)

where $u\in \mathbb {V}$ and $e\in \mathbb {E}$.

The weight matrix of the hypergraph represents the importance of each hyperedge. A commonly used method for the hyperedge weight measurement is based on the Gaussian kernel, where the scores of each pair of vertices belonging to the same hyperedge are calculated using the distance between the vertices in the pair and the average score can be used as the weight of the hyperedge, i.e.,

$$\displaystyle \begin{aligned} w(e) = \sum_{u,v\in e} \exp \left( -\frac{d( X(u), X(v) )}{\sigma^2} \right), \end{aligned} $$

(4.7)

where w(e) denotes the weight of hyperedge e, and σ is the band width of the Gaussian kernel.

In this way, if the vertices connected by a hyperedge are with relatively higher similarity, the corresponding hyperedge weight could be larger and vice versa. Then, the hyperedge weights can represent whether this hyperedge is trustable for further processing.

In practice, σ can be set as the median value of the distances among all vertices by

$$\displaystyle \begin{aligned} \sigma = \text{median}_{u,v\in \mathbb{V}} d\left({ X\left({u}\right), X\left({v}\right) }\right), \end{aligned} $$

(4.8)

where median denotes the median value. It is noted that the hyperedge weight can be set in other ways following the purpose of evaluating the importance of each hyperedge.

The main limitation of the distance-based hypergraph generation method is the inaccurate distances due to noise and outliers of data, which may further introduce noise to the structure of hypergraphs. In practice, the feature representation for the data is still a challenging task. It is not easy to conduct effective feature extraction under certain application scenario. The metric for distance calculation also matters. Although the Euclidean distance is commonly used, there still exist some other metrics, such as the L ₁-norm and the negative cosine distance. The decision making of these metrics requires experimental evaluation. Therefore, the distance-based hypergraph generation method may suffer under such circumstances.

The nearest-neighbor-based hyperedge generation strategy is the most simple one to be deployed in practice. The limitations of this strategy are as follows. First, the hyperparameter, i.e., k for the k-NN neighbors and 𝜖 for the 𝜖-ball neighbors, may significantly affect the structure of the hypergraph and further influence the performance of hypergraph learning. Unfortunately, there are still no general principles for the selection of k and 𝜖, and the adaptive justification of these hyperparameters is not trivial in practice. Second, the calculation of the k-NN neighbors is expensive for large scaled data in both time and memory.

Regarding the clustering-based hyperedge generation strategy, there is no common way to determine how many clusters should the vertex set be divided into, as the scale of the clustering results also affects the structure of the hypergraph. A possible solution is to conduct clustering multiple times in different scales, which generates multiple hypergraphs with different k values and then composes these hypergraphs together for multilevel representation.

4.2.2 Representation-Based Hypergraph Generation

As introduced above, the distance-based hypergraph generation has some disadvantages. For the kNN hypergraph, the hypergraph, which connects the centroid sample and its k nearest samples, is uniform. Its structure may not be sufficiently adaptive. Also, the distance-based hypergraph is sensitive to noise. To solve this problem, the hypergraph can be generated by the representation-based methods.

Different from the distance-based methods, which generate hyperedges through some metrics in the feature space, the relations among the vertices in representation-based methods are from the feature reconstruction, as shown in Fig. 4.5. In reconstruction, different strategies have different generation effects. Here we introduce three representation-based main branches to construct hypergraphs, i.e., l ₁-hypergraph [7], l ₂-hypergraph [8], and the combination of them both. The details of these methods are described as follows.

Fig. 4.5

An illustration of the representation-based methods

(1) l ₁-Hypergraph Generation

For the l ₁-hypergraph construction, as introduced in [7], sparse representation method can be used to formulate the relation between the hyperedge and its vertices, and the sparse representation is embodied in the coefficients that linearly combine the basic vectors to reconstruct the input vector. In the hyperedge construction, the centroid vertex is reconstructed by the other vertices in the same hyperedge. We use the coefficients to present the incidence matrix of hypergraph. Mathematically, we denote the centroid vertex in the l ₁-hypergraph by v _c, and it can be represented as

$$\displaystyle \begin{aligned} & \mathop{\arg\min\limits_{\mathbf{z}}}\lVert\mathbf{Bz}-\mathbf{X}(v_c)\rVert_2^2+\gamma\lVert\mathbf{z}\rVert_1, \\ & {s.t.}~~\forall i, {\mathbf{z}}_i \ge 0, \\ \end{aligned} $$

(4.9)

where X(v _c) denotes the feature vector of the centroid vertex, B denotes the feature of its k nearest vertices, and z _i is the reconstruction coefficient vector. The first term in Eq. (4.9) is the reconstruction term that makes a good representation of input vector X(v _c) with the basic vectors B. The second term is the l ₁-regularization, which forces the coefficient z to sparse. γ is a hyperparameter that balances the influences of the two terms. The constraint z _i ≥ 0 makes the reconstruction coefficients non-negative. Note that each sample may act as a centroid vertex to generate a hyperedge. For the dataset containing n samples, the optimization problem is solved for n times. The non-zero reconstruction coefficients in the representation can be seen as the connection weights of the neighborhood vertices in the hyperedge, and the neighborhood vertices with zero reconstruction coefficients are outside of the hyperedge. The connection weight between the hyperedge and the neighborhood vertices can be set as the vector of coefficients z _i. The incident matrix H of this hypergraph is defined as

$$\displaystyle \begin{aligned} \begin{array}{rcl} {} \mathbf{H}(v_j, e_i) = \begin{cases} {\mathbf{z}}_i^j & \text{if } v_j \in e_i \\ 0 & \text{otherwise} \end{cases}, \end{array} \end{aligned} $$

(4.10)

where e _i is generated with the centroid vertex v _i, and ${\mathbf {z}}_i^j$ is the jth element of representation coefficients z.

(2) Elastic-Hypergraph Generation

The l ₁-regularization in l ₁-hypergraph can generate sparse and effective hypergraphs, despite that fact that it is hard to reveal the grouping information of samples. To enhance the effect of grouping, the elastic net [9] is introduced to combine an l ₂-norm penalty with the l ₁-norm constraint. The objective function of elastic net can be formulated as

$$\displaystyle \begin{aligned} & \mathop{\arg\min\limits_{\mathbf{z}}}\lVert\mathbf{Bz}-\mathbf{X}(v_c)\rVert_2^2+\gamma\lVert\mathbf{z}\rVert_1+\beta\lVert\mathbf{z}\rVert_2^2, \\ & {s.t.}~~\forall i. {\mathbf{z}}_i \ge 0. \\ \end{aligned} $$

(4.11)

The elastic net can create a hyperedge whose weight can be determined by the reconstruction coefficients by using both the l ₂-norm and the l ₁-norm penalties to group more relevant and important neighbors.

(3) l ₂-Hypergraph Generation

Note that there are two drawbacks of the above two representation-based approaches: (1) They use a l ₂-norm-based metric to measure the reconstruction errors, which makes them still sensitive to sparse reconstruction errors. (2) Since these methods create hyperedges by linearization, they are unable to handle nonlinear data. By eliminating the sparse noise component from the original data, integrating the locality, and maintaining the constraint to the linear regression framework, the l ₂-hypergraph [9] is created to address these issues as

$$\displaystyle \begin{aligned} & \mathop{\arg\min\limits_{\mathbf{z}}}\lVert\mathbf{X}-\mathbf{XC}-\mathbf{E}\rVert_F^2+\frac{\gamma_1}{2}\lVert\mathbf{C}\rVert_F^2+\frac{\gamma_2}{2}\lVert\mathbf{Q}\odot\mathbf{C}\rVert_F^2+\beta\lVert\mathbf{E}\rVert_1, \\ & {s.t.}~~{\mathbf{C}}^T\mathbf{1}=\mathbf{1}, \mathtt{Diag}(\mathbf{C})=\mathbf{0}, \\ \end{aligned} $$

(4.12)

where ⊙ stands for element-wise multiplication, C is the coefficient matrix, E is the data error matrix, and Q is the locality adapter matrix used to retain the local manifold structures. Hyperedges can then be created using the coefficient matrix C.

The ability of each vertex being able to be reconstructed in the feature space can be evaluated via representation-based hyperedges. It is possible to calculate and use the correlation between the feature vectors to create connections among the vertices. Similar to the distance-based methods, this field of study may encounter the issue of data noise and outliers. Another drawback of this type of hypergraph generation methods is that only a portion of the relevant samples is chosen for reconstruction during the computing process, and the resulting hyperedge may not be able to accurately capture the data correlation through the complete data distribution.

4.3 Explicit Hypergraph Modeling

Different from implicit hypergraph modeling, in some cases, there are existing connections among data. Explicit hypergraph modeling focuses on these scenarios and generates hypergraph structure using attribute information or networks.

4.3.1 Attribute-Based Hypergraph Generation

The data in real world may be associated with attributes in many cases. For example, the users in social network could have profiles, such as gender, age, and interests. The visual objects in images could have different characteristics, such as color, shape, and texture. Given the data assigned with different attributes, attribute-based hypergraph generation methods can be adopted to construct the hypergraph based on the attribute information, which provides an explicit way to encode semantic properties and diffuse knowledge [10]. As such a construction schema leverages the apparent correlations among objects directly, it can be categorized as explicit hyperedge methods.

To generate a hypergraph using attributes, the following steps are needed: the hypergraph structure construction and the hyperedge weight assignation. The first step is to generate the vertex set $\mathbb {V}$ and hyperedge set $\mathbb {E}$ based on the attribute information, and the second step is to assign different weights to the hyperedges and acquire the weight matrix W.

When constructing the hypergraph from the attribute data, the samples to be explored are first modeled as vertices in a hypergraph, denoted as the vertex set $\mathbb {V}$. The same attribute shared by different vertices effectively indicates that these samples share common characteristics, which may be an objective tag or a subjective evaluation. Therefore, each attribute can be regarded as the semantic information on a connection, i.e., a hyperedge. In attribute-based hypergraph generation methods, a group of hyperedges (called a hyperedge group) are generated by linking all the vertices associated with the attribute space. It is obvious that the number of hyperedges equals to the number of attributes in this way. Such a hyperedge group generated based on the attribute information is denoted by

$$\displaystyle \begin{aligned} \mathbb{E}_{\text{attribute}} = \Big \{ N_{\text{att}}(a) \mid a \in \mathbb{A} \Big \}, \end{aligned} $$

(4.13)

where N _att(a) is the subset of vertex set $\mathbb {V}$ sharing the attribute a, and $\mathbb {A}$ is a set containing all defined attributes. Sometimes the attribute could be hierarchical, e.g., the car within the vehicles. In this case, the $\mathbb {A}$ and $\mathbb {E}_{\text{attribute}}$ can be extended to involve the subtypes of the attributes.

Here we give one simple example to show how to construct the hypergraph structure using the attribute information, as shown in Fig. 4.6. Given a social network data with user profiles, the users in the social network are first modeled as vertices $\mathbb {V}$. The user profiles contain the objective reality such as gender and age as well as the subjective characteristics such as interests and knowledge, both of which can be adopted to generate the hyperedge groups. For example, we can have e _female hyperedge connecting all female users and e _sports linking users who like sports. Additionally, as discussed above, sometimes the attributes are hierarchical. Under such circumstances, we can generate hyperedges with different levels to characterize multiple-scale attribute connections. For instance, we have users A, B, C, and D who all like sports, among them both users A and B like playing basketball, and users C and D like playing tennis. In this case, we first generate e _sports connecting users A, B, C, and D, and then e _basketball and e _tennis are generated to link A, B and C, D, respectively. The hyperedge set in this example can be written as

$$\displaystyle \begin{aligned} \mathbb{E} = \{e_{female}, e_{sports}, e_{basketball}, e_{tennis}.\}.\end{aligned}$$

Fig. 4.6

An illustration of the attribute-based hyperedge generation method

The hyperedge weights are also important here. For attribute-based hypergraph, the number of shared attributes among the samples connected by the hyperedge can quantitatively reflect the relative correlation strength. Specifically, the more the attributes that the samples share, the stronger connections exist among these corresponding vertices, and the bigger weight that the hyperedges are assigned. Here each hyperedge e here can be seen as a clique. The mean of the heat kernel weights w(e) of the pairwise edges in this clique is considered as the corresponding hyperedge weight:

$$\displaystyle \begin{aligned} w(e)=\frac{1}{\delta(e)(\delta(e)-1)} \sum_{u,v \in e} \exp \left(-\frac{\left\|\mathbf{X}(u)-\mathbf{X}(v)\right\|{}_2^{2}}{\sigma^2}\right), \end{aligned} $$

(4.14)

where δ(e) indicates the degree of hyperedge e, and X(u) and X(v) denote the feature vectors of vertices u and v, respectively.

The attribute-based hypergraph generation method can capture the semantic properties apart from the structural information conveyed by the hypergraph structures themselves. The attributes serve as a type of intermediate-level feature representation of vertices and can provide another description for vertices beyond the low-level representations. However, the attributes are not available all the time. When there is no natural attribute descriptor for vertices, some extra solutions need to be applied to conduct attribute-based hypergraph generation. One possible solution is to manually design attribute tags, which may be both cumbersome and time-consuming. The other alternative is extracting attribute information from the raw low-level features by machine learning models [11]. Such a schema is more time-saving than manual definition, whereas the results rely heavily on the accuracy of the machine learning model. We also note that the attributes can be nameable, which indicates the semantic information can be directly understood, while they can also be non-nameable, which means the semantic information is not explicit.

4.3.2 Network-Based Hypergraph Generation

There are many applications of network data, including social networks [12], reaction networks [13], cellular networks [13], and human brain networks [3]. It is possible to generate subject correlations using the network information for these data. In a typical work of social media analysis [14], the vertices on hypergraph represent users and images. In addition to visual–textual relationships among images, hyperedges can be used to capture social links between users and images, also called homogeneous and heterogeneous hyperedges. The nearest-neighbor-based and attribute-based hyperedge generation methods are used to construct homogeneous hyperedges representing the visual and textual relations among images. Users and images are connected through social link relations to construct heterogeneous hyperedges. For example, both friendship and mobility information in location-based social networks can be used to generate hypergraphs using [12]. As a result, friendship hyperedges are generated within the social domain, and check-in hyperedges are generated across the social, semantic, temporal, and spatial domains. A protein–protein interaction network is naturally represented by a hypergraph [15], whose subsets (hyperedges) can be represented by tandem affinity purification (TAP) data.

Aside from the first-order correlation, high-order correlations, e.g., the second- and third-order correlations, within the network can also be used as a means for generating hyperedges. A center vertex can be connected with its first-order and high-order neighbors (i.e., vertices whose shortest path to the centroid is greater than 1) through a hyperedge. A vertex’s low-order neighbors need only to be considered if attention is focused on its local connection in the network. As an example, users who have similar preferences on items are able to be connected within the recommendation network [4] according to first-order and second-order correlations, which will be used in order to generate a hypergraph as well as to perform collaborative filtering for the recommendation. Alternatively, if information of a vertex travels a long distance in the network, higher-order correlation is required to generate hyperedges.

We then introduce two typical approaches to construct hyperedges from network/graph structure, i.e., pair-based and k-hop-based. Figure 4.7 illustrates the profile of these two approaches. In this example, $\mathbb {G}_s=(\mathbb {V}_s, \mathbb {E}_s)$ represents the graph structure with ${v_i} \in \mathbb {V}_s$ representing a vertex and $e_{s_{ij}}\in \mathbb {E}_s$ representing an edge connecting v _i and v _j. We let A indicate the adjacency matrix of $\mathbb {G}_s$. As a result of such a graph structure, two types of hyperedge groups can be generated (Fig. 4.7).

Fig. 4.7

An illustration of the network-based hyperedge generation method

(1) Pair-Based Hyperedge Generation Strategy

The $\mathbb {E}_{\text{pair}}$ is adopted to indicate the hyperedges constructed by pair correlations in the network/graph. $\mathbb {E}_{\text{pair}}$ targets at directly transforming the graph structure into a group of 2-uniform hyperedges, which can be formulated as follows:

$$\displaystyle \begin{aligned} {\mathbb{E}}_{\text{pair}} = \Big \{ \{v_i, v_j\} \mid (v_i, v_j) \in \mathbb{E}_s \ \Big \}. \end{aligned} $$

(4.15)

As a result, ${\mathbb {E}}_{\text{pair}}$ covers the low-order (pairwise) correlations in the graph structure, which is the basic information for high-order correlation modeling.

(2) k-Hop-Based Hyperedge Generation Strategy

$\mathbb {E}_{\text{hop}}$ is adopted to indicate the hyperedges constructed by the k-hop neighbors in the network/graph. First, we define the k-hop neighborhoods of a vertex v in graph $\mathbb {G}_s$ as follows:

$$\displaystyle \begin{aligned} N_{\text{hop}_k}(v) = \{ u \mid {\mathbf{A}}^k_{uv}\neq 0, u \in \mathbb{V}_s \}.\end{aligned}$$

Based on the k-hop’s reachable positions in the graph structure, $\mathbb {E}_{\text{hop}}$ aims to find the related vertices for a central vertex. The range of the values of k is [2, n _v], where n _v refers to the number of vertices in $\mathbb {G}_s$. The following is an example of a hyperedge group $\mathbb {E}_{\text{hop}}$ with k-hop:

$$\displaystyle \begin{aligned} \mathbb{E}_{\text{hop}_k} = \Big \{ N_{\text{hop}_k}(v) \mid v \in \mathbb{V} \Big \}. \end{aligned} $$

(4.16)

The hyperedge generated by ${\mathbb {E}}_{\text{hop}}$ can be exploited by extending the search radius to the external vertices, which also leads to groups of vertices rather than just two vertices, as opposed to two vertices only in the graph structure. As compared with just the pairwise correlation in ${\mathbb {E}}_{\text{pair}}$, it can provide more information about correlations.

Here, we discuss the advantages and limitations of the two types of hyperedges using network data, respectively. As far as the pair-based construction is concerned, clearly this type of hyperedge can only model low-order correlations, which cannot naturally explore high-order correlations in some scenarios. In contrast, hyperedges generated from the k-hop-based methods have the high-order information built-in of the original network. However, the high-order information in this type of hyperedges may be redundant and ambiguous. This is because the connection details in the k-hop-based hyperedges may be lost, which means that you cannot reconstruct the original network/graph from this type of hyperedge. Additionally, the k-hop-based hyperedges may lead to irreversible over-smoothing in each hyperedge, which is caused by the k-hop neighbors with exponential growth as k grows.

4.4 Typical Examples of Hypergraph Modeling

Here we give several examples of hypergraph modeling in real applications, including computer vision, recommender systems, computer-aided diagnosis, and brain networks, to demonstrate how to construct hypergraphs from data.

4.4.1 Computer Vision

Computer vision has attracted much attention in recent decades. In computer vision, there are multi-modal data, such as images, point clouds, etc. Both low-level vision tasks and high-level vision tasks have been deeply investigated. In these tasks, an important but challenging issue is the complex data correlation behind the vision data. For example, from the aspect of images, the pixels or patches are the elements of an image, while the semantic information for the image is represented by these pixels or patches. Terrence Joseph Sejnowski [16] mentioned that “In a task such as face recognition, in which important information may be contained in the high-order relationships among pixels , it seems reasonable to expect that better basis images may be found by methods sensitive to these high-order statistics.” Similar situations occur when facing multi-modal 3D object representation. Usually, a 3D object can be represented by different ways, such as one single image, multi-view, point clouds, voxel, and mesh. Under such circumstances, the correlation among these objects becomes even more complicated. To model such high-order relationship among pixels/patches in one image, or among different 3D objects, simple graph is not capable to conduct this task.

We first look into the high-order correlation modeling for an image. A 2D image is composed of a set of pixels, and each pixel owns a feature vector (channels). To generate a hypergraph to model the correlation behind this image, we can take each patch in the image as a vertex in the hypergraph, and the objective is to generate a group of hyperedges to connect these vertices (patches). Here we can employ the distance-based hypergraph generation method, in which each patch is selected as the centroid, and its nearest neighbors in the feature space are connected by a hyperedge. This process is shown in Fig. 4.8. Furthermore, we can also employ the spatial information to build connection among these patches. The patches with closed spatial locations in the image could be connected with a hyperedge. Figure 4.9 shows an example of hypergraph modeling for image patches using spatial information.

Fig. 4.8

An example of hypergraph modeling for image patches using feature information

Fig. 4.9

An example of hypergraph modeling for image patches using spatial information

For 3D visual objects, there are complex correlations among them. For example, different furniture, such as tables and chairs, have legs, and different vehicles, such as cars and bicycles, have wheels. Another challenging issue comes from the multi-modality aspect. Given different modal data of 3D objects, the correlations are composed of inter-modal correlations and the cross-modal correlations, as shown in Fig. 4.10. Given a large number of 3D objects, it is difficult to accurately and completely manually describe all these correlations.

Fig. 4.10

The complex correlations among multi-modal 3D objects

In order to efficiently build a hypergraph structure, we usually extract the features of 3D objects and then build implicit hypergraphs. 3D objects can be described by multiple modalities, including point clouds, views, grids, and voxels. We can extract the descriptors of their respective modalities through the corresponding deep neural networks, such as dynamic graph CNN (DGCNN) [17] and PointNet (PointNet) [18] for point cloud data, multi-view convolutional neural networks (MVCNN) [19], and group-view convolutional neural networks (GVCNN) [20] for the multi-view data. When multi-modal features have been obtained, we can build a hypergraph structure for each kind of features.

Here, each 3D object can be represented by a vertex in the hypergraph. Each time, one object is selected as the centroid in a feature space, and its nearest neighbors can be connected by a corresponding hyperedge. This process is repeated until all objects have been selected as the centroid once in this feature space. Every feature and possible feature combination can be used in this process. In this way, we can achieve multiple hypergraphs, represented by incidence matrices H ₁, H ₂, …, H _m to formulate their correlations under different modalities. The pipeline is demonstrated in Fig. 4.11. We can further concatenate these incidence matrices along the axes of hyperedges to integrate these hypergraphs and obtain the complete hypergraph structure, as shown in Fig. 4.12.

Fig. 4.11

An example of hypergraph modeling for multi-modal 3D objects

Fig. 4.12

An illustration of multi-hypergraph combination. This figure is from [5]

4.4.2 Recommender System

In a recommender system, the relationship between users and items can be represented by a bipartite graph, that is, if an item is in a user’s recommendation list, then we connect the user vertex with the item vertex. This bipartite graph can be simply transformed into a hypergraph, where vertices on one side remain and vertices on the other side become hyperedges, as shown in Fig. 4.13. In this way, each user can be represented as a vertex in the hypergraph, and the users shared the same items can be connected by a corresponding hyperedge here. If the item is regarded as a vertex, then the hyperedges are generated using shared users. This hypergraph generation procedure follows the attribute-based strategy.

Fig. 4.13

An example of hypergraph modeling for a recommender system

Mathematically, the ranking matrix of the recommender system equals to the incidence matrix of the corresponding hypergraph. With this transformation, we can solve the problem in recommender systems via hypergraph learning methods. In fact, undirected bipartite graph modeling and hypergraph modeling are interchangeable in some cases. If the edges in bipartite graph are weighted, we can use the hyperedge-dependent vertex weights accordingly.

4.4.3 Computer-Aided Diagnosis

In computer-aided diagnosis, the main objective is to measure whether a coming patient has some specific disease or not, or how serious the disease it is. For diagnosis, the experience and knowledge are from previous medical records. Case-based diagnosis has shown importance in practice. For AI-based computer-aided diagnosis, it is important to explore the existing labeled training data, which could be very few in some cases. These medical records may contain different examine files, MR images, CT images, and other types of data.

A conventional pipeline for computer-aided diagnosis is first extracting features from clinical text or medical imaging data and then applying computer programs to automatically categorize healthy people and patients. The commonly used techniques involve natural language processing, medical imaging analysis, machine learning, etc. It is worth noting that the existing methods mostly focus on individual subject classification. Under such circumstances, how to model the correlation among these subjects, including the training data and the coming patient (the testing data), is an important but difficult task.

Here, a hypergraph at the subject level, i.e., each vertex stands for a subject, can be generated, where the hyperedges can be created using the distance-based method or attribute-based method. Given the MR images or other medical data, the features can be used to measure the distance between each two subjects. Then, the k-NN scheme can be used to select nearest neighbors for a centroid vertex and then generate a corresponding hyperedge, as shown in Fig. 4.14.

Fig. 4.14

An example of hypergraph modeling for computer-aided diagnosis

Another type of applications is to model the inter-correlation in one medical image, such as gigapixel whole-slide histopathological images (WSIs). Survival prediction is an important task in medical image analysis, which targets on modeling the life duration of a patient using WSIs. Different from traditional images, WSIs are with very large size and rich details. Therefore, traditional image representation methods do not work well in this task. To formulate the inter-correlation inside a WSI, a hypergraph can be generated, which the patch correlations are generated. A group of patches can be sampled from the original WSI, such as 2000 or 8000 patches. Then, these patches are represented as vertices in the corresponding hypergraph. The hyperedges can be generated based on either the visual feature of these patches or the spatial information, or both of them, using the distance-based hypergraph generation methods.

4.4.4 Brain Network

Recently, the development of neuroimaging techniques has provided a way to understand the brain network on a large scale. Studies have shown that the interaction relationships in the brain, from neuronal information flow to whole-brain functional networks, are the basis of its functionality. Therefore, formulating the brain as a complex network and decoding its signals may further deepen our understanding of the human cognitive processes. The conventional functional network is usually modeled and represented based on pairwise correlations between two brain regions. However, neurologically, a brain region predominantly interacts with one or more other brain regions.

When using hypergraphs to model a single brain network, the vertices denote brain regions, and the hyperedges represent the interactions among multiple regions. Each element in the incidence matrix corresponds to the contribution of the brain region to the specific function, as shown in Fig. 4.15. In this process, each region can be selected as the centroid, and its nearest neighbor regions in the feature space can be selected and connected by a corresponding hyperedge.

Fig. 4.15

An example of hypergraph modeling for brain network

4.5 Hypergraph Modeling in Next Stage

In this part, we discuss future research topics of hypergraphs modeling to render them more accurate and flexible, including adaptive hypergraph modeling, generative hypergraph modeling, and knowledge hypergraph generation.

4.5.1 Adaptive Hypergraph Modeling

Having initialized the hypergraph structure, the structure is fixed during the learning process. However, the initial hypergraph structure constructed by existing hypergraph modeling methods contains many noisy connections that may be destructive for the learning process. Therefore, the original structure needs to be optimized according to the data and downstream tasks to cut down on structure noise. Although there are some existing work on hypergraph structure optimization, these efforts are still far from reaching the goal of accurately modeling of complex data correlations.

At this stage, the selection of hypergraph generation methods still depends on experience, rather than a theoretical strategy. A possible route to conduct automate hypergraph generation is to create various hypergraphs via different approaches and then group them together to obtain a more complex but relatively complete hypergraph. The grouping weights can be learned in the training stage. Another way is to update the incidence matrix of hypergraph structure, which can be either directly optimized as learnable parameters or indirectly optimized via metric learning.

4.5.2 Generative Hypergraph Modeling

The generative models are a set of models that learn the distribution from the observed data and generate new data instances based on probability. They have been widely used in different tasks such as generation, synthesis, translation, reconstruction, prediction, etc. In recent years, with the development of deep graph representation learning, deep graph generative models have attracted much attention. Given a series of training graph data (assumed to be taken from the same distribution), the neural network is trained as a graph generation model. Inspired by these generative models, building a hypergraph by estimating the distribution of latent structures from observed data may be a viable way. Given a set of training hypergraphs or sampling signals from every vertex, the distribution can be implicitly or explicitly derived by combining hypergraph embeddings and generative models.

However, there is still a long way to go for hypergraph generative models to become practical. Unlike simple graphs whose distributions are the joint distributions of all pairwise correlations between data, the distribution of a hypergraph structure is the joint distribution of all high-order correlations among data. Therefore, the joint distribution is high dimension, and the variables are dependent on each other. Estimating the density function is intractable with considerable complexity. Furthermore, due to the high-dimensional issue, a large amount of observed data is required to make the density estimate closer to the true distribution, which is difficult to obtain in practical applications. Despite the above obstacles, generative hypergraph modeling is an area worth exploring in the future and will become useful in many areas, such as simulations of complex physical systems, trajectory tracking system identification, and community detection.

4.5.3 Knowledge Hypergraph Generation

Knowledge hypergraph has attracted much attention in recent years since it can store facts using high-arity relations. In a knowledge hypergraph $\mathbb {H}=(\mathbb {V},\mathbb {E})$, the vertices represent the set of entities, and hyperedges demonstrate the high-arity relations. The basic unit is a fact based on a high-arity relation. Unlike knowledge graph that only uses binary relations, the relations in knowledge hypergraph are defined on any number of entities.

Although there have been several pieces of work targeting at knowledge hypergraph embedding and completion, such as Multi-fold TransH (m-TransH) [21], Hyper-relational Knowledge Graph Embedding (HINGE) [22], N-ary Link Prediction (NaLP) [23], they are mostly based on the assumption that there exists an initial knowledge hypergraph or some hyper-relational links. Few efforts have been made on the initial knowledge hypergraph generation. Actually, manually mining the hyper-relations among entities requires much time and effort. Therefore, it is of great significance to study the knowledge hypergraph generation methods for efficient and comprehensive knowledge inference.

4.6 Summary

In this section, we introduce the hypergraph modeling methods, which are categorized as the implicit type and the explicit type. The implicit hyperedges can be used in tasks in which we can represent each subject and develop metrics to evaluate sample similarity. By using the sparse representation, representation-based approaches might mitigate the impact of the noise vertices in comparison with distance-based ones. Explicit hyperedges are more appropriate when input data may already have certain structural details. In general, choosing a suitable hyperedge generation method is important for a specific task. Finally, adaptive and generative hypergraph modeling are worth further exploring to adjust hypergraph structures based on the data and the on-going tasks.