User group analytics: hypothesis generation and exploratory analysis of user data

Abstract

User data is becoming increasingly available in multiple domains ranging from the social Web to retail store receipts. User data is described by user demographics (e.g., age, gender, occupation) and user actions (e.g., rating a movie, publishing a paper, following a medical treatment). The analysis of user data is appealing to scientists who work on population studies, online marketing, recommendations, and large-scale data analytics. User data analytics usually relies on identifying group-level behavior such as “Asian women who publish regularly in databases.” Group analytics addresses peculiarities of user data such as noise and sparsity to enable insights. In this paper, we introduce a framework for user group analytics by developing several components which cover the life cycle of user groups. We provide two different analytical environments to support “hypothesis generation” and “exploratory analysis” on user groups. Experiments on datasets with different characteristics show the usability and efficiency of our group analytics framework.

Appendix: NP-hardness proofs

Theorem 2

The decision version of gDiscover problem is NP-Complete.

Proof

It is shown in [51] that a single-objective optimization problem for user group set discovery is NP-Complete by a reduction from the Exact 3-Set Cover problem (EC3). There, homogeneity is maximized and a threshold on coverage is satisfied. In our case, two new conflicting dimensions (diversity and coverage) are added. This means that the problem in [51] is a special case of ours, hence our problem is obviously harder. $\square$

For our proofs of hardness, we consider an infinite time limit in gNavigate since that does not affect the complexity of our problem.

Theorem 3

The exploration operation of gNavigate, i.e., opExplore, is NP-complete.

Proof

The decision version of the problem is as follows: For a given group g, a set of groups ${{\mathcal {G}}}$ and a positive integer k, an overlap threshold $\mu$, is there a subset of groups ${{\mathcal {G}}'} \subseteq explore (g,{{\mathcal {G}}},\mu )$ such that (i) $g' \in {{\mathcal {G}}'} \wedge g' \ne g \wedge overlap(g,g') \ge \mu$ and (ii) $\varSigma _{(g_1,g_2) \in {{\mathcal {G}}'} | g_1 \ne g_2}(1- overlap(g_1,g_2) )$ is maximized. A verifier v which returns true if both conditions (i) and (ii) are satisfied runs in polynomial time in the length of its input.

To verify NP-completeness, we reduce the Maximum Edge Subgraph (MES) [69] (also known as Dense k-subgraph) to the decision version of our problem. The problem of MES is defined as follows. Given an instance I consisting of a graph $G=(V,E)$, a weight function $w: E \rightarrow {\mathbb {N}}$, and a positive integer k, find a subset $V' \subseteq V$, $|V'|=k$ such that the total weight of the edges induced by $V'$, i.e., $\varSigma _{(v_i,v_j)} w(v_i,v_j)$ (where $(v_i,v_j) \in V' \times V'$) is maximized. This is an NP-complete problem [69] (originally reduced from the Clique problem).

Given I, we create an instance J of our problem as follows. J consists of a graph $G=(V,E)$ where the set of vertices $V= explore (g,{{\mathcal {G}}},\mu )$ are groups that satisfy (i). Every pair of groups $(g_1,g_2) \in V \times V$ is also connected with a labeled edge, i.e., $w(g_1,g_2)=1 - overlap(g_1,g_2)$. The subset $V' \subseteq V$ ($|V'|=k$) is then a subset of groups where the sum of the weights between each pair of groups in $V'$ is maximized, i.e.,  $|E(V')|=\frac{k \times (k-1)}{2}$. The set $V'$ is the most diverse subset of ${{\mathcal {G}}}$ that satisfies the overlap condition ($\forall g' \in {{\mathcal {G}}}, overlap(g,g') \ge \mu$). Therefore, a set $V'$ is a solution in instance I of MES $iff$ it is a solution in instance J of our problem. Hence, the exploration problem is NP-complete. $\square$

Theorem 4

The exploitation operation of gNavigate, i.e., opExploit, is NP-complete.

Proof

Similar to opExplore, a verifier v for exploitation runs in polynomial time in the length of its input. To verify NP-completeness, we reduce the Maximum Coverage Problem [70] to the decision version of our problem. The problem of Maximum Coverage Problem (MCP) is defined as follows. Given an instance I consisting of m sets S = { $S_1 \dots S_m$} where $S_i \in S_M$ ($S_M$ being a reference set), and a positive integer k, find a subset $S' \subseteq S$, such that $|S'|=k$ and the number of covered elements in $S_M$, i.e., $|\cup _{S_i \in S'} S_i| / |S_M|$ is maximized. This is an NP-complete problem [70]. Given I, we can create an instance J of our problem which consists of m sets $S = exploit (g,{{\mathcal {G}}},\mu )$ and a reference group, i.e., $S_M = g_{in}$. In opExploit, we are interested to have k groups $S' \subseteq S$ that cover maximum number of users in $S_M$, i.e., $|\cup _{S_i \in S'} S_i| / |S_M|$ is maximized. Therefore, a set $S'$ is a solution in instance I of MCP $iff$ it is a solution in instance J of opExploit. Hence opExploit is NP-complete. $\square$