Evaluating entity-description conflict on duplicated data

Abstract

Duplicated records, which describe the same entity in the real world, frequently generated by data integration. Ideally, the values on the same attributes of duplicated records should be identical. However, the duplicated records may have conflicting values on the same attributes due to ambiguity and data errors. Obviously, the more the conflicts there are among duplicated records in a data set, the poorer the quality of the data set is. To address the problem, we explore a new data quality measure, entity-description conflict, to evaluate the conflict on duplicated records. Since current entity resolution algorithms can hardly identify duplicated records correctly and completely, it brings challenges to compute the entity-description conflict. To this end, it is studied to compute the range of the entity-description conflict while the entity resolution result is not completely correct in this paper. (1) The mathematics model of the entity-description conflict is introduced. (2) Four primary operators for computing the range of the entity-description conflict are identified and are proved to be NP-hard, and thus it is proved that the problem of computing the range of the entity-description conflict is NP-hard. (3) Four approximation algorithms for the four primary operators are provided and a framework based on the four primary operators is proposed for computing the range of the entity-description conflict. (4) Using real-life data and synthetic data, the effectiveness and efficiency of the proposed algorithms are experimentally verified.

Appendix

Theorem 5 MaxDec-Comp is a polynomial-time 2-approximation algorithm.

Proof

From Proposition 1, MaxDec-Comp runs in polynomial time.

Suppose the solution obtained by MaxDec-Comp is $c_1$, the optimal solution is $c^*_1$, and the costs of $c_1$ and $c^*_1$ are denoted by $cost({c_1})$ and $cost({c^*_1})$ respectively such that $cost({c_1})=edc(c)-edc(c\backslash {c_1})$ and $cost({c^*_1})=edc(c)-edc(c\backslash {c^*_1})$. Now we prove $2cost({c_1})\ge {cost({c^*_1})}$.

Because the records in $c_1$ have the top $b$ weights, we obtain

$$\begin{aligned} \sum _{r_i\in {c_1}}{w_i}\ge {\sum _{r_i\in {c^*_1}}{w_i}} \end{aligned}$$

(11)

By the definition of $edc$ and the definition of weight $w_i$ of each record $r_i$, the following formula can be derived.

$$\begin{aligned} \nonumber \sum _{r_i\in {c_1}}{w_i}&= \sum _{r_i\in {c_1}}{\sum _{r_j\in {c}}dis(r_i,r_j)}\\ \nonumber&= {\sum _{r_i\in {c_1}}{\sum _{r_j\in {c_1}}dis(r_i,r_j)} +\sum _{r_i\in {c_1}}{\sum _{r_j\in {c\backslash {c_1}}}dis(r_i,r_j)}}\\ \nonumber&\le \sum _{r_i\in {c_1}}{\sum _{r_j\in {c_1}}dis(r_i,r_j)} +2\sum _{r_i\in {c_1}}{\sum _{r_j\in {c\backslash {c_1}}}dis(r_i,r_j)}\\ \nonumber&= \sum _{r_i\in {c}}{\sum _{r_j\in {c}}dis(r_i,r_j)} -\sum _{r_i\in {c\backslash {c_1}}}{\sum _{r_j\in {c\backslash {c_1}}}dis(r_i,r_j)}\\ \nonumber&= 2\left( \sum _{r_i,r_{j}\in {c},i\le {j}}{dis(r_i,r_{j})} -\sum _{r_i,r_{j}\in {c\backslash {c_1}},i\le {j}}{dis(r_i,r_{j})}\right) \\&= 2edc(c)-2edc(c\backslash {c_1})=2cost({c_1}) \end{aligned}$$

(12)

Similarly, we have

$$\begin{aligned} \nonumber \sum _{r_i\in {c^*_1}}{w_i}&= \sum _{r_i\in {c^*_1}}{\sum _{r_j\in {c}}dis(r_i,r_j)}\\ \nonumber&= \sum _{r_i\in {c^*_1}}{\sum _{r_j\in {c^*_1}}dis(r_i,r_j)} +\sum _{r_i\in {c^*}}{\sum _{r_j\in {c\backslash {c^*_1}}}dis(r_i,r_j)}\\ \nonumber&= 2\sum _{r_i,r_{j}\in {c^*_1},i\le {j}}{dis(r_i,r_j)} +\sum _{r_i\in {c^*_1}}{\sum _{r_j\in {c\backslash {c^*_1}}}dis(r_i,r_j)}\\ \nonumber&\ge {\sum _{r_i,r_{j}\in {c^*_1},i\le {j}}{dis(r_i,r_j)} +\sum _{r_i\in {c^*_1}}{\sum _{r_j\in {c\backslash {c^*_1}}}dis(r_i,r_j)}}\\&= {\sum _{r_i,r_{j}\in {c},i\le {j}}{dis(r_i,r_j)} -\sum _{r_i\in {c\backslash {c^*_1}}}{\sum _{r_j\in {c\backslash {c^*_1}}}dis(r_i,r_j)}} =cost({c^*_1}) \end{aligned}$$

(13)

From inequalities (11), (12) and (13), it follows that

$$\begin{aligned} 2cost({c_1})\ge {\sum _{r_i\in {c_1}}{w_i}}\ge {\sum _{r_i\in {c^*_1}}{w_i}} \ge {cost({c^*_1})} \end{aligned}$$

This completes the proof of Theorem 5. $\square$

Before proving Theorem 7, the following lemma is proved.

Lemma 1

Given an instance of $\mathsf {MaxInc^*}$, suppose the optimal solution contains $\{x_i|1\le {i}\le {n}\}$ and $\{y_{ij}|1\le {i}<j\le {n}\}$. $\{x_i|1\le {i}\le {n}\}$ and $\{y_{ij}|1\le {i}<j\le {n}\}$ satisfy that $\forall {i,j}, i\ne {j}, y_{ij}=min\{x_i,x_j\}$.

Proof

From inequalities (7) and (8), it follows that

$$\begin{aligned} y_{ij}\le {min\{x_i,x_j\}}. \end{aligned}$$

(14)

From inequality (14), we have

$$\begin{aligned} y_{ij}\ge {x_i+x_j-1}. \end{aligned}$$

(15)

As $0\le {x_j}\le {1}$ for $1\le j\le n$, it implies that

$$\begin{aligned} {x_i+x_j-1}\le {min\{x_i,x_j\}} \end{aligned}$$

(16)

Combining inequalities (14)-(16) gives

$$\begin{aligned} {x_i+x_j-1}\le {y_{ij}}\le {min\{x_i,x_j\}}. \end{aligned}$$

As $\{y_{ij}|1\le {i}<j\le {n}\}$ is the optimal solution for $\mathsf {MaxInc}$, $\sum _{1\le {i}<j\le {n}}{c_{ij} y_{ij}}$ is maximized. Thus $y_{ij}=min\{x_i,x_j\}$. $\square$

Theorem 7 MaxInc-Comp is a polynomial-time $\rho$ -approximation algorithm, where $\rho =\frac{(n-1)b}{b-1}$.

Proof

We have already shown that MaxInc-Comp runs in polynomial time.

Now we prove that MaxInc-Comp is a $\rho$-approximation algorithm. Suppose the optimal solution to $\mathsf {MaxInc^*}$ contains $\{x_i|1\le {i}\le {n}\}$ and $\{y_{ij}|1\le {i}<j\le {n}\}$. Then the cost of the optimal solution is $cost^*=\sum _{1\le {i}<j\le {n}}{c_{ij} y_{ij}}$ which is the upper bound of the cost of the optimal solution to $\mathsf {MaxInc}$ denoted by $cost$. Suppose the solution obtained by MaxInc-Comp is $(U,F)$. The cost of this solution is $\sum _{(i,j)\in {F}}c_{ij}$.

Applying Lemma , for all $1\le {i}<j\le {n}$, $y_{ij}\le {\frac{x_i+x_j}{2}}$, we have,

$$\begin{aligned} {\sum _{1\le {i}<j\le {n}}{c_{ij} y_{ij}}}\le {\sum _{1\le {i}<j\le {n}}{c_{ij} {\frac{x_i+x_j}{2}}}} =\sum _{i}{x_i \frac{\sum _{j\ne {i}}{c_{ij}}}{2}} \end{aligned}$$

(17)

Let $d_i={\frac{1}{2}} \sum _{j\ne {i}}{c_{ij}}$. Inequality (17) can be written as

$$\begin{aligned} \sum _{1\le {i}<j\le {n}}{c_{ij} y_{ij}}\le {\sum _{i}{x_id_i}} \end{aligned}$$

(18)

Suppose $d_i$ is sorted in a descending order, and the order is $d_1,d_2,\ldots ,d_{n-1}$. As $\sum _{i}{x_i}=b$, the following inequality can be easily proved by contradiction.

$$\begin{aligned} \sum _{i}{x_i d_i}\le {\sum _{1\le {i}\le {b}}d_i} \end{aligned}$$

(19)

The combination of inequalities (18) and (19) gives

$$\begin{aligned} \sum _{1\le {i}<j\le {n}}{c_{ij} y_{ij}}\le {\sum _{1\le {i}\le {b}}d_i} \end{aligned}$$

(20)

Because the algorithm sorts edges in $E$ in a descending order denoted by $e_{i_1j_1},e_{i_2j_2},\ldots ,e_{i_kj_k},\ldots$ and picks the edges which have larger weights and places their endpoints into $U$ until $|U|=b$. Let $E^{\prime }=\{e_{i_1j_1},e_{i_2j_2},\ldots ,e_{i_mj_m}\}$ be the edges which are picked during the loop of the algorithm. Since $E^{\prime }\subseteq {F}$ and $m\ge {\lfloor \frac{b}{2}\rfloor }\ge {\frac{b-1}{2}}$, we have

$$\begin{aligned} \sum _{(i,j)\in {E^{\prime }}}{c_{ij}}&\ge {\sum _{(i,j)\in {E^{\prime \prime }}}{c_{ij}}} \end{aligned}$$

(21)

$$\begin{aligned} \frac{\sum _{(i,j)\in {E^{\prime \prime }}}c_{ij}}{\lfloor \frac{b}{2}\rfloor }&\ge {\frac{\sum _{(i,j)\in {E^{\prime \prime }}}c_{ij}}{m}}\ge {\frac{\sum _{1\le {i} \le {b}}d_i}{\frac{b(n-1)}{2}}} \end{aligned}$$

(22)

From inequality (22), we have

$$\begin{aligned} \sum _{(i,j)\in {E^{\prime \prime }}}{c_{ij}}\ge {\frac{\sum _{1\le {i}\le {b}}d_i}{\frac{b (n-1)}{b-1}}} \end{aligned}$$

(23)

Combining inequalities (20) and (23) gives

$$\begin{aligned} \sum _{(i,j)\in {E^{\prime \prime }}}{c_{ij}}\ge {\frac{\sum _{1\le {i}<j\le {n}}{c_{ij} y_{ij}}}{\frac{b (n-1)}{b-1}}} \end{aligned}$$

(24)

From inequalities (21) and (24), we have

$$\begin{aligned} \sum _{(i,j)\in {E^{\prime }}}{c_{ij}}\ge {\frac{\sum _{1\le {i}<j\le {n}}{c_{ij} y_{ij}}}{\frac{b (n-1)}{b-1}}} \end{aligned}$$

(25)

which implies that

$$\begin{aligned} \sum _{1\le {i}<j\le {n}}{c_{ij}y_{ij}} \le {\rho \sum _{(i,j)\in {F}}c_{ij}} \end{aligned}$$

(26)

where $\rho =\frac{(n-1)b}{b-1}$. Hence MaxInc-Comp is a $\rho$-approximation algorithm. $\square$

Theorem 8 MinInc-Comp is a polynomial-time $\rho$ -approximation algorithm, where $\rho =\frac{(n-1)b}{b-1}$.

Proof

Obviously MinInc-Comp runs in polynomial time.

Now we prove that MinInc-Comp is a $\rho$-approximation algorithm. We denote the given $\mathsf {MinInc}$ instance as $I$. Suppose $I$ is encoded into a $\mathsf {MaxInc}$ instance denoted by $I^{\prime }$ and the relaxation IP instance of $I^{\prime }$ is denoted by $I^{\prime \prime }$ which is a $\mathsf {MaxInc^*}$ problem. Suppose the cost of the optimal solution to $I^{\prime }$ is $cost^{\prime }$ and the cost of the optimal solution to $I^{\prime \prime }$ is $cost^{\prime \prime }$. Applying Theorem 7, we have $\frac{cost^{\prime \prime }}{cost}\le {\rho }$. Since $cost^{\prime }=\sum _{(i,j)\in {F}}{(w-c_{ij})}$, the cost of the solution generated by MinInc-Comp denoted by $cost$ satisfies that $cost=\sum _{(i,j)\in {F}}{w}-cost^{\prime }$. We denote that $M=\sum _{(i,j)\in {F}}{w}$. Then we have,

$$\begin{aligned} cost=M-cost^{\prime }. \end{aligned}$$

(27)

Suppose the optimal solution to $I^{\prime \prime }$ involves $\{x_i|1\le {i}\le {n}\}$ and $\{y_{ij}|1\le {i}<j\le {n}\}$. Since $cost^{\prime \prime }=\sum _{1\le {i}<j\le {n}}{(w-c_{ij}) y_{ij}}$ is maximized and the cost $cost^*$ of the optimal solution to $I$ satisfies that $cost^*=\sum _{1\le {i}<j\le {n}}{c_{ij} y_{ij}}$ is minimized. Then we have

$$\begin{aligned} cost^*=\sum _{(i,j)\in {F}}{w}-\sum _{1\le {i}<j\le {n}}{(w-c_{ij}) y_{ij}}=M-cost^{\prime \prime }. \end{aligned}$$

(28)

Combining the equalities (27) and (28) gives

$$\begin{aligned} \frac{cost}{cost^*}=\frac{M-cost^{\prime }}{M-cost^{\prime \prime }}. \end{aligned}$$

(29)

By Theorem 7, we have

$$\begin{aligned} cost^{\prime \prime }\le {\rho cost^{\prime }} \end{aligned}$$

(30)

Since $w=(\rho +1) \max \{c_{ij}\}$, the following formula can be obtained.

$$\begin{aligned} \nonumber M&= \sum _{(i,j)\in {F}}w\\ \nonumber&= (\rho +1) \sum _{(i,j)\in {F}}{\max \{c_{ij}\}}\\ \nonumber&\ge {(\rho +1) \sum _{(i,j)\in {F}}c_{ij}}\\&\ge {(\rho +1) cost^{\prime }}. \end{aligned}$$

(31)

From the inequalities (30) and (31), it can be achieved that

$$\begin{aligned} \frac{M-cost^{\prime }}{M-cost^{\prime \prime }}\le {\frac{M-cost^{\prime }}{M-\rho cost^{\prime }}}\le {\frac{(\rho +1) cost^{\prime }-cost^{\prime }}{(\rho +1) cost^{\prime }-\rho cost^{\prime }}}=\rho \end{aligned}$$

which implies that

$$\begin{aligned} \frac{M-cost^{\prime }}{M-cost^{\prime \prime }}\le {\rho }. \end{aligned}$$

(32)

Combining the inequalities (29) and (32) gives

$$\begin{aligned} \frac{cost}{cost^*}\le {\rho }. \end{aligned}$$

(33)

Hence MinInc-Comp is a $\rho$-approximation algorithm. $\square$