Learning Approximate MRFs from Large Transactional Data

Abstract

In this abstract we address the problem of learning approximate Markov Random Fields (MRF) from large transactional data. Examples of such data include market basket data, co-authorship networked data, etc. Such data can be represented by a binary data matrix, with an entry (i, j) takes a value of one (zero) if the item j is (not) in the basket i. “Large” means that there can be many rows or columns in the data matrix. To model such data effectively in order to answer queries about the data efficiently, we consider the use of probabilistic models. In this abstract, we consider employing frequent itemsets to learn approximate global MRFs on large transactional data. We conduct an empirical study on real datasets to show the efficiency and effectiveness of our model on solving the query selectivity estimation problem, that is to approximately compute the marginal probability of sets of items (see [1] for the experimental results). Translated into the social network domain, this is the problem of computing the likelihood of seeing a particular combination of grocery items in the market basket domain, or the probability of a group of professors coauthoring a paper in a co-authorship network, etc. This marginal probability computation is also useful for anomalous link detection [2] in social network analysis. A link in a social network corresponds to a pair of items. The links whose associated marginal probabilities are significantly low can be thought of as anomalous.

This work is supported by DOE Award No. DE-FG02-04ER25611 and NSF CAREER Grant IIS-0347662. We refer the reader to a longer version of this paper for experimental results and complete proofs and discussions.