Mixed Membership Sparse Gaussian Conditional Random Fields

Abstract

Building statistical models to explain the association between responses (output) and predictors (input) is critical in many real applications. In reality, responses may not be independent. A promising direction is to predict related responses together (e.g. Multi-task LASSO). However, not all responses have the same degree of relatedness. Sparse Gaussian conditional random field (SGCRF) was developed to learn the degree of relatedness automatically from the samples without any prior knowledge. In real cases, features (both predictors and responses) are not arbitrary, but are dominated by a (smaller) set of related latent factors, e.g. clusters. SGCRF does not capture these latent relations in the model. Being able to model these relations could result in more accurate association between responses and predictors. In this paper, we propose a novel (mixed membership) hierarchical Bayesian model, namely M$^2$GCRF, to capture this phenomenon (in terms of clusters). We develop a variational Expectation-Maximization algorithm to infer the latent relations and association matrices. We show that M$^2$GCRF clearly outperforms existing methods for both synthetic and real datasets, and the association matrices identified by M$^2$GCRF are more accurate.