Scalable Inference of Attributes in Entity-Relationship Graphs

Опубликовано 12 августа 2016, 2:11

Most real-world applications involve massive databases comprising of a large number of entities and n-ary relationships, each of which in turn are associated with multiple attributes. Some of these attributes are readily available while others are rarely observed. Scalable and accurate inference of the predominantly missing, but high value attributes of entities and relationships using all the available information has far-reaching applications. Most of the existing scalable solutions adopt an application-specific approach combining simple supervised learning techniques and/or variants of the HITS algorithm with sophisticated feature engineering and careful pipe-lining to ensure accurate inference. In this talk, I'll describe our recent ongoing work on a fairly general and scalable inference framework for this scenario that can operate even with limited supervision. Our framework comprises of two main parts. First, we introduce a common attribute representation for the entities and relationships while separately encoding key inter-entity mappings, application-specific attribute dependencies and typing constraints. This representation allows one to succinctly describe a directed graphical model encoding all the desired dependencies between the various entities and relationship nodes in a declarative fashion. Secondly, to perform inference over this graphical model, we propose scalable variational EM based techniques based on judicious lazy propagation of information across the entity-relationship graph. Experiments on a large real world data set from online discussion forums demonstrate the efficacy and flexibility of our approach. I'll also briefly discuss potential future research directions on utility-based interactive collective inference. (joint work with Amit Kumar Singh and Dinesh Raghu from IBM Research)