Conditional Models for Combining Diverse Knowledge Sources in Information Retrieval

Опубликовано 6 сентября 2016, 6:21

Combining the outputs from multiple retrieval sources/engines is of great importance to a number of retrieval tasks such as multimedia retrieval, web retrieval and meta-search. For example, meta-search attempts to refine retrieval outputs by combining the ranked lists generated from different search engines. Despite the huge amount of combination strategies available, most of them are either completely independent on query topics or dependent on some manually defined query classes. To improve upon this, I first introduce a conditional probabilistic retrieval model as a principled framework for retrieval source combination. Based on this framework, I propose a novel combination approach called probabilistic latent query analysis (pLQA), which can discover latent query classes without prior human knowledge and merge retrieval sources adaptively according to query topics. To further adapt the combination function for individual queries, I also develop the probabilistic local context analysis(pLCA), which can automatically leverage unlearned retrieval sources via an undirected graphical model formalism. Experimental results on two large-scale retrieval tasks, i.e., multimedia retrieval and meta-search, demonstrate that the proposed methods can achieve considerable performance gains. Our future work includes extending the proposed methods to other applications such as question answering, cross-lingual IR, multi-engine machine translation, collaborative filtering and so forth.