Search: Use of Relevance Feedback and Estimating Effectiveness of Searches

Опубликовано 7 сентября 2016, 16:23

This talk consists of two parts focussed on two distinct but related themes. In the first part we look at the problem of searching on devices with small displays and explore relevance feedback approaches to improve effectiveness of search. We consider a minimal user interaction: for a screenful of search results the user is encouraged to indicate a single most relevant document. To study the problem we exploit the fact that for small display sizes and limited user actions we can construct a complete user decision tree that represents all possible outcomes of the user interaction with the system. By examining the tree we compute an upper limit on relevance feedback performance. We evaluate the performance of two strategies for displaying results: the document ranking that maximizes information gain from the user's choices and the standard ranking that presents top-K scored documents. Experimental results show the dependency between the utility of relevance feedback and the quality of the initial query. The second part of the talk deals with predicting the effectiveness of search for a given query. Two typically approaches are taken: by examining the properties of the search queries and by examining the properties of the retrieved document sets. We take the latter approach and devise four measures to characterize the retrieved document sets and estimate the quality of search: (i) the clustering tendency as measured by the Cox-Lewis statistic, (ii) the sensitivity to document perturbation, (iii) the sensitivity to query perturbation, and (iv) the local intrinsic dimensionality. We examine the usefulness of these measures for the task of ranking 200 queries according to the search effectiveness over the TREC (discs 4 and 5) dataset. Our ranking of queries is compared with the ranking based on the average precision using the Kendall statistic. A combination of our features results in the Kendall of 0.562 which, to our knowledge, is the highest correlation with the average precision reported to date on this dataset.