Is Query Performance Prediction With Multiple Query Variations Harder Than Topic Performance Prediction?

Accurately estimating the retrieval effectiveness of different queries representing distinct information needs is a problem in Information Retrieval (IR) that has been studied for over 20 years. Recent work showed that the problem can be significantly harder when multiple queries representing the same information need are used in predic- tion. By generalizing the existing evaluation framework of Query Performance Prediction (QPP) we explore the causes of these differ- ences in prediction quality in the two scenarios. Our empirical analy- sis demonstrates that for most predictors, this difference is solely an artifact of the underlying differences in the query effectiveness dis- tributions. Our detailed analysis also demonstrates key performance distribution properties under which QPP is most and least reliable.

Conference paper