Debiased offline evaluation of Active Learning in Recommender Systems

No Thumbnail Available
Carraro, Diego
Bridge, Derek G.
Journal Title
Journal ISSN
Volume Title
Association for the Advancement of Artificial Intelligence
Published Version
Research Projects
Organizational Units
Journal Issue
Active Learning (AL) when applied to Recommender Systems (RSs) aims at proactively acquiring additional ratings data from the RS users in order to improve subsequent recommendation quality. AL strategies are typically evaluated offline first, but the classic AL offline evaluation methodology does not take into account the bias problem in RS offline evaluation. This problem affects the evaluation of an RS, as brought to light by recent literature. But, we argue, it also affects the evaluation of AL strategies as well. For this reason, in paper, we propose a new AL offline evaluation methodology for RSs which mitigates the bias and thus facilitates a truer picture of the performances of the AL strategies under evaluation. We illustrate our proposed methodology on two datasets and with three simple and well-known AL strategies from the literature. Our experimental results differ from those reported previously in the literature, which shows the importance of our approach to AL evaluation.
Active Learning , Recommender Systems , Offline evaluation
Carraro, D. and Bridge, D. (2020) 'Debiased offline evaluation of Active Learning in Recommender Systems', Proceedings of the Thirty-Third International Florida Artificial Intelligence Research Society Conference (FLAIRS-33), pp.489-494. Available at: (Accessed: 3 September 2020)
© 2020, Association for the Advancement of Artificial Intelligence ( All rights reserved. The paper is posted here by permission of AAAI for your personal use. Not for redistribution.