Debiased offline evaluation of Active Learning in Recommender Systems

Carraro, Diego; Bridge, Derek G.

Debiased offline evaluation of Active Learning in Recommender Systems

Files

Debiased_Offline_Evaluation_of_Active_Learning_in_Recommender_Systems.pdf(167.36 KB)

Accepted Version

Date

2020-05-08

Authors

Carraro, Diego

Bridge, Derek G.

Publisher

Association for the Advancement of Artificial Intelligence

Abstract

Active Learning (AL) when applied to Recommender Systems (RSs) aims at proactively acquiring additional ratings data from the RS users in order to improve subsequent recommendation quality. AL strategies are typically evaluated offline first, but the classic AL offline evaluation methodology does not take into account the bias problem in RS offline evaluation. This problem affects the evaluation of an RS, as brought to light by recent literature. But, we argue, it also affects the evaluation of AL strategies as well. For this reason, in paper, we propose a new AL offline evaluation methodology for RSs which mitigates the bias and thus facilitates a truer picture of the performances of the AL strategies under evaluation. We illustrate our proposed methodology on two datasets and with three simple and well-known AL strategies from the literature. Our experimental results differ from those reported previously in the literature, which shows the importance of our approach to AL evaluation.

Keywords

Active Learning , Recommender Systems , Offline evaluation

Citation

Carraro, D. and Bridge, D. (2020) 'Debiased offline evaluation of Active Learning in Recommender Systems', Proceedings of the Thirty-Third International Florida Artificial Intelligence Research Society Conference (FLAIRS-33), pp.489-494. Available at: https://www.aaai.org/ocs/index.php/FLAIRS/FLAIRS20/paper/view/18486 (Accessed: 3 September 2020)