\documentclass[cs,phd,justified,12pt]{uccthesis}

%\usepackage{ragged2e} %To justify the text!!
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{mathtools}
\usepackage{graphicx}
\usepackage{amsthm}
% \usepackage{algorithm}
\usepackage{afterpage}

\newtheorem{theorem}{Theorem}[section]
\newtheorem{corollary}{Corollary}[theorem]
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{definition}[theorem]{Definition}
%\usepackage[utf8]{inputenc}
%\usepackage[english]{babel}

\usepackage{varwidth}
\usepackage{booktabs} % For formal tables
\usepackage{epsfig}
\usepackage{amsfonts}
\usepackage{array}
\usepackage[table, x11names, pdftex,dvipsnames]{xcolor}
\usepackage{rotating}
\usepackage{bbm}
\usepackage{tabularx}
\usepackage{color}
\usepackage{multirow}
\usepackage{url}
\usepackage{lscape}
\usepackage{subcaption}
\usepackage{xargs}

% Very useful abbrev
\def\prob{{\mathbb{P}}} % Probs
\def\norm{{\mathcal{N}}} % Normal
\def\real{{\mathbb{R}}} % Real numbers
\DeclareMathOperator*{\usup}{NEG-USER-SUPPORT}
\DeclareMathOperator*{\isup}{NEG-ITEM-SUPPORT}
\DeclareMathOperator*{\uvar}{USER-VARIANCE}
\DeclareMathOperator*{\ivar}{ITEM-VARIANCE}
\DeclareMathOperator*{\resample}{RESAMPLE}
\DeclareMathOperator*{\inject}{INJECT}
\DeclareMathOperator*{\fast}{FAST-RESAMPLE}
\DeclareMathOperator*{\ensemble}{ENSEMBLE}
\DeclareMathOperator*{\FunkSVD-CV}{EB-FunkSVD}
\DeclareMathOperator*{\Bias-CV}{EB-Linear}
\DeclareMathOperator*{\CPMF}{CPMF}
\DeclareMathOperator*{\BeMF}{BeMF}
\DeclareMathOperator*{\OrdRec}{OrdRec}
\DeclareMathOperator*{\divy}{div}

% Victor commands
\newcommand\ddfrac[2]{\frac{\displaystyle #1}{\displaystyle #2}}
\newcommand{\victor}[1]{\textbf{\color{magenta} *Victor: #1}}
\newcommand{\derek}[1]{\textbf{\color{brown} *Derek: #1}}
\newcommand{\db}[1]{\textbf{\color{green} **Derek: #1}}
\newcommand{\dgb}[1]{\textbf{\color{purple} **Derek: #1}}
\usepackage{tikz}
\usetikzlibrary{shapes, arrows}
\tikzstyle{block} = [rectangle, draw, text width=6em, text centered, rounded corners, minimum height=3em]
\tikzstyle{line} = [draw, -latex']
\usepackage[linesnumbered,ruled,vlined]{algorithm2e}
\usepackage{cleveref}
\setlength{\headheight}{46.48878pt}

% Acronyms
\usepackage[acronym]{glossaries}
\makeglossaries
\newacronym{rs}{RS}{Recommender System}
\newacronym{rss}{RSs}{Recommender Systems}
\newacronym{ml}{ML}{Machine Learning}
\newacronym{ers}{ERS}{Explicit feedback-based Recommender System}
\newacronym{erss}{ERSs}{Explicit feedback-based Recommender Systems}
\newacronym{irs}{IRS}{Implicit feedback-based Recommender System}
\newacronym{irss}{IRSs}{Implicit feedback-based Recommender Systems}
\newacronym{nn}{NN}{Neural Network}
\newacronym{nns}{NNs}{Neural Networks}
\newacronym{rmse}{RMSE}{Root Mean Square Error}
\newacronym{mse}{MSE}{Mean Squared Error}
\newacronym{mae}{MAE}{Mean Absolute Error}
\newacronym{ml1m}{ML1M}{MovieLens 1M}
\newacronym{pop}{POP}{Popularity}
\newacronym{bce}{BCE}{Binary Cross-Entropy}
\newacronym{mf}{MF}{Matrix Factorization}
\newacronym{bpr}{BPR}{Bayesian Personalized Ranking}
\newacronym{mlp}{MLP}{Multi-Layer Perceptron}
\newacronym{aur}{AUR}{Aleatoric Uncertainty-aware Recommendation}
\newacronym{gpr}{GPR}{Gaussian Pairwise Ranking}
\newacronym{gbr}{GBR}{Gaussian Binary Ranking}
\newacronym{rabr}{RaBR}{Rating-Based Ranking}
\newacronym{rebr}{ReBR}{Relevance-Based Ranking}
\newacronym{ubf}{UBF}{Uncertainty-Based Filtering}
\newacronym{prr}{PRR}{Probability-of-Relevance Ranking}
\newacronym{bnn}{BNN}{Bayesian Neural Network}
\newacronym{mcd}{MCDropout}{Monte Carlo Dropout}
\newacronym{al}{AL}{Active Learning}
\newacronym{cb}{CB}{Content-Based}
\newacronym{ca}{CA}{Context-Aware}
\newacronym{cf}{CF}{Collaborative Filtering}
\newacronym{ndcg}{NDCG}{Normalized Discounted Cumulative Gain}
\newacronym{mrr}{MRR}{Mean Reciprocal Rank}
\newacronym{map}{MAP}{Mean Average Precision}
\newacronym{mar}{MAR}{Mislabeling-Aware Ranking}
\newacronym{uri}{URI}{Uncertainty Recommendation Improvement}
\newacronym{uac}{UAC}{Uncertainty Accuracy Correlation}
\newacronym{hbr}{HBR}{Hierarchical Binary Ranking}
\newacronym{mcmc}{MCMC}{Monte-Carlo Markov Chains}
\newacronym{vi}{VI}{Variational Inference}
\newacronym{svi}{SVI}{Stochastic Variational Inference}
\newacronym{advi}{ADVI}{Automatic Differentiation Variational Inference}
\newacronym{bbb}{BBB}{Bayes By Back-propagation}
\newacronym{eb}{EB}{Error-Based}
\newacronym{cpmf}{CPMF}{Confidence-aware Probabilistic Matrix Factorization}
\newacronym{icpmf}{ICPMF}{Implicit Confidence-aware Probabilistic Matrix Factorization}
\newacronym{pmf}{PMF}{Probabilistic Matrix Factorization}
\newacronym{mnar}{MNAR}{Missing-not-at-random}

\begin{document}
\title[Uncertainty in\break Recommender Systems]{Uncertainty in Recommender Systems}
\author{Victor Azevedo Coscrato}
\qualifications{msc \\ 119222668}
\professor{Professor Utz Roedig}
\supervisor{Dr.\ Derek G.\ Bridge}%\\Dr. Steven Prestwich}
% \sponsor{SFI}
\date{April 2024}
\maketitle

\begin{dedication}
Para Rafaela, Silmara e Marcos
\end{dedication}

\begin{acknowledgements}
This PhD was a long journey, filled with achievements. But it also was by far the most difficult phase of my life. On several occasions, I needed help in many different ways. To those who provided much-needed support so that I could complete this dissertation, I would like to express my sincere thanks.

First, to my supervisor, Derek Bridge, for all the support provided, which greatly exceeded his academic responsibilities. Derek's dedication to research is very impressive, and can be confirmed by his academic record. Of much more importance is how caring a supervisor Derek can be. In many cases, supervisory meetings must be less about research and a lot more about the researcher, and that I learned from him. I was truly lucky to have Derek as my supervisor.

Rafaela, my fiancee, deserves much special thanks. Rafaela and I have been together long before I started the PhD. As a result of my choice to move to Ireland, we had to live apart for a long time. That was until she moved to join me, beginning an adventure that I am so happy to remember. Among all those who kept me strong during the times of adversity, Rafaela deserves the most recognition. Sincerely, thank you.

Another special thanks to all my colleagues from UCC. The PhD journey is lonely at times, but I was always able to push the loneliness away thanks to the friends I made along the way. To all of my friendships that started at the research office, at the handball court, or anywhere in Cork, cheers.

A minha família, agradeço a todo o suporte que me amparou até onde estou, em especial minha mãe Silmara e meu pai Marcos. Essa conquista também é de vocês. Obrigado.

Finally, I am grateful to the CRT-AI for this opportunity, in particular to the program coordinator, Janet Choi, who strives to meet the needs of students. This thesis has emanated from research conducted with the financial support of Science Foundation Ireland under Grant No.\ 18/CRT/6223, which is co-funded under the European Regional Development Fund. I am grateful for all the individuals who had some form of assistance with any part of this work.

\vspace{0.3cm}

\begin{flushright}
Victor Coscrato

\vspace{-0.25cm}
Cork, April 2024
\end{flushright}

\end{acknowledgements}

\sloppy
\overfullrule=0pt


\begin{abstract}

% Recommender Systems have emerged as a powerful tool in the information era. Due to the absurd availability of products and services currently offered on digital platforms, it is often necessary to use a system capable of filtering such items and offering those most relevant to each user. For this reason, there currently is a lot of value in developing increasingly accurate Recommender Systems.

% The pure pursuit of increasing recommendation accuracy through the development of new recommendation models receives a lot of attention. However, this is just one of the ways to improve a system. It is also possible to equip the system with new tools capable of extending its functionalities in different ways. The tool that we focus on in this dissertation are uncertainty estimators.

% \dgb{I would combine and abbreviate the previous two paras, and I would try to get a mention of their prediction models and their training data (since these are the foundation of prediction and label uncertainty). Then you can be clearer that researchers have been pursuing prediction model accuracy.} \victor{I couldn't abbreviate too much, I like all the sentences, but it follows a combined version that mentions feedback data and prediction modeling.}

Recommender Systems have emerged as a powerful tool in the information era. Due to the overwhelming number of items (products and services) currently offered on digital platforms, it is often necessary to use a system capable of ranking the items and offering those that are most relevant to each user. These systems typically use historical user-item interaction data to build models that can predict the relevance of each item to the user. 

There has long been a focus on increasing recommendation accuracy through the development of new prediction models. However, this is just one of the ways to improve these systems. It is also possible to equip them with new tools that extend their functionality in different ways. The tools that we focus on in this dissertation are uncertainty estimators.
% The problem of uncertainty is relevant to Recommender Systems in two ways: prediction uncertainty and label uncertainty. Prediction uncertainty is the expected imprecision of item scoring algorithms. Label uncertainty is the chance that an interaction gets mislabeled. This thesis conducts by far the most extensive study of such uncertainties, offering a varied set of methods for their estimation, that range from heuristic data metrics to novel prediction and label uncertainty prediction models. \dgb{I think if the first two para are combined and improved then the defns of prediction and label uncertainty can mention prediction models and the interaction data on which those models are trained, to give defns that are mot much more detailed, but are a little more precise.}

The problem of uncertainty is relevant to Recommender Systems in at least two ways: prediction uncertainty and label uncertainty. Prediction uncertainty is the expected imprecision of the predictions given by the system's model. Label uncertainty is the chance that interactions used to learn the prediction model are mislabeled. This dissertation reports by far the most extensive study of these two types of uncertainty, offering a varied set of methods for their estimation, ranging from heuristic data metrics to novel uncertainty prediction models.

In overview, this dissertation is the largest compilation of methods for estimating prediction uncertainty and label uncertainty in Recommender Systems to date. This collection includes already-existing methods -- that we survey, rewrite in a common notation, implement, make available under an open license, and compare in-depth -- and many original methods, some that derive directly from existing work, but others that involve complex modeling. We divide our work into three branches: prediction uncertainty in explicit feedback-based systems, prediction uncertainty in implicit feedback-based systems, and label uncertainty in implicit feedback systems.

While this dissertation proposes new uncertainty estimation methods, the novel work in this dissertation is not restricted to new estimation methods. We also propose new techniques for evaluating prediction uncertainty estimators. Furthermore, we present and validate novel ways of using uncertainty estimators to improve the operation of a Recommender System.

At the core of our research program, and for each of the three branches cited above, we have rigorous validation of our prediction and label uncertainty estimation methods through large-scale, reproducible empirical studies on publicly available recommendation datasets that unveil important insights into the performance and usefulness of the proposed methods. These studies include both the novel and surveyed uncertainty estimation methods, and make use of the novel uncertainty evaluation techniques that we propose.

This work can be an important mechanism for promoting new research on this topic that is still largely unexplored in the world of Recommender Systems. Thus, this dissertation is a contribution to the field of Recommender Systems, not just in terms of an all-encompassing compendium of uncertainty estimation methods for practitioners, but also in guiding future work. Given that the landscape of Recommender Systems continues to evolve, our work is poised to shape the discourse about uncertainty in the field.

%\dgb{The positivity is good. But, for me, it still lacks logic. It switches from one subject to another and back again and back again. It has phrases like ``these methodologies'' that don't seem to refer to anything. It says ``Beyond theoretical contributions'' but has already made the point that it is partly empirical -- indeed, it says that the empirical work is at the core. Suggestion: Maybe, the overview para should conclude with the stuff about the three branches. Then the next para should be about new estimators. Then something about new eval methods. Then something about empirical. The last para can stay but it needs to begin differently.}

%\victor{I see. In fact, there were a couple of phases out of place. I tried to follow your 'script' more closely, and made some small changes for cohesiveness.}

% \victor{I tried to be quite enthusiastic about the research here, do you think this will raise the expectations for positive results a bit too much?}

% \victor{Another comment: I tried not to cite specific stuff here, e.g. names of methods, for two reasons: first, there are too many, and second, I don't know if there is any value on citing something that we wont even try to explain here. Although, I am curious to see your opinion on this.}

% \dgb{(a) Yes, enthusiasm is good. The key is to emphasise contributions. (b) Provided you are stating the contribs that you want to state, then omitting names/details is fine. (c) However, I'm finding these last three paras to be lacking in structure/order.}

% \dgb{If you want to leave it, I will work some more on improving the English. If you agree with some of my comments, then try a second draft, and I will look at it.}

\end{abstract}

\clearpage

\input{chapters/1 - introduction}
\input{chapters/2 - recommenders}
\input{chapters/3 - explicit}
\input{chapters/4 - implicit}
\input{chapters/5 - implicit_learn}
\input{chapters/6 - conclusion}

\clearpage
\glsaddall
\printglossary[type=\acronymtype, nonumberlist]

\bibliography{bibliography}
% \nocite{*}

\appendix
\clearpage
\renewcommand{\thepage}{\Alph{chapter}\arabic{page}}
\setcounter{page}{1}
\include{chapters/appendix}

\end{document}


