\chapter{Introduction}\label{chapter:introduction}

%\db{You have some text overflow issues, which LaTeX shows by a black border on the right of the text. I have fixed many of them. But you should find out how to stop LaTeX from displaying the black border.} \victor{I added the command \\sloppy in the preamble,. It relaxes the fussy spacing rules of the template in the cases where text overflows. This should fix the black borders in almost every case.} \dgb{It does not fix it on the tables, e.g. Table 3.4. And it does not fix it in the List of Tables.} \victor{I think they are all fixed now, let me know if you find any other.}

%\db{At the end of the dissertation, you have a list of Acronyms with page numbers. It's nice but I would either exclude this completely, or include it but without the page numbers. The reason is that you have not been really careful to always use acrfull acrlong and acrshort commands for every acronym. There are lots of times when you have just written, e.g. BNN. There also some acronyms where you \emph{never} use the commands, e.g. MCMC.} \victor{Removed \checkmark} \dgb{Is there a way to include it without page numbers?} \victor{There certainly is, but still I'd have to make sure I write acronyms with the commands at least once. Maybe that's unnecessary work, but I will revisit this if I got some time remaining before submitting} \dgb{I think it is a good idea to have the list of acronyms. In desperation, you could tweak the title to say, e.g.: List of main acronyms!!! That way, if you miss MCMC, SVI, BCE, it won't matter.} \victor{So, I brought back the list of acronyms with no numbers, and I added another command that prints acronyms even if they were never typed as \\acrfull, \\acrshort or \\acrlong. This has the advantage that we may simply add acronyms to the thesis.tex file, making it easy to add stuff that we forgot to write with the commands. I added many that were missing, but it's likely I still miss some.}

\section{Background and Motivation}

Advances in technology have made information, products and services  easily accessible through digital platforms. Nowadays, numerous businesses offer a nearly uncountable number of news and media posts, products and services, collectively referred to as \textit{items}. Some examples are: streaming services, such as Spotify and Netflix; marketplaces, such as Amazon; and social media websites, such as Facebook, Instagram and TikTok. Due to the huge volumes involved, it is difficult for platform users to discover items they would like to consume and to choose between candidate items. 

\acrfull{rss} have emerged as tools to assist item discovery in these massive catalogs. They work by suggesting items that might be appealing according to the user's consumption behaviour. A successful \acrshort{rs} should help its users by exposing them to relevant items, especially ones that they may not have discovered for themselves, and it should help the business by increasing sales, revenue, customer retention and so forth. 

The economic interest around \acrshort{rss} has led to a rapid growth in the research field. The majority of the research conducted has focused on improving recommendation accuracy by developing novel, and more sophisticated, recommendation algorithms. Currently, there is a plethora of recommendation algorithms being used in real-world problems, achieving very high levels of accuracy. 

The main goal of a \acrshort{rs} is to recommended items that are relevant to its users. Typically, this requires that its recommendations are \emph{personalized}. That is, the system must provide recommendations that are tailored to what it infers about each user's preferences. Nevertheless, there are other important considerations. It is often also desirable, for example, for a set of recommendations to be \emph{diverse} and for at least some of its recommendations to be \emph{serendipitous}. We leave the explanation of \emph{diversity}, \emph{serendipity} and other \acrshort{rs} side goals to Chapter \ref{chapter:rs}.

There remain many open research topics, and the one we explore in this dissertation is the estimation of uncertainty in \acrshort{rss}.

Uncertainty is common to every machine learning (ML) task \cite{hammer2007process, hullermeier2021aleatoric}. In particular, model predictions carry a degree of uncertainty. This uncertainty is often neglected, but it can be beneficial if uncertainty is quantified and even exploited. The quantification of prediction uncertainty has different motivations in different domains. For example, in medical decision making, estimating the uncertainty of predictions is key to the management of risk \cite{begoli2019need}. In weather forecasting, estimating uncertainty and presenting the estimates to users can increase credibility \cite{wang2019deep}. \acrshort{rss} are no exception to the rule. For reasons that we explain below, recommendations can be uncertain and, as we also explore briefly below, estimating their uncertainty can improve the operation of a \acrshort{rs}. 

The literature in the area of uncertainty in ML often categorizes uncertainty into two types: \textit{aleatoric} and \textit{epistemic} \cite{hullermeier2021aleatoric}. The first refers to the randomness of the data itself, which cannot be reduced by additional knowledge. The second type, which is the focus of this dissertation, instead refers to the uncertainty caused by lack of knowledge. Therefore, epistemic uncertainty reduces our ability to know which of several prediction models is the correct one, which ultimately leads to uncertainty around the chosen model's predictions. For this reason, we can define \emph{prediction uncertainty} as the expected imprecision of a prediction. Some related work in the area uses terminology such as reliability \cite{bobadilla_reliability_2018} or confidence \cite{wang2018confidence, cleger2013being}, which, in the context of our work, we take to be the opposite of uncertainty, that is, the expected precision of a prediction. %\victor{Note that I introduce the terminology prediction uncertainty here now.}

In \acrshort{rss}, prediction uncertainty has many causes. First, to provide personalized recommendations, the system must infer the preferences of individual users. Naturally, each user interacts with the items in a particular way: they click, purchase, consume and/or rate in different ways. For example, some users interact with many items, while others interact with fewer items; some interact with very few items at all. Due to these particularities, the system becomes more knowledgeable about some users and less about others. This is one cause of uncertainty.

Similarly, each item has its own particular interaction pattern, which also affects the level of uncertainty in recommendations. For example, popular (frequently consumed) items tend to be fairly certain recommendations. On the other hand, a more niche item from the `long-tail' is often a more uncertain recommendation.

Furthermore, in practical \acrshort{rss}, the number of items is typically very large, meaning that users will have interacted with only a tiny fraction of them. This is known as \textit{sparsity}. It means the system must infer a user's preferences from a relatively small amount of information, leading sometimes to the generation of unsuccessful but also uncertain recommendations. 

Finally, there are challenges in collecting reliable user-item interaction data\linebreak \cite{samia2014handling}. In the case of \acrfull{erss} -- the class of systems that rely on the explicit input of user ratings (explained more fully in Chapter \ref{chapter:rs}) --- ratings data may be unreliable due to its high degree of subjectivity \cite{zenebe2009representation}, which can be explained by variations in user behaviour, personality and mood; change of preferences over time; and also by different interpretations of the rating scales by the users \cite{block1998exploring, Amatriain-Et-Al-2009}. Moreover, ratings are usually collected from open systems, being subject to natural noise (e.g.\ typing errors) and even malicious noise (e.g.\ from shilling attacks) \cite{o2006detecting}. Unreliable user-item interaction data may even be caused by accidents. This is particularly so in the case of \acrfull{irss} -- the class of systems that observe and record aspects of the user's interaction behaviour with the system, such as searches, clicks, downloads, purchases (again see Chapter \ref{chapter:rs}). A mistaken click, for example, means that the interaction data is not wholly reliable. 

In the case of \acrshort{irss}, there is another -- arguably much more problematic -- form of data unreliability, which is the lack of `negative' feedback. An \acrshort{irs} does not ask its users whether they liked or disliked an item they interacted with, or how much they liked/disliked it. These systems record simply the fact that the user interacted with the item. Therefore, there is no way to tell apart liked and disliked items, only observed and not observed interactions with items. The traditional way to overcome this challenge is to simply assume all items the user interacted with are relevant to them, and in turn items the user did not interact with are not relevant. This introduces another form of uncertainty in this type of system, that we denote \emph{label uncertainty}. %\dgb{You still haven't said what the uncertainty is!} \victor{new sentence:} In a nutshell, 
Label uncertainty is the chance that an interaction gets mislabeled, i.e.\ how likely it is that (i) an observed interaction is actually irrelevant or (ii) an unobserved interaction is actually relevant.

The presence of label uncertainty decreases the reliability of the input data that is used to learn the system. We leave further details for later (Chapter \ref{chapter:implicit_learn}). For now, it is enough to understand that \emph{label uncertainty} is another form of uncertainty and that it is rather different from \emph{prediction uncertainty}. %\dgb{Hmm. (a) A lack of flow because the last cause of prediction uncertainty (unreliable data) overlaps with the main cause of label uncertainty. (b) Yes, you can leave the details for later, but I don't think you are yet explaining the uncertainty here - another sentence (or two)is needed.} \victor{(a) I changed the beginning of the paragraph to improve the flow. (b) I also changed the end of it to include a slightly better explanation of this uncertainty.}

%\derek{Earlier you say that the focus of the dissertation is epistemic uncertainty. But this paragraph (or, at least some of it, especially the stuff about mistaken clicks) seems to me to be about aleatoric uncertainty. Would you agree? If so, should we more carefully split off the aleatoric ones from the epistemic ones?} \victor{I disagree. Aleatoric uncertainty can't be reduced by additional knowledge. If we knew which clicks were mistakes we could easily reduce uncertainty. You could even query users to confirm which interactions are indeed `real' clicks. From a statistical POV, aleatoric uncertainty means that you can never perfectly fit $Y \sim X$ even if $X$ is an infinite collection of data points and variables. I don't think any of the examples I mention here have anything related to aleatoric uncertainty.}

Quantifying the uncertainty of an \acrshort{rs}'s predictions and recommendations, either in the form of \emph{prediction uncertainty} or \emph{label uncertainty}, is important \cite{mcnee2003confidence,mazurowski_estimating_2013,cleger2013being,wang2022rethinking}. Ideally, uncertainty estimates should help detect which predictions and recommendations are more or less likely to be wrong \cite{bobadilla_reliability_2018, zhu2018assigning}. 

Focusing first on \emph{prediction uncertainty}, its estimation can improve the operation of the \acrshort{rs} in at least the following three ways.

First, estimates of prediction uncertainty can enhance the presentation of recommendations to users. This may make the RS more credible, increasing the user's trust and satisfaction. In the simplest case, we can present the degree of prediction uncertainty alongside each recommendation \cite{mcnee2003confidence, cleger2013being}. But there are other ways that uncertainty estimates could affect the presentation of a set of recommendations. For example, consider an RS that can offer explanations for its recommendations. The user is most likely to need explanations for recommendations where it is least clear why the item is being recommended. These are likely to be the recommendations about which the RS is least certain. The RS could use high uncertainty to trigger the offer of an explanation. This may avert the loss of credibility that can come from showing recommendations that do not make sense to the user.

Second, estimates of prediction uncertainty offer new strategies for selecting which items are suggested to the user \cite{mazurowski_estimating_2013, ortega2021providing}. Conventionally, the recommended items are those that the model predicts to have highest relevance to the user. But, equipped with estimates of uncertainty, an RS might discard highly uncertain recommendations, even when their predicted relevance is high: the user is shown items that the RS is certain are relevant. On the other hand, a \acrshort{rs} might deliberately include some items that it predicts as relevant but where the relevance is uncertain: from a user point-of-view, this coincides with the idea that \acrshort{rss} are tools for item discovery; from a system point-of-view, this helps the RS explore user tastes by seeking feedback on uncertain recommendations. An \acrshort{rs} might even combine these strategies: playing-it-safe by selecting some recommendations that are certain even though they may be obvious to the user; but taking-a-risk by also recommending some of the less certain ones. This is reminiscent of the exploration/exploitation trade-off that is widely recognized and studied in Reinforcement Learning and even in the RS literature, e.g.\ \cite{bouneffouf2013risk, barraza2017exploration}. In fact, controlling exploration/exploitation is arguably the application where uncertainty is most frequently used in \acrshort{rs} \cite{yang2021multi, zeldes2017deep}.

Third, these estimates may be useful internally to an \acrshort{rs}. A concrete example is found in the CoRec system, which uses co-training \cite{daCosta-Et-Al-2018}. CoRec comprises more than one model and the most certain predictions of one model are added to the training set of the other model. More generally, uncertainty measures might help a hybrid \acrshort{rs} decide which of several models to use in a particular circumstance \cite{Burke-2002}. There are also examples of Active Learning in \acrshort{rss} in which users are prompted to rate items that the \acrshort{rs} is uncertain about, e.g.\ \cite{Kohrs2001:smart_objects}.


Turning now to \emph{label uncertainty}, its estimation may also improve a \acrshort{rs}'s operation, and in two main ways. First, similarly to prediction uncertainty, label uncertainty may also be used to forge new strategies for recommendation selection. For example, items which a user has not previously interacted with that present high label uncertainty can be deemed potentially relevant. This is because they were likely mislabeled as irrelevant. Therefore, we may modify the system to encourage the recommendation of such items.

Second, label uncertainty estimates may be an important tool to counter-act popularity bias \cite{abdollahpouri2019unfairness} in \acrshort{rss}. Because unpopular items -- or `long-tail' items -- have few interactions, they are frequently deemed irrelevant, yet this may not be correct. It is likely that some unpopular items would be relevant to a user. %Although, due to their limited exposure, these may not be discovered by the user. 
By estimating label uncertainty, a \acrshort{rs} might find these unpopular mislabeled items and potentially recommend them to the user.

In the literature, there have been several attempts at estimating prediction uncertainty in \acrshort{rss}, and we explore them especially in Chapters \ref{chapter:explicit} and \ref{chapter:implicit} of this dissertation. However, we can identify several gaps in the field, deserving of more research. To begin with, most of the available prediction uncertainty estimation methods have not been compared to one another. An empirical comparison, with a standardized form of evaluation -- which the literature also lacks -- is crucial to assess the credibility of the different prediction uncertainty estimators. Another issue is the lack of clarity over their coverage. That is, there is no clear categorization of which systems each prediction uncertainty estimator is applicable to. For instance, some methods are model-agnostic, meaning that they can be applied to any \acrshort{rs}, while others are useful only for a restricted class of systems. Finally, many of these estimators are based on heuristics. We believe that a more sophisticated form of uncertainty modelling could lead to better uncertainty estimates.

Unlike prediction uncertainty, label uncertainty is a relatively new concept and has received very little attention in the \acrshort{rs} literature. Therefore, our goal in this branch of our research is to enlarge the literature with well formulated modelling methods --- see Chapter \ref{chapter:implicit_learn}.

%\db{I have a concern. Starting on page 2, you introduce uncertainty in RS. You define it: we ``define uncertainty as the expected imprecision of a prediction''. This sounds like you are defining uncertainty for this dissertation. You then give three use cases for this type of uncertainty in RS. However, this definition and these uses cases only apply to chapters 3 and 4. I wonder if we want to signal the alternative definition here too, and its use cases. Maybe we even need names for the two types. You have a name for the second type: mislabeling uncertainty (although I wonder whether label uncertainty would be a better name?). Maybe the first type is recommendation uncertainty or prediction uncertainty, and we would say that where-ever we use the word "uncertainty" on its own in the rest of the dissertation, we are talking about recommendation/prediction uncertainty. This is large-ish change to this section. Think about it, and we can discuss.} \victor{\checkmark} 


% We postpone a review of these systems to Chapter \ref{chapter:rs}. However, there is one distinction that we will draw now due its usefulness in presenting the remainder of this introductory chapter. 

% This distinction is based on the way that users provide feedback on items that the \acrshort{rs} exposes to them and that they consume. In this respect, \acrshort{rss} generally work in one of two settings: \emph{explicit feedback} or \emph{implicit feedback}.  In the explicit feedback setting, the system invites users to rate some or all of the items that the user has consumed. These known ratings are among the knowledge from which the \acrshort{rs} build a model that can predict the a user's ratings for unrated items. It uses the predicted ratings to help it decide which items to recommend to the user. On the other hand, in the implicit feedback setting, the \acrshort{rs} does not explicitly invite the user to provide consumption feedback. Instead, it . It is from these records of interaction behaviour that the system builds a model to predict the relevance of candidate items to the user.

% Although recommending relevant items is usually the main goal of a \acrshort{rs}, there are other important considerations. 
% \db{The remainder of this section needs changing but, so far, I have left the remainder of this section unchanged. I have two main concerns. (a) The first concern. For me, the example doesn't work for all sorts of reasons. First, if there's a small catalog, then who needs an RS? Second, even if there's a small catalog, then whether I will like today's dessert might still be very uncertain. (b) The second concern. This section's title includes ``Motivation''. And what you are doing is (i) trying to describe what recommendation uncertainty is (OK), and (ii) mention that it is under-explored and therefore presumably a ripe topic for your PhD (OK), but between (i) and (ii) we need to know why estimating uncertainty would be useful. That's the real motivation, but it is missing.}
% Consider, for example, two different \acrshort{rs} applications A and B. In application A, the catalog of items is not too large and just one purchase can be highly informative of what the user might consume next. One example is a coffee shop, where buying a coffee often suggests that the client might also want a desert alongside. On the other hand, the catalog on application B is huge, and it might take many inputs from the user before the system can accurately recommend more relevant items. An online marketplace is a good example of the latter.

% There is a clear distinction between systems A and B. To the former, recommendations are given with very little uncertainty, because there is not much to take into consideration when suggesting an item. On the other hand, system B needs to consider a much large pool of recommendation candidates, while working with little, and sometimes not very informative, user data. Therefore, recommendations are expected to be much less certain. In practice, most systems fall in between these extreme examples, and therefore, it is not straightforward to quantify what degree of certainty to expect from each recommendation.

% Furthermore, the recommendations given to each user, and each individual item recommended will have their own associated uncertainty. For example, an item frequently consumed tends to be a fairly certain recommendation. On the other hand, a more niche item is often a uncertain recommendation. Similarly, some users might have profiles that are easier to recommend for compared to others. Take for example a user that listens only to one music genre versus another that that consumes from several genres. Naturally, the uncertainty on the recommendations give to the former will be smaller than those for the later.

% For all these reasons, estimating uncertainty on individual recommendations is necessary. In the literature, there were several attempts to estimate uncertainty. Nevertheless, we identified a big scientific gap in the field: most of the available estimation methods haven't been empirically compared. Another issue is the lack of coverage clarity. For instance, some methods are model-agnostic, meaning that they can be applied to any \acrshort{rs}, while others are useful only to a restricted class of systems. In addition to this, many of these estimators are based on some heuristics. We believe that a more sophisticated form of uncertainty modelling could lead to better estimates. Finally, there is no standard in the way uncertainty estimates are evaluated.

\section{Contributions}
%\victor{I made several small changes to this section. But as you said, we can only finalize this once the whole thing is written}

This dissertation addresses prediction uncertainty and label uncertainty in recommender systems. More precisely, it discusses the various forms of uncertainty estimation in both explicit feedback-based \acrshort{rss} (Chapter \ref{chapter:explicit}) and implicit feedback-based \acrshort{rss} (Chapters \ref{chapter:implicit} and \ref{chapter:implicit_learn}). Moreover, it also explores how to properly evaluate such uncertainty estimates. Finally, it also explores use cases for uncertainty estimates. Below, we present the main contributions this dissertation makes to the field.

\subsection{Uncertainty estimators for explicit feedback \acrshort{rss}}
\label{sec:cont_exp}

We conduct by far the most extensive aggregation of methods for prediction uncertainty estimation in \acrfull{erss} (see Chapter \ref{chapter:explicit}). We define a taxonomy to classify the methods. Furthermore, we re-write all the methods using a common notation; for some of the methods, this was a considerable undertaking\footnote{It is instructive to compare, for example, our presentation of OrdRec in Section \ref{sec:multinomial_based} with the original in \cite{koren2011ordrec} to see one instance of just how much work was involved in adopting our common notation.}. We are positive that this contribution will substantially facilitate future research in the field. 

In addition, we also propose two new methods for prediction  uncertainty estimation in \acrshort{erss}, namely Ensemble (see Section \ref{sec:stability_based}) and CV-Bias (see Section \ref{sec:error_based}). These new methods are variants of the methods we surveyed. According to many of our evaluation metrics, these variants are effective, performing better than the ones that inspired them \cite{coscrato2022}. In Table \ref{tab:explicit_models}, we list all the uncertainty estimators for \acrshort{ers} that we compared, highlighting the novel ones introduced in this dissertation. 

\begin{table}[t]
\centering
\begin{tabular}{|c|c|c|}\hline
\textbf{Uncertainty estimator}     & \textbf{Introduced in} & 
\textbf{Defined in} \\ \hline
$\isup$       & \cite{mazurowski_estimating_2013} & \ref{sec:info_based} \\ \hline
$\ivar$       & \cite{adomavicius2007towards}     & \ref{sec:info_based} \\ \hline
$\resample$   & \cite{mazurowski_estimating_2013} & \ref{sec:stability_based} \\ \hline
\rowcolor{lightgray} $\ensemble$ & Novel          & \ref{sec:stability_based} \\ \hline
$\FunkSVD-CV$ & \cite{zhu2018assigning}           & \ref{sec:error_based} \\ \hline
\rowcolor{lightgray} $\Bias-CV$    & Novel        & \ref{sec:error_based} \\ \hline
$\CPMF$       & \cite{wang2018confidence}         & \ref{sec:distribution_based} \\ \hline
$\BeMF$       & \cite{ortega2021providing}        & \ref{sec:multinomial_based} \\ \hline
$\OrdRec$     & \cite{koren2011ordrec}            & \ref{sec:multinomial_based} \\ \hline
\end{tabular}
\caption{All the prediction uncertainty estimators for explicit feedback-based \acrshort{rss} that we compare. The ones that are original to this dissertation are highlighted in gray.}
\label{tab:explicit_models}
\end{table}

\subsection{Uncertainty estimators for implicit feedback \acrshort{rss}}
\label{sec:cont_imp}

We also conduct the largest aggregation of methods for uncertainty estimation in \acrfull{irss}. For \acrshort{irss}, we distinguish between two forms of uncertainty: prediction uncertainty and label uncertainty. %The difference will become more evident later on. 
For this reason, we further divide our aggregation of the methods into two chapters.

%\db{The titles of the two subsubsections below do not correspond, even loosely, to the title of the two chapters. I'm not saying they need to be identical but I would expect a greater connection. What's more they're not even consistent with what you have said so far. The first title uses the word ``inference'', but the second does not use the word ``learning''.} \victor{Giving names to each form of uncertainty greatly helped here. Both subsections were changed substantially.}

\subsubsection*{Prediction uncertainty in \acrshort{irss}}

In Chapter \ref{chapter:implicit}, we explore methods for prediction uncertainty estimation in \acrshort{irss}. The estimators in this chapter follow either from (i) heuristic data metrics or (ii) computing variability in the relevance estimator. In particular, we identify which uncertainty estimation methods from Chapter \ref{chapter:explicit} are extensible to \acrshort{irss}. Furthermore, we also explore methods for uncertainty quantification in \acrfull{nns}. 

Many high performing \acrshort{rss} in the literature use \acrshort{nns}. To the best of our knowledge, two of the methods for uncertainty quantification in \acrfull{nns}, \acrlong{bnn}s and Deep Ensembles, have not been previously applied to recommender systems, and so we include these in our investigations. Therefore, none of the methods from Chapter \ref{chapter:implicit} are original to this dissertation. What is novel is their application to \acrshort{irss}. Table \ref{tab:implicit_models4} lists all the prediction uncertainty estimators compared in the chapter.

\begin{table}[t]
\centering
\begin{tabular}{|c|c|c|}\hline
\textbf{Uncertainty estimator}     & \textbf{Introduced in} & 
\textbf{Defined in} \\ \hline
$\usup$       & \cite{mazurowski_estimating_2013}* & \ref{sec:info_based2} \\ \hline
$\isup$       & \cite{mazurowski_estimating_2013}* & \ref{sec:info_based2} \\ \hline
$\ensemble$ & \cite{mazurowski_estimating_2013}*  & \ref{sec:stability_based2} \\ \hline
\acrlong{bnn}s & \cite{blundell2015weight}* & \ref{sec:stability_based2} \\ \hline
\acrlong{mcd} & \cite{zeldes2017deep}             & \ref{sec:stability_based2} \\ \hline
Deep Ensemble & \cite{lakshminarayanan2017simple}*  & \ref{sec:stability_based2} \\ \hline
\end{tabular}
\caption{All the prediction uncertainty estimators for implicit feedback-based \acrshort{rss} that we compare in Chapter \ref{chapter:implicit}. We mark with * the methods that are to be found in the literature but, to the best of our knowledge, have never previously been applied to \acrshort{irss}.}
\label{tab:implicit_models4}
\end{table}

\subsubsection*{Label uncertainty in \acrshort{irss}}

In Chapter \ref{chapter:implicit_learn}, we discuss the learning of label uncertainty estimators in \acrshort{irss}. Although Chapter \ref{chapter:implicit_learn}, like Chapter \ref{chapter:implicit}, tackles the implicit feedback recommendation case, label uncertainty is very different from prediction uncertainty, both in motivation and in its use cases, which is why label uncertainty deserves its own chapter.

%\db{Perhaps a first contribution is your new mathematical modelling? Then the four novel methods are your second contribution?} \victor{I am not too sure what you mean by new mathematical modelling. I tweaked the following paragraph to acknowledge the modeling work more concretely.}

The main contribution of this chapter is the definition of four novel methods for label uncertainty estimation: ICPMF, Gaussian-MLP, \acrfull{gbr} and \acrfull{hbr}. This is a considerable undertaking considering that these models are quite different from each other. For instance, without jumping into details, Gaussian-MLP is neural-network based while \acrshort{hbr} is a hierarchical Bayesian model. In addition, we extensively compare these methods against \acrlong{aur}, the closest method from the literature. Table \ref{tab:implicit_models5} lists all the uncertainty estimators compared in the chapter.

\begin{table}[t]
\centering
\resizebox{\columnwidth}{!}{%
\begin{tabular}{|c|c|c|}\hline
\textbf{Uncertainty estimator}     & \textbf{Introduced in} & 
\textbf{Defined in} \\ \hline
\acrlong{aur}       & \cite{wang2022rethinking} & \ref{sec:aur} \\ \hline
\rowcolor{lightgray} ICPMF  & Novel & \ref{sec:icpmf} \\ \hline
\rowcolor{lightgray} Gaussian-MLP      & Novel & \ref{sec:neural_aur} \\ \hline
\rowcolor{lightgray} \acrlong{gbr}       & Novel & \ref{sec:gbr} \\ \hline
\rowcolor{lightgray} \acrlong{hbr}       & Novel & \ref{sec:hbr} \\ \hline
\end{tabular}}
\caption{All the label uncertainty estimators for implicit feedback-based \acrshort{rss} that we explore in Chapter \ref{chapter:implicit_learn}. The ones that are original to this dissertation are highlighted in gray.}
\label{tab:implicit_models5}
\end{table}

\subsection{Evaluation of prediction uncertainty estimates in \acrshort{rs}}

We present an extensive review of methods for evaluating prediction uncertainty estimates in both explicit and implicit feedback-based \acrshort{rss}. In addition, we extend and improve the techniques and metrics for evaluating prediction uncertainty estimates in \acrshort{rss}. In particular, we analyse how prediction uncertainty correlates with dataset statistics; we fix possible issues with existing metrics; and we propose new forms of evaluation for prediction uncertainty estimates in the top-$n$ recommendation task. As explained in contribution \ref{sec:cont_empirical} below, using these extended and improved techniques and metrics, we are able to conduct rigorous and reproducible empirical comparisons in \Cref{chapter:explicit,chapter:implicit,chapter:implicit_learn} 

\subsection{Uses for uncertainty estimates in \acrshort{rss}}

We review and expand on the methods to exploit prediction uncertainty estimates in both explicit and implicit feedback-based \acrshort{rss}. More specifically, for explicit feedback-based \acrshort{rs}, we introduce the idea of uncertainty-aware ranking, drawing in part on a recommendation strategy proposed in \cite{ortega2021providing}. In particular, in Chapter \ref{chapter:explicit}, we distinguish uncertainty-based filtering (UBF) and probability-of-relevance ranking (PRR). In Chapter \ref{chapter:implicit}, we extend the \acrshort{ubf} strategy to \acrshort{irss}. 

Similarly, in Chapter \ref{chapter:implicit_learn}, we explore \acrfull{mar}, that derives from \cite{wang2022rethinking}, another form of uncertainty-aware ranking, but based instead on label uncertainty.

We show that, in some cases, it is possible to provide more accurate recommendations by making use of uncertainty estimates through UBF or PRR. We also show that, in some cases, \acrshort{mar} increases overall recommendation accuracy and, on top of that, also helps reduce popularity bias.

%\db{I'm not sure this section is working well yet, particularly given that it is quite a big contribution of the work.} \victor{There were several changes to the secion. Mainly because chap 5 has new stuff now. See how you like it.}

\subsection{A reproducible empirical comparison of the uncertainty estimators}
\label{sec:cont_empirical}

Prior to this work, the literature in the field was largely disconnected. Most of the methods that we survey have never been empirically compared to each other, and therefore there was no clarity on what the state-of-the-art was. Of course, none of them had been compared with the methods that are original to this dissertation. We perform the most comprehensive comparison of uncertainty estimators for both \acrshort{erss} and \acrshort{irss} to date. In each case, we use popular and widely available datasets. 

We provide implementations for all the methods in Tables \ref{tab:explicit_models}, \ref{tab:implicit_models4} and \ref{tab:implicit_models5} under an open license. Therefore, all of our experiments are completely reproducible, and new developments in the field can benefit from our source code.

%\derek{The distinction between 1.2.3 and 1.2.5 is not clear.} \victor{I added a reference to 1.2.5 in 1.2.3 to show that the contributions are connected. I like to keep this contribution separate, because making it reproducible is a major advantage against the majority of our references.}

\section{Notation}
\label{sec:notation}

In this section, we introduce general notation --- notation that will be used throughout this dissertation. Other notation, which is specific to particular models, will be introduced on an as-needed basis in the remainder of the dissertation. Table \ref{tab:notation} summarizes the notation schematically.

\begin{table}[t]
\centering
\resizebox{\columnwidth}{!}{%
    \begin{tabular}{|l|l|} \hline
     \multicolumn{2}{|c|}{\rule{0pt}{20pt} \textbf{General Notation}} \\ \hline
    $U$ & The set of all users. \\ \hline
    $I$ & The set of all items. \\ \hline
    $F$ & %\begin{tabular}{@{}c@{}} 
    A rating or relevance scoring function, $F: U \times I \rightarrow \mathbb{R}$. \\ & For conciseness, $F_{ui} := F(u, i)$. %\end{tabular} 
    \\ \hline
    $G$ & %\begin{tabular}{@{}c@{}} 
    An uncertainty prediction function, $G: U \times I \rightarrow \mathbb{R}$. \\ & For conciseness, $G_{ui} := G(u, i)$. %\end{tabular} 
    \\ \hline
    % $\langle F_{ui}, G_{ui} \rangle$ & An uncertain prediction. \victor{check if being used}. \\ \hline
    $C_u$ & Candidate items, $C_u \subseteq I$, that might be recommended to user $u$. \\ \hline
    $Z_u^n$ & List of $n$ items that are recommended to user $u$. \\ \hline
    $\tau$ & The uncertainty threshold for \acrlong{ubf}. \\ \hline
    
    \multicolumn{2}{|c|}{\rule{0pt}{20pt} \textbf{\acrfull{erss}}} \\ \hline
    $r_{ui}$ & User $u$'s rating for item $i$. \\ \hline
    $S$ & The ordered set of $m$ possible rating values, $S = \{s_1, \ldots, s_m\}$. \\ \hline
    $R$ & The set of all user-item-rating-triples. \\ \hline
    $R_{u \cdot}$ & User $u$'s profile, i.e.\ $R_{u \cdot} = \{r_{ui} : \langle u, i, r_{ui} \rangle \in R\}$. \\ \hline
    $R_{\cdot i}$ & Item $i$'s support set, i.e.\ $R_{\cdot i} = \{r_{ui} : \langle u, i, r_{ui} \rangle \in R\}$. \\ \hline
    $\omega$ & Relevance threshold; item $i$ is deemed relevant to user $u$ if $r_{ui} \geq \omega$. \\ \hline

    \multicolumn{2}{|c|}{\textbf{\rule{0pt}{20pt} \acrfull{irss}}} \\ \hline
    $R$ & The set of all user-item interaction pairs. \\ \hline
    $R^-$ & %\begin{tabular}{@{}c@{}}  
    The set of all unobserved user-item interactions, \\ & i.e.\ $R^- = \{\langle u, i \rangle | u \in U, i \in I, (u, i) \notin R\}$. %\end{tabular} 
    \\ \hline
    $R_{u \cdot}$ & User $u$'s profile, i.e.\ $R_{u \cdot} = \{r_{ui} : \langle u, i \rangle \in R\}$. \\ \hline
    $R_{\cdot i}$ & Item $i$'s support set, i.e.\ $R_{\cdot i} = \{r_{ui} : \langle u, i \rangle \in R\}$. \\ \hline
    \end{tabular}}
    \caption{The dissertation's notation. Notice that the same notation may have a slightly different definition for \acrshort{erss} and \acrshort{irss}.}
    \label{tab:notation}
\end{table}

We will assume that our data consists of a set of users $U$, a set of items $I$, and a set $R$ of user-item interactions. When numeric ratings are available (explicit feedback), $R$ is formed by user-item-rating triples, where triple $\langle u, i, r_{ui}\rangle \in R$ means that user $u$'s rating of item $i$ is $r_{ui} \in S$. $S$ is the set of possible ratings -- e.g.\ $\{1\star, \cdots,  5\star\}$. Otherwise, we assume we have unary implicit feedback and so $R$ is a set of pairs $\langle u, i \rangle$, that contain the observed interactions. That is, if user $u$ has interacted with item $i$, then $\langle u, i \rangle \in R$. In addition, in this setting, the set of unseen pairs (unobserved interactions) is represented by the symbol $R^- = \{\langle u, i \rangle | u \in U, i \in I, (u, i) \notin R\}$.

A user's profile $R_{u \cdot}$ is the set of items that user $u$ has interacted with. Formally, $R_{u \cdot} = \{i : \langle u, i, r_{ui}\rangle \in R\}$ if the interactions are explicit ratings, or $R_{u \cdot} = \{i : \langle u, i \rangle \in R\}$ otherwise. Similarly, an item's support set $R_{\cdot i}$ is the set of users that interacted with item $i$: $R_{\cdot i} = \{u : \langle u, i, r_{ui} \rangle \in R\}$ in the case of explicit ratings, or $R_{\cdot i} = \{u : \langle u, i \rangle \in R\}$ for implicit feedback.

We will postpone a more extensive description of the remaining notation in the table (such as $\tau$ and $\omega$) for later chapters.

%\derek{I am glad you have restored the Table. But (i) It is strange to `bury' the table in a review chapter, esp.\ since the notation is used throughout the dissertation. Most people would put it into Chapter 1 or alongside the List of Figures and List of Tables in the frontispiece. If so, you would keep Section 2.3.1, but you would refer back to the Table. (ii) There is notation in the Table that Section 2.3.1 does not yet explain. I'm not suggesting that Section 2.3.1 should explain that notation. But maybe there needs to be a sentence that signals that this remaining notation will be explained later in this dissertation.}

%\victor{I looked up how to put the table upfront with the list of tables in stack exchange but I couldn't find an example that worked for me. Also, when I did it by simply moving it to the "thesis.txt" file, all the references to the table got broke, possibly because It was outside of a chapter and therefore had no number. My alternative was to move the entire section to Chapter 1. It breaks the flow a bit IMO, but it helps with the backwards references. I am happy enough.}

\section{Publications}

The following publications were produced from the work in this dissertation:

\begin{itemize}
    \item Victor Coscrato and Derek Bridge: \emph{Estimating and Evaluating the Uncertainty of Rating Predictions and Top-$n$ Recommendations in Recommender Systems}. ACM Trans. Recomm. Syst. 1, 2, Article 7 (June 2023), 34 pages. \url{https://doi.org/10.1145/3584021}
    \item Victor Coscrato and Derek Bridge: \emph{Recommendation Uncertainty in Implicit Feedback Recommender Systems}. In: Longo, L., O’Reilly, R. (eds) Artificial Intelligence and Cognitive Science. AICS 2022. Communications in Computer and Information Science, vol 1662. Springer, Cham. \url{https://doi.org/10.1007/978-3-031-26438-2_22}
\end{itemize}

\section{Outline of the Dissertation}

The remainder of this dissertation is structured as follows.

In Chapter \ref{chapter:rs}, we present a brief introduction to Recommender Systems. This chapter reviews, at a high level, the three stages of a recommender system: candidate selection, scoring and top-$n$ recommendations. Rather than being an exhaustive review, the chapter focuses on models and evaluation methods that will be used in the chapters to follow. The chapter ends on a brief introduction to uncertainty in \acrshort{rss}, that sets the ground to the following three chapters.

In \Cref{chapter:explicit,chapter:implicit,chapter:implicit_learn}, we will extensively research several methods for prediction and label uncertainty estimation in both Explicit and \acrlong{irss}. The literature in the field is very disconnected, and for this reason, we chose to present the most relevant related work alongside our own. In every chapter, we will emphasize which of the presented methods are original to this dissertation. In addition, Tables \ref{tab:explicit_models}, \ref{tab:implicit_models4} and \ref{tab:implicit_models5} list the methods we compare and highlight those that are novel. These chapters are organized as follows.

In Chapter \ref{chapter:explicit}, we explore and extend methods for prediction uncertainty estimation for explicit feedback-based \acrshort{rss}. This chapter includes a very extensive review and categorization of the research field. Furthermore, it also introduces two novel methods: $\ensemble$ and $\Bias-CV$. Finally, it presents a comprehensive empirical comparison between the methods.

Chapter \ref{chapter:implicit} is similar to the previous one, but it covers methods for prediction uncertainty estimation in \acrlong{irss}. It generalizes many of the methods from Chapter \ref{chapter:explicit} to the case of \acrshort{irss}. In addition, it discuss the use of methods for prediction uncertainty quantification in neural network based \acrshort{irss}, including two that have not been previously explored in the \acrshort{rs} literature: \acrlong{bnn}s and Deep Ensembles. It ends with another empirical comparison.

Chapter \ref{chapter:implicit_learn} introduces a collection of novel methods for label uncertainty estimation in implicit feedback-based \acrshort{rss}. More precisely, it introduces four new methods, ICPMF, Gaussian-MLP, \acrlong{gbr} and \acrlong{hbr}, some of which involve extensive mathematical modeling work. Again, the chapter culminates in an empirical comparison between the four novel models and another one from the literature.

%\db{Like when discussing contributions, you haven't chosen to mention your modelling work.}\victor{Similar comment as before. Not too sure about it, but tried to comment on it.}

Finally, Chapter \ref{chapter:conclusion} presents our conclusions and discusses possible directions for future research.

%\db{This chapter is much improved and close to being ready. Well done. I have one concern. See the end of Section 1.1.} \victor{\checkmark}