\chapter{Conclusions and Future Work}
\label{chapter:conclusion}

This dissertation presents by far the most comprehensive study of uncertainty in Recommender Systems to date. More precisely, we distinguish between prediction uncertainty and label uncertainty, and we present several methods for estimating each of them through the various chapters. In addition, we also discussed and extended the evaluation and the use cases for uncertainty estimates in both explicit and implicit feedback-based recommender systems.

In this chapter, we present a summary of our main findings. Then, we discuss potential paths for future research in the field.

\section{Conclusions}

\subsection{Prediction uncertainty in \acrshort{erss}}

In Chapter \ref{chapter:explicit}, we surveyed and expanded the literature in prediction uncertainty in \acrshort{erss}. First, we identified several existing methods that were never compared theoretically nor empirically. We re-wrote all of these methods to use a common notation. We also proposed new methods (ENSEMBLE and EB-Linear) by adapting existing methods. Then we investigated and extended the literature on the evaluation of prediction uncertainty estimates. Finally, we implemented all the methods and compared them. All this work should greatly help further research in the field, especially because all our implementations are open source and our results are therefore reproducible.

Our empirical study showed that several of the surveyed methods are successful at prediction uncertainty estimation, especially in the rating prediction case. We also found CPMF to be a powerful method for \acrlong{ubf} and \acrlong{prr}. Its use led to more accurate recommendations in both the datasets that we used.

\subsection{Prediction uncertainty in \acrshort{irss}}

In Chapter \ref{chapter:implicit}, we built on the work in Chapter \ref{chapter:explicit}, but now turning our attention to \acrshort{irss}. We noted that there was close to no literature in this case. Therefore, a major part of the chapter's contribution was the identification of the uncertainty estimation and evaluation methods from Chapter \ref{chapter:explicit} that were also extensible to \acrshort{irss}. In addition, we also have a strong emphasis on neural network-based \acrshort{rss} in this chapter. This is because (i) they are very prominent in the \acrshort{irss} literature, and (ii) we could take advantage of general methods for uncertainty estimation in neural networks. Even though there are no novel methods in this chapter, their use in the problem of estimating prediction uncertainty in \acrshort{irss} is novel. The chapter culminates in another extensive, and also reproducible, empirical study comparing all the explored methods.

Unfortunately, the takeaways from this chapter are not too positive. On the good side, we found that many of our explored methods (most notably, those based on ensembling) are more accurate at top-$n$ recommendation than our baseline methods. But, different from the explicit-feedback case, we found no further benefits to using any of our uncertainty estimators in the top-$n$ recommendations case. Most of the methods fail according to the \acrshort{uri} metric and were not beneficial when used for \acrlong{ubf}.

\subsection{Label uncertainty in \acrshort{irss}}

In Chapter \ref{chapter:implicit_learn}, we tackle label uncertainty in \acrshort{irss}. Although label uncertainty is a widely recognized issue, it is very unexplored. In fact, the chapter builds upon the only existing method for label uncertainty estimation, \acrfull{aur}. Apart from very extensively discussing the mislabeling issue, we also introduce four novel methods for label uncertainty estimation. The modeling work presented is very substantial, especially because our methods are very different from each other, including neural-networks and Bayesian hierarchical modeling. Again, we conclude the chapter with another empirical study, where the goal is to validate the use of \acrfull{mar} in top-$n$ recommendations and long-tail recommendations.

Our results had two major conclusions. First, we observed that ICPMF, one of our proposed methods, is better than AUR in both the datasets that we used. This is very impressive given that ICPMF is a lot less complex than all the other methods in the comparison, including AUR. Second, we found that, in some cases, we may greatly improve a system's accuracy in long-tail recommendations by exploiting label uncertainty estimates and using \acrshort{mar} to select the recommendations.

\section{Future work}

In this dissertation we shed light on the quite unexplored field of uncertainty in recommender systems. Although our work is very comprehensive, we hope it will encourage further extensions -- in particular, to tackle the limitations that we identified at the end of Chapters \ref{chapter:explicit}, \ref{chapter:implicit} and \ref{chapter:implicit_learn}  -- and the exploration of novel research questions in the field. Below, we give an overview of what we consider to be good directions for future exploration.

\subsection{Beyond-accuracy recommendation objectives}

In Chapter \ref{chapter:rs}, we briefly reviewed some of the most important beyond-accuracy recommendation objectives. In addition, we expressed our belief that uncertainty quantification might also be the key to improving beyond-accuracy recommendation goals, e.g.\ novelty, diversity and serendipity \cite{10.1145/2926720}. For example, a serendipitous recommendation can be one with high relevance yet high uncertainty. The subsequent chapters do not further explore these objectives. We consider this to be the most immediate and most important avenue for future research.

\subsection{Online evaluation and user trials}

In this dissertation, we restricted our experiments to offline evaluation in every case. As introduced in Chapter \ref{chapter:rs}, online evaluation is an alternative that better reflects the dynamic nature of \acrshort{rss}. Given the extensiveness of our experiments, it was impractical to perform online evaluation. Nevertheless, there is clear value in exploring the methods present here -- especially those with better performance -- in an online setup.

Taking one more step forward, the evaluation of uncertainty could arguably benefit even more from user trials. To give a tangible example, we already discussed our belief that uncertainty may correlate to serendipity. The difficulties in measuring serendipity offline \cite{10.1145/2926720} impose great challenges to our ability to test this hypothesis. For this reason, one important avenue for future research is to conduct user trials, in which users would provide feedback, not only on their item preferences, but also on beyond-accuracy questions, such as their surprise regarding recommendations that have different levels of uncertainty.

\subsection{Long-tail recommendations}

In Chapter \ref{chapter:implicit_learn}, we noted the value of models that estimate label uncertainty for long-tail recommendations, to counter-act popularity bias. We even compared the models that we defined in the chapter for their long-tail performance. But the RS literature offers a plethora of other methods, for counter-acting popularity bias \cite{liu2023mitigating, klimashevskaia2022mitigating, borges2021mitigating, abdollahi2018transparency, wei2021model}. A line of future work would be to conduct a more extensive comparison between the methods from Chapter \ref{chapter:implicit_learn} and these other methods from the literature.

%\victor{You suggested to cite something by Castells here. Maybe I am missing something, but most of his relevant papers on the matter focus on mitigating bias on the evaluation the evaluation side of recommendations, and not on the systems as a whole.}

\subsection{Label uncertainty in \acrshort{erss}}

In this dissertation, we have investigated label uncertainty for the implicit feedback case. Certainly, this is the most important case. But one could argue that label uncertainty is also present in explicit systems. On the one hand, we note that the problem is definitely not as big in the explicit case (where unobserved interactions are treated as missing data) as in the implicit case (where they are presumed irrelevent and can be sampled as negative interactions). Nevertheless, interactions in \acrshort{erss} may be `mislabeled' as being missing. This happens, for example, when the user consumes an item but chooses not to provide a rating. Therefore, label uncertainty could also be investigated for \acrshort{erss}.

\subsection{Novel forms of uncertainty-aware ranking}

Throughout this dissertation, we discussed methods for prediction uncertainty-aware ranking, namely \acrshort{ubf} and \acrshort{prr} and label uncertainty-aware ranking, namely \acrshort{mar}. We believe that there is a great creative space for other novel forms of uncertainty-aware ranking (for both prediction uncertainty and label uncertainty).

Some interesting ideas can be borrowed from the the related field of Information Retrieval. For example, Penha \& Hauff  explore uncertainty in neural learning-to-rank models \cite{penha2021calibration}. They obtain uncertainty estimates for a BERT ranker with the use of Monte-Carlo Dropout and Deep-Ensembles, which we explained in Chapter \ref{chapter:implicit}. Their uncertainty-aware ranking method combines the predicted interaction relevance with their estimated uncertainties. They found that `shrinking' the relevance of interactions with high relevance can sometimes improve the system's recommendation accuracy. In a similar vein, but now using models based on Gaussian Processes, Guiver \& Snelson proposed to either shrink or increase the relevance of items based on their uncertainty to make the model more conservative or more risk-taking \cite{guiver2008learning}.\footnote{We note, however, that their results are largely negative: they did not find this form of ranking to be more accurate.}

\subsection{Other relevant comparisons}

There is some work on uncertainty, or related to uncertainty, in the \acrshort{rss} literature that we did not cover in this dissertation. Mainly, we chose not to cover them because they tackle uncertainty from a different scope or purpose. Nevertheless, we believe that future research could try to bridge the gap between them and the research in this dissertation. We list the most relevant of them here and explain why we did not cover them.

\begin{itemize}
    \item Bernardis et al.\ showed that there is a strong correlation between the eigenvalues of the item similarity matrix and the accuracy of item-based recommenders \cite{bernardis2019estimating}. Because of this, they propose an eigenvalue confidence index to measure the confidence level of the recommendations given to each user. Their method is suitable for both explicit and implicit recommendation tasks, and confidence can be thought of as the inverse of prediction uncertainty. However, their method is applicable only to systems based on item similarity. Furthermore, like $\usup$, their confidence index is a user-centric measure, and therefore it lacks the granularity that is needed to differentiate the uncertainty of the individual items being recommended to a user.

    \item Another method, which is superficially related to the ones employed herein but is actually quite different in its purpose, is the use of Gaussian embeddings for collaborative filtering. Gaussian embeddings are a generalisation of the embeddings used in many recommender algorithms. In \cite{dos2017gaussian}, Gaussian embeddings are learned by a matrix factorization algorithm; in \cite{jiang2020convolutional}, they are learned by a convolutional neural model. Gaussian embeddings give us non-deterministic user and item representations, capturing the uncertainty that there is in learning these representations. However, they do not quantify the prediction or label uncertainty of interactions.

    \item Neupane et al.\ \cite{neupane2021metaedl} propose a method for quantifying the amount of evidence available when providing recommendations to cold-start users. Without jumping into details, their method combine traditional user and item embeddings with a meta evidential learning module that is able to predict what they call evidential uncertainty. We suspect that evidential uncertainty is tightly connected to our information-based methods. Therefore, we believe that future research could formalize the similarities and differences between prediction uncertainty and evidential uncertainty, and compare our methods against theirs, where applicable. %\dgb{Hmm. The last sentence is weak.} \victor{The whole paragraph was a bit weak, I changed it a bit.}

    %\item Another general method for uncertainty estimation is the use of conformal predictions. The literature applying conformal predictions within \acrshort{ml} is vast, but they have received received little attention in \acrshort{rss}. We know of just two papers, one by Kagitaa et.\ al.\ \cite{kagita2017conformal} and the other by Himabindu et.\ al.\ \cite{himabindu2018conformal}. These contain methods that could be compared against ours.

    %\dgb{Can you add even one sentence about why we did not already compare against them?} \victor{I don't like your changes to this paragraph. It's not true that we only know those 2. There are at least two other by some of the same authors. I propose the text below.}

    \item Another general method for uncertainty estimation are conformal predictions. The literature applying conformal predictions within \acrshort{ml} is vast, but so far they have received received little attention in \acrshort{rss}. Among the few existing works, we highlight two, one by Kagitaa et.\ al.\  \cite{kagita2017conformal} and the other by Himabindu et.\ al.\ \cite{himabindu2018conformal}. These contain methods that could, in theory, be compared with ours. However, unlike the methods in this dissertation, their conformal predictions are not point estimates  of label uncertainty. Instead, conformal predictions provide a set of possible labels (e.g.\ a set of points in a rating scale) for each user-item interaction. For this reason, it is not too straightforward to compare their methods against ours.
\end{itemize}

%\dgb{(a) I note you've chosen not to mention the missing-not-at-random/ causal/ counterfactual work. (b) The dissertations ends with a negative para.} \victor{I borrowed an idea from Diego's thesis. It ends with a `conclusion to the future work' paragraph that also cites some papers. I think this kills two birds with one stone.}

%\victor{new paragraph:} 
The sections above provide numerous possible paths for future research. But they are only a sample of everything that can be studied in the field. Many other directions can be suggested. For example, one could investigate the connection between label uncertainty and the definition of Missing-Not-At-Random (MNAR) data in \acrshort{rss} \cite{steck2010training, wang2019doubly}. Or, going one step further, one could compare label uncertainty with causal \cite{bonner2018causal} or counterfactual recommendations \cite{wang2021clicks, xu2020adversarial}, since they all tackle similar implicit feedback reliability issues. The fact is that this dissertation could be the starting point for much future research.

%\victor{About the refs: I found a very nice python script that capitalized all the titles for me. Although, they don't appear capitalized here, I believe the thesis template forces them to be lower case. At least, what I did fixed the cases after commas.}

%\dgb{Have you run it? The bib here looks no better to me. (a) You still have the duplicate citation. (b) the paper titles are still inconsistent in use of upper/lower case (c) The journal/ book/ conference names are still inconsistent in upper/lower case but also in use of Procs vs Proceedings, use of numbers (3rd third Third). }

%\victor{Sorry if I gave the impression that bib was done. I was just communicating the exquisite behaviour of the UCC template. It won't allow me to fully capitalize the full title unless I do it in a hacky way. The default is only first letter capital.}

%\victor{I've now gone through the bib a couple of times, I don't see problems any longer.}

%\dgb{Think about the order of the previous subsections. Is there any order that is more logical or more positive?} \victor{I gave this some though, but honestly I don't know if there's any better ordering. Maybe this means there isn't. But if you spot any ordering that should change, I am all for it.}

%\dgb{I'm quite pleased with this chapter. :)} \victor{:)}

% \dgb{Another idea for FW: the relationship between label uncertainty and the work on (not-)missing-at-random (see Diego's thesis and, relatedly, causal reasoning/counterfactuals.} \victor{I see the relationship. I will leave this here and give it a second thought.}



% We refrain from further exploring methods of such nature because they are not straight-forward to compare in a model-agnostic manner. Depending on the loss function of choice for learning the baseline model, the interpretation and range of the relevance values are very different. One concrete example is the difference between models learned with BCE and BPR. For the former, relevance values are probabilities contained in $[0, 1]$. For the latter, relevance values are latent scores in $(-\infty, \infty)$. Therefore, care must be taken when comparing models from different nature.


%\dgb{Acronyms: I see HBR but not HBR-R and HBR-Y. That's probably OK. I don't see CPMF and ICPMF.}