\section{Possible paragraph on introduction}

An RS may filter and rank candidate items in order to recommend a list of the top-$n$ candidate items. Often, the filtering and ranking are \emph{personalized}: the top-$n$ on any occasion depends on the user and the context in which the items are to be consumed. An RS will often use a model that has been learned from knowledge about the users, knowledge about the items, and records of interactions between users and items (such as purchases, downloads, clicks and ratings). While there are many different types of  models, content-based and collaborative-filtering (CF) recommenders are two major categories. The first uses descriptions of the items in the catalog allied with records of the active user's item consumption history. On the other hand, CF recommenders learn from historical user-item interactions; in this case, the recommendations to a user are affected by the preferences of other users in the system.




data: \cite{liu2020towards} \cite{gadanho2007addressing}






\section{MF}

In this work, we want, not only to predict ratings, but also the \textit{uncertainty level}, $\rho_{ui}$, present in a prediction. We will refer to a pair $\langle\hat{r}_{ui}$, $\rho_{ui}\rangle$ as an \textit{uncertain prediction}. In all the work that we report in this paper, uncertainty levels are real-valued, $\rho_{ui} \in \real$. In some cases, they are probabilistic, which bounds them in $[0,1]$, but this is not always the case. The higher the uncertainty level, the more uncertain the prediction.


The conventional way of selecting the top-$n$ is to use the predicted ratings alone (see \textit{rating-based ranking}, below). But, given that we are considering models that can estimate prediction uncertainty, then there is an opportunity to consider other ways of selecting the top-$n$. These use what we can collectively refer to as \textit{uncertainty-aware ranking}. If the estimates of uncertainty are good enough, we may achieve better (or, at least, usefully different) recommendation performance by recommendation strategies that rank the candidates using some combination of the predicted ratings and their estimated uncertainties. We propose two such strategies: \textit{uncertainty-based filtering} and \textit{probability-of-relevance ranking}.


\begin{table}[t]
\centering
    \begin{tabular}{ll} 
    $U$ & set of all users \\
    $I$ & set of all items \\
    $r_{ui}$ & user $u$'s rating for item $i$ \\
    $S$ & ordered set of $m$ possible rating values, $S = \{s_1, \ldots, s_m\}$ \\
    $R$ & set of all user-item interactions \\
    $R_{u \cdot}$ & user $u$'s profile, i.e.\ $R_{u \cdot} = \{r_{ui} : \langle u, i, r_{ui}\rangle \in R\}$ \\
    $R_{\cdot i}$ & item $i$'s support set, i.e.\ $R_{\cdot i} = \{r_{ui} : \langle u, i, r_{ui}\rangle \in R\}$ \\
    $F$ & a prediction model, i.e.\ $\hat{r}_{ui} = F(u, i)$ \\
    $C_u$ & candidate items, $C_u \subseteq I$, that might be recommended to user $u$ \\
    $Z_u^n$ & list of $n$ items that are recommended to user $u$ \\
    \end{tabular}
    \caption{General notation}
    \label{tab:notation}
\end{table}



\paragraph*{Future work} One avenue for future work is, of course, to address the limitations described above. But, there are at least two further additional avenues for exploration.

First, this work is deliberately limited to the case of explicit feedback; more specifically, we focus on uncertainty estimation for rating prediction. An obvious avenue for future work is to investigate uncertainty estimation for RSs that use implicit feedback (based on user behaviour). We are aware of only a few papers that explore uncertainty estimation in RS that use implicit feedback, e.g.\ \cite{dos2017gaussian, wang2022rethinking, jiang2020convolutional, neupane2021metaedl}. We are already developing our own models for this case, e.g.\ \cite{coscrato2022}, building on the foundation that this paper provides.












To answer \textbf{Q1}, we compute the accuracy of the top-$K$ recommendations for different recommendation list sizes $K = 1, 2 \ldots, 10$. Figure \ref{fig:accuracy} shows the MAP@K obtained in both datasets. Note that MF-NUS \& MF-NIS and MLP-NUS \& MLP-NIS are omitted because their MAP is the same as MF or MLP.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{img/AICS/accuracy.pdf}
    \caption{$MAP@K$ for $K = 1, 2 \ldots, 10$ for the Movielens (left) and Pinterest (right) datasets.}
    \label{fig:accuracy}
\end{figure}

The ensemble models, MF-ENSEMBLE and MLP-ENSEMBLE, show a remarkable MAP improvement over the baselines, MF and MLP. On the other hand, the BayesianMLP has similar performance to the deterministic MLP, and MCDropout has the worst performance on both datasets. Therefore, we found that some models that perform uncertainty estimation improve accuracy, others worsen it. Clearly, ensembling emerges as the most beneficial method with respect to accuracy.

To answer \textbf{Q2}, we analyze the accuracy of the models for users, grouped according to their average recommendation uncertainty. More precisely, we calculated the average uncertainty on each user's recommendation list, and split the users into 10 equal-sized uncertainty bins, where bin 1 will have the 10\% of users with the smallest average recommendation uncertainty and bin 10 will have those with the highest. Our intuition is that accuracy will fall as uncertainty grows. Figure \ref{fig:uncertainty} shows the results.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{img/AICS/MAP-Uncertainty.pdf}
    \caption{$MAP@10$ for users grouped according to the average uncertainty of their recommendation. The higher the bin index, the higher is uncertainty.}
    \label{fig:uncertainty}
\end{figure}

In line with our intuition, we see that MAP has a strong negative correlation with some of the uncertainty estimation methods. In particular, MF-Ensemble, MLP-Ensemble, MF-NUS and MLP-NUS appear to be those more strongly reflecting the expected behaviour. The results for $\usup$ show that mature users tend to get more accurate recommendations. In fact, $\usup$ has the strongest correlation to MAP in the Movielens dataset. On the other hand, in the Pinterest data, the MF-Ensemble is the one to achieve the strongest correlation. This, together with the earlier results in Figure \ref{fig:accuracy}, show that ensembling is not only a technique that can boost the accuracy of recommenders, but can also offer uncertainty estimates that correspond with the expected recommendation accuracy.

Other models to follow the expected behaviour are MCDropout in both datasets, and the BayesianMLP in the Pinterest dataset. Oddly, the BayesianMLP shows a growing MAP curve in the Movielens data, meaning that users with higher recommendation list uncertainties are getting higher accuracy, which is a result that needs further investigation. 

Models using $\isup$ do not show a very strong correlation between uncertainty and MAP with the exception of the last uncertainty bins in the Pinterest case. This too is a result that needs further investigation.

Largely, we believe that we have obtained a positive answer to \textbf{Q2}. For this research question, ensembling has, again, proven to be a great tool. Nevertheless, the simple and cheap $\usup$ metric can also provide good value with no computational cost added, in contexts where user-centric estimates suffice.
