\chapter[Label Uncertainty in \acrshort{irss}]{Label Uncertainty in Implicit Feedback-Based Recommender Systems}
\label{chapter:implicit_learn}

%\victor{There are several changes to this intro, mainly due to the renaming to label uncertainty. I finally got down to a version that I am happy with.}

Learning a model for a \acrlong{irs} generally involves assuming that unobserved interactions are equally relevant to each other and less relevant than observed interactions (see Section \ref{sec:learning_imp}). On the one hand, this general assumption is necessary to furnish the training dataset with negative samples. On the other hand, it is well known that observations are not an oracle for user preferences. For instance, there may be items that are relevant to a user but which the user has not consumed simply because the user is not aware of them, or due to other barriers, such as high price or limited availability. Users may also actually dislike some of the items they interacted with. This uncertainty regarding the true relevance of interactions is what we refer as \emph{label uncertainty}.\footnote{This focuses on label uncertainty, and not prediction uncertainty. For this reason, in this chapter, where we use the word `uncertainty' on its own, it is label uncertainty that we are referring to.}

Noting this disconnect between user preferences and observed interactions, Hu et al.\ use additional information about the observed interactions to signify levels of confidence \cite{hu2008collaborative}. For example, in a song recommender, the confidence may derive from the number of times each song has been played by the user. 

While Hu et al.\ allow observed interactions to have varying levels of confidence, they still treat all the unobserved interactions as equally irrelevant -- in the example above, all unobserved interactions refer to songs that the user has played zero times. However, as explained above, there are several reasons why a user might not interact with a potentially relevant item. Therefore, the assumption of homogeneous irrelevance is not compelling. In practice, there should also be varying levels of confidence associated with each unobserved interaction, including those used as negative samples -- that is, how plausible it is that the negative sample is indeed correctly labeled as negative. Mislabeled instances may impact recommendations drastically, especially for items that are not popular: these are almost always labeled as negatives \cite{saito2020unbiased}.

Label uncertainty is very different from prediction uncertainty, which we have explored in Chapters \ref{chapter:explicit} and \ref{chapter:implicit}. We defined prediction uncertainty as the expected imprecision of a prediction -- in other words, the stochasticity of the model's \emph{output}. Now, label uncertainty is the imprecision of the interaction labeling mechanism, which serves as \emph{input} for the learning of recommendation models.

Note that even though prediction uncertainty and label uncertainty are defined differently, there is a connection between them. It is likely that label uncertainty to which a model is subject will manifest as prediction uncertainty in the model's output.

In Chapter 4, we directly tacked the problem of prediction uncertainty estimation in \acrshort{irss}. Note that, in that chapter, the uncertainty estimators are based upon the output of the relevance model. For example, MF-ENSEMBLE's uncertainty follows from averaging several versions from the same uncertainty estimator. None of the methods in that chapter involve the direct \emph{learning} of an uncertainty estimator\footnote{To clarify, although the methods in Chapter \ref{chapter:implicit} are all doing learning, they are learning $F$ (the relevance model), they are not learning $G$ (the uncertainty model): Eqs.\ \ref{eq:ensemble_unc} and \ref{eq:uncertainty} show that $G$ is computed from $F$.}. %\dgb{See the comment I left in chapter 4 about needing a footnote here to clarify.}\victor{\checkmark} 
This is for a good reason: in \acrshort{irss}, unobserved interactions -- that is, those whose uncertainty we want to predict -- are part of the training set due to the use negative samples. This is paradoxical from the point of view of predicting uncertainty. There is no sense in learning to predict the uncertainty of an interaction that is part of the training set. %\dgb{This isn't quite working. I need to be told (a) why ch4 had no learning and (b) why ch5 only has learning. I think I'm getting half an explanation of each.}\victor{I feel like (a) is done properly, specially with the addition of the footnote. The paragraph to explain (b) is supposed to be the one following this, but I understand it may not be great, I gave it a second go (see below). You can argue that I still don't explain why we only do learning. In fact, I suppose it is possible to somehow estimate label uncertainty empirically, so I want to communicate that we do learning in here, but I don't want to be assertive that this is the only way.}

So, while we observe that the use of negative samples in \acrshort{irss} limits our ability to estimate prediction uncertainty, we find that there is another form of uncertainty in \acrshort{irss} that can be measured and exploited: label uncertainty. In this chapter, we discuss this form of uncertainty in depth.

% \victor{Alternative to the para above:} What makes sense instead is to predict the chance of mislabeling --- in other words, label uncertainty. Label uncertainty can be more easily understood through the lens of user similarity. Assume two users $u$ and $w$ are very similar in the way they interact with the items in the system. If there is a small set of items that $u$ has interacted with but $w$ has not, then we can assume that these items are of potential relevance to $w$\footnote{This is the premise of memory-based \acrshort{rss}.}. Treating them as irrelevant during training is therefore to mislable them. Here, we generalize this reasoning from a prediction point of view: we build models to predict the chance of mislabeling.

% \db{I've left both para above. Personally, despite my reservations about it (which I've now commented out), I prefer the original para.}

We note in advance that this labeling problem is of especial concern for items in the long tail -- that is, the items least interacted with. Because these items have few observed interactions, they are assumed irrelevant to almost every user. Naturally, some or many of these assumed irrelevant interactions will be mislabeled. Therefore, estimating label uncertainty may be particularly useful for these long-tail items. %\db{Is `irrelevant' (used twice) the right word?} \victor{I think it is, because I say they are "assumed irrelevant". I've pointed to the disconnect between observation and relevance many times, so this should be fine IMO.}

% Finally, we highlight that the uncertainty estimation methods in Chapter \ref{chapter:implicit} are \emph{inferred} -- either directly from the data, or from the method used to predict relevance, in which case they measure the stochasticity of the predicted relevance scores. In this respect, they follow the same uncertainty definitions used in Chapter \ref{chapter:explicit}. In contrast, the methods that we will introduce in this chapter \emph{learn} mislabeling uncertainty estimation models. Therefore, even though Chapter \ref{chapter:implicit} and this chapter both tackle \acrshort{irss}, their uncertainty estimates are unrelated and not comparable. \db{Hmm. Not sure about the last sentence. The way it's written makes it sound like the move from inference to learning is what makes them incomparable. But to me it's change of type of uncertainty that makes the incomparable. Maybe I'd move this sentence earlier.}

The only piece of work that we are aware of that models label uncertainty in \acrshort{irs} is by Wang et al.\  \cite{wang2022rethinking}. Their approach, which they call \acrfull{aur}, estimates the label uncertainty associated with each given relevance prediction. They show that, by estimating label uncertainty, it is possible to design an \acrshort{irss} that is (i) more accurate and (ii) less prone to popularity bias.

Building on Wang et al.'s work, in this chapter we propose new methods for \emph{learning} label uncertainty estimators in \acrlong{irss}. We begin with an extensive review of Wang et al.'s \acrshort{aur} (Section \ref{sec:aur}). Then, we present alternatives to \acrshort{aur} (see Sections \ref{sec:icpmf}, \ref{sec:neural_aur}, \ref{sec:gbr} and \ref{sec:hbr}). These alternatives aim to fix some shortcomings that we observe in \acrshort{aur}. Finally, Section \ref{sec:results5} presents the results of an empirical study, comparing \acrshort{aur} against the novel methods that we propose.

% It's worth noting that, in the case of \acrlong{erss} (see Section \ref{sec:learn_exp}), uncertainty estimators are learned only from observed interactions, and then used to predict the uncertainty of unobserved interactions. This is because, contrarily to implicit feedback, explicit feedback provides tangible information regarding the relevance of the rated items.

% \victor{add: This is because the \emph{scoring} step in \acrshort{erss} is based upon a \emph{rating prediction} model, which in turn is based only on observed ratings (see Section \ref{sec:learn_exp}).} \derek{Do you want a sentence about this being possible because of the variation in the observed interactions (ratings)?} \victor{done above.} \db{Your new sentence is not the sentence I was expecting, and doesn't add anything.} \victor{I'm sorry, but then I don't understand what you are expecting, can you explain it? Or even better, suggest a sentence?}

%\victor{I removed a lot of confusing text from here (kept in the thesis file as a comment). I believe I incorporated the important bits of it in my two novel paragraphs (above in magenta).}
% Although this may seem similar to the methods in Chapter \ref{chapter:implicit}, the interpretation of the uncertainties estimated by \acrshort{aur} differs drastically from the the interpretation of uncertainty in the previous chapters of this dissertation. \derek{So, do you want to say how it differs, otherwise this feels incomplete.} It also means that the previous forms of evaluation are no longer suitable and that the uncertainty estimates do not share the same use cases.\footnote{We highlight that the uncertainty estimation methods in Chapter \ref{chapter:implicit} are not affected by the negative sampling because they are \emph{inferred} -- either directly from the data, or from the employed scoring methods -- rather than \emph{learned}. \derek{This points feels under-explained.}} 

%\derek{This chapter is complex. I feel that it needs a much stronger roadmap here. It needs to lay out what we are going to do, step by step, along with (high-level) motivation.} \victor{First round of improvements went fine, but I agree motivation can still be stronger.}

\section{\acrlong{aur}}
\label{sec:aur}

The method proposed in \cite{wang2022rethinking} is the first to tackle the labeling issue in \acrshort{irs} from a predictive perspective. Their proposal, named \acrfull{aur}, is a model-agnostic learning procedure that allows for a vast class of \acrlong{cf} models to be employed for estimation of label uncertainty in \acrshort{irs}.

\subsection{The \acrshort{aur} model}

The \acrshort{aur} method is a point-wise learning procedure (see Section \ref{sec:point}) that assign a numeric value to the implicit feedback data. Identically to Eq.\ \ref{eq:imp}, \acrshort{aur} sets $Y_{ui} = 1$ if user $u$ has interacted with item $i$ and $Y_{ui} = 0$ otherwise. In other words, the observed relevance of the interaction is set to the numeric value of 1 (0) if the interaction is observed (not observed). Then, by assuming $Y_{ui} \sim N(\mu_{ui}, \sigma^2_{ui})$, the likelihood of a given interaction $Y_{ui}$ is given by,

\begin{equation}
\label{eq:likelihood_aur}
    \prob(Y_{ui}|\mu_{ui}, \sigma^2_{ui}) = N(Y_{ui}|\mu_{ui}, \sigma^2_{ui}) = \frac{1}{\sigma_{ui}\sqrt{2\pi}} \exp \left[ - \frac{(\mu_{ui} - Y_{ui})^2}{2\sigma_{ui}^2} \right]
\end{equation}
Interestingly, \acrshort{aur} closely resembles the formulation of $\CPMF$ (Eq.\ \ref{eq:cpmf}). 

The choice of the Gaussian distribution is compelling, due to this distribution being parameterized by a tendency and a dispersion measure, which can be understood as the average relevance of the interaction and its uncertainty. On the other hand, this model disrespects the binary nature of implicit feedback by treating it as a continuous random variable\footnote{These random variables have two possible values according the training set (0 or 1) -- and therefore should be discrete. But, \acrshort{aur} treats them as continuous: at prediction time, the values are not restricted to $\{0, 1\}$.}. Later in this chapter, we will present alternative approaches that respect the binary nature of implicit feedback. %\dgb{Some of the problems I had later concerning $\{0, 1\}$ versus $[0, 1]$ stem from your decision to use the word `continuous'. Can it be deleted? Would that avoid people making the same misinterpretations that I was making?} \victor{I don't think it can. the fact that it is assumed continuous is precisely what makes it wrong. The correct way would be a Bernoulli (discrete) random variable. I tried to counter-act potential misinterpretations with a footnote.}

The negative log-likelihood, used to learn \acrshort{aur}, is,

\begin{equation}
\label{eq:aur_neg_log_likelihood}
    -\log P(Y_{ui}|\mu_{ui}, \sigma^2_{ui}) \propto \frac{(\mu_{ui} - Y_{ui})^2}{2\sigma_{ui}^2} + \log(\sigma_{ui})
\end{equation}

From Eq.\ \ref{eq:aur_neg_log_likelihood}, we note that \acrshort{aur} unravels the interaction relevance estimation problem into two: estimating the distribution's mean and variance, $\mu$ and $\sigma^2$, respectively. A mean prediction model $F_\theta: U \times I \xrightarrow{} \mathcal{R}$ is defined, such that $\mu_{ui} = F_\theta(u, i)$, where $\theta$ is the model's parameter set. Similarly, a variance prediction model $G_\phi: U \times I \xrightarrow{} \mathcal{R}$ is defined, parameterized by $\phi$. From here onwards, for ease of notation, we omit the subscripts $\theta$ and $\phi$. Also, following from Table \ref{tab:notation}, we write $F_{ui} = F(u, i)$ and $G_{ui} = G(u, i)$.

Now, the variance parameter $\sigma^2_{ui}$ must be positive. To ensure this property, Wang et al.\ \cite{wang2022rethinking} use an exponential form $\sigma^2_{ui} = e^{(G_{u i})}/K$, where $K$ is a scale hyper-parameter. In addition, they propose to use a regularizing prior for the variance $G_{ui} \sim N(0, \tau^2)$. Therefore, the posterior negative log-likelihood of the interaction variance is,

\begin{equation}
\label{eq:aur_prior}
    -\log \prob(G_{ui}|Y_{ui}, F_{ui}) \propto \frac{(F_{ui} - Y_{ui})^2}{\frac{2}{K}e^{G_{ui}}} + \frac{1}{2} G_{ui} + \frac{G_{ui}^2}{\tau^2}
\end{equation}

Following from Eq.\ \ref{eq:likelihood_aur}, and combined with the prior variances, after some simplification, the loss function for a batch of training instances $\mathcal{B}$ is given by, 

\begin{equation}
\label{eq:aur_neg_log_likelihood_prior}
    \mathcal{L}(\mathcal{B}|\Theta) = \frac{1}{\#\mathcal{B}} \sum_{(u, i) \in \mathcal{B}} \frac{(F_{ui} - Y_{ui})^2}{e^{G_{u, i}}} + \beta G_{u, i} + \gamma {G_{u, i}^2} + \lambda_\theta ||\theta||
\end{equation}

\noindent where $\Theta = \theta \, \cup \, \phi$ is the set containing all the parameters that compose $\mu_{ui}, \sigma_{ui} \, \forall \, u \in U, i \in I$, and $\beta$, $\gamma$ are hyper-parameters. The additional term $||\theta||$ regularizes the relevance estimator, with regularization strength controlled by $\lambda_\theta$.

\subsection{Learning \acrshort{aur}}
\label{sec:learn_aur}

%\victor{This subsec is full of small text improvements. I didn't keep the original versions, but please check again.}

Eq.\ \ref{eq:aur_neg_log_likelihood_prior} offers a general loss function that allow models $F$ and $G$ to be learned. That is, given two parametric collaborative filtering models $F_\theta$ and $G_\phi$, their parameters $\theta$ and $\phi$ can be learned simultaneously by stochastic gradient descent on the negative log-likelihood. Nevertheless, Wang et al.\ highlight that $F$ and $G$ tend to be strongly correlated: the higher the training error on learning an interaction relevance, the larger the uncertainty will be. In other words, the learner can select for higher uncertainty as a `shortcut' to account for a poor relevance fit. For this reason, learning $F$ and $G$ simultaneously may lead to an inability to learn the best relevance estimator.

Hence, \cite{wang2022rethinking} propose to learn \acrshort{aur}'s $F$ and $G$ sequentially. Their proposal consists of two steps: first, to learn $F_\theta$, the variance of every instance is fixed $\sigma^2_{ui} = 1$; then, to learn the uncertainty estimator $G_\gamma$, with the parameters $\theta$ fixed to the values learned in the first step.

Wang et al.\ compare their sequential learning method to the simultaneous learning approach on two datasets, measuring Recall@50. Interestingly, in many cases, confounding expectations, the simultaneous learning method outperformed the sequential method. Moreover, they found that the optimal learning method depended on the underlying \acrshort{cf} model. In particular, sequential learning was slightly outperformed by simultaneous learning when using a matrix factorization model (Eq.\ \ref{eq:MF}).

Overall, the differences in performance observed between sequential and simultaneous learning in \cite{wang2022rethinking} were small and not decisive. That is, there is no empirical evidence to support the claim that the success (or otherwise) of \acrshort{aur} is related to the sequential learning algorithm introduced by Wang et al. For this reason, in our experiments, we will stick to simultaneous learning in all cases.

\subsection{Top-$n$ recommendations with \acrshort{aur}}
\label{sec:aur_rec}

%\derek{So perhaps here I'd want to use the phrase ``use cases'' again, and refer back to earlier. And, you may disagree, but I'd want to be clearer. I would want to mention top-n rec (ReBR), and uncertainty-aware (UBF and PRR) again, and explain which of these remain applicable and which does not (and why). And, if we are about to define a new method, should it have a name/acronym?} \victor{I rebuild the whole section. I disagree that we should follow the previous chapters too closely, but it doesnt hurt explaining why we shouldnt do the same stuff as before.}

\acrshort{aur} allows us to score candidate items. There is a novel form of ranking that may be used with \acrshort{aur}, referred to as \acrfull{mar}, described below. It includes \acrfull{rebr} (as in Chapter \ref{chapter:implicit}) as a special case. We note that other forms of ranking such as \acrfull{ubf} and \acrfull{prr} do not make sense with \acrshort{aur}. \acrshort{ubf} discards the most uncertain candidates. In the case of \acrshort{aur}, this would result in us discarding candidates (items that are recorded as being unobserved) that are most likely to have been mislabeled, and therefore potentially relevant. %\db{Actually, this logic is questionable. If a negative is mislabeled, then, yes, it is in actuality relevant, but also it was actually observed, and we don't in practice to recommend observed items.} \victor{Although I understand your reasoning, this concern is only relevant if you take this sentence out of context. I think for a second time (there is another similar comment somewhere back) you are disregarding the dynamic between observation and relevance that we discussed in the intro of this chapter.} 
As for \acrshort{prr}, it favours candidates with high probability of being relevant. As in Chapter \ref{chapter:implicit}, it is not applicable to \acrshort{aur} because \acrshort{aur}'s scoring method is relevance-based by default. 

% Next we would rank the items, based on the scores, and select the top-$n$ from the ranked candidates for display to the user.
% There are two forms of ranking that may be used with \acrshort{aur}: \acrfull{rebr}, which is the same as it was in Chapter \ref{chapter:implicit}, and \acrfull{mar}, which is novel to \acrshort{aur}. Below, we discuss both. 

%\db{Sorry if I am going back on myself. But, now, I think you should remove the nest little ReBR subsection.} \victor{Not a problem. I agree.}

% \subsubsection*{\acrfull{rebr}}

% Identically to chapter \ref{chapter:implicit}, we use \acrfull{rebr} to refer to the conventional way of ranking candidates --- based solely on relevance. In the case of \acrshort{aur}, this means that we ignore $G$ and recommend by selecting the top-$n$ items with the highest estimated average relevance $F_{ui}$ to the user from a set of candidate items $C_u \subseteq I$. 

% \db{Assuming you removed the subsection above, then I would also remove the heading, i.e. the next line:}

% \subsubsection*{\acrfull{mar}}

% \db{And, assuming you're still with me, then I would add maybe one more sentence: Let us turn  now to the the \acrshort{aur} scoring function.}

Let us turn now to the \acrshort{aur} scoring function. Given that \acrshort{aur} estimates label uncertainty in addition to relevance, it is possible to combine these to give mislabeling-aware recommendations. The form of ranking that we denote \acrfull{mar}, introduced by Wang et al., takes the uncertainty into account when ranking the candidate items. 

Given a weighting parameter $\lambda \in [0, 1]$ (a hyper-parameter), the method scores user-item interactions according to the following rule,

\begin{align}
\label{eq:mar}
\lambda F_{ui} + (1 - \lambda) \sqrt{G_{ui}}    
\end{align}

The lower the value of $\lambda$, the higher the importance given to the label uncertainty. Note that if $\lambda = 1$, we obtain \acrshort{rebr}.

Using lower values of $\lambda$ to favour uncertain interactions make sense for this form of uncertainty. The set of recommendation candidates for a user is typically drawn from the unobserved interactions for that user. Therefore, the higher their uncertainties, the more plausible it is that the interactions should not be recorded as being unobserved, and therefore are relevant. %\db{See my earlier point about UBF. Is this logic really sound. Yes, they're likely to be relevant, but mislabeled means they were observed and should not have been candidates.} \victor{Again, I think this is fine. But I am happy to discuss it further.}

In addition, in our opening remarks to this chapter, we noted the importance of label uncertainty estimation for long-tail items. According to Wang et al.\, recommending through \acrshort{mar} should increase recommendation accuracy on long-tail items. Because these items are infrequently interacted with, they tend to have higher label uncertainty. Therefore, by decreasing $\lambda$, we expect to favour the recommendation of long-tail items. If so, \acrshort{mar} may also assist to reduce the popularity bias, which is a well-known issue in many recommendation algorithms \cite{saito2020unbiased}.

\section{The choice of $F$ and $G$ in \acrshort{aur}}
\label{sec:model_miss}

%\derek{I'm not sure about the title: very vague. I feel like it introduces alternatives to Wang et al.'s AUR } \victor{you are right, the previous title "Modeling Relevance and Mislabeling Uncertainty" was uninformative. Now it is as clear as it gets. Also, there are minor text adjustments in the section.}

%\derek{So now you're introducing alternatives to Wang et al.'s models. So (a) I would say so. (b) It looks strange that one model has its own subsection (the only subsection in this section), so I would give each of them their own subsection (yes, I know the first subsection will be very short, but you can add a little waffle) (c) I would make sure that the names of the models appear in the titles of the subsections.} \victor{see below.}

As previously stated, AUR offers a general loss function (see Eq.\ \ref{eq:aur_neg_log_likelihood_prior}) that allows for a vast family of choices for the relevance and uncertainty estimators, $F$ and $G$ respectively. In principle, both can be any parameterized \acrshort{cf} model.

In the original paper \cite{wang2022rethinking}, Wang et al.\ employ \acrfull{mf} (Eq.\ \ref{eq:MF}), LightGCN \cite{he2020lightgcn} and VAE \cite{liang2018variational}, motivated, it seems, only by the popularity of these models. As we already mentioned, their experiments included variants in which $F$ and $G$ were learned either sequentially or simultaneously. Surprisingly, it was the simplest model, designated MF-AUR-J, that achieved the highest overall recall in both datasets that they analyzed. MF-AUR-J uses \acrshort{mf} and learns $F$ and $G$ simultaneously (or jointly, J).

In our empirical study (Section \ref{sec:exp5}), we include MF-AUR-J -- simply referred to as \acrshort{aur} there. Here, we propose two new alternatives: Implicit CPMF and Gaussian-MLP. 

\subsection{Implicit CPMF}
\label{sec:icpmf}

The fact that Wang et al.'s simplest model achieved their best results, led us to question how complex $G$ must be in order to accurately estimate label uncertainty. Moreover, we saw a similar picture in Chapter \ref{chapter:explicit}, where $\CPMF$, an architecture that models each user and item uncertainty with a single parameter, proved better than models with many more parameters.

As one much simpler alternative to \acrshort{aur}, and following the same principles, we introduce a new model that we name ICPMF (Implicit CPMF). For this model, we define $G$ as a simple product between a user variance parameter and an item variance parameter. $F$ on the other hand is a traditional \acrshort{mf} model. That is, 

\begin{align}
    \label{eq:ICPMF}
    F_\theta(u, i) &= p_u^T q_i \\
    \label{eq:ICPMF2}
    G_\phi(u, i) &= \sigma_u \sigma_i
\end{align}

where $\sigma_u$ and $\sigma_i$ are learnable scalar parameters for each user and item, respectively, and $p_u$, $q_i$ are user and item latent factors as in Eq.\ \ref{eq:MF}. To ensure positivity of the Gaussian variance, we use the same transformation as AUR, $\sigma^2_{ui} = e^{\sigma_u \sigma_i} / K$. The model parameters -- that is, all $p_u$, $\sigma_u$ for $u \in U$ and $q_i$, $\sigma_i$ for $i \in I$ -- are learned simultaneously through stochastic gradient descent on the negative log-likelihood (Eq.\ \ref{eq:aur_neg_log_likelihood_prior}), exactly as in \acrshort{aur}.

%\derek{So you can make this longer by first emphasising that it still has the same motivation as AUR, and learn F and g, just like they do etc etc etc. But also give details of how $\sigma_u$ and $\sigma_i$ will be learned. It's not clear to me. What's the loss function?} \victor{fixed.}

\subsection{Gaussian-MLP}
\label{sec:neural_aur}

%\derek{So the title is grand but vague. How about: Gaussian-ML: a model that unified relevance and uncertainty?} \victor{Now I follow the same title idea as before. Also, the section is packed with small text improvements (in addition to the major paragraph change (see below).}

%\derek{Again, emphasise that you are introducing a new model, in the same areas, but as  competitor for AUR.}

%So far, we have described the problem of learning an uncertainty-aware \acrshort{rs} as a two level problem \derek{I'm not sure what you are referring to when you say ``uncertainty-aware rs (do you mean everything we've done so far in this dissertation, or do you mean AUR?) and I'm not sure what ``two level'' refers to (is there some reference to AUR's sequential model or is it just that we are learning to predict F and to predict G, or is it something else?)}: the uncertainty estimator must be learned in addition to the relevance estimator. But in this section, we propose an approach that unifies the relevance and uncertainty models using a neural network. \derek{I think this is still basically trying to do the same as AUR, yes? If so, say so. (Maybe we even need a phase for this kind of uncertainty modelling, separate from AUR, which is the name of a family of models for doing this kind of uncertainty.}

%\victor{Replace the previous paragraph with this: 
The two methods introduced so far, MF-AUR-J and ICPMF, tackle relevance and label uncertainty estimation as a two model problem: $F$ and $G$ are two separate models, learned simultaneously. But in this section, we propose a novel approach that unifies $F$ and $G$ using a two-output neural network. This method, that we denote Gaussian-MLP, serves the same purpose as MF-AUR-J and ICPMF, but with the particularity of defining a single neural network-based model that replaces $F$ and $G$.

We define a two-output neural network, as follows. We start with a latent factor model, where users and items are model by $d$-sized embeddings $p_u$ and $q_i$, respectively. Then, similarly to the MLP (Eq.\ \ref{eq:MLP}), we define a feed-forward neural network that takes the interaction's concatenated embeddings. The final layer of this feed-forward neural network will output a pair $\langle \mu_{ui}, \sigma^2_{ui} \rangle$. More precisely, our proposed network comprises a set of feed-forward layers $f_1, \ldots, f_L$ defined as,

\begin{align}
    f_0 &= p_u || q_i \\
    f_l &= \text{ReLU}(W_l f_{l-1}), \quad \text{for} \, l \in 1, \ldots, L-1; \\
    f_L &= \langle w_{L\mu}^t f_{L-1}, g(w_{L\sigma}^t f_{L-1}) \rangle 
\end{align}
\noindent
The network parameters are: $W_l$, the weight matrix for hidden layer $l$, for $l=1, \ldots, L-1$; $w_{L\mu}$, the output layer's weight vector for the relevance output; and $w_{L\sigma}$, the output layer's weight vector for the uncertainty output. For the uncertainty output, activation function $g$ is used to ensure positivity. In this work, we employ a scaled exponential form $g(x) = e^{x}/K$, where $K$ is a scaling hyper-parameter.

Thus, the full set of parameters for the model is $\Theta = \{\{p_u\}_{u \in U}, \{q_i\}_{i \in I}\,$\linebreak $\{W_l\}_{l \in 1,\ldots,L-1}, w_{L\mu}, w_{L\sigma}\}$. It is straightforward to learn these parameters using stochastic gradient descent and same loss function that \acrshort{aur} uses (Eq.\ \ref{eq:aur_neg_log_likelihood_prior}). Note that, by using \acrshort{aur}'s loss function, we assume that $\mu_{ui}$ and $\sigma^2_{ui}$ are the parameters of a Gaussian distribution, as \acrshort{aur} does. That is, $Y_{ui} \sim N(\mu_{ui}, \sigma^2_{ui})$.

%\derek{We need roadmap here and/or at the start of the next section. Tell the reader what we have done and what we are about to do (and why): We have summarized AUR and proposed two alternatives (ICPMF and Gaussian-MLP) (and we will later be using them in our experiments) but now we want to... step back (?), reformulate? what? and why?} \victor{done in the next section.}

\section{Binary Mislabeling-aware Recommenders}
\label{sec:binary}
%\derek{Depending on what you say in the roadmap, you may want to re-consider the title of this section. You may even want to merge it with the next section. I that case, there would be one section maybe called Binary Mislabeling-aware Recommenders, with three subsections (a) rethinking, (b) Gaussian (c) Hierarchical.} \victor{I like your proposal. This section changed a lot, I couldn't keep all commented, but here it goes.}

The models described in the previous two sections (\acrshort{aur}, Implicit CPMF and Gaussian-MLP) all share the same assumption of normality $Y_{ui} \sim N(\mu_{ui}, \sigma^2_{ui})$. However, by making this assumption, these models treat implicit feedback data as continuous -- that is, they are assumed to be numerical values assuming 0 (non-observed interaction) or 1 (observed interaction). %\db{In the previous sentence, am I right to be dissatisfied that you don't use the word "between"??} \victor{There are two ways to look at this: I was taking the POV of the training data, where labels are 0s and 1s. I believe you are thinking in terms of the numerical relevance output, which is indeed between 0 and 1. To be fair, I believe my writing is more precise.} \dgb{Again maybe it's the word `continuous' that misleads me. Would it be terrible to say: these models treat implicit feedback data as numerical values, assuming 0 (non-observed interaction) or 1 (observed interaction).}
In Section \ref{sec:point}, we argued that a more sound approach is to predict each interaction's probability of observation $P(Y_{ui} = 1)$. We can say that the formulation used by these three models use a continuous variable to approximate the (true) binary implicit feedback variable $Y_{ui}$.

In this section, we develop two novel methods for learning mislabeling-aware systems. They are shaped by the same motivation behind \acrshort{aur}. But both methods respect the binary nature of implicit feedback: they tackle the problem of directly estimating the interaction probabilities $P(Y_{ui} = 1)$. Before presenting these two new methods, we present a discussion of the use of implicit feedback that serves several purposes: it deepens our reasoning about mislabeling uncertainty; it critiques \acrshort{aur}; and it motivates the two new methods that follow.

\subsection{Rethinking implicit feedback data}
\label{sec:rethinking}

%\derek{At the risk of repeating myself, this section needs to open with a statement that tells the reader what you are about to do, and why - even if it's a high level statement. And the ``why' needs to relate us back to AUR - because presumably (a) we are still in the same setting as AUR (remember, I said it would be useful to have name for this kind of uncertainty) and (b) it is criticisms of AUR that motivates this rethink.} \victor{I now do it before starting the section. I don't get what you mean by naming this uncertainty, its mislabeling uncertainty all along, do you want an acronym?}

Implicit feedback data consists of a set of observed interactions $R$ between a set of users $U$ and a set of items $I$. The goal of an \acrshort{irs} is to utilize the knowledge in $R$ to infer the relevance of non-observed interactions $\{u, i\} \notin R$. Formally, let $Y_{ui} = 1$ denote the event that user $u$ has interacted with item $i$. Then, traditional \acrshort{irss} (see Section \ref{sec:learning_imp}) reduce the problem of relevance inference to estimating the probability of future interactions $P(Y_{ui} = 1 | {u, i} \in R^-)$.

Now, as previously noted, there is a disconnect between relevance and observations. First of all, it is likely that every user has disliked some items they interacted with. That is, if we let $\zeta_{ui} = 1$ denote the event that user $u$ likes item $i$ -- or, synonymously, item $i$ is relevant to user $u$ ---, then, our reasoning implies that $\prob(\zeta_{ui} = 1 | Y_{ui} = 1) \neq 1$.

%\victor{I agree, a bit too repetitive. I wrote an alternative version to this paragraph (below) -- (a) I understand your point, but I would counter argue that repeated feedback is a particular case of implicit feedback (b) Technically this is all consistent. I don't really mention repeated feedback in chapter 2.} To address this issue, Hu et al.\  assign confidence levels $c_{ui}$ to each observed interaction \cite{hu2008collaborative}. Higher $c_{ui}$ values indicate that the observation is more likely to be relevant. Their proposal is to use $c_{ui} \propto \rho_{ui}$, where $\rho_{ui}$ is some additional information about the interaction relevance. Borrowing their example, in a TV recommender, $\rho_{ui}$ may be the number of times the user has watched a show. Although such confidence estimates help to address the relevance-observation disconnect, it relies on additional data, other than the pure \victor{add: binary} implicit feedback \victor{add: -- in the example case, repeated feedback.} \derek{Hmm. (a) Arguably, the viewing count \emph{is} an example of pure implicit feedback; it's not additional data. (Maybe I shoould have made this criticism earlier too.) (b) How are you defining implicit feedback in Chapter 2: are you requiring it to be binary? In other words, are you being consistent?}

% %\victor{Alternative version to the paragraph above: 
% As we noted in this chapter's introduction, one way to mitigate this issue is to attach confidence levels to each observed interaction \cite{hu2008collaborative}. However, it is not clear how to assign such confidence levels in general. In a TV show recommender, we can use the number of times the user has watched a show \cite{hu2008collaborative} or show much a show a viewers actually plays. But confidence estimates like these rely on additional data, other than pure binary implicit feedback. \db{I still have some discomfort. Do we really need to remind the reader about Hu et al? Could we drop this para?}

By the same reasoning, and of much greater importance, is that $\prob(\zeta_{ui} = 0 | Y_{ui} = 0) \neq 1$. In fact, this is what motivates \acrshort{irss}: discovering those interactions that are labeled as 0 at training time that are the most likely to be relevant. 

%\derek{I remain a little it concerned by the repetition from earlier. I guess the reason you are repeating is because you are introducing some notation. (a) Can you reduce the repetition, e.g. do you need to repeat the stuff about Hu et al? (b) can you explicitly tell the reader that you are about to recap stuff from earlier but add some notation and extend it.} \victor{Fixed the most repetitive part above. Also, the notation is not really used after, so I cut some of it off. I think the last paragraph is fine.}

It is sensible to assume that interactions have a certain probability of being observed, and that this probability is guided by their relevance $\prob(Y_{ui} = 1 | \zeta_{ui})$. 
In addition, the interaction probability also depends on exposure. Let $O_{ui}$ be an exposure indicator, that is $O_{ui} = 1$ if user $u$ has been exposed to item $i$, and $0$ otherwise. One heuristic, which can be found, for example, in Liang et al.'s ExpoMF model \cite{liang2016modeling} is $O_{ui} = 0 \implies Y_{ui} = 0$ -- in other words, users must be aware of an item, prior to consuming it.

Saito et al.\ couple relevance and exposure \cite{saito2020unbiased}. They propose to model the observations as, 

\begin{align}
    \prob(Y_{ui} = 1) &= \prob(\zeta_{ui} = 1) \, \prob(O_{ui} = 1) \label{eq:saito}
\end{align}

According to Eq.\ \ref{eq:saito}, interactions have a certain probability of being observed, and that probability is proportional to both the probability of relevance and the probability of exposure. %\derek{You say proportional, but your equation shows it as equality.}\victor{There are two comments here: First, if I am to be fussy, equals contains proportional, so technically this is accurate anyway. But more importantly, my sentence states that the prob of relevance is proportional to both quantities (yes, it is equal to them multiplied but that is not what I am stating). I think the sentence is fine, but if you want suggest something more understandable, please do.} 
Turning our attention back to \acrshort{aur}, we observe that,

\begin{align}
    F_{ui} &\propto \prob(Y_{ui} = 1) \\
    G_{ui} &\propto \prob(Y_{ui}^{actual} \neq Y_{ui})
\end{align}
where $Y^{actual}$ denotes the actual relevance indicator, that is not subject to mislabeling, and the symbol $\propto$ denotes direct proportionality.
%\derek{So you have some obvious but undefined notation $Y^{actual}$.}

We note that, if $(u, i) \in R^-$, then $\prob(Y_{ui}^{actual} \neq Y_{ui})$ indicates how likely it is that the interaction is mislabeled as negative. If this probability is high, then it is plausible that the user is unaware of the item  -- that is, $O_{ui} = 0$ --, because if the user were aware of an item that was relevant to them, they would very likely interact with it.\footnote{We say plausible because mislabeling may have causes other than lack of awareness.} Hence, for $(u, i) \in R^-$, we suppose that $G_{ui} \propto \prob(O_{ui} = 0)$. Reordering Eq.\  \ref{eq:saito}, we have,

\begin{align}
\label{eq:rel_exp}
    \prob(\zeta_{ui} = 1) &= \frac{\prob(Y_{ui} = 1)}{\prob(O_{ui} = 1)}
\end{align}
%\derek{Same comment: proportional or equal to?}
%\victor{Same answer as well.}

%Now, we can compare Eq.\ \ref{eq:rel_exp} to AUR's scoring function $r_{ui} = \lambda F_{ui} + (1 - \lambda) \sqrt{G_{ui}}$. \derek{Well, this is its scoring function for a particular use case. If you think it is the main (only?) use case, then that's something to be clearer about when you introduced it earlier. (Also, as I said it might help if it had an acronym.)} Because $F_{ui} \propto \prob(Y_{ui} = 1)$ and following our assumption $G_{ui} \propto \prob(O_{ui} = 0)$, we conclude that AUR's scores, for $(u, i) \in R^-$, follow the same relevance modeling principle as proposed in Eq.\ \ref{eq:saito}. \derek{Am I missing something? In this use case, AUR has a linear combination of F and G, so, we replace F and G by these two probs, but how then does that give us Eq 5.11? }

%\victor{Alternative to the paragraph above:
Now, we can compare Eq.\ \ref{eq:rel_exp} to Eq.\ \ref{eq:mar} -- $\lambda F_{ui} + (1 - \lambda) \sqrt{G_{ui}}$ --, which is the scoring function that \acrshort{aur} uses for \acrfull{mar}. Because $F_{ui} \propto \prob(Y_{ui} = 1)$ and following our assumption that $G_{ui} \propto \prob(O_{ui} = 0)$, we observe that there is a strong connection between \acrshort{mar} and the modeling principle proposed by Saito et al.\ (Eq. \ref{eq:saito}). 
% That is, the relevance probability and the \acrshort{mar} ranking scores grow with the like probability $\zeta_{ui}$ (and thus, with $F_{ui}$), and with the observation probability $O_{ui}$ (and thus with $G_{ui}$). \db{I'm still struggling with your reasoning her.}
The relevance probability $\prob(Y_{ui} = 1)$ grows with the probability that the item is liked, $P(\zeta_{ui} = 1)$. Hence, so does MAR's score, because $F_{ui}$ is directly proportional to $\prob(Y_{ui} = 1)$. Similarly, the relevance probability $\prob(Y_{ui} = 1)$ also grows with the observation probability $\prob(O_{ui} = 1)$. Therefore, so does MAR's score, because $G_{ui}$ is \emph{inversely} proportional to $\prob(O_{ui} = 1)$. %\dgb{This new para does seem like it may be helpful. But (a) it needs to be integrated into the previous one. (b) I'm still not seeing why relevance prob grows with observation prob.} \victor{(a) Yeah, I agree they should be the same para, I split them just cause there was a comment in between. (b) To be honest, I ran out of ideas for this, I modified it multiple times already, but I feel like my wording reflects the conclusion of the comparison between eq 5.11 (or 5.14 -- they are the same) and MAR in a relatively clear way. If you still disagree, we can discuss.}

%Notwithstanding the soundness of AUR, in particular due to its similarity to the observation model from Eq.\ \ref{eq:saito}, it has the flaw of treating implicit feedback as numerical data in $\{0, 1\}$. For this reason, $F_{ui}$ is not an estimate for $P(Y_{ui} = 1)$, but instead a correlated quantity. In the following two sections, we formulate two novel methods that respect the binary nature of the feedback data, and offer estimators for $P(Y_{ui} = 1)$, while estimating mislabeling uncertainty similarly to AUR. \derek{I'm uneasy with this para, maybe because it does too many things and therefore doesn't do them explicitly enough (a) we conclude that AUR is sound in some fashion. (b) Separately, we have a criticism about it being continuous. Why does that matter. Do we think that predicting a correlated quantity will be worse? (c) We want to see if we can come up with models that avoid this criticism. What do we hope this will achieve?} \victor{Replace this paragraph with the following two. They should fix all three comments.}

We conclude that \acrshort{mar} is sound, in particular due to its similarity to the observation model from Eq.\ \ref{eq:saito}. However, \acrshort{aur} (the method that motivates \acrshort{mar}) has the flaw of treating implicit feedback as numerical data in $\{0, 1\}$. %\db{If it's continuous, don't you mean $[0, 1]$?} \victor{Same as the comment from 2 pages ago.} 
For this reason, $F_{ui}$ is not an estimate for $P(Y_{ui} = 1)$, but instead a correlated quantity. It is not clear whether this flaw compromises recommendation accuracy.

In the next two sections, we propose two novel models that respect the binary nature of the implicit feedback and explicitly estimate $P(Y_{ui} = 1)$. Our goal is to asses whether these alternative methods, which do not share the aforementioned flaw with \acrshort{aur}, will achieve higher recommendation accuracy.

\subsection{Gaussian binary ranking}
\label{sec:gbr}

A standard principle for learning an \acrshort{rs} model for implicit feedback data is to treat the problem as one that involves binary classification and to predict, for each interaction, the probability that it is relevant; we see this, for example, in approaches that use \acrlong{bce} (Eq.\ \ref{eq:bce_loss}). Inspired by this, we introduce \acrfull{gbr}: a classification oriented method to estimate probabilities of relevance and the uncertainties associated with them. 

Let $r_{ui} \sim \norm (\mu_{ui}, \sigma^2_{ui})$ be a Gaussian variable that models the relevance of item $i \in I$ to user $u \in U$. The probability that the user $u$ interacts with item $i$ is modeled as,

\begin{equation}
\label{eq:gbr}
    \prob (Y_{ui} = 1) = \prob (r_{ui} > 0) = \prob (\norm (\mu_{ui}, \sigma^2_{ui}) > 0) = 1 - \Phi \left( \frac{-\mu_{ui}}{\sqrt{\sigma^2_{ui}}}\right) 
\end{equation}

\noindent where $\Phi$ is the standard Gaussian cumulative distribution function. Naturally, the probability for a pair to be unobserved is,

\begin{equation}
    \prob (Y_{ui} = 0) = \prob (r_{ui} \leq 0) = \Phi \left( \frac{-\mu_{ui}}{\sqrt{\sigma^2_{ui}}}\right) 
\end{equation}

Therefore, the likelihood of a given interaction is given by,

\begin{equation}
\label{eq:likelihood_gbr}
    \prob(Y_{ui}|\mu_{ui}, \sigma^2_{ui}) = \prob(Y_{ui} = 1)^{\delta_{ui}} \prob(Y_{ui} = 0)^{1 - \delta_{ui}}
\end{equation}

\noindent where $\delta_{ui} = 1$ if $u$ has interacted with $i$, and $0$ otherwise. Notice that, although we employ the notation $Y_{ui} = 1$ and $Y_{ui} = 0$, these values are only labels, indicating whether or not the user-item interaction was observed, and not numeric values, as they were in the case of \acrshort{aur}. 

The negative log-likelihood for one interaction is,

\begin{align}
\label{eq:gbr_neg_log_likelihood}
    -\log \prob(Y_{ui}|\mu_{ui}, \sigma^2_{ui}) &= - {\delta_{ui}} \log\left(1 - \Phi \left( \frac{-\mu_{ui}}{\sqrt{\sigma^2_{ui}}}\right)\right) - (1 - \delta_{ui}) \log \Phi\left( \frac{-\mu_{ui}}{\sqrt{\sigma^2_{ui}}}\right)
\end{align}

Now, we borrow from \acrshort{aur}. We define two models $F_\theta(u, i)$ and $G_\phi(u, i)$ such that $\mu_{ui} = F_{ui}$ and $\sigma^2_{ui} = e^{G_{ui}}/K$. Additionally, we employ the same regularizing prior $G_{ui} \sim N(0, \tau^2)$. It follows that the posterior negative log-likelihood of the variance is,

\begin{align}
    -\log \prob(G_{ui}|Y_{ui}, F_{ui}) \propto & \,- {\delta_{ui}} \log\left(1 - \Phi \left( \frac{-\mu_{ui}}{\sqrt{e^{G_{ui}}/K}}\right)\right) - \notag \\ 
    & \,\,(1 - \delta_{ui}) \log \Phi\left( \frac{-\mu_{ui}}{\sqrt{e^{G_{ui}}/K}}\right) + \frac{G_{ui}}{\tau^2} \label{eq:gbr_prior}
\end{align}

%\acrshort{gbr} is, arguably, more sound than AUR since it handles implicit feedback more authentically. But, as soon as we begin our experimentation with it, we noticed an identifiability problem. Because $\prob(Y_{ui} = 0) = \prob (r_{ui} \leq 0)$, then $\mu_{ui} = 0 \implies \prob (Y_{ui} = 0) = 0.5$ regardless of the $\sigma^2_{ui}$ value. \derek{You will have explained the previous sentence to me, but seeing it here, I cannot reconstruct the reasoning. Can you be more explicit?} During learning, this causes interactions with average relevance in the vicinity of 0 to have a random variance. \derek{Same}

%\victor{Alternative to the paragraph above: 
\acrshort{gbr} is, arguably, more sound than \acrshort{aur} since it handles implicit feedback more authentically. But, as soon as we began our experimentation with it, we noticed an identifiability problem. Note that, by construction, we have $\prob(Y_{ui} = 0) = \prob(r_{ui} \leq 0)$. Note also that the probability density function of $r_{ui}$ is symmetrical around its average $\mu_{ui}$.\footnote{This is a property of the Gaussian distribution.} As a consequence, if $\mu_{ui} = 0$ we have,

\begin{align}
    \prob(Y_{ui} = 0) = \prob (r_{ui} \leq 0) = \prob (r_{ui} > 0) = \prob(Y_{ui} = 1)
\end{align}

\noindent where the second equality follows from the symmetry around 0 (the assumed value of $\mu_{ui}$). Note that, the equation above holds regardless of the $\sigma^2_{ui}$ value. This causes $\sigma^2_{ui}$ to be unidentifiable in the case of $\mu_{ui} = 0$. In practice, during learning, this causes the uncertainty of interactions with average relevance in the vicinity of 0 to not converge.

For this reason \acrshort{gbr} is not able to provide proper label uncertainty estimates for every instance. We chose to leave the theoretical formulation of the method here, since it was an important step in the development of our next model (Section \ref{sec:hbr}). But we did not include it in our experiments. %\derek{OK. Is there something more positive we can write? (a) Maybe we can write something about the remedy to this problem? (b) Maybe we can say that, while it didn't give a practical solution, it gave us something. What? Maybe it helped us formulate the next model, HBR?} \victor{add: As a silver lining, the development of \acrshort{gbr} gave us ideas on how to formulate another method that serves the same purpose, but does not suffer from the same identifiability issue.} \victor{I added this sentence to end on a better note, but I don't love it. Mainly because it gives the impression that we will parallel HBR with GBR, which we do only up to some extent. Answering (a): I don't think this may be remedied.}

\subsection{Hierarchical binary ranking}
\label{sec:hbr}

% %\derek{Again we need some roadmap. We're going to do another model, but why? how will it differ from AUR and from GBR?} \victor{add: 
% Unfortunately, the identifiability problem found in GBR makes it impractical for estimating label uncertainty. Nevertheless, we believe the method has some interesting ideas. First, it defines Gaussian latent relevance variables $r_{ui}$. The choice for the Gaussian distribution is great, because it is parameterized by the two quantities of interest to us -- the average relevance and the uncertainty. In addition, \acrshort{gbr} provides a straightforward way to connect such latent continuous relevances to the implicit feedback (Eq.\ \ref{eq:hbr}). Although, we remark that the chosen link is also responsible for the identifiability issue.

% As follows, we propose \acrfull{hbr}, an alternative hierarchical Bayesian model that follows a setup similar to \acrshort{gbr}. Although, it employs a more sophisticated form of linking between the latent relevances and the binary implicit feedback that turn out to not suffer from the same problem than \acrshort{gbr}.

In this section, we propose \acrfull{hbr}, a hierarchical Bayesian model as an alternative to \acrshort{gbr}. Like \acrshort{gbr}, it defines Gaussian latent relevance variables $r_{ui}$. The choice of the Gaussian distribution seems appropriate, because it is parameterized by the two quantities of interest to us -- the average relevance and the uncertainty.
Where \acrshort{hbr} differs from \acrshort{gbr} is in the way it connects the latent continuous relevances to the implicit feedback. \acrshort{gbr} uses a straightforward connection (Eq.\ \ref{eq:gbr}), but this is also responsible for \acrshort{gbr}'s identifiability issue. \acrshort{hbr} employs a more sophisticated form of connection (Eq.\ \ref{eq:hbr}) that turns out to not suffer from the identifiability problem.

%\derek{I notice we have a subsection coming up much later: Inference and ranking with HBR. It's not good to have a section with a lone subsection. So, I suggest you try to use multiple subsections.Maybe they can be similar to the ones in the AUR section?} \victor{Good call.}

%\derek{Do you want to say (somewhere) why this is hierarchical, and why it is Bayesian? Also do you want to include the influence diagram (if that's what it is called?)} \victor{Done (in many different places in the following subsec).}

%\derek{Do you want to say something (somewhere) about Bayesian inference in general - methods for doing it. Kind of a para of lit review?} \victor{I am not sure. Because this is the only method relying on Bayesian inference, I believe we should introduce stuff on the fly. This should work well with proper writing. I think current version is a lot more clear about a lot of the Bayesian stuff.}

\subsubsection{The \acrshort{hbr} model}

Let $r_{ui} \sim \mathcal{N}(\mu_{ui}, \sigma^2_{ui})$ be a latent relevance score for user $u$'s interaction with item $i$, %\derek{Notation. I am uneasy that $r_{ui}$ has played two roles in the dissertation. It has been something the user gives a numeric rating (explicit), or a binary rating (implicit), but it has also been a latent score, to be estimated.} 
and let $Y_{ui} = 1$ denote the event that user $u$ has interacted with item $i$. We propose a novel method, named \acrfull{hbr}, that models such probabilities using the standard logistic function, %\victor{I understand your concern. Although, I have two arguments: First, the notation don't overlap within the same chapter. Second, I kind of exploit the notation abuse in my thoughts. After all, relevance is a "parallel" to ratings in the implicit case. Thus, I kind of like this abuse.}

\begin{align}
\label{eq:hbr}
    \prob_{ui} = \prob (Y_{ui} = 1|r_{ui}) = \frac{1}{1 + e^{r_{ui}}} 
\end{align}

In other words, we say,

\begin{align*}
    Y_{ui}|r_{ui} \sim \text{Bernoulli}\left(\frac{1}{1 + e^{r_{ui}}}\right) \\
\end{align*}

Our proposal casts the scoring problem into the task of estimating the parameters of each interaction's latent relevance distribution, similarly to \acrshort{aur}. The difference with \acrshort{aur}, as previously stated, is that we now offer an explicit estimator for $P(Y_{ui} = 1)$, acknowledging the binary nature of the implicit feedback. 

We note that $r_{ui}$ is a random variable, and therefore Eq.\ \ref{eq:hbr} defines a hierarchical Bayesian model \cite{gelman1995bayesian}. That is, there are two random variables (for each interaction) $Y_{ui}$ and $r_{ui}$ that are connected through a dependency structure. More precisely, their joint distribution follows from the Bayes theorem,%} \derek{See above comments about hierarchical. Say what a hierarchical model is, and why this is one?}

\begin{equation}
    \prob(Y_{ui} \cap r_{ui}) = \prob(Y_{ui} | r_{ui}) \prob(r_{ui})
\end{equation}

%Now, these parameters $\mu_{ui}, \sigma^2_{ui}$ can be estimated in many ways. \derek{There's a kind of a step missing because you're now mentioning parameters but they have not been properly introduced here.} In particular, we can follow AUR's approach and define a mean prediction model $F_\theta$ and a variance prediction model $G_\phi$. We will further illustrate a model defining $F$ and $G$ as in ICPMF (Eq.\ \ref{eq:ICPMF}). \derek{The previous sentence is confusing. I think you mean: There are many ways of defining models F and G. For concreteness, we will show one approach which defines them similar to the way they were defined in ICPMF. But why this choice?} \derek{To be honest, I don't see how what follows is related to ICPMF. Something to do with having a $\sigma_u$ and $\sigma_i$. Be clearer. Go slower.} Although, we reinforce that this is only one of many possible choices. We assume normality for all the parameters in the model. It follows the hierarchical model,

%\victor{alternative to the paragraph above: 
Now, as in all the previous methods in the chapter, our goal is to parameterize $\mu_{ui}$ and $\sigma^2_{ui}$, the parameters of the (Gaussian) latent relevances $r_{ui}$, and subsequently learn the parameters from the implicit feedback data. %\db{I still feel that you're talking about parameterizing $mu$ and $\sigma^2$ but they do not so far feature in HBR, e.g. not in Eq 5.21 or 5.22. Hence I still feel there is a step missing.} \victor{Maybe it's because $\mu$ and $\sigma$ appear explicitly only on the first phrase of this subsec. I added one phrase in this paragraph to remind the reader that $\mu$ and $\sigma$ do appear on the equations implicitly (through $r_{ui}$).} \dgb{Can we not just say: our goal is to learn $\mu_{ui}$ and $\sigma^2_{ui}$, the parameters of the (Gaussian) latent relevances $r_{ui}$, from the implicit feedback data.}
In particular, we can follow the same approach as before and define a mean prediction model $F_\theta$ and a variance prediction model $G_\phi$. The parameters $\theta$ and $\phi$ must now be treated as Bayesian parameters -- that is, they must follow some prior probabilistic distribution. This is because \acrshort{hbr} relies on approximate Bayesian inference, for reasons that will become clear shortly.

Because we must define a prior distribution for the parameters, it is cumbersome to formalize \acrshort{hbr} for general parameterizations of $F$ and $G$. For this reason, we opt to show the formal development with one particular parameterization that closely resembles ICPMF. More precisely, we will follow the same parameterization for $F$ and $G$ as ICPMF (Eqs.\ \ref{eq:ICPMF} and \ref{eq:ICPMF2}). This choice has two motivations: first, the simplicity of ICPMF's uncertainty estimator translates into easier prior definitions, and secondly, ICPMF has shown good performance in our experiments, as will be shown in Section \ref{sec:results5}. We emphasise, however, that this parameterization is only one of many possible choices. 

We assume normality for all the parameters in the model. Then the hierarchical model is as follows,

\begin{align}
    \text{Stage I (Observations):} &\, Y_{ui}|r_{ui} \sim \text{Bernoulli}\left(\frac{1}{1 + e^{r_{ui}}}\right) \notag \\
    \text{Stage II (Latent relevances):} &\, r_{ui} | p_u, q_i, \sigma_u, \sigma_i  \sim \mathcal{N}(p_u^T q_i, e^{\sigma_u \sigma_i}) \notag \\
    \text{Stage III (Priors):} &\, p_u \sim \mathcal{N}^d(0, \alpha_u I_{d \times d});
    q_i \sim \mathcal{N}^d(0, \alpha_i I_{d \times d}); \notag \\
    &\sigma_u \sim \mathcal{N}(0, \alpha_{\sigma u}); \sigma_i \sim \mathcal{N}(0, \alpha_{\sigma i})) \notag
\end{align}

\noindent where $\mathcal{N}^d$ denotes a $d$-dimensional multivariate Gaussian.\footnote{We abuse the notation by writing 0 as the multivariate distributions' average. In fact, the averages are $d$-dimensional zero vectors.} 

We let $Y = [Y_{ui}]$ denote the implicit feedback matrix, $Z = [r_{ui}]$ denote the latent interaction relevances, $P^{\#U \times d}$, $Q^{\#I \times d}$ denote the user and item embeddings matrices, respectively, and $\sigma_U^{\#U \times 1}$, $\sigma_I^{\#I \times 1}$ denote the variance parameter vectors for users and items, respectively. For ease of notation, we write the latent parameter set $\Theta = \{P, Q, \sigma_U, \sigma_I\}$. 

The joint probability distribution for the observed interactions is,

\begin{align}
    \prob(Y, Z, \Theta) &= \prob(Y, Z| \Theta) \, \prob(\Theta) \notag \\ 
                        &= \prob(Y | Z) \, \prob(Z | \Theta) \, \prob(\Theta) \notag \\
                        &= \prob(Y | Z) \, \prob(Z | \Theta) \, \prob(P) \, \prob(Q) \, \prob(\sigma_U) \, \prob(\sigma_I) \label{eq:joint}
\end{align}

Eq.\ \ref{eq:joint} makes evident the hierarchical structure of the model. The responses $Y$ depend on the latent parameters in $\Theta$ only through the latent relevances $Z$. Figure \ref{fig:hbr} illustrates \acrshort{hbr} as a probabilistic graphical model in the style of Chapter 8 of  \cite{bishop2006pattern}.

\begin{figure}
    \centering
    \includegraphics{img/Chap 5/HBR graph.pdf}
    \caption{HBR's probabilistic graphical model. The white (non-filled) circles indicate non-observed (latent) random variables, while the grey (filled) circle indicates the observed variable -- in this case, the binary implicit feedback. The arrows indicate probabilistic dependencies.}
    \label{fig:hbr}
\end{figure}

\subsubsection{Learning \acrshort{hbr}}

The posterior distribution of the latent variables -- that is, their distribution given the implicit feedback $Y$ -- can be expressed as,

\begin{align}
    \prob(\Theta, Z | Y) = \frac{\prob(Y, Z, \Theta)}{\int \prob(Y, Z, \Theta) d\Theta dZ}
\end{align}

However, working with this posterior distribution is infeasible, because the denominator (i.e.\ the evidence) is intractable due to the multidimensional integral involved. For this reason, we must resort to using approximate inference.

The tools for approximate Bayesian inference problems such as this one revolve around Markov Chain Monte Carlo methods (MCMC) \cite{geyer1992practical} and Stochastic Variational Inference (SVI) \cite{hoffman2013stochastic}. In fact, MCMC has been previously explored for \acrlong{erss} \cite{salakhutdinov2008bayesian}. In particular, CPMF has a Bayesian extension based on MCMC \cite{wang2018confidence}. However, it is unclear how a similar MCMC algorithm could be employed in the case of \acrshort{irss} due to the necessity of negative sampling.

For this reason, we use SVI. Unlike MCMC, which is a sampling method, SVI is an optimization method. For this reason, the learning can be performed in batches, which can include randomly-selected negative samples, as in Algorithm \ref{alg:point}.

%SVI consists in searching for a variational distribution $v(Z, \Theta | \lambda)$, parameterized by $\lambda$, and minimizing the Kullback-Leiber (KL) divergence between $q(Z, \Theta | \lambda)$ and the true posterior $\prob(Z, \Theta | Y)$. \derek{Notation! (a) Is it good to be using $q$? (b) What is this lambda? Is it the one from 5.5? Or is the one that you only define after 5.24? Not helpful to have two different definitions of lambda in the same chapter.} More specifically, the objective is to maximize the evidence lower bound (ELBO), \victor{Interesting. I tunneled visioned on the usual notation from variational inference books and I never noticed that it is very confusing in the thesis. I rewrote all of this part (not only the commented bits). Remove this paragraph after review.}

SVI consists in searching for a variational distribution $v(Z, \Theta | \eta)$ that serves as an approximation to the true posterior $\prob(\Theta, Z | Y)$. In practice we pick a family of distributions parameterized by $\eta$, that are called variational parameters -- that is, for each value of $\eta$, we recover a different distribution for the chosen family. Then, we choose $\eta$ such that $v(Z, \Theta | \eta)$ is as close as possible to the true posterior $\prob(\Theta, Z | Y)$. This distribution similarity is measured according to the Kullback-Leiber (KL) divergence. It can be shown that minimizing the KL divergence is equivalent to minimizing the Evidence Lower Bound (ELBO). In fact, the ELBO is equal to the negative KL divergence up to an additive constant, and is defined as,

\begin{align}
\label{eq:elbo}
    \text{ELBO} \coloneqq \mathbb{E}_{v(Z, \Theta | \eta)}[\log \prob(\Theta, Z | Y) - \log v(Z, \Theta | \eta)]
\end{align}

%\derek{Again moving too quickly. We wanted KL divergence. Is that what we have here?} \victor{I am trying to go slower now, but I disagree with the KL divergence point. Most of my references mention KL divergence, then mention its relation with the ELBO and never come back to it. I think adding the formal definition of KL divergence is a bit of math waffling.}

Typically, the true posterior is very complex and is not contained in the variational family. Therefore, the quality of the approximated solution through SVI depends on the choice of the variation family over which we search for the optimal $v(Z, \Theta | \eta)$. %\derek{I don't know what a variational family is.} \victor{Now it should be clear because of the previous improvements.}

For \acrshort{hbr}, we consider variational distributions that factorize similarly to the joint distribution Eq.\ \ref{eq:joint}. More precisely, we assume,

\begin{align}
    v(Z, \Theta | \eta) &= v_Z(Z | \eta) v_\Theta(\Theta | \eta) \notag \\
    &= v_Z(Z | \eta) v_P(P | \eta) \, v_Q(Q | \eta) \, v_{\sigma_U}(\sigma_U | \eta) \, v_{\sigma_I}(\sigma_I | \eta)
\end{align}

\noindent where $v_\star$ denotes the variational distribution of the variable matrix $\star$. The choice for this family of factorizable distributions is in line with the construction of our priors, that lead to an also factorizable joint distribution (see Eq.\ \ref{eq:joint}).

%\victor{The following two paragraphs are novel, and are replacements to the one that come after. They should give a much better explanation to some of the obscure reasoning that you pointed to.}
At this point, it is useful to recap that, although we employ a full Bayesian treatment of our parameters -- that is, $\Theta$ is a set of random variables rather than a set of deterministic parameters --, our goal is not to study the uncertainty around the parameters, but instead only the uncertainty of the relevances $r_{ui}$ and probabilities of relevance $Y_{ui}$. For this reason, we can use point mass distributions (or delta distributions) for all $v_P, v_Q, v_{\sigma_I}, v_{\sigma_I}$. A point mass distribution is such that all the probability mass is concentrated around a single value, which is the distribution's parameter.

Notice that by choosing point mass distributions, the number of variational parameters is equal to the dimensions of the latent matrix. To give a concrete example, if $P$ is a $10 \times 10$ matrix -- that is, the dataset has 10 users and we employ latent embeddings of size 10 -- then $v_P$ is a delta distribution parameterized also by a $10 \times 10$ parameters matrix. In this example, the learned variational parameters of the delta distribution will be deterministic estimates for $P$. In the next section, we will see that the choice of point mass distributions also greatly reduces prediction time.

%\victor{remove this paragraph}
%Now, we assume $q(\Theta | \lambda)$ is a point mass distribution (or delta distribution). \derek{Another assumption. What does it mean? What justifies it?} That is, it assigns probability one to its parameter value $\lambda$ \derek{But lambda is a set!} and probability zero to every other value in $\mathbb{R}$\derek{I'm lost. Maybe again the problem that R and r are playing different roles in the dissertation.}. In practice, this means that the optimal variational parameters $\lambda^*$ serve as a point estimate (MAP) for $\Theta$. That is, the posterior distribution of $\Theta$ is reduced to a single point (its maximum), which are the chosen estimates for $\Theta$. \derek{The steps of your reasoning by now are obscure to me.}

Learning the variational parameters $\eta$ follows through stochastic gradient decent. On each step, the ELBO term (Eq.\ \ref{eq:elbo}) is optimized for a data batch. We emphasise that each data batch contains positive and negative interactions, as per Algorithm \ref{alg:point}. 

% The following part was complicated, and to be fair, unnecessary, so I removed it.

% The use of point mass probabilities means that the maximization objective from Eq.\ \ref{eq:elbo} is reduced to, \derek{If you say so! I don't know why. 5.23 and 5.25 don't even look like the same kind of thing}

% \begin{align}
%     \lambda^* &= \arg_\lambda \max \mathbb{E}_{q(Z, \Theta | \lambda)}[\log \prob(Y, Z, \Theta)  - \log q(Z, \Theta | \lambda)] \notag \\ 
%               &= \arg_\lambda \max \mathbb{E}_{q(\Theta | \lambda)}[\log \prob(Y, Z, \Theta) - \log q(Z|\Theta)]
% \end{align}

Our implementation of \acrshort{hbr} takes advantage of Automatic Differentiation Variational Inference (ADVI) \cite{kucukelbir2017automatic}, and is implemented in Pyro \cite{bingham2018pyro}.

\subsubsection{Top-$n$ recommendations with \acrshort{hbr}}

Our choice for point mass variational distributions leads to point estimates for the model parameters $p_u$, $q_i$, $\sigma_u$, $\sigma_i$. Therefore, even though \acrshort{hbr} is based on variational inference, we do not need to do any posterior inference to make recommendations. Instead, we have a straightforward way to get point estimates for the average relevance $F_{ui} = \mu_{ui}$ and uncertainty $G_{ui} = \sigma^2_{ui}$. We simply follow Equations \ref{eq:ICPMF} and \ref{eq:ICPMF2}. Now, having $\mu_{ui}$ and $\sigma^2_{ui}$, we may follow, at least, the following two ranking criteria. %\db{Note that you say ``three''. I have a comment about this below.}

%\db{Sorry, again,. if this is my fault, but I suggest deleting the next little subsubsection on ReBR.} \victor{Here the addition of ReBR was a bit more committal unfortunately. I tried to mirror the previous chapter and it ended up messing a potentially finer version that I had previously. But let me whine less and present a better version.}

% \subsubsection*{Relevance-based ranking}

% Identically to \acrshort{aur}'s \acrshort{rebr}, we may ignore $G_{ui}$ and perform ranking solely based on the average relevance $F_{ui}$. That is, \acrshort{rebr} recommends the top-$n$ items with the highest estimated average relevance $F_{ui}$ to the user from a set of candidate items $C_u \subseteq I$.

%\db{Now I have a problem. Above, you said three criteria. I presume you mean MAR, HBR-R and HBR-Y. This makes me expect three sets of results in the experiments. But, in fact, there is only HBR-R and HBR-Y.  So I'm confused. If there are truly 3, then I'd expect another set of results.} \victor{Now that we wrote ReBR as a special case of MAR, I can more confidently write these as only two cases.}

\subsubsection*{Ranking based on $r_{ui}$}

Our first possible ranking criterion follows directly from \acrshort{mar}. Because HBR offers estimates for $\mu_{ui}$ and $\sigma^2_{ui}$ ($F_{ui}$ and $G_{ui}$), it can borrow MAR's ranking criterion from Eq.\ \ref{eq:mar}. The reasoning is analogous to \acrshort{aur}: the value of $\lambda$ controls (inversely) the importance given to label uncertainty -- that is, the lower is $\lambda$, the higher is the importance given to $G_{ui}$, and therefore more emphasis is given to recommending potentially relevant items that are mislabeled during training. To facilitate referencing, we will use HBR-R to refer to the HBR model with MAR's ranking criterion\footnote{We use the suffix R to reinforce that ranking is based on the latent relevances $r_{ui}$.}. Again, we note that \acrshort{rebr} is the special MAR case where $\lambda = 0$.

\subsubsection*{Ranking based on $\prob_{ui}$}
\label{sec:hbr_rank}

Above, we discussed ranking through the latent relevance ($r_{ui}$) parameters ($\mu_{ui}, \sigma^2_{ui}$), intentionally leaving out the interaction probability $\prob_{ui}$. Now, since we are aiming to treat implicit feedback as binary, let us examine the meaning of $\mu_{ui}$ and $\sigma^2_{ui}$ in relation to $\prob_{ui}$.

By construction, we have $r_{ui} \sim \mathcal{N}(\mu_{ui}, \sigma^2_{ui})$, and as a consequence, $\prob_{ui} = \frac{1}{1+e^{r_{ui}}} \sim \text{Logit-Normal}(\mu_{ui}, \sigma^2_{ui})$ \cite{atchison1980logistic}. Therefore, we can use some known results about the logit-normal distribution.

First, let $s$ denote the standard logistic function. Then, $s(\mu_{ui})$ is the median of the interaction probability, $s(\mu_{ui}) = \text{Med}(\prob_{ui})$ \cite{holmes2022moments}. Therefore, $\mu_{ui}$ on its own is a robust point estimate for the interaction probability. Unfortunately, the logit-normal distribution has no analytical solution for any measure of dispersion, such as variance. But, as shown in \cite{frederic2008two}, there is a positive correlation between the variance of $\prob_{ui}$ and $\sigma^2_{ui}$.

Second, there are some known approximations for the mean and variance of $\prob_{ui}$.  Again, let $s$ denote the standard logistic function. Then, following from \cite{daunizeau2017semi}, one possible set of approximations is,

\begin{align}
\label{eq:approx_logitnormal}
    E[\prob_{ui}] &= s\left(\frac{\mu_{ui}}{\sqrt{1+\frac{3}{\pi^2}\sigma^2_{ui}}}\right) \\
    \text{Var}[\prob_{ui}] &= s\left(\frac{\mu_{ui}}{\sqrt{1+\frac{3}{\pi^2}\sigma^2_{ui}}}\right) \left(1-s\left(\frac{\mu_{ui}}{\sqrt{1+\frac{3}{\pi^2}\sigma^2_{ui}}}\right)\right)s\left(1-\frac{1}{\sqrt{1+\frac{3}{\pi^2}\sigma^2_{ui}}}\right) \notag
\end{align}

Based on these, we can design another ranking rule analogous to MAR. In this case, we rank items according to the following, 

\begin{align}
\label{eq:hbr_rule}
    \lambda E[\prob_{ui}] + (1 - \lambda) \text{Var}[\prob_{ui}]
\end{align}

In our experiments, we will use HBR-Y to refer to a system that uses the HBR model but with the ranking rule from Equation \ref{eq:hbr_rule}\footnote{We choose the suffix Y because ranking in this case is based on the interaction probability $\prob(Y_{ui} = 1|r_{ui})$.}. 

Note again that, if $\lambda = 1$, then we obtain a ranking based only on $E[\prob_{ui}]$ -- a ranking that is parallel to \acrshort{rebr}. Nevertheless, we also note that $E[\prob_{ui}]$ is a function of both $\mu_{ui}$ and $\sigma_{ui}$. Therefore, ranking based solely on $E[\prob_{ui}]$ is already in a way accounting for the uncertainty in the latent relevances.

Before we finish, we make two final remarks. First, we emphasize that HBR-R and HBR-Y are the same model, the only difference is in the form of item ranking employed at the top-$n$ recommendation stage. In the light of this, we will also abuse names a little: for simplicity in what follows, we say that HBR-Y, like HBR-R, uses MAR, although, in fact, only the latter uses MAR (Eq.\ \ref{eq:mar}); the former uses something that is similar to MAR (Eq.\ \ref{eq:hbr_rule}). Secondly, as noted by \cite{daunizeau2017semi}, $\text{Var}[\prob_{ui}]$ is a monotonically increasing function of $\sigma_{ui}$, and therefore HBR-Y should not differ too drastically from HBR-R.

\section{Empirical Study}
\label{sec:exp5}

In this section, we compare the methods for mislabeling-aware ranking that were introduced in this chapter. In particular, our goal is to contrast the recommendation accuracy of the new models proposed in this chapter with \acrshort{aur}, the method taken from the literature.

As with previous chapters, we ensure reproducibility of the experiments by making all the code available in \url{github.com/vcoscrato/uncertain}.

\subsection{Models}

Our experiments comprise the following models: \acrshort{aur}, ICPMF, Gaussian-MLP and \acrshort{hbr} -- the last of these, in fact, being further subdivided into HBR-R and HBR-Y. With the exception of Gaussian-MLP, which is a unified model, all these models use estimators of relevance and of uncertainty, denoted $F$ and $G$ above. In all cases, we use \acrshort{mf} for $F$; and, in most cases, we use \acrshort{mf} for $G$ also. The exceptions to the latter are the uncertainty estimators $G$ for ICPMF and for the \acrshort{hbr} models, where, by design, $G$ estimates uncertainty as the product of user and item latent variances (Eq.\ \ref{eq:ICPMF2}). Using the same underlying estimator for $F$ and $G$ wherever possible gives us a fair comparison. 

Apart from seeing which of the models has the highest accuracy, the experiments should be revealing in two other ways. First, we may evaluate the necessity of the more complex (higher dimensional) uncertainty estimator employed by \acrshort{aur} against the much simpler one of ICPMF and \acrshort{hbr}. In addition, these estimators of varying complexity are compared against Gaussian-MLP, a method that is rather different from the previous two because it unifies $F$ and $G$.

Second, we can investigate the effect of giving a correct treatment to the implicit feedback, as discussed in Section \ref{sec:binary}. AUR, ICPMF and Gaussian-MLP approximate the implicit feedback with a numerical formulation whereas \acrshort{hbr} (both HBR-R and HBR-Y) respect the binary nature of the implicit feedback, but learns an approximation for the true posterior through variational inference. In this respect, one particular comparison is of special interest: because we use the exact same relevance and uncertainty representation for both ICPMF and \acrshort{hbr}\footnote{We remind the reader that we developed HBR to use the same relevance and uncertainty representation as ICPMF. But this was a choice. Variants of HBR, with other representations, can be derived.}, we are able to isolate this difference and can see its effect on the results. %\db{For more clarity, after the first sentence, the para needs to remind the reader of which ones do not use the `correct' treatment (AUR, ICPMF, Gaussian-MLP) and which ones do (the two HBR approaches). Then it seems strange that the rest of the para only contrasts ICPMF with HBR.} \victor{I listed now all methods in each "class". In sequence, the contrast between only ICPMF and HBR is well explained IMO.}

We add to our results the best MF baselines for each dataset, according to Section \ref{sec:baselines}. More precisely, what we refer to as MF (Baseline) in the following results is MF-BPR for the MovieLens dataset and MF-MSE for Pinterest.


\subsection{Datasets}

We use the same datasets as in Section \ref{sec:data4} of Chapter \ref{chapter:implicit}. Moreover, we use the same data partitions as before.% Thus, the accuracy reported for the systems in this chapter may be compared against those therein. 

%\dgb{I thought you were going to remove this `tease', because you only compare with baselines.} \victor{Ops... fixed.}

\subsection{Methods}

\subsubsection{Tuning}
The models that we compare have a number of hyper-parameters that need to be tuned. To do so, we use a Bayesian parameter search, as per Section \ref{sec:tuning4}. With each method, we learn 20 models with different values for the hyper-parameters, sampled as follows:

\begin{itemize}
    \item For all models, the learning rate is sampled from [0.00001, 0.001]. The number of negative samples per positive interaction in each data batch is sampled from $\{1,2,\ldots,30\}$.
    \item For all models based on \acrshort{aur}'s objective (Eq.\ \ref{eq:aur_neg_log_likelihood_prior}) -- that is, \acrshort{aur}, ICPMF and Gaussian-MLP -- we sample $\beta$ from $\{1e-3, 1\}$ and $\gamma$ from $\{1e-4, 1e-1\}$.
    \item \acrshort{aur} and ICPMF: The L2 penalty factor applied to the user and item factors for the relevance estimator ($\lambda_\theta$ in Eq.\ \ref{eq:aur_neg_log_likelihood_prior}) is taken from $[10^{-5}, 10^{-3}]$.
    \item Gaussian-MLP: Again, we use the same three-layer MLP as in \cite{he2017neural}, with dropout rate during training taken from $[0, 0.2]$.
    \item \acrshort{hbr}: For HBR, we must choose the prior variances $\alpha_u, \alpha_i, \alpha_{\sigma i}, \alpha_{\sigma i}$. In every case, we sample their values from $[1e-5, 1e-1]$.
\end{itemize}

These candidate hyper-parameter values were chosen based on pilot runs. Using results from the pilot runs, we ensured that `winning' values did not fall close to the boundaries of the range of candidate values, which would suggest the optimal value might lie outside of the range. When a `winning' value did fall close to a boundary, we extended the range of values.

Following the same reasoning from Chapter \ref{chapter:implicit}, to ensure fairness in our comparisons, we set the user and item latent embeddings size $d$ to 128 for all models. Notice that this means that both the relevance and uncertainty estimators are based on latent factors that are 128 wide.

Again, to avoid the need to tune the number of training iterations, we employ early-stopping to end the learning phase when the MAP@10 on the validation set does not improve for three consecutive iterations.

\subsubsection{Expected correlations evaluation}

Before evaluating the methods according to recommendation accuracy, for each model, we first look at the correlation between the average relevance $\mu_{ui}$ and uncertainty $\sigma^2_{ui}$. (In the case of HBR-Y, we look at the correlation between the average probability of interaction $E[\prob_{ui}]$ and the uncertainty $\text{Var}[\prob_{ui}]$). Having a positive correlation between the two is what motivates the additive form of ranking employed by \acrshort{mar} \cite{wang2022rethinking}

We follow the same procedure from previous chapters. That is, we obtain the average relevance and label uncertainty from a randomly sampled set of 100,000 user-item pairs. Then, we plot the dispersion of the obtained values, together with a least squares tendency line, and analyse the results visually.

\subsubsection{Top-$n$ recommendation evaluation}

We examine the recommendation accuracy for each model according to MAP and Recall at $n=10$ as a function of $\lambda$. In practice, $\lambda$ is a parameter that should be fixed in production. Here, we plot the results for a range of $\lambda$ values to illustrate how it affects accuracy. Again, we remind the reader that \acrshort{rebr} is the particular case of MAR with $\lambda=1$.

In this part of the evaluation, we consider as the candidates for recommendation to the user all items except those the user has interacted with (in the training or validation set). In the next part of the evaluation, we will restrict the candidates to long-tail items.

% \db{One idea is to move the material that you have here into the Results section (5.5), putting some of it as the intro to 5.5 and putting some of it into 5.5.1 and some of it into 5.5.2. You might even add a section prior to 5.5.1 and 5.5.2 called Expected correlations. The advantages are that it puts discussion closer to where it is needed and makes this chapter closer in structure to the previous one. The disadvantage is that the Methods section 5.4.3 would now contain a lone subsection 5.4.3.1, which is ugly. To overcome that, you could merge so that 5.4.2 is called Datasets and Methods, and it would have no subsections.} 

% \victor{I implemented your ideas partially. I like having the long-tail evaluation here, both for consistency (specially with chapter 3), and to avoid displaying metric equations in the results section. I also added the expected correlations subsection in the results.}

%\victor{old} In the beginning of the chapter, we explained how the mislabeling uncertainties differ from the previously discussed forms of uncertainty explored in chapters \ref{chapter:explicit} and \ref{chapter:implicit}. For this reason, the analytical techniques that we developed for uncertainty evaluation in the previous chapters are no longer applicable. Thus, our evaluation for the usefulness of measuring mislabeling uncertainty will be accuracy-based. More specifically, we measure the MAP and Recall at $n$ for each method. \derek{Hopefully, this para can be improved once there are improvements to the earlier part of the chapter, e.g.\ maybe there's an acronym that refers to the form of uncertainty-aware ranking that we are doing here.}

%\victor{old} But there are some other aspects that need investigation. First, the usefulness of mislabeling uncertainty \derek{Again we need a clearer way of referring to this whole form of uncertainty, but one that avoids ``AUR'', because that's the name of a model} through the scoring rule $\lambda F_{ui} + (1 - \lambda) \sqrt{G_{ui}}$  \derek{(a) again, if this also had a name/acronym that might help (b) isn't the assumed correlation really just a fact about AUR, rather than a fact about AUR + this rule?} employ for AUR is based on the premise that $\mu_{ui}$ and $\sigma^2_{ui}$ are strongly correlated. Therefore, before evaluating accuracy, for each model, we will also measure, and compare, the correlation between $\mu_{ui}$ and $\sigma^2_{ui}$.

%\victor{The following two paragraphs should replace the previous two. You can delete the previous when reviewed: 

\subsubsection{Long-tail recommendation evaluation}
\label{sec:lt}

In Section \ref{sec:aur_rec}, we argued that \acrshort{mar} may lead to recommendations that exhibit less popularity bias. To evaluate this, we will employ an additional evaluation protocol, introduced in \cite{wang2022rethinking}, which focuses on long-tail items. We will refer to this as long-tail (LT) recommendation. This protocol forces the system to recommend only tail items, and measures the accuracy of these recommendations.

For long-tail recommendation, we must define what we mean by tail items. We adopt the same method as in \cite{wang2022rethinking}. That is, tail items are those with fewest interactions such that those interactions comprise 50\% of all the observed interactions. We denote the set of tail items as $I_{LT}$. Then, for long-tail recommendation evaluation, the candidate items for recommendation to each user are given by $R_{u\cdot} \cap I_{LT}$. Similarly, the test items for each user are also confined to long-tail items: $T_{uLT} = T_u \cap I_{LT}$, where $T_u$ are $u$'s test set items (Section \ref{sec:eval_topn_acc}). Let $Z_{uLT}^n$ denote the top-$n$ recommended to user $u$ based on candidates $R_{u\cdot} \cap I_{LT}$. Then, we define the MAP and Recall at $n$ relative to the long-tail items as,

\begin{align*}
    \text{Recall Relative@n}_u &= \frac{\#\{Z_{uLT}^n \cap T_{uLT} \cap \}}{\#T_{uLT}} \\
    \text{Precision Relative@n}_u &= \frac{\#\{Z_{uLT}^n \cap T_{uLT}\}}{n} \\
    \text{AP Relative@n}_u &= \frac{1}{\#T_u} \sum_{k=1}^{n} \text{Precision Relative@k}_u \times \delta(Z_u^n(k) \in T_{uLT}) \\
    \text{MAP Relative@n} &= \frac{1}{\#U} \sum_{u \in U} \text{AP Relative@n}_u \\
\end{align*}
These are analogous to definitions in Section \ref{sec:eval_topn_acc}, but confined to long-tail items.

\section{Results}
\label{sec:results5}

% In this section, we present the results. Our evaluation is largely accuracy-oriented. After all, the measurement of label uncertainty is meant to help us identify potentially relevant items that have been mislabeled. Through the use of \acrshort{mar}, such items can be given a higher ranking score, making them more liable to be recommended. Hence, the approach that we adopt to the evaluation of label uncertainty estimators is to analyse their accuracy as a function of $\lambda$, the weighting parameter that controls the importance given to label uncertainty during ranking\footnote{Again, we remind the reader that \acrshort{rebr} is the particular MAR case with $\lambda=1$.}. We evaluate both general top-$n$ recommendations (Section \ref{sec:eval_top_5}) and long-tail recommendations (Section \ref{sec:eval_lt}). Before that, we begin with a discussion about the correlation between relevance and label uncertainty. 

%\dgb{This para feels very much a repeat of what you have said already -- except perhaps the footnote. Can you move the footnote and delete the para? Or shorten it all. Or...?} \victor{I agree. I moved the footnote to 5.4.3.3 (in the main text, not as a footnote anymore). It follows a simpler version of the paragraph just so this is not blank.}

In this section, we present the results. We evaluate both general top-$n$ recommendations (Section \ref{sec:eval_top_5}) and long-tail recommendations (Section \ref{sec:eval_lt}). Before that, we begin with a discussion of the correlation between relevance and label uncertainty. 

\subsection{Expected correlations}

We begin by examining the correlation between the average relevance and label uncertainty. Figure \ref{fig:corr5} illustrates the dispersion diagrams with least squares tendency lines added.

\begin{figure}[tp]
\centering
\begin{subfigure}{\linewidth}
    \centering
    \includegraphics[width=0.8\linewidth]{img/Chap 5/Movielens/relevance_uncertainty5.pdf}
    \caption{MovieLens dataset.}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \centering
    \includegraphics[width=0.8\linewidth]{img/Chap 5/Pinterest/relevance_uncertainty5.pdf}
    \caption{Pinterest dataset.}
\end{subfigure}
\caption{Dispersion diagrams between the estimated average relevances and mislabeling uncertainties for each method and dataset. The black line in each diagram is a least squares tendency line to assist visualization.}
\label{fig:corr5}
\end{figure}

We observed the expected correlation with \acrshort{aur}, ICPMF and Gaussian-MLP for the MovieLens dataset. On the other hand, the same does not hold for AUR and ICPMF for the Pinterest dataset. One case that draws attention is AUR for the Pinterest data, where there are several interactions with low relevance and high uncertainty. When ranking with \acrshort{mar}, such items will be more likely to be recommended for smaller values of $\lambda$.

HBR-R and HBR-Y show no apparent correlation in almost every case: the tendency lines are approximately flat for both models in the MovieLens dataset and for HBR-R on Pinterest. The exception is HBR-Y on Pinterest, where, although the tendency line still points to a positive correlation, we see that the correlation is not strictly increasing. This is perhaps to be expected. The uncertainties estimated by HBR-Y approximate 0 as $E[\prob_{ui}]$ approximate either 0 or 1. This is a consequence of the binary formulation. If $E[\prob_{ui}]$ is close to 0 (or 1), then all the probability mass for $\prob_{ui}$ is concentrated around (0 or 1), and therefore, its variance is small.

%As we pointed out earlier, even though we have found that we do not always observe the anticipated correlations, this does not necessarily signal any problem with the uncertainty estimators. Instead, it only gives an intuition that \acrshort{mar} may not lead to more accurate recommendations than \acrshort{rebr}. To give more insight, we have observed that the relevance-uncertainty dispersion is affect by several hyper-parameters, specially $\beta$ and $\gamma$ from Eq.\ \ref{eq:aur_neg_log_likelihood_prior} and the size of the negative sample.

%\db{Let's discuss. Maybe when I kow what you're trying to say, I can help you say it better.} \victor{So, my idea is to discourage the reader from taking strong conclusions based on this section. The last sentence argue what i've seen on my trial runs: this plots have some run-to-run variation depending of the hyperparams. I would appreciate some help to make the sentence better understandable.}

%\victor{Alternative paragraph:} 
As we pointed out earlier, even though we have found that we do not always observe the anticipated correlations, this does not necessarily signal any problem with the uncertainty estimators. It shows only that using MAR with those estimators may not lead to more accurate recommendations than \acrshort{rebr}. Another reason why these results need not be too concerning is because we have observed (on pilot runs) that the relevance-uncertainty dispersion is affect by several hyper-parameters, especially $\beta$ and $\gamma$ from Eq.\ \ref{eq:aur_neg_log_likelihood_prior} and the size of the negative sample. It may be possible to have another hyper-parameter configuration that leads to models with similar accuracy but different relevance-uncertainty dispersion. %\victor{Maybe this paragraph should be split in two (at 'another reason why'...)}

\subsection{Top-$n$ recommendations}
\label{sec:eval_top_5}

Now we examine the recommendation accuracy for each model according to MAP and Recall at $n=10$ as a function of $\lambda$. In practice, $\lambda$ is a parameter that should be fixed in production. Here, we plot the results for a range of $\lambda$ values to illustrate how it affects accuracy. We note that the MF baseline accuracy, of course, has no label uncertainty estimator, and that is why its accuracy does not vary with $\lambda$.

\begin{figure}[tp]
\centering
\begin{subfigure}{\linewidth}
    \centering
    \includegraphics[width=0.8\linewidth]{img/Chap 5/Movielens/accuracy5.pdf}
    \caption{MovieLens dataset.}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \centering
    \includegraphics[width=0.8\linewidth]{img/Chap 5/Pinterest/accuracy5.pdf}
    \caption{Pinterest dataset.}
\end{subfigure}
\caption{MAP@10 and Recall@10 for every model compared in the MovieLens (top) and Pinterest (bottom) datasets as a function of $\lambda$.}
\label{fig:acc5}
\end{figure}

%\derek{You have included the regular results and the long tail results all in the same large figure. I recommend splitting into two figures. The you can re-order the text below so that it discusses each figure separately.} \victor{\checkmark}

%\derek{I also feel that you are not saying enough about the results. There are a lot of results but very little analysis. E.g when lambda is high, then this becomes top-n rec (ignoring uncertainty); maybe we should compare the models for high lambda to see which is most accurate in this special case. Then we can see what happens as lambda is reduced: does it help accuracy? Then separately discuss it all over again for the long tail case.} \victor{There are two new paragraphs in here, and some minor changes to those previously existing, to implement the changes you proposed.}

%\db{One other thought. Maybe we should be extracting the results for lambda = 1, i.e. ReBR, and showing them in a separate table. We could also include the ReBR results from the prev chapter (you did say in sec 5.4.2 (Datasets) that you have the same splits and this allows a comparison with the prev chapter!). This might be especially worth doing if any of the results from this chapter improve on results from the previous chapter.} \victor{I think this though were related to the past version where we had sections dedicated to ReBR. Now that we defined it as a particular MAR case, I don't feel this should be done.}

Let's initially focus on the choice of $\lambda$. We note that for many models, the choice $\lambda=1$ -- that is, recommendations given through \acrshort{rebr} -- are the most accurate. In fact, this is the case for all models with the Pinterest data. This is in line with what we might expect from the correlation plots. The only exceptions are ICPMF and HBR-Y on MovieLens. For the former, MAP@10 is at its maximum at $\lambda = 0.3$ and Recall@10 at $\lambda = 0.4$. For the latter, both MAP and Recall are at their maximum at $\lambda = 0.6$. Therefore, we conclude that in some specific cases \acrshort{hbr} may be beneficial, although it is often worse than \acrshort{rebr}.

Now, comparing the models against each other, we note the great overall performance of ICPMF. For every metric and dataset, ICPMF is always among the most accurate systems. This is a remarkable result because ICPMF is arguably the simplest method: compared to AUR, its uncertainty estimator is of much lower dimensionality; and, compared to HBR, its learning is exact (i.e.\ no approximation such as that given by variational inference is required); in general, it is a lot more straightforward. Gaussian-MLP, on the other hand, achieved poor overall results. In particular, it achieved very low recommendation accuracy compared to all the other models on the Pinterest dataset. From these results, we conclude that unifying $F$ and $G$, as discussed in Section \ref{sec:neural_aur}, looks unpromising.

In addition, ICPMF achieved better recommendation performance than the baseline MF (MovieLens data) or almost equivalent to the baseline (Pinterest data) . Moreover, AUR is also more accurate then the baseline for MovieLens, although it is worse on Pinterest.

We also observe a difference between the results for the two datasets. On the one hand, AUR is better than HBR-R and HBR-Y in the MovieLens dataset. On the other hand, the opposite is the case for Pinterest. In particular, HBR-Y is very close to ICPMF (the top performer) in the Pinterest data according to both metrics. This may be indicative that the variation approximation needed to learn HBR performs better on bigger datasets.
 
Interestingly, we note that if $\lambda = 0$, then the ranking scores are defined solely by $G_{ui}$ -- that is, $F_{ui}$ is ignored. Even in this extreme case, we see that AUR, ICPMF and Gaussian-MLP are still able to recommend with decent accuracy in the MovieLens data. Therefore, another conclusion is that the label uncertainties, at least in these cases, are in fact related to relevance as we supposed.

\subsection{Long-tail recommendations}
\label{sec:eval_lt}

As described in Section \ref{sec:lt}, we also consider the alternative setting where candidate items for recommendation are restricted to long-tail items. %With this setup, we monitor MAP Relative@10 and Recall Relative@10 as a function of $\lambda$. 
Figure \ref{fig:acc_rel5} displays the results.

\begin{figure}[tp]
\centering
\begin{subfigure}{\linewidth}
    \centering
    \includegraphics[width=0.8\linewidth]{img/Chap 5/Movielens/accuracy_relative5.pdf}
    \caption{MovieLens dataset.}
\end{subfigure}
\begin{subfigure}{\linewidth}
    \centering
    \includegraphics[width=0.8\linewidth]{img/Chap 5/Pinterest/accuracy_relative5.pdf}
    \caption{Pinterest dataset.}
\end{subfigure}
\caption{MAP Relative@10 and Recall Relative@10 for every model compared in the MovieLens (top) and Pinterest (bottom) datasets as a function of $\lambda$ but for log-tail items only.}
\label{fig:acc_rel5}
\end{figure}

Again, let's focus initially on the choice of $\lambda$. When restricting recommendations to the long-tail items, we observe $\lambda$ having a higher impact in many cases, especially in the MovieLens dataset. For this dataset we observe a major advantage of \acrshort{mar} over \acrshort{rebr} for ICPMF -- in this case, MAP Relative@10 is at its highest at $\lambda=0.7$ and Recall Relative@10 at $\lambda=0.8$.  The same is observed for HBR-Y: both metrics are at their maximum when $\lambda=0.5$. Interestingly, Gaussian-MLP is best when $\lambda=0$. Unfortunately, the \acrshort{mar} results for the Pinterest are not as positive. In fact \acrshort{rebr} -- the case $\lambda=1$ -- is better or very close to \acrshort{mar} in almost every case. The only exception is Gaussian-MLP, but its accuracy is by far the lowest. Therefore, similarly to the general top-$n$ recommendation case, \acrshort{mar} was also not proven advantageous in the long-tail recommendation case for the Pinterest dataset.

Next, let's compare the models. In the MovieLens dataset, following the same trend from the overall top-$n$ recommendation case, the performance of ICPMF is also remarkable for long-tail recommendation. We observe that ICPMF has a big advantage against every other method. Moreover, ICPMF is also a lot more accurate than the MF baseline in this dataset. For Pinterest, we found that the MAP and Recall of all methods, except Gaussian-MLP, are comparable to those of the baseline. That is, even though \acrshort{mar} is not beneficial in this dataset, the use of our methods with \acrshort{rebr} provides long-tail recommendations that are at least close in accuracy to the baseline, and, in the case of ICPMF, sometimes slightly better.

\section{Conclusions and Limitations}

In this chapter, we explored label uncertainty in \acrshort{irss}. By learning to estimate this form of uncertainty, which is rather different from the prediction uncertainty that we explored in Chapters \ref{chapter:explicit} and \ref{chapter:implicit}, we aim to counter the mislabeling problem in the learning of \acrshort{irss} models, which is a consequence of negative sampling. We have introduced four novel methods for label uncertainty estimation. Our main outcomes and findings are:

\begin{itemize}
    \item We drew a parallel between label uncertainty (and, in particular, its use in \acrlong{mar}) and the observation model from \cite{saito2020unbiased}. This extends the discussion in \cite{wang2022rethinking} and provides a stronger theoretical support to the exploration of label uncertainty.
    \item We found that ICPMF, which has a very simple label uncertainty model, is generally a good choice. In fact, this method was uniformly better than every other across all our experiments. That is, it exhibited leading accuracy in both top-$n$ recommendations and long-tail recommendations in both the MovieLens and Pinterest datasets according to every metric. The effectiveness of ICPMF recalles the effectiveness of CPMF in Chapter \ref{chapter:explicit}.
    \item We also found that \acrshort{mar} may in fact be a powerful tool to combat popularity bias in some cases. For the MovieLens dataset, we showed that the use of \acrshort{mar}, with the correct choice of $\lambda$, can sometimes lead to major accuracy improvements. The improvement is most remarkable with ICPMF. Unfortunately, we did not observe the same for the Pinterest dataset.
\end{itemize}

Again, we acknowledge that our empirical study has some limitations, listed below. From these limitations, we may derive new research questions for exploration in future work (see Chapter \ref{chapter:conclusion}). 

\begin{itemize}
    \item First, we have restricted ourselves to MF models as the standard choices for $F$ and $G$. Although this choice allows for a fair comparison between the methods, MF is only one of many possible choices for both. We believe there is clear value in expanding the comparison to other choices. In particular, one could explore the `interaction effect' between $F$ and $G$. For instance, the impact on $G$ given different choices of $F$, and vice versa.
    \item Second, as in the previous chapters, we considered only two datasets in our study. Although we observed a clear winner across our results (ICPMF), we also noticed that certain methods -- especially HBR -- performed differently in the two datasets. To explore this further, it would be useful to extend the comparison to other recommendation datasets.
    \item Finally, the only use we have made of label uncertainty estimates is in \acrshort{mar}. Although, we suppose that label uncertainty may be exploited in other ways. For example, there may be ranking criteria other than MAR, that could be tried. There may even be uses outside of top-$n$ recommendations, such as uses in active learning.
\end{itemize}

%\dgb{Again the limitations feel a bit rushed.} \victor{I expanded a bit.}



% \victor{Unused text from here onwards: ------------------------------------}

% & = \left( \prod_{u = 1}^U \prod_{i = 1}^I q(r_{ui} | p_u, q_i, \sigma_u, \sigma_i) \right) \left( \prod_{u = 1}^U q(p_u) q(\sigma_U) \right) \left( \prod_{i = 1}^I q(q_i) q(\sigma_I) \right)



% The expected value can be estimated by taking Monte-Carlo samples from the variational distribution $q(\Theta)$. Then, we take gradient steps in $\Theta$ and repeat until convergence. Note that, due to the factorisable priors of choice, and the conditional independences imposed, the variational distribution may be easily defined,

% \begin{align}
%     q(\Theta) = q(\sigma_U) q(\sigma_I) \prod_{u = 1}^U \prob(p_u) \prod_{i = 1}^I \prob(q_i) \prod_{u = 1}^U \prod_{i = 1}^I \prob (r_{ui} | p_u, q_i, \sigma_u, \sigma_i)
% \end{align}



% \begin{align}    
%     \frac{1}{1 + e^{r_{ui}}} \sim \text{Logit-normal}(\mu_{ui}, \sigma^2_{ui}) \\
%     \prob (r_{ui}|Y_{ui}) \propto \prob(Y_{ui}|r_{ui}) \prob(r_{ui})
% \end{align}

% Therefore, the negative log-posterior of $r_{ui}$ is,

% \begin{align}
%     -\log \prob(r_{ui}|Y_{ui}) \propto - Y_{ui} \log \left(\frac{1}{1 + e^{r_{ui}}}\right) - (1 - Y_{ui}) \log \left(1 - \frac{1}{1 + e^{r_{ui}}}\right) + \notag \\
%     \frac{\mu_{ui} - Y_{ui}}{2\sigma^2_{ui}} + \log(\sigma_{ui})
% \end{align}

% The loss function to be optimized, for a batch of training instances $\mathcal{B}$ is,

% \begin{align}
%     \mathcal{L}(\mathcal{B}|\theta) = \frac{1}{\#\mathcal{B}} \sum_{(u, i) \in \mathcal{B}} - Y_{ui} \log \left(\frac{1}{1 + e^{r_{ui}}}\right) - (1 - Y_{ui}) \log \left(\frac{e^{r_{ui}}}{1 + e^{r_{ui}}}\right)  \notag \\ + \frac{\mu_{ui} - Y_{ui}}{2\sigma^2_{ui}} + \log(\sigma_{ui})
% \end{align}


% \begin{align}
%     \text{Second: \,\,\,\,} \prob (Y_{ui} = 1|r_{ui}) = F_{r_{ui}}(r_{ui}), \quad Y_{ui}|r_{ui} \sim \text{Bernoulli}\left(F_{r_{ui}}(r_{ui})\right) \\
%     F_{r_{ui}}(r_{ui}) \sim \text{Uniform}(0, 1) \\
%     \prob (r_{ui}|Y_{ui}) \propto \prob(Y_{ui}|r_{ui}) \prob(r_{ui})
% \end{align}




% \section{Setup}
% \label{sec:setup5}

% More specifically, we will confine our attention to gaussian methods, in which the relevance of a user-item interaction is modeled as $r_{ui} \sim \norm (\mu_{ui}, \sigma^2_{ui})$. In such case, $\mu_{ui}$ denotes the expected relevance of the interaction, and $\sigma^2_{ui}$ the uncertainty associated with it. The models' task is to predict these gaussian parameters for unseen user-item interaction. The gaussian recommenders in this chapter will following the form,

% \begin{align}
%     \mu_{ui} &= F_\theta(u, i) \\
%     \sigma^2_{ui} &= \frac{1}{K} \exp \left( G_\gamma(u, i) \right)
% \end{align}

% Therefore, we need to learn two functions. First, the expectancy predictor $F$, parameterized by $\theta$ to predict the distribution's average for each interaction. And secondly, the variance predictor $G$, parameterized by $\gamma$ to predict the interaction variances. The exponential of $G$ is taken to ensure positive variances, and $K$ is a hyper-parameter that controls the exponential scale.

% The functions $F$ and $G$ can be any CF model, such as FunkSVD (Eq.\ \ref{eq:MF}) or MLP (Eq.\ \ref{eq:MLP}), the function of choice is referred to as the underlying model. Now, to complete the framework, we need an algorithm that allow $F$ and $G$ to be learned from implicit signals. Later in this Chapter, we describe three methods for this task: \acrshort{aur}, \acrshort{gbr} and \acrshort{gpr}.






% \subsection{\acrlong{gpr}}
% \label{sec:gpr}
% Another compelling objective used to learn implicit \acrshort{rs} is \acrshort{bpr} (Eq.\ \ref{eq:bpr}). Motivated by it, we propose a pair-wise learning principle that allows for uncertainty estimation in \acrshort{irs}. Let $r_{ui}$ denote the relevance of item $i$ to user $u$. Then, $r_{uij} = r_{ui} - r_{uj}$ models how much user $u$ prefers item i over j.  Given a user $u$ and a pair of items $(i, j)$, the probability that the user $u$ will prefer item $i$ over $j$ can be modeled as

% \begin{equation}
%     P(i >_u j) = S(r_{uij})
% \end{equation}

% \noindent
% where $S$ denotes the sigmoid function is is meant to constraint the probability to the $[0, 1]$ interval. Again, we let $r_{ui} \sim \norm (\mu_{ui}, \sigma^2_{ui})$. Then, we model the pairwise preferences as,

% \begin{equation}
%     P(i >_u j) = P(r_{uij} > 0) = P(r_{uji} \leq 0)
% \end{equation}

% Now, because $r_{uji}$ is also normally distributed, we can explicitly calculate this probability,

% \begin{align}
%     P(r_{uji} \leq 0) &= P\left(\frac{r_{uji} - (\mu_{uj} - \mu_{ui})}{\sqrt{\sigma^2_{uj} + \sigma^2_{ui}}} \leq \frac{- (\mu_{uj} - \mu_{ui})}{\sqrt{\sigma^2_{uj} + \sigma^2_{ui}}} \right) \\ 
%     &= \Phi \left( \frac{\mu_{ui} - \mu_{uj}}{\sqrt{\sigma^2_{uj} + \sigma^2_{ui}}}\right)
% \end{align}

% Where $\Phi$ is the standard gaussian distribution cumulative distribution function. Therefore, the likelihood of the training data (the known pairwise preferences) is given by,

% \begin{equation}
% \label{eq:likelihood_gpr}
%     \prob(R) = \prod_{u \in U} \prod_{i \in I_u^+} \prod_{i \in I_u^-} \Phi \left( \frac{\mu_{ui} - \mu_{uj}}{\sqrt{\sigma^2_{uj} + \sigma^2_{ui}}}\right)
% \end{equation}

% Where $I_u^+ = \{i \in I | (u, i) \in D^+\}$ and $I_u^- = \{i \in I | (u, i) \in D^-\}$.


