Related papers: Leveraging Multi-Method Evaluation for Multi-Stake…

Multistakeholder recommender systems are those that account for the impacts and preferences of multiple groups of individuals, not just the end users receiving recommendations. Due to their complexity, these systems cannot be evaluated…

Recommender systems have been successfully applied to assist decision making by producing a list of item recommendations tailored to user preferences. Traditional recommender systems only focus on optimizing the utility of the end users who…

Recommender systems are personalized information access applications; they are ubiquitous in today's online environment, and effective at finding items that meet user needs and tastes. As the reach of recommender systems has extended, it…

While machine learning (ML) technology affects diverse stakeholders, there is no one-size-fits-all metric to evaluate the quality of outputs, including performance and fairness. Using predetermined metrics without soliciting stakeholder…

Given that there are a variety of stakeholders involved in, and affected by, decisions from machine learning (ML) models, it is important to consider that different stakeholders have different transparency needs. Previous work found that…

Recommender systems are personalized information systems. However, in many settings, the end-user of the recommendations is not the only party whose needs must be represented in recommendation generation. Incorporating this insight gives…

One of the most essential parts of any recommender system is personalization-- how acceptable the recommendations are from the user's perspective. However, in many real-world applications, there are other stakeholders whose needs and…

Multimodal recommendation systems are increasingly popular for their potential to improve performance by integrating diverse data types. However, the actual benefits of this integration remain unclear, raising questions about when and how…

There are many potential measures that one might use when evaluating mixed-reality experiences. In this position paper I will argue that there are various stances to take for evaluation, depending on the framing of the experience within a…

Recommender Systems today are still mostly evaluated in terms of accuracy, with other aspects beyond the immediate relevance of recommendations, such as diversity, long-term user retention and fairness, often taking a back seat. Moreover,…

Traditionally, especially in academic research in recommender systems, the focus has been solely on the satisfaction of the end-user. While user satisfaction has, indeed, been associated with the success of the business, it is not the only…

We propose a new method for analyzing a set of parameters in a multiple criteria ranking method. Unlike the existing techniques, we do not use any optimization technique, instead incorporating and extending a Segmenting Description…

This study aims to optimize the evaluation metric of multimodal multi-objective optimization problems using a Regionalized Metric Framework, which provides a certain boost to research in this field. Existing evaluation metrics usually use…

The creation of benchmarks to evaluate the safety of Large Language Models is one of the key activities within the trusted AI community. These benchmarks allow models to be compared for different aspects of safety such as toxicity, bias,…

Current practice for evaluating recommender systems typically focuses on point estimates of user-oriented effectiveness metrics or business metrics, sometimes combined with additional metrics for considerations such as diversity and…

Recommender Systems have proliferated as general-purpose approaches to model a wide variety of consumer interaction data. Specific instances make use of signals ranging from user feedback, item relationships, geographic locality, social…

There is growing research interest in recommendation as a multi-stakeholder problem, one where the interests of multiple parties should be taken into account. This category subsumes some existing well-established areas of recommendation…

Academic research in recommender systems has been greatly focusing on the accuracy-related measures of recommendations. Even when non-accuracy measures such as popularity bias, diversity, and novelty are studied, it is often solely from the…

Contemporary recommender systems act as intermediaries on multi-sided platforms serving high utility recommendations from sellers to buyers. Such systems attempt to balance the objectives of multiple stakeholders including sellers, buyers,…

This paper describes a generalizable model evaluation method that can be adapted to evaluate AI/ML models across multiple criteria including core scientific principles and more practical outcomes. Emerging from prediction competitions in…