Related papers: Meterstick: Benchmarking Performance Variability i…
How can applications be deployed on the cloud to achieve maximum performance? This question has become significant and challenging with the availability of a wide variety of Virtual Machines (VMs) with different performance capabilities in…
While there is rapid progress in video-LLMs with advanced reasoning capabilities, prior work shows that these models struggle on the challenging task of sports feedback generation and require expensive and difficult-to-collect finetuning…
Mobile gaming presents a number of main issues which remain open. These are concerned mainly with connectivity, computational capacities, memory and battery constraints. In this paper, we discuss the design of a fully distributed approach…
The system mentioned in the title belongs to the family of the so-called massively multi-player online social games (MMOSG). It features a scoring system for the elements of the game that is prone to herding effects. We analyze in detail…
The intersection of Mean Field Games (MFGs) and Reinforcement Learning (RL) has fostered a growing family of algorithms designed to solve large-scale multi-agent systems. However, the field currently lacks a standardized evaluation…
Assistance games are a promising alternative to reinforcement learning from human feedback (RLHF) for training AI assistants. Assistance games resolve key drawbacks of RLHF, such as incentives for deceptive behavior, by explicitly modeling…
In distributed multimedia applications, content is often delivered to users in a degraded form due to network-induced lossy compression. Real-time and interactive use cases like cloud gaming, which render content on the fly, require low…
This study investigates the concept of flexibility within League of Legends, a popular online multiplayer game, focusing on the relationship between user adaptability and team success. Utilizing a dataset encompassing players of varying…
What makes a good Large Language Model (LLM)? That it performs well on the relevant benchmarks -- which hopefully measure, with some validity, the presence of capabilities that are also challenged in real application. But what makes the…
Blockchain is a paradigm derived from distributed systems, protocols, and security concepts. However, can blockchain applications provide services in industrial environments, especially concerning performance issues? In blockchains, long…
In this work we proposing adapting the Minecraft builder task into an LLM benchmark suitable for evaluating LLM ability in spatially orientated tasks, and informing builder agent design. Previous works have proposed corpora with varying…
Cloud benchmarks suffer from performance fluctuations caused by resource contention, network latency, hardware heterogeneity, and other factors along with decisions taken in the benchmark design. In particular, the sampling strategy of…
The potential data contamination issue in contemporary large language models (LLMs) benchmarks presents a fundamental challenge to establishing trustworthy evaluation frameworks. Meanwhile, they predominantly assume benign, resource-rich…
Researchers and engineers are increasingly adopting cloud-native technologies for application development and performance evaluation. While this has improved the reproducibility of benchmarks in the cloud, the complexity of cloud-native…
There is a broad consensus that the inability to form long-term plans is one of the key limitations of current foundational models and agents. However, the existing planning benchmarks remain woefully inadequate to truly measure their…
In recent years, the video game industry has experienced substantial growth, presenting players with a vast array of game choices. This surge in options has spurred the need for a specialized recommender system tailored for video games.…
Multiplayer online battle arena games provide an excellent opportunity to study team performance. When designing a team, players must negotiate a \textit{proficiency-congruency dilemma} between selecting roles that best match their…
Game-based learning shows real promise for engaging students in well-resourced classrooms, but what about the millions who study in schools with far fewer opportunities? We propose a practical framework for bringing Minecraft Education…
Task environments developed in Minecraft are becoming increasingly popular for artificial intelligence (AI) research. However, most of these are currently constructed manually, thus failing to take advantage of procedural content generation…
This paper presents the first industry-standard open-source machine learning (ML) benchmark to allow perfor mance and accuracy evaluation of mobile devices with different AI chips and software stacks. The benchmark draws from the expertise…