Related papers: Statistical Challenges in Online Controlled Experi…
Online controlled experiments (a.k.a. A/B testing) have been used as the mantra for data-driven decision making on feature changing and product shipping in many Internet companies. However, it is still a great challenge to systematically…
Cloud services today are increasingly built using functionality from other running services. In this paper, we question whether legacy Quality of Services (QoS) metrics and enforcement techniques are sufficient as they are producer centric.…
How diverse are sharing economy platforms? Are they fair marketplaces, where all participants operate on a level playing field, or are they large-scale online aggregators of offline human biases? Often portrayed as easy-to-access digital…
Online data sources offer tremendous promise to demography and other social sciences, but researchers worry that the group of people who are represented in online datasets can be different from the general population. We show that by…
Large technology firms face the problem of moderating content on their online platforms for compliance with laws and policies. To accomplish this at the scale of billions of pieces of content per day, a combination of human and machine…
In recent years, the interest in developing adaptive solutions for online testing has grown significantly in the industry. While the advances related to this relative new technology have been developed in multiple domains, it lacks in the…
Online controlled experiments are a crucial tool to allow for confident decision-making in technology companies. A North Star metric is defined (such as long-term revenue or user retention), and system variants that statistically…
Social media platforms have become a hub for political activities and discussions, democratizing participation in these endeavors. However, they have also become an incubator for manipulation campaigns, like information operations (IOs).…
In this paper, we examine the biases that arise when firms run A/B tests on continuous parameters to estimate global treatment effects on performance metrics of interest; we particularly focus on price experiments to measure the price…
In order to make technological systems and platforms more equitable, organizations must be able to measure the scale of potential inequities as well as the efficacy of proposed solutions. In this paper, we present a system that measures…
As artificial intelligence increasingly influences our world, it becomes crucial to assess its technical progress and societal impact. This paper surveys problems and opportunities in the measurement of AI systems and their impact, based on…
"Data" is becoming an indispensable production factor, just like land, infrastructure, labor or capital. As part of this, a myriad of applications in different sectors require huge amounts of information to feed models and algorithms…
This paper demonstrates the successful application of Off-Policy Evaluation (OPE) to accelerate recommender system development and optimization at Adyen, a global leader in financial payment processing. Facing the limitations of traditional…
Online experiments (A/B tests) are widely regarded as the gold standard for evaluating recommender system variants and guiding launch decisions. However, a variety of biases can distort the results of the experiment and mislead…
IoT edge computing positions computing resources closer to the data sources to reduce the latency, relieve the bandwidth pressure on the cloud, and enhance data security. Nevertheless, data security in IoT edge computing still faces…
The sharing economy is sprawling across almost every sector and activity around the world. About a decade ago, there were only a handful of platform driven companies operating on the market. Zipcar, BlaBlaCar and Couchsurfing among them.…
Randomized experiments play a major role in data-driven decision making across many different fields and disciplines. In medicine, for example, randomized controlled trials (RCTs) are the backbone of clinical trial methodology for testing…
The Internet service provider industry is currently experiencing intense competition as companies strive to provide top-notch services to their customers. Providers are introducing cutting-edge technologies to enhance service quality,…
Modern data centers suffer from immense power consumption. As a result, data center operators have heavily invested in capacity scaling solutions, which dynamically deactivate servers if the demand is low and activate them again when the…
Sharing economy platforms have become extremely popular in the last few years, and they have changed the way in which we commute, travel, and borrow among many other activities. Despite their popularity among consumers, such companies are…