Related papers: The future of forecasting competitions: Design att…
Over-fitting is a dreaded foe in challenge-based competitions. Because participants rely on public leaderboards to evaluate and refine their models, there is always the danger they might over-fit to the holdout data supporting the…
In this paper, we propose AutoCompete, a highly automated machine learning framework for tackling machine learning competitions. This framework has been learned by us, validated and improved over a period of more than two years by…
Computer modeling is essential to research on Advanced Accelerator Concepts (AAC), as well as to their design and operation. This paper summarizes the current status and future needs of AAC systems and reports on several key aspects of (i)…
Ontologies are known for their ability to organize rich metadata, support the identification of novel insights via semantic queries, and promote reuse. In this paper, we consider the problem of automated planning, where the objective is to…
We introduce the fundamental ideas and challenges of Predictable AI, a nascent research area that explores the ways in which we can anticipate key validity indicators (e.g., performance, safety) of present and future AI ecosystems. We argue…
Affordances are key attributes of what must be perceived by an autonomous robotic agent in order to effectively interact with novel objects. Historically, the concept derives from the literature in psychology and cognitive science, where…
A number of scientific competitions have been organised in the last few years with the objective of discovering innovative techniques to perform typical High Energy Physics tasks, like event reconstruction, classification and new physics…
The increasing recognition of the association between adverse human health conditions and many environmental substances as well as processes has led to the need to monitor them. An important problem that arises in environmental statistics…
Assessing forecasting performance is a time intensive activity, often requiring months or years before we know whether or not the reported forecasts were accurate. Cognitive tests can be quickly administered and are predictive of…
A long noted difficulty when assessing the reliability (or calibration) of forecasting systems is that reliability, in general, is a hypothesis not about a finite dimensional parameter but about an entire functional relationship. A…
A key aspect of the design of evolutionary and swarm intelligence algorithms is studying their performance. Statistical comparisons are also a crucial part which allows for reliable conclusions to be drawn. In the present paper we gather…
Predicting the outcome of sports events is a hard task. We quantify this difficulty with a coefficient that measures the distance between the observed final results of sports leagues and idealized perfectly balanced competitions in terms of…
We explore the fundamental problem of sorting through the lens of learning-augmented algorithms, where algorithms can leverage possibly erroneous predictions to improve their efficiency. We consider two different settings: In the first…
We study the design of effort-maximizing grading schemes between agents with private abilities. Assuming agents derive value from the information their grade reveals about their ability, we find that more informative grading schemes induce…
A desirable property of an intelligent agent is its ability to understand its environment to quickly generalize to novel tasks and compose simpler tasks into more complex ones. If the environment has geometric or arithmetic structure, the…
Combining quantum computers with classical compute power has become a standard means for developing algorithms that are eventually supposed to beat any purely classical alternatives. While in-principle advantages for solution quality or…
We present a new way of estimation of the role of chance in achieving success, by comparing the empirical data from 100-meter dash competitions (one of the sports disciplines with the most stringent controls of external randomness), with…
We consider the problem of constructing optimal designs for model discrimination between competing regression models. Various new properties of optimal designs with respect to the popular $T$-optimality criterion are derived, which in many…
Attribute-driven software architecture design aims to provide decision support by taking into account the quality attributes of softwares. A central question in this process is: What architecture design best fulfills the desirable software…
Prediction problems often admit competing models that perform almost equally well. This effect challenges key assumptions in machine learning when competing models assign conflicting predictions. In this paper, we define predictive…