Related papers: Designing and Deploying Online Field Experiments
Conventional wisdom holds that model-based planning is a powerful approach to sequential decision-making. It is often very challenging in practice, however, because while a model can be used to evaluate a plan, it does not prescribe how to…
We describe the experimental methodology developed and employed in a series of experiments within the Defense Advanced Research Projects Agency (DARPA) Conflict Modeling, Planning, and Outcomes Exploration (COMPOEX) Program. The primary…
A/B experiments are commonly used in research to compare the effects of changing one or more variables in two different experimental groups - a control group and a treatment group. While the benefits of using A/B experiments are widely…
Policymakers must often act under conditions of deep uncertainty, such as emergency response, where predicting the specific impacts of a policy apriori is implausible. Large Language Model (LLM) agent simulations have been proposed as tools…
Performing controlled user experiments on small devices in naturalistic mobile settings has always proved to be a difficult undertaking for many Human Factors researchers. Difficulties exist, not least, because mimicking natural small…
Thought experiments are considered valuable tools in science, enabling the exploration of hypotheses and the examination of complex ideas in a conceptual, non-empirical framework. These thought experiments can be useful in design fiction…
Scientific laboratories are among the most challenging course components to integrate into online instruction. Available technology restricts the design and nature of experiments and it can be hard to replicate the collaborative lab…
A discourse strategy is a strategy for communicating with another agent. Designing effective dialogue systems requires designing agents that can choose among discourse strategies. We claim that the design of effective strategies must take…
A cognitive handoff is a multipurpose handoff that achieves many desirable features simultaneously; e.g., seamlessness, autonomy, security, correctness, adaptability, etc. But, the development of cognitive handoffs is a challenging task…
This paper sets out a process of app analysis intended to support understanding of use but also redesign. From usage logs we infer activity patterns - Markov models - and employ probabilistic formal analysis to ask questions about the use…
Web Data Extraction is an important problem that has been studied by means of different scientific tools and in a broad range of applications. Many approaches to extracting data from the Web have been designed to solve specific problems and…
Socio-technical design embeds social investigations and inquiries into (Information) Technology Design processes. In this position paper, we propose, by using the aforementioned approach the design of technology and policies can…
We consider the joint problem of online experiment design and parameter estimation for identifying nonlinear system models, while adhering to system constraints. We utilize a receding horizon approach and propose a new adaptive input design…
Recent years have seen substantial progress in automated design-to-code generation, with many methods proposed for generating HTML and CSS from webpage screenshots. However, the absence of a standardized evaluation platform makes it…
The quality of software is enhanced by using the design patterns. The design patterns are the reusable component used in the development of the software, which delivers improved quality software to the end users. The researchers have…
Change-based testing is a key component of continuous integration at Facebook. However, a large number of tests coupled with a high rate of changes committed to our monolithic repository make it infeasible to run all potentially-impacted…
In recent years, the research community has raised serious questions about the reproducibility of scientific work. In particular, since many studies include some kind of computing work, reproducibility is also a technological challenge, not…
In data analysis, unexpected results often prompt researchers to revisit their procedures to identify potential issues. While some researchers may struggle to identify the root causes, experienced researchers can often quickly diagnose…
Generating plans of action, and reasoning about change have long been considered a core competence of intelligent agents. It is thus no surprise that evaluating the planning and reasoning capabilities of large language models (LLMs) has…
One type of question that is commonly found in day-to-day scenarios is ``fan-out'' questions, complex multi-hop, multi-document reasoning questions that require finding information about a large number of entities. However, there exist few…