Related papers: mage: Fluid Moves Between Code and Graphical Work …
Data science workflows are human-centered processes involving on-demand programming and analysis. While programmable and interactive interfaces such as widgets embedded within computational notebooks are suitable for these workflows, they…
Computational notebooks, while essential for data science, are limited by their one-dimensional interface, which poorly aligns with non-linear developer workflows and complicates collaboration and human-AI interaction. In this work, we…
Graphical User Interface (GUI) agents, powered by Large Foundation Models, have emerged as a transformative approach to automating human-computer interaction. These agents autonomously interact with digital systems or software applications…
Computer-Use Agents (CUA) are becoming increasingly capable of autonomously operating digital environments through Graphical User Interfaces (GUI). Yet, most GUI remain designed primarily for humans--prioritizing aesthetics and…
The increasing integration of artificial intelligence (AI) in everyday life brings with it new challenges and questions for regarding how humans interact with autonomous agents. Multi-agent experiments, where humans and AI act together, can…
This paper presents MagicGUI, a foundational mobile GUI agent designed to address critical challenges in perception, grounding, and reasoning within real-world mobile GUI environments. The framework is underpinned by following six key…
This study examines the role of vagueness in the design process and its strategic management for the effective human-AI interaction. While vagueness in the generation of design ideas promotes diverse interpretations and prevents fixation,…
The introduction of new tools in people's workflow has always been promotive of new creative paths. This paper discusses the impact of using computational tools in the performance of creative tasks, especially focusing on graphic design.…
Generative models are now capable of producing natural language text that is, in some cases, comparable in quality to the text produced by people. In the computing education context, these models are being used to generate code, code…
In this paper, we present the computational task-management tool Ganga, which allows for the specification, submission, bookkeeping and post-processing of computational tasks on a wide set of distributed resources. Ganga has been developed…
Interactive visualization can support fluid exploration but is often limited to predetermined tasks. Scripting can support a vast range of queries but may be more cumbersome for free-form exploration. Embedding interactive visualization in…
Highly customized graphical user interfaces play a major role in today's software applications. There exist many technologies that support developing them. But these technologies demand writing a significant amount of unintuitive code to…
The Grid and Cloud User Support Environment (gUSE) enables users convenient and easy access to grid and cloud infrastructures by providing a general purpose, workflow-oriented graphical user interface to create and run workflows on various…
Game development-based approaches are increasingly used to design curricula that can engage students, as these can help them apply and practice learnt computer science concepts. However, it can become complex to develop a minimum working…
Technological progress has persistently shaped the dynamics of human-machine interactions in task execution. In response to the advancements in Generative AI, this paper outlines a detailed study plan that investigates various human-AI…
AI-powered code assistants, such as Copilot, are quickly becoming a ubiquitous component of contemporary coding contexts. Among these environments, computational notebooks, such as Jupyter, are of particular interest as they provide rich…
Artificial Intelligence (AI) has been increasingly applied to creative domains, leading to the development of systems that collaborate with humans in design processes. In Graphic Design, integrating computational systems into co-creative…
Software development of modern, data-driven applications still relies on tools that use interaction paradigms that have remained mostly unchanged for decades. While rich forms of interactions exist as an alternative to textual command…
Graphical User Interface (GUI) agents have the potential to assist users in interacting with complex software (e.g., PowerPoint, Photoshop). While prior research has primarily focused on automating user actions through clicks and…
Divergent thinking in the ideation stage of creative problem-solving demands that individuals explore a broad design space. Yet this exploration rarely follows a neat, linear sequence; problem-solvers constantly shift among searching,…