Related papers: GuiLiner: A Configurable and Extensible Graphical …

We create a Python GUI scripting interface working under Windows in addition to (UNIX/Linux). The GUI has been built around the Python open-source programming language. We use the Python's GUI library that so called Python Mega Widgets…

Unlike static and rigid user interfaces, generative and malleable user interfaces offer the potential to respond to diverse users' goals and tasks. However, current approaches primarily rely on generating code, making it difficult for…

In this paper, we define visual log of a software system as data capturing the interactions between its users and its graphic user interface (GUI), such as screen-shots and screen recordings. We vision that mining such visual log could be…

Graphical User Interface (GUI) agents possess significant commercial and social value, and GUI agents powered by advanced multimodal large language models (MLLMs) have demonstrated remarkable potential. Currently, existing GUI agents…

This paper contains a brief discussion of the Application Development Environment (ADE) that is used to build database applications involving the graphical user interface (GUI). ADE computing separates the database access and the user…

Graphical User Interface (GUI) agents adopt an end-to-end paradigm that maps a screenshot to an action sequence, thereby automating repetitive tasks in virtual environments. However, existing GUI agents are evaluated almost exclusively on…

Simulating nuanced user experiences within complex interactive search systems poses distinct challenge for traditional methodologies, which often rely on static user proxies or, more recently, on standalone large language model (LLM) agents…

The enabling of scientific experiments that are embarrassingly parallel, long running and data-intensive into a cloud-based execution environment is a desirable, though complex undertaking for many researchers. The management of such…

Recent years have witnessed a rapid development of mobile GUI agents powered by large language models (LLMs), which can autonomously execute diverse device-control tasks based on natural language instructions. The increasing accuracy of…

We aim to increase the flexibility at which a data worker can choose the right tool for the job, regardless of whether the tool is a code library or an interactive graphical user interface (GUI). To achieve this flexibility, we extend…

We present a workflow and associated toolkit to automate the creation of graphical user interfaces (GUI) for executables run from command line interfaces (CLI). The workflow consists of three phases, namely (Step 1) the plugin design, (Step…

Mobile GUI agents execute user commands by directly interacting with the graphical user interface (GUI) of mobile devices, demonstrating significant potential to enhance user convenience. However, these agents face considerable challenges…

When designing a new presentation front-end called FlexiView for requirements modeling tools, we encountered a general problem: designing such an interface requires a lot of experimentation which is costly when the code of the tool needs to…

Recently, Multimodal Large Language Models (MLLMs) have been used as agents to control keyboard and mouse inputs by directly perceiving the Graphical User Interface (GUI) and generating corresponding commands. However, current agents…

User-driven applications are the programs, in which the full control is given to the users. Designers of such programs are responsible only for developing an instrument for solving some task, but they do not enforce users to work with this…

Software testing is a fundamental process of software development, and prior work has shown that visualizations of test results support testers' decision-making. However, Human-Computer Interaction research on software testing has yet to…

Software frameworks for neural networks play a key role in the development and application of deep learning methods. In this paper, we introduce the Chainer framework, which intends to provide a flexible, intuitive, and high performance…

Graphical User Interface (GUI) has become one of the most significant parts of mobile applications (apps). It is a direct bridge between mobile apps and end users, which directly affects the end user's experience. Neglecting GUI quality can…

Developing and testing user interfaces (UIs) and training AI agents to interact with them are challenging due to the dynamic and diverse nature of real-world mobile environments. Existing methods often rely on cumbersome physical devices or…

Oracles used for testing graphical user interface (GUI) programs are required to take into consideration complicating factors such as variations in screen resolution or color scheme when comparing observed GUI elements to expected GUI…