Related papers: Interacto: A Modern User Interaction Processing Mo…
User interaction (UI) logs are high-resolution event logs that record low-level activities performed by a user during the execution of a task in an information system. Each event in a UI log corresponds to a single interaction between the…
As mobile devices are becoming ubiquitous, regularly interacting with a variety of user interfaces (UIs) is a common aspect of daily life for many people. To improve the accessibility of these devices and to enable their usage in a variety…
Human-Computer Interaction with the traditional User Interface is done using a specified in advance script dialog menu, mainly based on human intellect and unproductive use of navigation. This approach does not lead to making qualitative…
In an era where user interaction with technology is ubiquitous, the importance of user interface (UI) design cannot be overstated. A well-designed UI not only enhances usability but also fosters more natural, intuitive, and emotionally…
We address the problem of accurate capture of interactive behaviors between two people in daily scenarios. Most previous works either only consider one person or solely focus on conversational gestures of two people, assuming the body…
The automobile is always a point of interest where new technology has been deployed. Because of this interest, human-vehicle interaction has been an appealing area for much research in recent years. The current in-vehicle design has been…
Traditional recommender systems present a relatively static list of recommendations to a user where the feedback is typically limited to an accept/reject or a rating model. However, these simple modes of feedback may only provide limited…
The availability of interaction devices has raised interest in techniques to support the user interface (UI). A UI specification describes the functions that a system provides to its users by capturing the interface details and includes…
Understanding user interface (UI) functionality is a useful yet challenging task for both machines and people. In this paper, we investigate a machine learning approach for screen correspondence, which allows reasoning about UIs by mapping…
Computer-use agents (CUAs) automate on-screen work, as illustrated by GPT-5.4 and Claude. Yet their reliability on complex, low-frequency interactions is still poor, limiting user trust. Our analysis of failure cases from advanced models…
With the large language model showing human-like logical reasoning and understanding ability, whether agents based on the large language model can simulate the interaction behavior of real users, so as to build a reliable virtual…
We address the problem of accurate capture and expressive modelling of interactive behaviors happening between two persons in daily scenarios. Different from previous works which either only consider one person or focus on conversational…
Today, software industry has a rapid growth. In order to resist the competition increased by this growth, software projects need to be developed with higher quality and especially user friendly. Therefore, the importance of human-computer…
Graphical User Interface (GUI) agents, powered by Large Foundation Models, have emerged as a transformative approach to automating human-computer interaction. These agents autonomously interact with digital systems or software applications…
To harness the potential of advanced computing technologies, efficient (real time) analysis of large amounts of data is as essential as are front-line simulations. In order to optimise this process, experts need to be supported by…
Ideas about how to increase the unconscious participation in interaction between 'a human' and 'a computer' are developed in this paper. Evidence of impact of the unconscious functioning is presented. The unconscious is characterised as…
Human computer interaction is shifting from screen-based systems to multimodal interfaces where artificial intelligence powered systems increasingly interpret user intent through speech, gesture, and gaze. Yet users rarely understand how…
Text prompt is the most common way for human-generative AI (GenAI) communication. Though convenient, it is challenging to convey fine-grained and referential intent. One promising solution is to combine text prompts with precise GUI…
The current software development tools show the same form of interaction as when they started back, in the mid 70's. However, since the appearance of visual languages and due to their own nature, they can be handled by tools which have…
Following established tradition, software engineering today is rooted in a conceptually centralized way of thinking. The primary SE artifact is a specification of a machine -- a computational artifact -- that would meet the (elicited and)…