Related papers: Vrengt: A Shared Body-Machine Instrument for Music…

Music is both an auditory and an embodied phenomenon, closely linked to human motion and naturally expressed through dance. However, most existing audio representations neglect this embodied dimension, limiting their ability to capture…

Painting and music therapy approaches can help to foster social interaction for autistic people. However, the tools sometimes lack of flexibility and fail to keep people's attention. Unknowns also remain about the effect of combining these…

Visually impaired people encounter many challenges in their everyday life, especially when it comes to navigating and representing space. The issue of shopping is addressed mostly on the level of navigation and product detection, but…

We introduce Multimodal DuetDance (MDD), a diverse multimodal benchmark dataset designed for text-controlled and music-conditioned 3D duet dance motion generation. Our dataset comprises 620 minutes of high-quality motion capture data…

Drumming belongs to a family of musical instruments whose practice, whether as an amateur or at a high level, is associated with an increased risk of musculoskeletal disorders (MSD), particularly of the upper limbs and lumbar spine. The…

We present DuetGen, a novel framework for generating interactive two-person dances from music. The key challenge of this task lies in the inherent complexities of two-person dance interactions, where the partners need to synchronize both…

Since 2014, we have been conducting experiments based on a multidisciplinary collaboration between specialists in theatrical staging and researchers in virtual reality, digital art, and video games. This team focused its work on the…

Dance and music typically go hand in hand. The complexities in dance, music, and their synchronisation make them fascinating to study from a computational creativity perspective. While several works have looked at generating dance for a…

We present AlignNet, a model that synchronizes videos with reference audios under non-uniform and irregular misalignments. AlignNet learns the end-to-end dense correspondence between each frame of a video and an audio. Our method is…

People may perform diverse gestures affected by various mental and physical factors when speaking the same sentences. This inherent one-to-many relationship makes co-speech gesture generation from audio particularly challenging.…

Breathing is a spontaneous but controllable body function that can be used for hands-free interaction. Our work introduces "iBreath", a novel system to detect breathing gestures similar to clicks using bio-impedance. We evaluated iBreath's…

Music-driven choreography is a challenging problem with a wide variety of industrial applications. Recently, many methods have been proposed to synthesize dance motions from music for a single dancer. However, generating dance motion for a…

In this study, we aim to determine if generalized sounds and music can share a common emotional space, improving predictions of emotion in terms of arousal and valence. We propose the use of multiple datasets as a multi-domain learning…

Robotic sensorimotor extensions (supernumerary limbs, prosthesis, handheld tools) are worn devices used to interact with the nearby environment, whether to assist the capabilities of impaired users or to enhance the dexterity of industrial…

The thud of a bouncing ball, the onset of speech as lips open -- when visual and audio events occur together, it suggests that there might be a common, underlying event that produced both signals. In this paper, we argue that the visual and…

In recent years, deep learning algorithms have become increasingly more prominent for their unparalleled ability to automatically learn discriminant features from large amounts of data. However, within the field of electromyography-based…

Optical vibration sensing enables recovering the scene sound directly from the surface vibration of nearby objects, turning everyday objects into ``visual microphones''. However, most prior methods had focused on capturing the vibrations of…

Music source separation aims to separate polyphonic music into different types of sources. Most existing methods focus on enhancing the quality of separated results by using a larger model structure, rendering them unsuitable for deployment…

In order to collaborate and co-create with humans, an AI system must be capable of both reactive and anticipatory behavior. We present a case study of such a system in the domain of musical improvisation. We consider a duo consisting of a…

The task of music-driven dance generation involves creating coherent dance movements that correspond to the given music. While existing methods can produce physically plausible dances, they often struggle to generalize to out-of-set data.…