(M/F) Postdoctoral Researcher Vacancy - - Interactive Learning of Gestural Control of Intonation

Job Description

Missions

In oral interactions, speech prosody, which includes intonation and rhythm, is a dedicated channel of communication that carries both speech structuring information (e.g., syntactic boundaries, contrastive focus) and expressiveness (e.g., attitudes or emotions). For patients with laryngeal impairments, the loss over intonation control severely impacts speech intelligibility and social interactions. Related voice substitution solutions aim to reconstruct degraded speech waveforms using alternative sources of information for intonation. For example, it has been shown in previous studies that participants can control very accurately the intonation contour of a sentence by drawing a given example contour on a graphic tablet. Nevertheless this has been proved much challenging when controlling expressive intonation patterns, and when control was done spontaneously (without any reference) in synchrony with articulation. In parallel, a line of study dedicated to gesture-to-sound-mapping has developed advanced Mapping-by-Demonstration techniques, a user-centred paradigm where individuals perform example gestures while demonstrating the desired sound output. The system then learns and generalises these associations, allowing personalised and adaptive control over real-time audio generation.
Objectives
We propose in this Post Doctoral project to leverage Mapping-by-Demonstration techniques for the gestural control of expressive intonation variations in a real-time speech reconstruction system. The two following steps will be addressed:
1. Interactive learning of intonation control: The Post Doctoral candidate will implement Mapping-by-Demonstration techniques using the MuBu library to map users' gestures to typical expressive intonation patterns extracted from an in-lab expressive dataset. The first step will include the control of temporality: the gesture will serve as control to synchronise the intonation pattern to the sentence articulated by the user. The second step will add the control of variants of the pre-defined intonation patterns leveraging gesture variability. Various interfaces for hand gesture acquisition will be explored, including 2D (tablet) and 3D (accelerometer) movements.
2. Evaluation in a behavioural study: The Post Doctoral candidate will design experimental protocols to evaluate the performance of the system. This will include (1) the ease of learning new gestures with Mapping-by-Demonstration ; (2) the quality of control after learning. Experiments involving a number of testing participants will be conducted accordingly. For both cases, evaluation criteria will be defined, and metrics will be extracted using objective and/or perceptual tests.

Activity

- Getting familiar with the MuBu library in the Max/MSP environment for Mapping-by-Demonstration
- Identifying typical intonation patterns in our corpus of expressive speech
- Implementing Mapping-by-Demonstration to control these intonation patterns through gesture (temporality and variability)
- Integration with our real-time speech reconstruction module
- Design of an evaluation protocol in terms of communication tasks and objective and/or subjective metrics
- Evaluation on a group of users and analysis of the results
- Writing a journal or conference paper

Your Profil

Skills

Those without expertise in some of the listed areas are nevertheless encouraged to apply.
- Experience in human-computer interfaces / gesture-sound mapping
- Knowledge of machine learning and signal processing
- Knowledge of speech science and prosody
- Real-time Max/MSP programming
- Strong motivation to conduct behavioural experiments involving human participants

Your Work Environment

The Gipsa-lab is a joint research laboratory of the CNRS, Grenoble-INP -UGA and the University of Grenoble Alpes. It is under agreement with Inria and the Observatory of Sciences of the Universe of Grenoble. He conducts theoretical and applied research on AUTOMATICS, SIGNAL, IMAGES, SPEECH, COGNITION, ROBOTICS and LEARNING.
Multidisciplinary and at the interface between the human, the physical and digital worlds, our research is confronted with measurements, data, observations from physical, physiological and cognitive systems. They focus on the design of methodologies and algorithms for processing and extracting information, decisions, actions and communications that are viable, efficient and compatible with physical and human reality. Our work is based on mathematical and computer theories for the development of models and algorithms, validated by hardware and software implementations.
By relying on its platforms and partnerships, Gipsa-lab maintains a constant link with applications in a wide variety of fields: health, environment, energy, geophysics, embedded systems, mechatronics, processes and industrial systems, telecommunications, networks, transport and vehicles, operational safety and security, human-computer interaction, linguistic engineering, physiology and biomechanics, etc.
Due to the nature of its research, Gipsa-lab is in direct and constant contact with the economic environment and society.
Its potential as teacher-researchers and researchers is invested in training at the level of universities and engineering schools on the Grenoble site (Grenoble Alpes University).
Gipsa-lab develops its research through 16 teams or themes organized into 4 divisions:
• Automatic and Diagnosis (PAD)
• Data Science (PSD)
• Speech and Cognition (PPC)
• Geometries, Learning, Information and Algorithms (GAIA).
The staff supporting research (38 engineers and technicians) is distributed in the common services distributed within 2 divisions:
• The Administrative and Financial Pole
• The Technical Pole
Gipsa-lab has around 150 permanent staff, including 70 teacher-researchers and 41 researchers. It also welcomes guest researchers and post-docs.
Gipsa-lab supervises nearly 150 theses, including around 50 new ones each year. All the theses carried out in the laboratory are financed and supervised by teacher-researchers and researchers, including 50 holders of an HDR.
Finally, around sixty Master's trainees come each spring to swell the ranks of the laboratory.
This Post Doctoral position is part of the SilentPitch ANR project which involves a puri-disciplinary team of researchers including machine learning, speech science, cognition and behavioural studies.
This Post Doctoral position will be co-supervised by Olivier Perrotin (GIPSA-lab, Grenoble) and Jules Françoise (LISN, Orsay). The post-doctoral researcher will be based in the ComLearn team within the Speech and Cognition department at GIPSA-lab, and will work primarily in Grenoble. Occasional visits to LISN will be scheduled throughout the duration of the project..

Constraints and risks

None

Compensation and benefits

Compensation

Between €3,041.58 and €4,216.70 depending on experience

Annual leave and RTT

44 jours

Remote Working practice and compensation

Pratique et indemnisation du TT

Transport

Prise en charge à 75% du coût et forfait mobilité durable jusqu’à 300€

About the offer

About the CNRS

The CNRS is a major player in fundamental research on a global scale. The CNRS is the only French organization active in all scientific fields. Its unique position as a multi-specialist allows it to bring together different disciplines to address the most important challenges of the contemporary world, in connection with the actors of change.

CNRS

The research professions