PhD position M/F : Multimodal Perception and Hybrid Motion Control Generation in Robot Manipulation

Job Description

Thesis Subject

Robots are increasingly expected to leave structured industrial cells and operate in open environments such as farms, workshops, warehouses, hospitals, homes or outdoor intervention sites. Recent progress in humanoid locomotion and manipulation has shown that robots can become increasingly capable of execution complex tasks in unstructured environments [6]. In parallel, advances in foundation models are opening new possibilities for integrating semantic understanding, task planning, and robot action generation. In particular, vision-language-action models offer a promising direction towards more versatile and generalizable robotic systems. But, beyond understanding task objectives and generating actions, robots must be able to handle complex physical interactions and account for the properties and motion of manipulated objects. Achieving these capabilities in unstructured environments remains an open research challenge, requiring further advances in multimodal perception, dexterous manipulation, and adaptive control.
This PhD project addresses robotic manipulation in less controlled environments, where classical rigid-object manipulation based mainly on 6D pose estimation is only a partial answer. Many objects encountered in practice are deformable, articulated, flexible, partially occluded or affected by contacts during interaction. Their state cannot always be reduced to a rigid pose: shape, local geometry, contact conditions, grasp stability, material response and task-related affordances may all become relevant for action.
Recent reviews on robotic cloth manipulation [2], as well as benchmark efforts on grasp selection such as [5], show that deformable like cloth-like objects clearly expose the limitations of pose-based manipulation. The robot must reason about shape evolution, contacts and partial observability rather than only estimating a rigid transformation.
This is where hybridization becomes essential. Model-based approaches bring structure, physical consistency and control-oriented constraints, but they are often incomplete when the robot interacts with objects whose geometry, material properties and contact conditions are only partially known. As discussed in [1], data-driven approaches can learn visual descriptors, latent states, residual dynamics or action priors from experience, but they may require large datasets and do not naturally provide safety or stability guarantees. Real-world manipulation therefore calls for methods that combine both: models to constrain and guide action, and data to adapt to phenomena that are difficult to model explicitly.
The thesis is positioned within the PEPR Robotique, and more specifically within HAMMER project, dedicated to the hybridization of model-based and data-driven methods for motion generation in robotics. The proposed work will contribute to this PEPR objective through the context of manipulation. Recent multimodal affordance representations [3] also illustrate the interest of combining external visual perception with local feedback for manipulation. The central idea is to develop hybrid perception-control strategies that combine geometric and physical priors, learned representations, multimodal sensing and safety-aware control, so that the robot can act with both adaptability and physical consistency.
[1] Ai, B., et al. “A review of learning-based dynamics models for robotic manipulation.” Science Robotics (2025)
[2] Longhini, A., et al. “Unfolding the Literature: A Review of Robotic Cloth Manipulation.” Annual Review of Control, Robotics, and Autonomous Systems (2025).
[3] Wu, Q., et al. “TARS: Tactile Affordance in Robot Synesthesia for Dexterous Manipulation.” IEEE Robotics and Automation Letters (2025)
[4] Zhao, W., et al. “VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation.” Robotics: Science and Systems. (2024)
[5] De Gusseme, V.-L., et al. “A Dataset and Benchmark for Robotic Cloth Unfolding Grasp Selection: The ICRA 2024 Cloth Competition.” The International Journal of Robotics Research. (2026)
[6] Gu, Zhaoyuan et al. “Humanoid Locomotion and Manipulation: Current Progress and Challenges in Control, Planning, and Learning” IEEE/ASME Transactions on Mechatronics 31.2 (2026)

Your Work Environment

This PhD student will involve on the target project HAMMER of the PEPR Robotic
The thesis will focus on three main contributions:
• Represent object high-dimensional states. Build compact multimodal representations combining RGB-D geometry, local shape, contact cues, proprioceptive signals and learned descriptors.
• Hybridize model-based and data-driven control. Develop closed-loop manipulation strategies where physical/geometric constraints guide learning, and learned components improve adaptation to uncertainty.
• Ground high-level action priors in physical feedback. Use foundation models or vision-language-action models to propose goals, actions or manipulation primitives, and refine them through multimodal feedback, model-based constraints and safety-aware control.
Experimental work will be carried out on a Franka Research 3 (FR3) equipped with different sensing modalities. The expected outcome is a hybrid perception-control framework for real-world robotic manipulation, where the robot adapts its actions from rich object-state representations and real-time physical feedback.

Compensation and benefits

Compensation

2300 € gross monthly

Annual leave and RTT

44 jours

Remote Working practice and compensation

Pratique et indemnisation du TT

Transport

Prise en charge à 75% du coût et forfait mobilité durable jusqu’à 300€

About the offer

About the CNRS

The CNRS is a major player in fundamental research on a global scale. The CNRS is the only French organization active in all scientific fields. Its unique position as a multi-specialist allows it to bring together different disciplines to address the most important challenges of the contemporary world, in connection with the actors of change.

CNRS

The research professions