PhD in Humanoid Robotics - Safe Renforcement Learning (M/F)

Job Description

Thesis Subject

2. Thesis Statement
This thesis aims to develop a hybrid dynamic locomotion architecture in which a reinforcement-learning (RL) policy makes discrete, high-level decisions — such as contact mode, footstep or gait-pattern selection, and behavior transitions — while an online-formulated and online-parametrized Model Predictive Controller enforces the robot's physical constraints and produces the continuous, whole-body motion. The MPC guarantees feasibility, safety, and dynamic consistency, while the RL component provides adaptation, anticipation, and strategic decision-making in complex, uncertain, or previously unseen situations.
3. State of the Art
Legged and humanoid locomotion over unstructured or dynamic terrain requires two qualitatively different kinds of reasoning. On one hand, choosing where and how to make or break contact (which foot, which surface, which timing, which gait) is fundamentally a combinatorial, discrete decision problem that is difficult to encode as a smooth cost or convex constraint. On the other hand, once a contact plan or behavior mode is chosen, generating a dynamically consistent, torque- and constraint-feasible trajectory is a continuous optimization problem for which model-based whole-body control and Model Predictive Control (MPC) are mature and well understood. This section reviews the relevant state of the art along five axes: model-based whole-body and predictive control, contact-implicit and hybrid MPC, safe reinforcement learning and execution-time safety filters, approaches that couple RL and MPC, and learned discrete mode selection for hybrid legged locomotion.
3.1 Model-based whole-body control and predictive control for legged robots
Quadratic-Program-based whole-body control has become the standard formulation for prioritized task execution under joint, torque, and contact constraints on legged and humanoid robots [1]–[3], with Hierarchical QP providing strict task prioritization for fast online motion generation [4]. A comprehensive recent survey consolidates the numerical algorithms and modeling choices behind optimization-based control for dynamic legged robots, including centroidal, whole-body, and contact-implicit formulations [5]. These model-based approaches provide strong constraint-satisfaction guarantees and generalize well across tasks, but they depend on accurate dynamic models and require an externally supplied contact plan or sequence, which is precisely the combinatorial decision this thesis proposes to delegate to a learned policy.
3.2 Contact-implicit and hybrid Model Predictive Control
A growing body of work seeks to let MPC reason directly about contact rather than relying on a pre-specified contact sequence. Hybrid iterative-LQR MPC modifies the cost formulation to handle non-aligned contact modes and has been shown to outperform centroidal approaches under large perturbations on hardware [6]. Fast contact-implicit MPC formulations exploit smoothed or relaxed complementarity constraints to make contact-implicit optimization tractable at control-loop rates [7], and related work demonstrates contact-implicit MPC producing diverse quadruped motions without any pre-planned contact mode or trajectory [8]; inverse-dynamics formulations of contact-implicit MPC further improve numerical robustness [9]. These methods show that reasoning jointly about contact and motion is possible in a receding-horizon setting, but they remain purely model-based: the contact mode emerges from a local numerical relaxation rather than from a policy that has learned to anticipate which mode is strategically preferable, which is the gap this thesis targets by letting a trained RL policy supply that discrete decision.
3.3 Safe reinforcement learning and execution-time safety filters
Safe Reinforcement Learning typically embeds constraints during training, for instance through Constrained Policy Optimization, which enforces a per-iteration trust-region safety guarantee [12], or through Constraints as Terminations (CaT), which converts constraint violations into early episode terminations for legged locomotion RL [14]. A recent survey positions these training-time approaches alongside execution-time methods and highlights the persistent gap between statistical, training-time safety and the hard guarantees required for real-world deployment [11]. Execution-time safety filters address this gap directly: predictive safety filters wrap an arbitrary controller with a model-based corrective layer [13], multi-layered schemes combine control barrier functions with MPC to guarantee legged-robot safety even under large disturbances [10], and, most directly relevant to this thesis, a second-order Quadratic-Program safety filter has recently been demonstrated to constrain RL policy outputs at runtime, in acceleration space, on both a torque-controlled manipulator and a position-controlled Unitree H1 humanoid, using a torque or forward-dynamics task to reproduce the RL policy's intended behavior together with a disturbance observer to compensate model error [15]. This filter builds on an underlying physical-human-robot-interaction control framework developed for the same class of platforms [16]. The QP safety filter in [15] guarantees strict, single-step constraint satisfaction for any RL policy without retraining, but it acts only on the continuous action already produced by the policy; it neither reasons about, nor reshapes, the discrete contact or behavior decisions that precede that action — which is exactly the extension this thesis pursues, moving from single-step QP filtering to an online, horizon-based MPC shaped by the policy's own discrete choices.
3.4 Coupling reinforcement learning and Model Predictive Control
A separate line of work seeks to combine RL and MPC more tightly than a post-hoc filter. Policy-search-for-MPC treats hard-to-optimize MPC decision variables as the object of an offline-learned, adaptively-selected policy, demonstrated on agile drone flight [17], and this idea has been extended into an Actor-Critic MPC architecture that embeds a differentiable MPC layer inside an actor-critic loop, improving out-of-distribution robustness and sample efficiency [18]. These approaches build on the broader family of differentiable MPC methods, which back-propagate gradients through the optimization layer itself to enable end-to-end learning of cost or model parameters [19]. Contact scheduling has also been treated as a bi-level learning problem, for instance by using Bayesian optimization to select a contact schedule for a downstream trajectory optimizer from high-level task descriptors [20]. Across this literature, the pattern that emerges is that the greatest gains come from using RL to select or shape the small set of hard, combinatorial, or hard-to-differentiate decision variables of an MPC problem, while leaving continuous constraint satisfaction to the MPC itself — which is precisely the division of labor targeted by this thesis, applied specifically to contact-mode and behavior-transition decisions in legged and wheeled-legged locomotion.
3.5 Learned discrete mode selection for hybrid legged and wheeled-legged locomotion
A final and directly relevant strand of work studies how RL policies can represent and select discrete locomotion modes. Discrete-time hybrid automata learning embeds an explicit, unsupervised mode selector inside the RL pipeline itself, allowing a quadruped to identify and switch between physically distinct dynamic modes without hand-labeled segmentation, demonstrated on a contact-rich skateboarding task [21]. These results confirm both that discrete mode information is a powerful and learnable signal for legged and wheeled-legged RL policies, and that mode switching remains handled internally to the policy, with no formal guarantee that the resulting behavior respects the robot's physical constraints once realized on hardware — the gap this thesis addresses by routing the learned discrete decision through an online MPC rather than a black-box low-level controller.
3.6 Positioning of this thesis
Taken together, the state of the art shows mature, separate solutions to two different halves of the problem: model-based MPC and QP methods that guarantee constraint satisfaction given a contact plan [1]–[9], and RL methods that can learn to select or represent discrete contact and behavior decisions but without formal safety guarantees [21], [22], with safety filters offering strict guarantees only for already-continuous policy outputs [10], [11], [13], [15]. Coupled RL-MPC architectures show that learned components can profitably shape a small set of hard MPC decision variables [17]–[20], but this idea has not yet been applied to online contact-mode and behavior-transition decisions validated on real legged or wheeled-legged hardware. This thesis is positioned precisely at that intersection: it extends the single-step, continuous-action QP safety filter already demonstrated on manipulator and humanoid hardware [15], [16] into a receding-horizon MPC whose contact schedule and cost structure are shaped online by an RL policy's discrete decisions, targeting validation on the TIRREX/RENOIR Kangaroo platform [22] and Unitree humanoids, whose wheeled-legged and bipedal morphologies respectively echo the discrete mode-switching phenomena studied in [21].
The central scientific bet of this thesis is that these two paradigms are complementary rather than competing: RL is well suited to discrete, sequential, and anticipatory decision-making under uncertainty, while MPC is well suited to enforcing hard physical constraints (joint limits, torque saturation, friction cones, self- and environment-collision avoidance) over a receding horizon. Rather than filtering a single-step RL action a posteriori, as in the existing QP safety filter, this thesis proposes to let RL decisions shape the MPC problem itself — its contact schedule, cost weights, horizon structure, or reference behavior — online, at each replanning cycle.
4. Research Questions
• How can discrete or categorical RL decisions (contact mode, footstep region, gait/behavior transition) be translated, online and at control-loop rate, into MPC parameters — contact sequence, task weights, terminal constraints, or reference trajectories — without introducing infeasibility?
• How should the RL policy, its observation space, and its reward be designed so that it learns to anticipate the feasibility limits of the downstream MPC, rather than proposing decisions the MPC can only partially satisfy?
• What interface and information flow between RL and MPC (e.g., warm-starting, shared value functions, learned cost shaping, or differentiable MPC layers) best preserve both the reactivity of RL and the guarantees of MPC?
• How can the coupled architecture remain compatible with real-time constraints, given that RL policies, MPC re-formulation/re-solve, and low-level torque control naturally operate at different frequencies, as in existing RL+QP pipelines?
• How can disturbance and model-error estimation (in the spirit of generalized-momentum observers and tilt observers used in current RL+QP work) be extended to a receding-horizon setting to preserve safety under imperfect dynamic models?
• Does the resulting hybrid controller improve robustness, generalization, and constraint satisfaction over both pure end-to-end RL and pure model-based contact planning, in simulation and on hardware?
5. Proposed Methodology and Work Packages
WP1 — State of the art and problem formulation
Structured review of (i) model-based contact planning and whole-body MPC/QP for legged and humanoid robots, (ii) constrained and safe reinforcement learning for locomotion, including constraints-as-terminations and execution-time safety filters, and (iii) existing hybrid RL–MPC architectures (e.g., RL for MPC cost/parameter tuning, learned terrain classifiers feeding contact planners, differentiable MPC). This work package will formalize the RL–MPC interface as an explicit optimization and control problem and define evaluation metrics for feasibility, constraint violation, and task performance.
WP2 — RL policy for discrete, high-level decision-making
Design and training, in simulation, of an RL policy whose action space is discrete or hybrid (contact mode, footstep target region, gait or behavior mode), rather than continuous joint commands. The policy will be trained with the online MPC in the loop, so that its reward reflects the true dynamic feasibility and cost of its decisions once realized by the MPC, extending the constraints-as-terminations philosophy to decisions rather than raw actions.
WP3 — Online MPC formulation and parametrization module
Development of a receding-horizon MPC module that can be re-formulated and re-parametrized at runtime from the RL policy's discrete decisions: contact scheduling, task and constraint weighting, and terminal conditions. This work package will build on the acceleration-based QP formulation, torque and forward-dynamics task alternatives, and velocity-damper constraint handling already validated on manipulator and humanoid hardware, extending them from a single-step safety filter to a multi-step predictive horizon with warm-starting for real-time performance.
WP4 — Coupled real-time architecture and disturbance handling
Integration of the RL decision layer, the online MPC, and low-level torque or position control into a single real-time pipeline with well-characterized control frequencies for each layer. Extension of disturbance-observer-based external force and modeling-error estimation to the horizon setting, so that the MPC's constraint enforcement remains reliable despite imperfect dynamic models, mirroring the mitigation strategy used for the Unitree H1's identified mass mismatch.
WP5 — Simulation and hardware validation
Progressive validation in simulation (e.g., mc_mujoco) and then on real hardware, on tasks that intrinsically require discrete contact-mode reasoning: stepping-stone or discontinuous terrain crossing, stair climbing, dynamic obstacle avoidance, and gait-transition scenarios. The experimental platform will be either the TIRREX/RENOIR platform, based on the PAL Robotics Kangaroo wheeled-legged robot, or a Unitree humanoid (H1v1, H1v2, or R1), depending on availability and on which platform best matches the discrete contact-mode phenomena under study (e.g., wheel/leg mode switching on Kangaroo versus bipedal contact scheduling on the Unitree platforms). Results will be compared against (i) a pure end-to-end RL baseline, (ii) a pure model-based contact planner, and (iii) the existing single-step RL+QP safety-filter architecture.
6. Expected Contributions
• A general architecture coupling RL-based discrete decision-making with an online-formulated, online-parametrized MPC for hybrid dynamic locomotion, with a formalized interface between the two layers.
• Training methodology for RL policies that anticipate MPC feasibility, extending constraints-as-terminations from continuous actions to discrete, high-level decisions.
• A real-time-compatible MPC re-formulation and re-parametrization scheme, together with an extension of disturbance-observer techniques to the receding-horizon setting.
• Experimental validation and quantitative comparison against model-based and end-to-end learned baselines, in simulation and on hardware, on the TIRREX/RENOIR platform (PAL Robotics Kangaroo) and/or a Unitree humanoid (H1v1/H1v2/R1).
• An open-source implementation, in continuity with the open-source tools already released by the team (Pinocchio, Crocoddyl, Stack-of-Tasks, and the RL+QP safety-filter codebase).
7. Supervision and Environment
The thesis will be hosted at LAAS-CNRS (Gepetto team, Toulouse), under the supervision of Olivier Stasse, CNRS Directeur de Recherche specializing in humanoid robotics, whole-body and model predictive control, and reinforcement-learning-based locomotion, and co-author of the Constraints as Terminations (CaT) framework. It will be co-supervised by Mehdi Benallegue, CNRS-AIST JRL (Joint Robotics Laboratory, Tsukuba), whose recent work on second-order QP safety filtering for RL policy execution directly motivates and grounds the online MPC formulation targeted by this thesis. This LAAS/JRL co-supervision reflects the collaborative, multi-partner structure of the HAMMER project and will give the candidate combined access to the Gepetto team's model-based software ecosystem (Pinocchio, Crocoddyl, Stack-of-Tasks) and to JRL's RL+QP safety-filter framework. Experimental validation will be carried out on the TIRREX/RENOIR platform, based on the PAL Robotics Kangaroo wheeled-legged robot, and/or on a Unitree humanoid (H1v1, H1v2, or R1), with further hardware access possible through the wider HAMMER consortium.
8. Required Profile
• Master's degree (or equivalent) in robotics, automatic control, or machine learning.
• Solid background in optimal control and numerical optimization (MPC, QP, DDP/FDDP, SQP).
• Working knowledge of reinforcement learning (policy-gradient and actor-critic methods; familiarity with constrained or safe RL is a plus).
• Strong programming skills in C++ and Python; familiarity with rigid-body dynamics libraries (Pinocchio) and simulation environments (MuJoCo, Gazebo) is an asset.
• Interest in hardware experimentation and real-time robotic software architectures (ROS/ROS2, mc_rtc).
• Good written and spoken English; French is a plus but not required.
References
[1] M. Charbonneau, V. Modugno, F. Nori, G. Oriolo, D. Pucci, and S. Ivaldi, "Learning robust task priorities of QP-based whole-body torque-controllers," in Proc. IEEE-RAS Int. Conf. Humanoid Robots (Humanoids), 2018, pp. 1–9.
[2] R. Cisneros, M. Benallegue, A. Benallegue, M. Morisawa, H. Audren, P. Gergondet, A. Escande, A. Kheddar, and F. Kanehiro, "Robust humanoid control using a QP solver with integral gains," in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), 2018.
[3] K. Bouyarmane, K. Chappellet, J. Vaillant, and A. Kheddar, "Quadratic programming for multirobot and task-space force control," IEEE Trans. Robotics, vol. 35, no. 1, pp. 64–77, 2018.
[4] A. Escande, N. Mansard, and P.-B. Wieber, "Hierarchical quadratic programming: Fast online humanoid-robot motion generation," Int. J. Robotics Research, vol. 33, no. 7, pp. 1006–1028, 2014.
[5] P. M. Wensing, M. Posa, Y. Hu, A. Escande, N. Mansard, and A. Del Prete, "Optimization-based control for dynamic legged robots," IEEE Trans. Robotics, vol. 40, pp. 43–63, 2024.
[6] N. J. Kong, C. Li, G. Council, and A. M. Johnson, "Hybrid iLQR model predictive control for contact implicit stabilization on legged robots," IEEE Trans. Robotics, vol. 39, no. 6, pp. 4712–4727, 2023.
[7] S. Le Cléac'h, T. A. Howell, M. Schwager, and Z. Manchester, "Fast contact-implicit model predictive control," IEEE Trans. Robotics, vol. 40, pp. 1617–1629, 2024.
[8] G. Kim, D. Kang, J.-H. Kim, S. Hong, and H.-W. Park, "Contact-implicit model predictive control: Controlling diverse quadruped motions without pre-planned contact modes or trajectories," Int. J. Robotics Research, 2025.
[9] V. Kurtz, A. Castro, A. Ö. Önol, and H. Lin, "Inverse dynamics trajectory optimization for contact-implicit model predictive control," Int. J. Robotics Research, 2025 (in press), doi: 10.1177/02783649251344635.
[10] R. Grandia, A. J. Taylor, A. D. Ames, and M. Hutter, "Multi-layered safety for legged robots via control barrier functions and model predictive control," in Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2021, pp. 8352–8358.
[11] L. Brunke, M. Greeff, A. W. Hall, Z. Yuan, S. Zhou, J. Panerati, and A. P. Schoellig, "Safe learning in robotics: From learning-based control to safe reinforcement learning," Annu. Rev. Control, Robotics, and Autonomous Systems, vol. 5, pp. 411–444, 2022.
[12] J. Achiam, D. Held, A. Tamar, and P. Abbeel, "Constrained policy optimization," in Proc. 34th Int. Conf. Machine Learning (ICML), 2017, pp. 22–31.
[13] K. P. Wabersich and M. N. Zeilinger, "A predictive safety filter for learning-based control of constrained nonlinear dynamical systems," Automatica, vol. 129, art. 109597, 2021.
[14] E. Chane-Sane, P.-A. Leziart, T. Flayols, O. Stasse, P. Souères, and N. Mansard, "CaT: Constraints as terminations for legged locomotion reinforcement learning," in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), 2024, pp. 13303–13310.
[15] A. Cariou, B. Muraccioli, P.-A. Leziart, M. Celerier, A. Demont, G. Venture, and M. Benallegue, "Safe execution of RL policies via second-order QP constraint enforcement for real-world robotic deployments," to appear in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), 2026.
[16] B. Muraccioli, M. Celerier, M. Benallegue, and G. Venture, "Demonstrating a control framework for physical human-robot interaction toward industrial applications," in Proc. Robotics: Science and Systems (RSS), 2025.
[17] Y. Song and D. Scaramuzza, "Policy search for model predictive control with application to agile drone flight," IEEE Trans. Robotics, vol. 38, no. 4, pp. 2114–2130, 2022.
[18] A. Romero, E. Aljalbout, Y. Song, and D. Scaramuzza, "Actor-critic model predictive control: Differentiable optimization meets reinforcement learning for agile flight," in Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2024, pp. 14777–14784; extended version in IEEE Trans. Robotics, vol. 42, pp. 673–692, 2025.
[19] B. Amos, I. D. J. Rodriguez, J. Sacks, B. Boots, and J. Z. Kolter, "Differentiable MPC for end-to-end planning and control," in Advances in Neural Information Processing Systems (NeurIPS), 2018.
[20] T. Seyde, J. Carius, R. Grandia, F. Farshidian, and M. Hutter, "Locomotion planning through a hybrid Bayesian trajectory optimization," in Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2019, pp. 5544–5550.
[21] H. Liu, S. Teng, B. Liu, W. Zhang, and M. Ghaffari, "Discrete-time hybrid automata learning: Legged locomotion meets skateboarding," in Proc. Robotics: Science and Systems (RSS), 2025.
[22] E. Mingo Hoffman, A. Curti, N. Miguel, S. K. Kothakota, A. Molina, A. Roig, and L. Marchionni, "Modeling and numerical analysis of Kangaroo lower body based on constrained dynamics of hybrid serial-parallel floating-base systems," Robotics and Autonomous Systems, vol. 182, art. 104827, 2024.

Your Work Environment

This thesis is proposed within HAMMER, Target Project 2 of the priority research and acceleration-equipment program in robotics funded by France 2030 and managed by the Agence Nationale de la Recherche (ANR). HAMMER is a large collaborative project led by CNRS–Université Côte d'Azur I3S, gathering CNRS-AIST JRL, LAAS-CNRS, CNRS-UBE ICB, ONERA DTIS, CNRS-INP GIPSA-lab, CNRS-Sorbonne Université ISIR, Inria-ENS Willow, Inria ACENTAURI, Inria Larsen, and Mines Paris PSL. The project aims to endow robots with advanced locomotion and mobility capabilities by developing a hybrid approach that combines the complementary strengths of model-based and data-driven methods.
Within this context, the LAAS-CNRS Gepetto team brings a long-standing expertise in model-based whole-body control for humanoid robots (Pinocchio, Crocoddyl, Stack-of-Tasks, TALOS) and, more recently, in reinforcement-learning-based locomotion, notably through the Constraints as Terminations (CaT) framework for constrained legged locomotion RL. This thesis builds directly on that lineage and on recent work at CNRS-AIST JRL that filters RL policy outputs through a second-order Quadratic Program (QP) safety layer to guarantee constraint satisfaction at execution time (Cariou, Muraccioli et al., accepted at IROS 2026). The proposed subject extends this line of research from single-step QP safety filtering toward a fully hybrid, horizon-based architecture in which RL and Model Predictive Control (MPC) are tightly coupled.

Constraints and risks

This PhD will take place partly at the Gepetto team at LAAS/CNRS Toulouse France and partly at the JRL, UMI/CNRS AIST, Tsukuba, Ibaraki, Japon
The selected candidat is expected to run experiments on human size humanoid robots (1m70)

Compensation and benefits

Compensation

2300 € gross monthly

Annual leave and RTT

44 jours

Remote Working practice and compensation

Pratique et indemnisation du TT

Transport

Prise en charge à 75% du coût et forfait mobilité durable jusqu’à 300€

About the offer

About the CNRS

The CNRS is a major player in fundamental research on a global scale. The CNRS is the only French organization active in all scientific fields. Its unique position as a multi-specialist allows it to bring together different disciplines to address the most important challenges of the contemporary world, in connection with the actors of change.

CNRS

The research professions