Reinforcement Learning in Robotics

Neueste Paper

• Dropout-GRPO: Variational Stochasticity for Continuous Latent Reasoning

  Wooil Jung · 10. Juni 2026

  Group Relative Policy Optimization (GRPO) relies on the diversity of $K$ rollouts within each group; otherwise, the group-mean advantage $A^{(k)} = r^{(k)} - \mu_r$ collapses to zero. This presents a structural challenge for latent-reasoning models like Coconut, which feed continuous hidden states r…

• N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimization

  Xukun Zhu, Hang Yu, Peng Di, Linchao Zhu · 10. Juni 2026

  The success of Large Language Models in mathematical reasoning relies heavily on the generation of diverse and valid solution paths during the rollout phase. However, current rollout techniques face a fundamental trade-off: token-level sampling often yields redundant trajectories that differ only in…

• Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learning

  Thanh Nguyen, Tri Ton, Hongbin Choe, Tung M. Luu, Chang D. Yoo · 10. Juni 2026

  Diffusion-based Q-learning has emerged as a powerful paradigm for offline reinforcement learning, but its reliance on multi-step denoising makes both training and inference computationally expensive and brittle. Recent efforts to accelerate diffusion Q-learning toward single-step action generation t…

• Embodiment-conditioned Generalist Control for Multirotor Aerial Robots

  Orestis Konstantaropoulos, Welf Rehberg, Mihir Kulkarni, Kostas Alexis · 10. Juni 2026

  We present a generalist position control policy capable of controlling arbitrary multirotor configurations of a certain rotor count (e.g., hexarotors or quadrotors) with a single set of network weights. The policy is conditioned on a physics-grounded embodiment descriptor: a mass and inertia-normali…

• Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning

  Zhiyuan Zhou, Andy Peng, Charles Xu, Qiyang Li, Tobias Springenberg, Kevin Frans, Sergey Levine · 10. Juni 2026

  Expressive continuous control policies, such as diffusion and flow models, form the backbone of recent advances in scaling imitation learning for simulated and real robot control. While they are known to scale stably in the supervised imitation learning setting, incorporating them into reinforcement…

• SHAPO: Sharpness-Aware Policy Optimization for Safe Exploration

  Kaustubh Mani, Yann Pequignot, Vincent Mai, Liam Paull · 10. Juni 2026

  Safe exploration is a prerequisite for deploying reinforcement learning (RL) agents in safety-critical domains. In this paper, we approach safe exploration through the lens of epistemic uncertainty, where the actor's sensitivity to parameter perturbations serves as a practical proxy for regions of h…

• MODIP: Efficient Model-Based Optimization for Diffusion Policies

  Zakariae El Asri, Philippe Gratias-Quiquandon, Nicolas Thome, Olivier Sigaud · 10. Juni 2026

  Diffusion policies (DPs) have emerged as expressive policy representations for robot learning, often used with imitation learning methods such as behavioral cloning (BC). However, while their success has largely been confined to BC, direct reinforcement learning (RL) fine-tuning remains challenging …

• Convergence of Monte Carlo Optimistic Policy Iteration: Beyond Uniform State-Action Updates

  Octave Oliviers, Glenn Vinnicombe · 10. Juni 2026

  The asymptotic behaviour of Monte Carlo optimistic policy iteration (MC-O-PI) is a long-standing open question. When the model of the environment is unknown, as is common in practice, the only known condition that guarantees convergence to optimality is impractical. In its canonical form, this condi…

• Reasoning or Memorization? Direction-Aware Diversity Exploration in LLM Reinforcement Learning

  Jiangnan Xia, Yucheng Shi, Yu Yang, Kishan Panaganti, Zhenwen Liang, Ninghao Liu · 10. Juni 2026

  Reinforcement learning has become a key paradigm for eliciting reasoning abilities in large language models, where exploration is crucial for discovering effective solution trajectories. Existing exploration methods typically encourage diversity in semantic or gradient spaces, without distinguishing…

• Failure Modes of Deep Multi-Agent RL in Asynchronous Pricing: Reproducible Triggers, Trace Diagnostics, and a Partial Fix

  Shree Murthy, Rohan Pandey · 10. Juni 2026

  We study two reproducible failure modes of deep multi-agent reinforcement learning in continuous-time pricing markets: (i) tacit cartel formation between competing DDPG agents, and (ii) actor--critic instability at high event rates. We instantiate both inside a single CT-MARL benchmark (Poisson-cloc…

• Geometrically Averaged Hard Target Updates for Linear Q-Learning

  Donghwan Lee · 10. Juni 2026

  Periodic hard target updates are among the most common stabilization devices in modern deep Q-learning. Recent studies suggest that target updates can improve stability in Q-learning with function approximation, including linear function approximation. We introduce and analyze the so-called $\lambda…

• Effective Reinforcement Learning for Agentic Search by Recycling Zero-Variance Queries During Training

  Jo\~ao Coelho, Jo\~ao Magalh\~aes, Bruno Martins, Chenyan Xiong · 10. Juni 2026

  The use of GRPO-style algorithms has become the standard strategy for training LLM search agents under outcome-only rewards. With these algorithms, a query contributes to parameter updates only when its rollout group mixes successes and failures; all-correct (too-easy) and all-incorrect (too-hard) g…

• ActProbe: Action-Space Probe for Early Failure Detection of Generative Robot Policies

  Bingjia Huang, Xiangyu Li, Xiang Wang, Liang Mi, Zixu Hao, Weijun Wang, Hao Wu, Kun Li, Yunxin Liu, Ting Cao · 9. Juni 2026

  Generative robot policies fail unpredictably at deployment: they hesitate at critical moments, drift off-task, or commit to unrecoverable actions. Existing online failure detectors either require white-box access to policy internals or add runtime overhead through resampling and observation-side sig…

• MC-CPO: Mastery-Conditioned Constrained Policy Optimization for Pedagogically Safe Intelligent Tutoring Systems

  Oluseyi Olukola, Nick Rahimi · 9. Juni 2026

  Intelligent tutoring systems increasingly rely on reinforcement learning to personalise instruction, yet optimising for observable engagement signals can systematically decouple learner activity from genuine knowledge acquisition. Analysing over 21 million student interactions across two deployed pl…

• Reinforcement Learning for Flow-Matching Policies with Density Transport

  Boshu Lei, Kostas Daniilidis, Antonio Loquercio · 9. Juni 2026

  We present an online reinforcement learning (RL) algorithm for fine-tuning flow-matching policies in continuous-control problems. Our key insight is to view RL-based policy improvement as a transport of action densities towards regions of high reward, which naturally aligns with the transport formul…

• HARBOR: A Harness Framework for Agentic Robot Reinforcement Learning

  Zechu Li, Yufeng Jin, Xiaoyang Liu, Puze Liu, Vignesh Prasad, Carlo D'Eramo, Georgia Chalvatzaki · 9. Juni 2026

  Reinforcement learning (RL) has become a powerful paradigm for robot learning, particularly in sim-to-real settings, but its broader adoption remains limited by the engineering pipeline surrounding the algorithms. Building tasks, shaping rewards, and tuning hyperparameters require substantial expert…

• Stabilizing On-Policy Distillation for MLLM Reasoning with Global Normalization

  Dongze Hao, Zhiwei Jin, Chen Chen, Haonan Lu · 9. Juni 2026

  On-policy distillation (OPD) has recently emerged as an important post-training paradigm. By using a stronger teacher model to provide dense, fine-grained supervision for sampled trajectories, OPD offers a clear advantage over reinforcement learning with verifiable rewards (RLVR), which typically de…

• Counterfactual Transport Flows for Offline Conservative Trajectory Refinement

  Lena Krieger, Xuan Zhao, Zhuo Cao, Qin Wang, Hanno Scharr, Ira Assent · 9. Juni 2026

  Offline reinforcement learning (RL) offers a path to policy improvement from logged data alone, using historical returns or other measurable outcomes as world feedback. A key difficulty is improving observed behavior without extrapolating beyond what the offline data supports. We propose \emph{count…

• Autonomous Aerial Manipulation via Contextual Contrastive Meta Reinforcement Learning

  Lixuan Jin, Bingxuan Lan, Xinyi Bao, Xiangyuan Xie, Chunjie Zhang, Zheng Chen, Tianshuo Liu, Ruijie Tian, Jinyu Ru, Gang Wang, Lei Yuan, Yang Yu · 9. Juni 2026

  Unmanned aerial vehicles (UAVs) are increasingly being deployed in logistics, service robotics, and other real-world applications, creating a growing demand for autonomous payload acquisition and delivery. Existing approaches typically assume pre-attached payloads or rely on specialized grippers, le…

• PriFT: Prior-Support Guided Supervised Fine-Tuning

  Ke Wang, Shuangqi Li, Mathieu Salzmann, Pascal Frossard · 9. Juni 2026

  Supervised fine-tuning (SFT) is an efficient approach for downstream task adaptation and often serves as the initialization stage for reinforcement learning (RL), but it can show weaker generalization than RL. A key limitation is its off-policy objective: SFT fits fixed demonstrations token by token…

• Shape Formation for the Cooperative Transportation of Arbitrary Objects Using Multi-Agent Reinforcement Learning

  Mohamed Sayed, Wolfram Burgard, Tanja Katharina Kaiser · 9. Juni 2026

  Cooperative object transportation is essential in numerous domains, including industrial to domestic services. A popular transportation strategy is to carry objects on top of multi-robot systems. The corresponding task is typically solved by decomposing it into three interconnected subproblems: form…

• Reformulate LLM Reinforcement Learning for Efficient Training under Black-box Discrepancy

  Jiashun Liu, Runze Liu, Xu Wan, Jing Liang, Hongyao Tang, Ling Pan · 9. Juni 2026

  Reinforcement Learning (RL) has emerged as a pivotal post-training paradigm, yet it frequently suffers from unpredictable sub-optimum performance or even training collapses. Recent findings attribute these failures to a hidden train-inference discrepancy (or mismatch), stemming from the disparate un…

• A Unifying Lens on Reward Uncertainty in RLHF

  Ely Hahami, Yoel Zimmermann, Ray Zhou, Jack Benarroch Jedlicki · 9. Juni 2026

  Reinforcement learning from human feedback (RLHF) is bottlenecked by \emph{reward hacking}, where the policy exploits errors in a proxy reward model (RM) and produces high RM scores without genuine quality gains. A natural mitigation is \emph{pessimism}: penalizing rewards in regions where the RM is…

• In-Context Reinforcement Learning via Communicative World Models

  Fernando Martinez-Lopez, Tao Li, Yingdong Lu, Juntao Chen · 9. Juni 2026

  Reinforcement learning (RL) agents often struggle to generalize to new tasks and contexts without updating their parameters, mainly because their learned representations and policies are overfit to the specifics of their training environments. To boost agents' in-context RL (ICRL) ability, this work…

• UNIQ: Conformal Calibration for Adaptive Conservatism in Offline Reinforcement Learning

  Aditya Upadhyay · 9. Juni 2026

  Offline reinforcement learning requires careful conservatism to mitigate distribution shift, yet most existing methods apply a fixed penalty uniformly across all states regardless of local data coverage. We present UNIQ (Uncertainty-Informed Quantile), an offline RL method that introduces state-adap…