Reinforcement Learning in Robotics

Latest papers

• RASFT: Rollout-Adaptive Supervised Fine-Tuning for Reasoning

  Yongliang Miao, Fengyuan Liu, Wei Shi, Yanguang Liu, Fei Sun, Na Zou, Mengnan Du · 8 June 2026

  Supervised fine-tuning (SFT) is a prevailing method for adapting large language models to reasoning tasks by imitating offline expert demonstrations, often treating a single expert trajectory as the target behavior. However, reasoning is not simple path imitation: rigidly following one demonstrated …

• Learning Explicit Behavioral Models with Adaptive Questions and World-Model Probes

  Hikaru Shindo, Yu Deng, Teng Cao, Quentin Delfosse, Christopher Tauchmann, Jannis Bl\"uml, Gopika Sudhakaran, Kristian Kersting · 8 June 2026

  Interactive agents trained only against task return can achieve high scores while failing to represent the mechanisms that make their actions succeed. This makes brittle behavior difficult to diagnose and limits adaptation when environment dynamics change. Existing LLM reflection and policy-code rep…

• Uncertainty-Aware LLM-Guided Policy Shaping for Sparse-Reward Reinforcement Learning

  Ujjwal Bhatta, Utsabi Dangol, Sumaly Bajracharya, Rodrigue Rizk, KC Santosh · 8 June 2026

  Sparse rewards and heterogeneous task sequences remain persistent challenges in Reinforcement Learning (RL), often resulting in slow convergence, weak generalization, and inefficient exploration. We propose Uncertainty-Aware LLM-Guided Policy Shaping (ULPS), a novel framework that integrates a calib…

• Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learning

  Yijin Zhou, Linqian Zeng, Xiaoya Lu, Wenyuan Xie, Dongrui Liu, Junchi Yan, Jing Shao · 8 June 2026

  Large language model (LLM)-based agents often make suboptimal tool-use decisions, including unsupported tool invocation and hallucinated direct responses, which may accumulate errors throughout multi-step interactions. Existing approaches mainly improve these behaviors through inference-time correct…

• AdaGRPO: A Capability-Aware Adaptive Enhancement for Flow-based GRPO

  Jiazi Bu, Pengyang Ling, Yujie Zhou, Yibin Wang, Yuhang Zang, Tianyi Wei, Xiaohang Zhan, Jiaqi Wang, Tong Wu, Xingang Pan, Dahua Lin · 8 June 2026

  Group Relative Policy Optimization (GRPO) has demonstrated remarkable success in aligning text-to-image (T2I) flow models with human preferences. However, we have identified that the learning loop of current flow-based GRPO is fundamentally decoupled from the learner's current capability, suffering …

• The Sim-to-Real Gap of Foundation Model Agents: A Unified MDP Perspective

  Xiaoou Liu, Tiejin Chen, Weibo Li, Xiyang Hu, Hua Wei · 8 June 2026

  Foundation model agents are increasingly deployed for real-world decision-making, but suffer from the sim-to-real gap. While robotics and classical control have mature frameworks to address this gap, the foundation model community is treating agent robustness as an entirely novel phenomenon. Our pap…

• GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratios

  Ke Hu, Shutong Ding, Panxin Tao, Jingya Wang, Ye Shi · 8 June 2026

  Generative policies provide expressive and multimodal action distributions, making them attractive for reinforcement learning (RL) in complex continuous-control tasks. Among them, flow-based policies are especially appealing because they generate actions through deterministic transport maps. However…

• OpenSkill: Open-World Self-Evolution for LLM Agents

  Zhiling Yan, Dingjie Song, Hanrong Zhang, Wei Liang, Yuxuan Zhang, Yutong Dai, Lifang He, Philip S. Yu, Ran Xu, Xiang Li, Lichao Sun · 8 June 2026

  Self-evolving agents requires adaptation after deployment, but existing approaches assume a usable learning loop, such as curated skills, successful trajectories, or verifier signals. Real open-world deployments may provide none of these, offering only a task prompt. In this work, we study open-worl…

• Robust Instruction Compliance in Cooperative Multi-Agent Reinforcement Learning

  Wo Wei Lin, Ethan Rathbun, Enrico Marchesini, Xiang Zhi Tan · 8 June 2026

  Multi-agent reinforcement learning (MARL) in real-world use cases may need to adapt to external natural language instructions that interrupt ongoing behavior and conflict with long-horizon objectives. However, conditioning rewards on instructions introduces a fundamental failure mode as Bellman upda…

• Performance Variation in Deep Reinforcement Learning

  Haruto Tanaka, A. Rupam Mahmood · 8 June 2026

  Deep reinforcement learning (RL) algorithms often suffer from low run-to-run robustness, manifesting as significant performance variation across independent runs of identically configured agents. Although this issue poses a spectrum of challenges across research and practice, relatively few studies …

• Residual-Controlled Multiplier Learning for Stochastic Constrained Decision-Making

  Kang Liu, Jianchen Hu, Ziyu Qu · 8 June 2026

  Stochastic constrained decision-making requires optimizing performance objectives while enforcing statistical requirements such as safety or fairness. However, standard primal--dual methods struggle to update multipliers robustly under stochastic mini-batch feedback, as the noise of mini-batch gradi…

• Representation Learning Enables Scalable Multitask Deep Reinforcement Learning

  Johan Obando-Ceron, Lu Li, Scott Fujimoto, Pierre-Luc Bacon, Aaron Courville, Pablo Samuel Castro · 5 June 2026

  Scaling reinforcement learning (RL) to diverse multitask settings remains a central challenge. While recent advances in model-based RL achieve strong performance, they rely on planning and complex training pipelines, making it unclear which components are essential for scalability. We revisit this q…

• Agentic Monte Carlo: Simulating Reinforcement Learning for Black-Box Agents

  Dae Yon Hwang, Raunaq Suri, Valentin Villecroze, Anthony L. Caterini, Jesse C. Cresswell, No\"el Vouitsis, Brendan Leigh Ross · 5 June 2026

  LLM agents operate in two distinct regimes: open-weight agents amenable to reinforcement learning (RL) and black-box agents whose behaviour must be controlled purely at test time. Although black-box agents are often backed by state-of-the-art proprietary LLMs, API-only access precludes parameter-lev…

• Scalable Reinforcement Learning via Adaptive Batch Scaling

  Jongchan Park · 5 June 2026

  Conventional wisdom holds that large-batch training is fundamentally incompatible with Reinforcement Learning (RL) - beyond a modest threshold, increasing batch sizes typically yields diminishing returns or performance degradation due to the inherent non-stationarity of the data distribution. We cha…

• MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following

  Mohammad Mahdi Salmani-Zarchi, Zahra Rahimi, Heshaam Faili, Mohammad Javad Dousti · 5 June 2026

  Reinforcement learning with verifiable rewards is ideal for multi-constraint instruction following, yet standard group-relative policy optimization (GRPO) becomes unstable under discrete, low-dispersion rewards, where within-group reward distributions are frequently homogeneous. We identify and form…

• On Advantage Estimates for Max@K Policy Gradients

  Shota Takashiro, Soichiro Nishimori, Paavo Parmas, Yongmin Kim, Kohsei Matsutani, Gouki Minegishi, Yusuke Iwasawa, Takeshi Kojima, Yutaka Matsuo · 5 June 2026

  Reinforcement learning with verifiable rewards is widely used for post-training reasoning models, but sparse outcome rewards make exploration difficult. A complementary approach is to optimize inference-time objectives such as pass@K and max@K directly, yet existing policy-gradient estimators for th…

• What Objects Enable, Not What They Are: Functional Latent Spaces for Affordance Reasoning

  Rohan Siva, Neel P. Bhatt, Yunhao Yang, Seoyoung Lee, Nishant Gadde, Christian Ellis, Alvaro Velasquez, Zhangyang Wang, Ufuk Topcu · 5 June 2026

  Existing robot planning systems rely on appearance-based reasoning, where visual observations are encoded into latent spaces organized around object appearances (e.g., recognizing a "cart" based on how it looks). However, planning requires reasoning about task-relevant functionalities of objects (e.…

• Selective-Advantage Entropy-Adaptive Horizon GRPO: Asymmetric Token-Level Discounting for Efficient Reinforcement Learning of Language Models

  Chirag Chawla, Rohan Charudatt Salvi, Madhav S. Baidya · 5 June 2026

  Group Relative Policy Optimisation (GRPO) has emerged as an effective reinforcement-learning algorithm for aligning language models on reasoning tasks, but it treats every token position and every sampled rollout symmetrically. We introduce two complementary extensions: (i) Adaptive-Horizon GRPO (AH…

• Learning of Robot Safety Policies via Adversarial Synthetic Scenarios

  Nikolai Dorofeev, Alexey Odinokov, Rostislav Yavorskiy · 5 June 2026

  In this work, we propose an agentic gamification framework for hazard-informed learning of robot safety policies through synthetic scenarios. We model scenario generation as an adversarial game between two agents: a Red Team that explores the space of potential failures by constructing hazardous sit…

• Retry Policy Gradients in Continuous Action Spaces

  Soichiro Nishimori, Paavo Parmas · 5 June 2026

  Retry-based objectives such as pass@K and max@K optimize the best return obtained from multiple sampled trajectories, and recent work has shown that they can promote exploration without explicit exploration bonuses. In discrete action spaces, ReMax was shown to do so by adapting to return uncertaint…

• Towards Healthy Evolution: Exploring the Role and Mechanisms of Human-Agent Interaction in Self-Evolving Systems

  Dianxing Shi, Junqi He, Junhao Chen, Bowen Wang, Yuta Nakashima · 5 June 2026

  Self-evolving agents improve through continual self-play and self-generated learning signals, but autonomous evolution can also cause capability degradation and safety drift. Although human feedback has proven effective for static and post-trained agents, its role in self-evolving systems remains un…

• Semi-Offline Reinforcement Learning for Optimized Text Generation

  Changyu Chen, Xiting Wang, Yiqiao Jin, Victor Ye Dong, Li Dong, Jie Cao, Yi Liu, Rui Yan · 5 June 2026

  In reinforcement learning (RL), there are two major settings for interacting with the environment: online and offline. Online methods explore the environment at significant time cost, and offline methods efficiently obtain reward signals by sacrificing exploration capability. We propose semi-offline…

• Sample-efficient Low-level Motion Planning for Robotic Manipulation Tasks via Zero-shot Transfer Learning

  Yuanzhi He, Victor Romero-Cano, Jos\'e J. Pati\~no, Juan David Hern\'andez, William Sawtell, Gualtiero Colombo · 5 June 2026

  As robotic systems become more sophisticated, the growing complexity of their motion planning models and the longer training times pose substantial challenges. Evolutionary algorithms such as the Sample-efficient Cross-Entropy Method (iCEM) have recently demonstrated promising potential for low-leve…

• RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training

  Rachit Bansal, Clara Mohri, Tian Qin, David Alvarez-Melis, Sham Kakade · 4 June 2026

  The standard LLM training pipeline applies reinforcement learning (RL) only after pre-training and supervised fine-tuning (SFT). We question this status quo by training a LLM from scratch and applying RL, SFT, and SFT followed by RL directly to intermediate pre-training checkpoints. We find that RL …

• Self-Distilled Policy Gradient

  Yifeng Liu, Shiyuan Zhang, Yifan Zhang, Quanquan Gu · 4 June 2026

  On-policy self-distillation, where a language model conditions on privileged context to supervise its own generations, is a promising source of dense supervision for sparse-reward reinforcement learning. Actually, it can be instantiated as an auxiliary full-vocabulary student-to-teacher reverse Kull…