Reinforcement Learning in Robotics

Latest papers

• Wavelet Fourier Diffuser: Frequency-Aware Diffusion Model for Reinforcement Learning

  Yifu Luo, Yongzhe Chang, Xueqian Wang · 3 June 2026

  Diffusion probability models have shown significant promise in offline reinforcement learning by directly modeling trajectory sequences. However, existing approaches primarily focus on time-domain features while overlooking frequency-domain features, leading to frequency shift and degraded performan…

• Libra: Efficient Resource Management for Agentic RL Post-Training

  Kaiwen Chen, Xin Tan, Jingzong Li, Hong Xu · 3 June 2026

  Reinforcement learning (RL) has become a standard post-training paradigm for large language models (LLMs), extending beyond preference alignment to complex reasoning and multi-turn agentic behaviors. In agentic RL, the rollout stage generates trajectories while invoking tools, producing long-tailed …

• When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming

  Zelalem Abahana · 3 June 2026

  Reinforcement learning from human feedback (RLHF) makes large-scale post-training possible by replacing an underspecified human objective with learned and scalable proxies. The same substitution creates a structured failure surface: optimization can raise the learned reward while external quality fa…

• Planning with Uncertainty: Symmetries, Policy Inference, and Solution Compression

  Frederico Messa, Andr\'e Grahl Pereira · 3 June 2026

  Fully-observable non-deterministic (FOND) planning is at the core of artificial intelligence planning with uncertainty. It models uncertainty through actions with non-deterministic effects. In this work, we present a collection of techniques that establish explicit best-first policy-space search as …

• Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill

  Tao Chen, Gangwei Jiang, Pengyu Cheng, Siyuan Huang, Yihao Liu, Jingwei Ni, Jiaqi Guo, Mengyu Zhou, Kai Tang, Junling Liu, Qinliang Su, Xiaoxi Jiang, Guanjun Jiang · 3 June 2026

  Reward models (RMs) provide critical feedback signals for LLM post-training, notably in reinforced fine-tuning (RFT) and reinforcement learning (RL) pipelines. However, current reward evaluation relies on heterogeneous criteria such as rule-based verifiers, ground-truth references, procedural checkl…

• Physics-Guided Policy Optimization with Self-Distillation

  Ke Wang, Yuning Wu, Haoran Liu, Chaoqun Jia, Devin Chen, Kai Wei · 3 June 2026

  Self-distilled policy optimization (SDPO) has become a popular paradigm for LLM post-training, where a model learns from its own predictions conditioned on privileged information. SDPO, however, is sensitive to how much each update step should be trusted: corrections from a self-teacher can be highl…

• Too Much of a Good Thing: When sim2real Efforts Impede Policy Learning (And What to Do About It)

  Kyle Morgenstein, Bharath Masetty, Stephen Welch, Luis Sentis · 3 June 2026

  While sim2real efforts are necessary for effective policy transfer to hardware, there is such a thing as too much of a good thing. We argue that sim2real efforts have led to misaligned incentives with policy learning, resulting in simulator lock in and poor policy exploration due to the unreasonable…

• Reinforcement Learning from Cross-domain Videos with Video Prediction Model

  Zhao Yang, Xinrui Zu, Jacob E. Kooi, Thomas Delliaux, He Liu, Shujian Yu, Kevin Sebastian Luck, Vincent Fran\c{c}ois-Lavet · 3 June 2026

  Reinforcement learning from expert videos across visually distinct domains is challenging due to the absence of reward signals and the presence of domain gaps. We introduce XIPER (Cross-domain Video Prediction Reward), a reward model for learning from expert videos collected in a visually different …

• Consistency Training Can Entrench Misalignment

  David Demitri Africa, Arathi Mani · 3 June 2026

  Consistency training encourages a model to produce similar outputs across related inputs or sampling procedures. Such methods are simple, scalable, and largely label-free, but their effects on model alignment remain poorly understood. Could the self-bootstrapping nature of these methods amplify unde…

• ConTraIRL: Factorized Contrastive Abstractions for Transferable IRL

  Yikang Gui, Bikramjit Banerjee, Prashant Doshi · 3 June 2026

  Reward transfer in Inverse Reinforcement Learning (IRL) is unreliable when policies must generalize to unseen combinations of environment dynamics and task goals. We propose Factorized Contrastive Abstractions for Transferable IRL (ConTraIRL), a framework that enables compositional reward transfer b…

• Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learning

  Anthony GX-Chen, Ankit Anand, Gheorghe Comanici, Zaheer Abbas, Eser Ayg\"un, David Smalling, Shibl Mourad, Doina Precup, Andr\'e Barreto, Mark Rowland · 3 June 2026

  Classical reinforcement learning (RL) typically seeks a deterministic policy that maximizes the expected sum of a scalar reward. Yet, modern applications such as language model fine-tuning or scientific discovery demand diversity. Existing remedies such as entropy regularization or diversity bonuses…

• Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions

  Bingxu Liu, Jiashun Liu, Johan Obando-Ceron, Hao Wang, Runze Liu, Pablo Samuel Castro, Aaron Courville, Ling Pan · 3 June 2026

  While Proximal Policy Optimization (PPO) demonstrates strong performance in stationary settings, we show that its standard optimization paradigm struggles in continual and non-stationary environments. The failure does not stem from insufficient model capacity or overly restrictive clipping. Instead,…

• FGRPO: Federated GRPO with Adaptive Aggregation on Non-IID Data

  Pengyu Chen, Shaowei Li, Kai Wang, Yunsheng Yuan, Kai Han, Jun Luo, Feng Li · 3 June 2026

  Recent advances in language models have established reinforcement learning as the primary paradigm for eliciting self-correction and long-chain reasoning. While group relative policy optimization (GRPO) offers superior scalability by eliminating the critic network, deploying it on a central infrastr…

• Tool-Aware Optimization with Entropy Guidance for Efficient Agentic Reinforcement Learning

  Hongye Cao, Nuo Yan, Haoyuan Deng, Ziwei Wang, Tianpei Yang, Jing Huo, Yuyao Zhang, Yang Gao · 3 June 2026

  Agentic reinforcement learning (RL) equips large language models (LLMs) with tool-use capabilities that substantially improve reasoning on complex tasks. However, integrating external tools often destabilizes training: over-reliance on tools can induce input distribution shift, while overly conserva…

• EvoTrainer: Co-Evolving LLM Policies and Training Harnesses for Autonomous Agentic Reinforcement Learning

  Guhong Chen, Yingcheng Shi, Yongbin Li, Binhua Li, Xander Xu, Hu Wei, Shiwen Ni, Min Yang, Jieping Ye · 3 June 2026

  Autonomous LLM training is often framed as recipe search, which leaves the training harness largely static. This limitation sharpens in agentic RL, where shifting bottlenecks and scalar rewards mask diverse failure modes. We introduce EvoTrainer, an autonomous training framework that co-evolves LLM …

• SkillPyramid: A Hierarchical Skill Consolidation Framework for Self-Evolving Agents

  Yuan Xiong, Ziqi Miao, Qian Chen, Lijun Li, Yequan Wang, Shizhu He, Jun Zhao, Kang Liu · 3 June 2026

  Recent AI agents can flexibly invoke skills to solve complex tasks, but their long-term improvement is fundamentally constrained by a lack of systematic skill construction, accumulation, and transfer. In particular, without a unified framework for skill consolidation, agents tend to redundantly cons…

• ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Information

  Zehua Liu, Yuxuan Yao, Xiaojin Fu, Tao Zhong, Mingxuan Yuan · 3 June 2026

  Asynchronous reinforcement learning can improve language-model post-training throughput by decoupling response generation from policy optimization, but stale responses introduce distribution drift. Standard behavior-corrected methods control this drift with behavior-policy probabilities, importance …

• ARCA: Adapter-Residual Credit Assignment When Token Signals Degenerate

  Rodney Lafuente-Mercado · 2 June 2026

  Token-level credit assignment for language-model reinforcement learning is usually formulated as if the policy were fully trainable, while practical LLM-RL pipelines often rely on parameter-efficient fine-tuning, especially LoRA. We argue that this separation hides a structural failure mode. Under L…

• Agentic Transformers Provably Learn to Search via Reinforcement Learning

  Tong Yang, Yu Huang, Yingbin Liang, Yuejie Chi · 2 June 2026

  Tree search is a central abstraction behind many language-agent reasoning and decision-making tasks: agents must explore actions, remember failures, and backtrack toward promising alternatives. Yet, we lack a theoretical understanding of how transformer-based policies acquire such search capabilitie…

• Decoupled Behavioral Cloning for Scalable Inductive Generalization in RL from Specifications

  Vignesh Subramanian, Subhajit Roy, Suguman Bansal · 2 June 2026

  Inductive generalization is a framework for reinforcement learning (RL) generalization in which inductively related task instances admit inductively related policies. Prior work captures this structure via a higher-order policy-evolution function learned directly with RL, but suffers from poor train…

• SIRI: Self-Internalizing Reinforcement Learning with Intrinsic Skills for LLM Agent Training

  Zhongyu He, Yuanfan Li, Fei Huang, Tianyu Chen, Siyuan Chen, Xingyang Li, Meng Hsuan Yu, Xiangrong Liu, Leyi Wei, Lu Pan, Ke Zeng, Xunliang Cai · 2 June 2026

  Long-horizon LLM agents can benefit from reusable skills, yet existing skill-based methods often rely on external skill generators during training or persistent skill retrieval at inference, increasing engineering complexity, context length, and deployment latency. We propose Self-Internalizing Rein…

• ToolSelf: Unifying Task Execution and Self-Reconfiguration via Tool-Driven Emergent Adaptation

  Jingqi Zhou, Sheng Wang, Dezhao Deng, Junwen Lu, Junwei Su, Qintong Li, Jiahui Gao, Hao Wu, Jiyue Jiang, Lingpeng Kong, Dunhong Jin, Chuan Wu · 2 June 2026

  LLM-powered agentic systems excel at complex long-horizon tasks, but remain constrained by static configurations fixed before execution. Such rigidity forces a trade-off between domain-specific performance and cross-task generalization: strong priors and compact tool spaces aid specialization but we…

• Coordination Graphs for Constrained Multi-Agent Reinforcement Learning

  Santiago Amaya-Corredor, Miguel Calvo-Fullana, Anders Jonsson · 2 June 2026

  Constrained Multi-agent reinforcement learning (CMARL) faces two intertwined challenges: the joint action space grows exponentially with the number of agents, and additional requirements couple agents in ways that reward structure alone does not capture. We introduce Coordination Graphs for Constrai…

• Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief

  Hongqiang Lin, Pengfei Wang, Nenggan Zheng · 2 June 2026

  Offline reinforcement learning (RL) aims to optimize policies from pre-collected datasets. A bottleneck of this paradigm is managing epistemic uncertainty, which arises from limited data coverage (sample-level) and the ambiguity in identifying transition dynamics from finite data (model-level). To p…

• HALO: Learning Human-Robot Collaboration via Heterogeneous-Agent Lyapunov Policy Optimization

  Hao Zhang, Yaru Niu, Yikai Wang, Ding Zhao, H. Eric Tseng · 2 June 2026

  To improve generalization and resilience in human-robot collaboration (HRC), robots must contend with diverse combinations of human behaviors and contexts, motivating multi-agent reinforcement learning (MARL). However, inherent heterogeneity between robots and humans creates a rationality gap (RG), …