Reinforcement Learning in Robotics

Latest papers

• Drop the Act: Probe-Filtered RL for Faithful Chain-of-Thought Reasoning

  Swapnil Parekh · 13 May 2026

  Reasoning models post-hoc rationalize answers they have already committed to internally, producing chains of *reasoning theater*: deliberative-looking steps that contribute nothing to correctness. This wastes inference tokens, pollutes interpretability, and obscures what the model actually computed.…

• EvoNav: Evolutionary Reward Function Design for Robot Navigation with Large Language Models

  Zhikai Zhao, Chuanbo Hua, Federico Berto, Zihan Ma, Kanghoon Lee, Jiachen Li, Jinkyoo Park · 13 May 2026

  Robot navigation is a crucial task with applications to social robots in dynamic human environments. While Reinforcement Learning (RL) has shown great promise for this problem, the policy quality is highly sensitive to the specification of reward functions. Hand-crafted rewards require substantial d…

• Missingness-MDPs: Bridging the Theory of Missing Data and POMDPs

  Joshua Wendland, Markel Zubia, Roman Andriushchenko, Maris F. L. Galesloot, Milan Ceska, Henrik von Kleist, Thiago D. Simao, Maximilian Weininger, Nils Jansen · 13 May 2026

  We introduce missingness-MDPs (miss-MDPs), a novel subclass of partially observable Markov decision processes (POMDPs) that incorporates the theory of missing data. A miss-MDP is a POMDP whose observation function is a missingness function, specifying the probability that individual state features a…

• Internalizing Curriculum Judgment for LLM Reinforcement Fine-Tuning

  Han Zheng, Yining Ma, Karthick Gunasekaran, Bharathan Balaji, Zheng Du, Shiv Vitaladevuni, Cathy Wu · 13 May 2026

  In LLM Reinforcement Fine-Tuning (RFT), curriculum learning drives both efficiency and performance. Yet, current methods externalize curriculum judgment via handcrafted heuristics or auxiliary models, risking misalignment with the policy's training dynamics. In this paper, we introduce METIS (METaco…

• Shaping Zero-Shot Coordination via State Blocking

  Mingu Kang, Sunwoo Lee, Yonghyeon Jo, Seungyul Han · 13 May 2026

  Zero-shot coordination (ZSC) aims to enable agents to cooperate with independently trained partners without prior interaction, a key requirement for real-world multi-agent systems and human-AI collaboration. Existing approaches have largely emphasized increasing partner diversity during training, ye…

• GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation

  Sijia Li, Yuchen Huang, Zifan Liu, Yanping Li, Jingjing Fu, Li Zhao, Jiang Bian, Ling Zhang, Jun Zhang, Rui Wang · 13 May 2026

  Reinforcement learning has become a widely used post-training approach for LLM agents, where training commonly relies on outcome-level rewards that provide only coarse supervision. While finer-grained credit assignment is promising for effective policy updates, obtaining reliable local credit and as…

• EVOCHAMBER: Test-Time Co-evolution of Multi-Agent System at Individual, Team, and Population Scales

  Yaolun Zhang, Tianyi Xu, Shengyu Dai, Zhenwen Shao, Qingyun Wu, Huazheng Wang · 13 May 2026

  We argue that multi-agent test-time evolution is not single-agent evolution replicated N times. A single-agent learner can only evolve its own context and memory. A multi-agent system additionally evolves who collaborates, how they collaborate, and how knowledge flows across the population. These co…

• RankQ: Offline-to-Online Reinforcement Learning via Self-Supervised Action Ranking

  Andrew Choi, Wei Xu · 13 May 2026

  Offline-to-online reinforcement learning (RL) improves sample efficiency by leveraging pre-collected datasets prior to online interaction. A key challenge, however, is learning an accurate critic in large state--action spaces with limited dataset coverage. To mitigate harmful updates from value over…

• When to Ask a Question: Understanding Communication Strategies in Generative AI Tools

  Charlotte Park, Kate Donahue, Manish Raghavan · 13 May 2026

  Generative AI models differ from traditional machine learning tools in that they allow users to provide as much or as little information as they choose in their inputs. This flexibility often leads users to omit certain details, relying on the models to infer and fill in under-specified information …

• Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates

  Anish Diwan, Davide Tateo, Christopher E. Mower, Haitham Bou-Ammar, Jan Peters, Oleg Arenz · 13 May 2026

  Inverse reinforcement learning (IRL) is typically formulated as maximizing entropy subject to matching the distribution of expert trajectories. Classical (dual-ascent) IRL guarantees monotonic performance improvement but requires fully solving an RL problem each iteration to compute dual gradients. …

• Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization

  Huimin Xu, Shuai Zhao, Xiaobao Wu, Anh Tuan Luu · 13 May 2026

  Reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for improving the reasoning ability of large language models. However, widely used RLVR algorithms, such as GRPO, often suffer from entropy collapse, leading to premature determinism and unstable optimization. Exi…

• Vulnerable Agent Identification in Large-Scale Multi-Agent Reinforcement Learning

  Simin Li, Zihao Mao, Zheng Yuwei, Linhao Wang, Ruixiao Xu, Chengdong Ma, Zhiqian Liu, Xin Yu, Yuqing Ma, Xin Wang, Jie Luo, Bo An, Yaodong Yang, Weifeng Lv, Xianglong Liu · 13 May 2026

  Partial agent failure becomes inevitable when systems scale up, making it crucial to identify the subset of agents whose failure causes worst-case system performance degradations. We study this Vulnerable Agent Identification (VAI) problem in large-scale multi-agent reinforcement learning (MARL). We…

• TMRL: Diffusion Timestep-Modulated Pretraining Enables Exploration for Efficient Policy Finetuning

  Matthew M. Hong, Jesse Zhang, Anusha Nagabandi, Abhishek Gupta · 13 May 2026

  Fine-tuning pre-trained robot policies with reinforcement learning (RL) often inherits the bottlenecks introduced by pre-training with behavioral cloning (BC), which produces narrow action distributions that lack the coverage necessary for downstream exploration. We present a unified framework that …

• Debiased Model-based Representations for Sample-efficient Continuous Control

  Jiafei Lyu, Zichuan Lin, Scott Fujimoto, Kai Yang, Yangkun Chen, Saiyong Yang, Zongqing Lu, Deheng Ye · 13 May 2026

  Model-based representations recently stand out as a promising framework that embeds latent dynamics information into the representations for downstream off-policy actor-critic learning. It implicitly combines the advantages of both model-free and model-based approaches while avoiding the training co…

• Aligning Flow Map Policies with Optimal Q-Guidance

  Christos Ziakas, Alessandra Russo, Avishek Joey Bose · 13 May 2026

  Generative policies based on expressive model classes, such as diffusion and flow matching, are well-suited to complex control problems with highly multimodal action distributions. Their expressivity, however, comes at a significant inference cost: generating each action typically requires simulatin…

• TOPPO: Rethinking PPO for Multi-Task Reinforcement Learning with Critic Balancing

  Yuanpeng Li, Gefei Lin, Annie Qu, Rui Miao · 13 May 2026

  Soft Actor-Critic (SAC) and its variants dominate Multi-Task Reinforcement Learning (MTRL) due to their off-policy sample efficiency, while on-policy methods such as Proximal Policy Optimization (PPO) remain underexplored. We diagnose that PPO in MTRL suffers from a previously overlooked issue: crit…

• Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control

  Jiazheng Zhang, Ziche Fu, Junrui Shen, Yunbin Zhao, Yunke Zhang, Zhiheng Xi, Long Ma, Chenxin An, Zhihao Zhang, Shichun Liu, Dingwei Zhu, Shihan Dou, Shaofan Liu, Han Li, Wiggin Zhou, Aiden Adams, Tao Gui, Fei Huang, Qi Zhang, Xuanjing Huang · 13 May 2026

  Policy entropy has emerged as a fundamental measure for understanding and controlling exploration in reinforcement learning with verifiable rewards (RLVR) for LLMs. However, existing entropy-aware methods mainly regulate entropy through global objectives, while the token-level mechanism by which sam…

• ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network

  Qian Chen, Junqiao Zhao, Hongtu Zhou, Hang Yu, Yanping Zhao, Chen Ye, Guang Chen · 13 May 2026

  Long-horizon, sparse-reward tasks pose a fundamental challenge for reinforcement learning, since single-step TD learning suffers from bootstrapping error accumulation across successive Bellman updates. Actor-critic methods with action chunking address this by operating over temporally extended actio…

• fg-expo: Frontier-guided exploration-prioritized policy optimization via adaptive kl and gaussian curriculum

  Mingxiong Lin, Zhangquan Gong, Maowen Tang, Qian Li, Chuangchuang Wang, Jian Ma, Sutian Huang, Kai Tang, Haonan Lu · 13 May 2026

  Reinforcement Learning with Verifiable Rewards (RLVR) has become the standard paradigm for LLM mathematical reasoning, with Group Relative Policy Optimization (GRPO) serving as the dominant algorithm. We identify two overlooked inefficiencies inherent in GRPO. First, a fixed KL coefficient overly re…

• Offline Constrained Reinforcement Learning under Partial Data Coverage

  Seokmin Ko, Ambuj Tewari, Kihyuk Hong · 13 May 2026

  We study offline constrained reinforcement learning with general function approximation in discounted constrained Markov decision processes. Prior methods either require full data coverage for evaluating intermediate policies, lack oracle efficiency, or requires the knowledge of data-generating dist…

• Optimal Policy Learning under Budget and Coverage Constraints

  Giovanni Cerulli · 13 May 2026

  We study optimal policy learning under combined budget and minimum coverage constraints. We show that the problem admits a knapsack-type structure and that the optimal policy can be characterized by an affine threshold rule involving both budget and coverage shadow prices. We establish that the line…

• Focusing Influence Mechanism for Multi-Agent Reinforcement Learning

  Yisak Park, Sunwoo Lee, Seungyul Han · 13 May 2026

  Cooperative multi-agent reinforcement learning (MARL) under sparse rewards remains fundamentally challenging because agents often fail to concentrate their influence, leading to insufficiently coordinated exploration. To address this, we propose the Focusing Influence Mechanism (FIM), a framework th…

• Breaking $\textit{Winner-Takes-All}$: Cooperative Policy Optimization Improves Diverse LLM Reasoning

  Haoxuan Chen, Tianming Liang, Wei-Shi Zheng, Jian-Fang Hu · 13 May 2026

  Reinforcement learning with verifiers (RLVR) has become a central paradigm for improving LLM reasoning, yet popular group-based optimization algorithms like GRPO often suffer from exploration collapse, where the models prematurely converge on a narrow set of high-scoring patterns, lacking the abilit…

• Selective Off-Policy Reference Tuning with Plan Guidance

  Duc Anh Le, Tien-Phat Nguyen, Thien Huu Nguyen, Linh Ngo Van, Trung Le · 13 May 2026

  Reinforcement learning with verifiable rewards helps reasoning, but GRPO-style methods stall on hard prompts where all sampled rollouts fail. SORT adds a repair update for those failures without changing rollout generation: it derives a plan from the reference solution, compares token probabilities …

• H\"older Policy Optimisation

  Yuxiang Chen, Dingli Liang, Yihang Chen, Ziqin Gong, Chenyang Le, Zhaokai Wang, Jiachen Zhu, Lingyu Yang, Jianghao Lin, Weinan Zhang, Jun Wang · 13 May 2026

  Group Relative Policy Optimisation (GRPO) enhances large language models by estimating advantages across a group of sampled trajectories. However, mapping these trajectory-level advantages to policy updates requires aggregating token-level probabilities within each sequence. Relying on a fixed aggre…