Reinforcement Learning in Robotics

Neueste Paper

• Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents

  Qirui Mi, Zhijian Ma, Mengyue Yang, Haoxuan Li, Yisen Wang, Haifeng Zhang, Jun Wang · 17. April 2026

  LLM-driven agents demonstrate strong performance in sequential decision-making but often rely on on-the-fly reasoning, re-deriving solutions even in recurring scenarios. This insufficient experience reuse leads to computational redundancy and execution instability. To bridge this gap, we propose Ski…

• Step-level Denoising-time Diffusion Alignment with Multiple Objectives

  Qi Zhang, Dawei Wang, Shaofeng Zou · 17. April 2026

  Reinforcement learning (RL) has emerged as a powerful tool for aligning diffusion models with human preferences, typically by optimizing a single reward function under a KL regularization constraint. In practice, however, human preferences are inherently pluralistic, and aligned models must balance …

• Does RL Expand the Capability Boundary of LLM Agents? A PASS@(k,T) Analysis

  Zhiyuan Zhai, Wenjing Yan, Xiaodan Shao, Xin Wang · 17. April 2026

  Does reinforcement learning genuinely expand what LLM agents can do, or merely make them more reliable? For static reasoning, recent work answers the second: base and RL pass@k curves converge at large k. We ask whether this holds for agentic tool use, where T rounds of interaction enable compositio…

• Model-Based Reinforcement Learning under Random Observation Delays

  Armin Karamzade, Kyungmin Kim, JB Lanier, Davide Corsi, Roy Fox · 17. April 2026

  Delays frequently occur in real-world environments, yet standard reinforcement learning (RL) algorithms often assume instantaneous perception of the environment. We study random sensor delays in POMDPs, where observations may arrive out-of-sequence, a setting that has not been previously addressed i…

• Reinforcement Learning via Value Gradient Flow

  Haoran Xu, Kaiwen Hu, Somayeh Sojoudi, Amy Zhang · 17. April 2026

  We study behavior-regularized reinforcement learning (RL), where regularization toward a reference distribution (the dataset in offline RL or the base model in LLM RL finetuning) is essential to prevent value over-optimization caused by erroneous out-of-distribution extrapolation. Existing methods e…

• Blazing the trails before beating the path: Sample-efficient Monte-Carlo planning

  Jean-Bastien Grill, Michal Valko, R\'emi Munos · 17. April 2026

  You are a robot and you live in a Markov decision process (MDP) with a finite or an infinite number of transitions from state-action to next states. You got brains and so you plan before you act. Luckily, your roboparents equipped you with a generative model to do some Monte-Carlo planning. The worl…

• LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking

  Lukas Helff, Quentin Delfosse, David Steinmann, Ruben H\"arle, Hikaru Shindo, Patrick Schramowski, Wolfgang Stammer, Kristian Kersting, Felix Friedrich · 17. April 2026

  As reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant paradigm for scaling reasoning capabilities in LLMs, a new failure mode emerges: LLMs gaming verifiers. We study this phenomenon on inductive reasoning tasks, where models must induce and output logical rules. We find t…

• Targeted Exploration via Unified Entropy Control for Reinforcement Learning

  Chen Wang, Lai Wei, Yanzhi Zhang, Chenyang Shao, Zedong Dan, Weiran Huang, Ge Lan, Yue Wang · 17. April 2026

  Recent advances in reinforcement learning (RL) have improved the reasoning capabilities of large language models (LLMs) and vision-language models (VLMs). However, the widely used Group Relative Policy Optimization (GRPO) consistently suffers from entropy collapse, causing the policy to converge pre…

• Mean Flow Policy Optimization

  Xiaoyi Dong, Xi Sheryl Zhang, Jian Cheng · 17. April 2026

  Diffusion models have recently emerged as expressive policy representations for online reinforcement learning (RL). However, their iterative generative processes introduce substantial training and inference overhead. To overcome this limitation, we propose to represent policies using MeanFlow models…

• MARS$^2$: Scaling Multi-Agent Tree Search via Reinforcement Learning for Code Generation

  Pengfei Li, Shijie Wang, Fangyuan Li, Yikun Fu, Kaifeng Liu, Kaiyan Zhang, Dazhi Zhang, Yuqiang Li, Biqing Qi, Bowen Zhou · 17. April 2026

  Reinforcement learning (RL) paradigms have demonstrated strong performance on reasoning-intensive tasks such as code generation. However, limited trajectory diversity often leads to diminishing returns, which constrains the achievable performance ceiling. Search-enhanced RL alleviates this issue by …

• Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning

  Can Xie, Ruotong Pan, Xiangyu Wu, Yunfei Zhang, Jiayi Fu, Tingting Gao, Guorui Zhou · 17. April 2026

  Reinforcement Learning with Verifiable Rewards (RLVR) has shown significant promise for enhancing the reasoning capabilities of large language models (LLMs). However, prevailing algorithms like GRPO broadcast a uniform advantage signal across all tokens in a sequence. This coarse-grained approach ov…

• Pareto-Optimal Offline Reinforcement Learning via Smooth Tchebysheff Scalarization

  Aadyot Bhatnagar, Peter M{\o}rch Groth, Ali Madani · 16. April 2026

  Large language models can be aligned with human preferences through offline reinforcement learning (RL) on small labeled datasets. While single-objective alignment is well-studied, many real-world applications demand the simultaneous optimization of multiple conflicting rewards, e.g. optimizing both…

• Bridging MARL to SARL: An Order-Independent Multi-Agent Transformer via Latent Consensus

  Zijian Zhao, Jing Gao, Sen Li · 16. April 2026

  Cooperative multi-agent reinforcement learning (MARL) is widely used to address large joint observation and action spaces by decomposing a centralized control problem into multiple interacting agents. However, such decomposition often introduces additional challenges, including non-stationarity, uns…

• Jump-Start Reinforcement Learning with Vision-Language-Action Regularization

  Angelo Moroncelli, Roberto Zanetti, Marco Maccarini, Loris Roveda · 16. April 2026

  Reinforcement learning (RL) enables high-frequency, closed-loop control for robotic manipulation, but scaling to long-horizon tasks with sparse or imperfect rewards remains difficult due to inefficient exploration and poor credit assignment. Vision-Language-Action (VLA) models leverage large-scale m…

• Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellation

  Fei Ding, Yongkang Zhang, youwei wang, Zijian Zeng · 16. April 2026

  In sparse termination rewards, intra-group comparisons have become the dominant paradigm for fine-tuning reasoning models via reinforcement learning. However, long-term training often leads to issues like ineffective update accumulation (learning tax), solution probability drift, and entropy collaps…

• Representation over Routing: Overcoming Surrogate Hacking in Multi-Timescale PPO

  Jing Sun · 16. April 2026

  Temporal credit assignment in reinforcement learning has long been a central challenge. Inspired by the multi-timescale encoding of the dopamine system in neurobiology, recent research has sought to introduce multiple discount factors into Actor-Critic architectures, such as Proximal Policy Optimiza…

• On the Fundamental Limitations of Dual Static CVaR Decompositions in Markov Decision Processes

  Mathieu Godbout, Audrey Durand · 16. April 2026

  It was recently shown that dynamic programming (DP) methods for finding static CVaR-optimal policies in Markov Decision Processes (MDPs) can fail when based on the dual formulation, yet the root cause of this failure remains unclear. We expand on these findings by shifting focus from policy optimiza…

• Chain of Uncertain Rewards with Large Language Models for Reinforcement Learning

  Shentong Mo · 16. April 2026

  Designing effective reward functions is a cornerstone of reinforcement learning (RL), yet it remains a challenging and labor-intensive process due to the inefficiencies and inconsistencies inherent in traditional methods. Existing methods often rely on extensive manual design and evaluation steps, w…

• Minimax Optimality and Spectral Routing for Majority-Vote Ensembles under Markov Dependence

  Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma · 16. April 2026

  Majority-vote ensembles achieve variance reduction by averaging over diverse, approximately independent base learners. When training data exhibits Markov dependence, as in time-series forecasting, reinforcement learning (RL) replay buffers, and spatial grids, this classical guarantee degrades in way…

• RANDPOL: Parameter-Efficient End-to-End Quadruped Locomotion via Randomized Policy Learning

  Zhuochen Liu, Rahul Jain, Quan Nguyen · 16. April 2026

  Modern learning-based locomotion controllers typically rely on fully trainable deep neural networks with a large number of parameters. This paper studies a different design point for end-to-end control: whether effective quadruped locomotion can be achieved with a drastically reduced trainable param…

• SFT-GRPO Data Overlap as a Post-Training Hyperparameter for Autoformalization

  Xiaole Su, Kasey Zhang, Andy Lyu · 16. April 2026

  Supervised fine-tuning (SFT) followed by Group Relative Policy Optimization (GRPO) is a common post-training recipe. We conduct a controlled ablation over SFT-GRPO data overlap, evaluating Qwen3-8B (thinking disabled) post-trained for Lean 4 autoformalization under six conditions that differ solely …

• Adaptive Memory Crystallization for Autonomous AI Agent Learning in Dynamic Environments

  Rajat Khanda, Mohammad Baqar Sambuddha Chakrabarti, Satyasaran Changdar · 16. April 2026

  Autonomous AI agents operating in dynamic environments face a persistent challenge: acquiring new capabilities without erasing prior knowledge. We present Adaptive Memory Crystallization (AMC), a memory architecture for progressive experience consolidation in continual reinforcement learning. AMC …

• Soft $Q(\lambda)$: A multi-step off-policy method for entropy regularised reinforcement learning using eligibility traces

  Pranav Mahajan, Ben Seymour · 16. April 2026

  Soft Q-learning has emerged as a versatile model-free method for entropy-regularised reinforcement learning, optimising for returns augmented with a penalty on the divergence from a reference policy. Despite its success, the multi-step extensions of soft Q-learning remain relatively unexplored and l…

• From $P(y|x)$ to $P(y)$: Investigating Reinforcement Learning in Pre-train Space

  Yuqiao Tan, Minzheng Wang, Bo Liu, Zichen Liu, Tian Liang, Shizhu He, Jun Zhao, Kang Liu · 16. April 2026

  While reinforcement learning with verifiable rewards (RLVR) significantly enhances LLM reasoning by optimizing the conditional distribution P(y|x), its potential is fundamentally bounded by the base model's existing output distribution. Optimizing the marginal distribution P(y) in the Pre-train Spac…

• Memory Transfer Learning: How Memories are Transferred Across Domains in Coding Agents

  Kangsan Kim, Minki Kang, Taeil Kim, Yanlai Yang, Mengye Ren, Sung Ju Hwang · 16. April 2026

  Memory-based self-evolution has emerged as a promising paradigm for coding agents. However, existing approaches typically restrict memory utilization to homogeneous task domains, failing to leverage the shared infrastructural foundations, such as runtime environments and programming languages, that …