Reinforcement Learning in Robotics

Latest papers

• Action Shapley: A Training Data Selection Metric for World Model in Reinforcement Learning

  Rajat Ghosh, Debojyoti Dutta · 19 January 2026

  Numerous offline and model-based reinforcement learning systems incorporate world models to emulate the inherent environments. A world model is particularly important in scenarios where direct interactions with the real environment is costly, dangerous, or impractical. The efficacy and interpretabil…

• CTHA: Constrained Temporal Hierarchical Architecture for Stable Multi-Agent LLM Systems

  Percy Jardine · 19 January 2026

  Recently, multi-time-scale agent architectures have extended the ubiquitous single-loop paradigm by introducing temporal hierarchies with distinct cognitive layers. While yielding substantial performance gains, this diversification fundamentally compromises the coordination stability intrinsic to un…

• A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning

  Siyuan Guo, Yanchao Sun, Jifeng Hu, Sili Huang, Hechang Chen, Haiyin Piao, Lichao Sun, Yi Chang · 19 January 2026

  Offline reinforcement learning (RL) provides a promising solution to learning an agent fully relying on a data-driven paradigm. However, constrained by the limited quality of the offline dataset, its performance is often sub-optimal. Therefore, it is desired to further finetune the agent via extra o…

• Factored Value Functions for Graph-Based Multi-Agent Reinforcement Learning

  Ahmed Rashwan, Keith Briggs, Chris Budd, Lisa Kreusser · 19 January 2026

  Credit assignment is a core challenge in multi-agent reinforcement learning (MARL), especially in large-scale systems with structured, local interactions. Graph-based Markov decision processes (GMDPs) capture such settings via an influence graph, but standard critics are poorly aligned with this str…

• Evidence-Augmented Policy Optimization with Reward Co-Evolution for Long-Context Reasoning

  Xin Guan, Zijian Li, Shen Huang, Pengjun Xie, Jingren Zhou, Jiuxin Cao · 16 January 2026

  While Reinforcement Learning (RL) has advanced LLM reasoning, applying it to long-context scenarios is hindered by sparsity of outcome rewards. This limitation fails to penalize ungrounded "lucky guesses," leaving the critical process of needle-in-a-haystack evidence retrieval largely unsupervised. …

• Self-Organizing Dual-Buffer Adaptive Clustering Experience Replay (SODASER) for Safe Reinforcement Learning in Optimal Control

  Roya Khalili Amirabadi, Mohsen Jalaeian Farimani, Omid Solaymani Fard · 16 January 2026

  This paper proposes a novel reinforcement learning framework, named Self-Organizing Dual-buffer Adaptive Clustering Experience Replay (SODACER), designed to achieve safe and scalable optimal control of nonlinear systems. The proposed SODACER mechanism consisting of a Fast-Buffer for rapid adaptation…

• Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics

  Dongyoung Kim, Sumin Park, Huiwon Jang, Jinwoo Shin, Jaehyung Kim, Younggyo Seo · 16 January 2026

  Large Vision-Language Models (LVLMs) have recently shown great promise in advancing robotics by combining embodied reasoning with robot control. A common approach involves training on embodied reasoning tasks related to robot control using Supervised Fine-Tuning (SFT). However, SFT datasets are ofte…

• DeFlow: Decoupling Manifold Modeling and Value Maximization for Offline Policy Extraction

  Zhancun Mu · 16 January 2026

  We present DeFlow, a decoupled offline RL framework that leverages flow matching to faithfully capture complex behavior manifolds. Optimizing generative policies is computationally prohibitive, typically necessitating backpropagation through ODE solvers. We address this by learning a lightweight ref…

• SuS: Strategy-aware Surprise for Intrinsic Exploration

  Mark Kashirskiy, Ilya Makarov · 16 January 2026

  We propose Strategy-aware Surprise (SuS), a novel intrinsic motivation framework that uses pre-post prediction mismatch as a novelty signal for exploration in reinforcement learning. Unlike traditional curiosity-driven methods that rely solely on state prediction error, SuS introduces two complement…

• Scalable Oversight for Superhuman AI via Recursive Self-Critiquing

  Xueru Wen, Jie Lou, Xinyu Lu, Junjie Yang, Yanjiang Liu, Yaojie Lu, Debing Zhang, Xing Yu · 16 January 2026

  As AI capabilities increasingly surpass human proficiency in complex tasks, current alignment techniques, including SFT and RLHF, face fundamental challenges in ensuring reliable oversight. These methods rely on direct human assessment and become impractical when AI outputs exceed human cognitive th…

• Projected Microbatch Accumulation yields reference-free proximal policy updates for reinforcement learning

  Nilin Abrahamsen · 16 January 2026

  This note introduces Projected Microbatch Accumulation (PROMA), a proximal policy update method for large language model fine-tuning. PROMA accumulates policy gradients across microbatches by projecting out sequence-wise gradient components before microbatch aggregation. The projection is applied la…

• Provably Safe Reinforcement Learning for Stochastic Reach-Avoid Problems with Entropy Regularization

  Abhijit Mazumdar, Rafal Wisniewski, Manuela L. Bujorianu · 16 January 2026

  We consider the problem of learning the optimal policy for Markov decision processes with safety constraints. We formulate the problem in a reach-avoid setup. Our goal is to design online reinforcement learning algorithms that ensure safety constraints with arbitrarily high probability during the le…

• SRT: Accelerating Reinforcement Learning via Speculative Rollout with Tree-Structured Cache

  Chi-Chih Chang, Siqi Zhu, Zhichen Zeng, Haibin Lin, Jiaxuan You, Mohamed S. Abdelfattah, Ziheng Jiang, Xuehai Qian · 15 January 2026

  We present Speculative Rollout with Tree-Structured Cache (SRT), a simple, model-free approach to accelerate on-policy reinforcement learning (RL) for language models without sacrificing distributional correctness. SRT exploits the empirical similarity of rollouts for the same prompt across training…

• GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR

  Jiaying Zhang, Lei Shi, Jiguo Li, Jun Xu, Jiuchong Gao, Jinghua Hao, Renqing He · 15 January 2026

  Reinforcement Learning with Verifiable Rewards (RLVR) is crucial for advancing large-scale reasoning models. However, existing parameter-efficient methods, such as PiSSA and MiLoRA, are designed for Supervised Fine-Tuning (SFT) and do not account for the distinct optimization dynamics and geometric …

• Reinforcement Learning with Exogenous States and Rewards

  George Trimponias, Thomas G. Dietterich · 15 January 2026

  Exogenous state variables and rewards can slow reinforcement learning by injecting uncontrolled variation into the reward signal. This paper formalizes exogenous state variables and rewards and shows that if the reward function decomposes additively into endogenous and exogenous components, the MDP …

• MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweighting

  Kangda Wei, Ruihong Huang · 15 January 2026

  Group Relative Policy Optimization (GRPO) has become a standard approach for training mathematical reasoning models; however, its reliance on multiple completions per prompt makes training computationally expensive. Although recent work has reduced the number of training steps required to reach peak…

• Imagine-then-Plan: Agent Learning from Adaptive Lookahead with World Models

  Youwei Liu, Jian Wang, Hanlin Wang, Beichen Guo, Wenjie Li · 15 January 2026

  Recent advances in world models have shown promise for modeling future dynamics of environmental states, enabling agents to reason and act without accessing real environments. Current methods mainly perform single-step or fixed-horizon rollouts, leaving their potential for complex task planning unde…

• GIFT: Unlocking Global Optimality in Post-Training via Finite-Temperature Gibbs Initialization

  Zhengyang Zhao, Lu Ma, Yizhen Jiang, Xiaochen Ma, Zimo Meng, Chengyu Shen, Lexiang Tang, Haoze Sun, Peng Pei, Wentao Zhang · 15 January 2026

  The prevailing post-training paradigm for Large Reasoning Models (LRMs)--Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL)--suffers from an intrinsic optimization mismatch: the rigid supervision inherent in SFT induces distributional collapse, thereby exhausting the exploration sp…

• Reward Learning through Ranking Mean Squared Error

  Chaitanya Kharyal, Calarina Muslimani, Matthew E. Taylor · 15 January 2026

  Reward design remains a significant bottleneck in applying reinforcement learning (RL) to real-world problems. A popular alternative is reward learning, where reward functions are inferred from human feedback rather than manually specified. Recent work has proposed learning reward functions from hum…

• Structure Detection for Contextual Reinforcement Learning

  Tianyue Zhou, Jung-Hoon Cho, Cathy Wu · 14 January 2026

  Contextual Reinforcement Learning (CRL) tackles the problem of solving a set of related Contextual Markov Decision Processes (CMDPs) that vary across different context variables. Traditional approaches--independent training and multi-task learning--struggle with either excessive computational costs …

• Safe Heterogeneous Multi-Agent RL with Communication Regularization for Coordinated Target Acquisition

  Gabriele Calzolari (Lulea University of Technology), Vidya Sumathy (Lulea University of Technology), Christoforos Kanellakis (Lulea University of Technology), George Nikolakopoulos (Lulea University of Technology) · 14 January 2026

  This paper introduces a decentralized multi-agent reinforcement learning framework enabling structurally heterogeneous teams of agents to jointly discover and acquire randomly located targets in environments characterized by partial observability, communication constraints, and dynamic interactions.…

• Cross-Domain Imitation Learning via Optimal Transport

  Arnaud Fickinger, Samuel Cohen, Stuart Russell, Brandon Amos · 14 January 2026

  Cross-domain imitation learning studies how to leverage expert demonstrations of one agent to train an imitation agent with a different embodiment or morphology. Comparing trajectories and stationary distributions between the expert and imitation agents is challenging because they live on different …

• ORBIT: On-policy Exploration-Exploitation for Controllable Multi-Budget Reasoning

  Kun Liang, Clive Bai, Xin Xu, Chenming Tang, Sanwoo Lee, Weijie Liu, Saiyong Yang, Yunfang Wu · 14 January 2026

  Recent Large Reasoning Models (LRMs) achieve strong performance by leveraging long-form Chain-of-Thought (CoT) reasoning, but uniformly applying overlong reasoning at inference time incurs substantial and often unnecessary computational cost. To address this, prior work explores various strategies t…

• Scalable Multiagent Reinforcement Learning with Collective Influence Estimation

  Zhenglong Luo, Zhiyong Chen, Aoxiang Liu, Ke Pan · 14 January 2026

  Multiagent reinforcement learning (MARL) has attracted considerable attention due to its potential in addressing complex cooperative tasks. However, existing MARL approaches often rely on frequent exchanges of action or state information among agents to achieve effective coordination, which is diffi…

• The End of Reward Engineering: How LLMs Are Redefining Multi-Agent Coordination

  Haoran Su, Yandong Sun, Congjia Yu · 14 January 2026

  Reward engineering, the manual specification of reward functions to induce desired agent behavior, remains a fundamental challenge in multi-agent reinforcement learning. This difficulty is amplified by credit assignment ambiguity, environmental non-stationarity, and the combinatorial growth of inter…