Reinforcement Learning in Robotics

Latest papers

• The Weight of Silence: A Causal Case for Weights Over the Scratchpad in Latent Chess Reasoning

  Ishan S. Kshirsagar · 24 July 2026

  Latent, or silent, reasoning lets language models carry out intermediate computation in continuous vector space instead of words, and is widely assumed to function as an internal scratchpad the model actively consults during inference. Whether that assumption survives reinforcement learning has not …

• Adaptive Multi-Horizon Reinforcement Learning

  Manoosh Samiei, Doina Precup, Paul Masset · 24 July 2026

  Effective decision-making in complex and changing environments requires balancing short-term and long-term consequences. In reinforcement learning (RL), this trade-off is typically controlled through a fixed discount factor, which imposes a single exponentially discounted temporal horizon. However, …

• Relative Value Learning

  Marc H\"oftmann, Jan Robine, Stefan Harmeling · 24 July 2026

  In reinforcement learning, critics typically estimate absolute state values $V(s)$, estimating how good a particular situation is in isolation. However, it turns out that only differences in value are relevant for control. Motivated by this, we propose Relative Value Learning (RV), a framework that …

• PATS: Policy-Aware Training Scaffolding for Agentic Reinforcement Learning

  Yipeng Shi, Zhipeng Ma, Yue Wang, Qitai Tan, Yang Li, Peng Chen, Zhengzhou Zhu · 24 July 2026

  In long-horizon LLM agent reinforcement learning, weak policies often repeat similar failures, producing uninformative rollout trajectories and limiting effective policy optimization. Existing skill-centric methods improve exploration by optimizing, filtering, or internalizing reusable skills. Howev…

• Emergent Compositional Skills in Mixture-of-Experts VLAs

  Shlok Shah, Rhiaan Jhaveri, Tharun Kumar Tiruppali Kalidoss, Chirayu Nimonkar, Ishaan Javali · 24 July 2026

  We consider the problem of learning compositional robot policies end-to-end from expert demonstrations, without any pre-specified notion of task decomposition or hierarchy. We ask whether a VLA trained with a simplified Mixture-of-Experts (MoE) action head can emergently learn to decompose tasks int…

• Expert Behavior Prior Reinforcement Learning

  Gong Gao, Weidong Zhao, Xianhui Liu, Ning Jia · 24 July 2026

  Behavior prior reinforcement learning (BPRL) has emerged as a promising paradigm to improve sample efficiency in online reinforcement learning (RL) by leveraging policy priors derived from offline demonstrations. However, most existing BPRL methods rely on static offline datasets, which often suffer…

• Offline RL with Hierarchical Action Chunking

  Ahad Jawaid · 24 July 2026

  Offline goal-conditioned reinforcement learning (RL) holds the promise of learning general-purpose policies from static datasets. However, scaling these methods to long-horizon tasks remains a challenge due to the curse of horizon, where value estimation errors can compound through long chains of bo…

• Preference Tuning as Spectral Update Reorganization

  Peiyan Zhang, Haibo Jin, Liying Kang, Haohan Wang · 24 July 2026

  Preference-based post-training is usually understood through endpoint behavior, yet the learned update that produces this behavior remains largely opaque. We study RLHF and related preference optimization through the spectral structure of their induced parameter updates. By decomposing effective LoR…

• Koopman Dreamer: Spectrally Constrained Latent Dynamics for Stable World-Model Imagination

  Jiaqi Li, Xinglong Zhang, Haibin Xie, Yixing Lan, Wei Pan, Xin Xu · 23 July 2026

  Latent world models improve sample efficiency in continuous control by optimizing policies over imagined latent trajectories, but common neural transitions offer limited direct control over modal persistence and error accumulation in long rollouts. We propose Koopman Dreamer, a Dreamer-style world m…

• Memory Merge DQN: Sensitivity Weighted Target Updates for Stable Value Learning

  Adrian Ly, Richard Dazeley, Peter Vamplew, Sunil Aryal, Francisco Cruz · 23 July 2026

  Deep Q-networks use target networks to stabilise bootstrapped value learning, but the standard hard copy update also introduces a tradeoff. Holding the target network fixed, improves short term stability, yet each hard update abruptly replaces the target parameters with the newest online network and…

• Leveraging Offline Supervision for Efficient and Generalizable Reinforcement Learning in Large-Scale Vision-Language-Action Models

  Dmitriy Poyarkov, Aleksei Staroverov, Aleksandr I. Panov · 23 July 2026

  It is commonly observed that online reinforcement learning (RL) produces better-performing strategies than offline methods across a broad range of performance measures. In particular, RL-trained policies exhibit stronger out-of-distribution (OOD) behavior, where models trained only with imitation le…

• Toward Adaptable Multi-Agent Reinforcement Learning: An Assumption-Aware Review

  Siyi Hu, Mohamad A Hady, Jianglin Qiao, Jimmy Cao, Mahardhika Pratama, Ryszard Kowalczyk · 23 July 2026

  Multi-Agent Reinforcement Learning (MARL) has achieved strong performance in simulated benchmarks, yet real deployments often violate the assumptions under which algorithms are designed and evaluated. Agent populations may change, objectives may shift, centralized information may be unavailable, exe…

• Diffusion ReRoll: Revisable Denoising for Robotic Sequential Prediction

  Seonsoo Kim, Seongil Hong, Jun-Gill Kang · 23 July 2026

  We propose Diffusion ReRoll, a diffusion-based framework for robotic sequential prediction that enables revisable denoising over horizons. Existing diffusion-based sequence predictors typically perform a single monotonic denoising process. In contrast, Diffusion ReRoll selectively re-noises regions …

• The World Model Remembers, the Actor Forgets: Dream Rehearsal for Continual Model-Based RL

  Gurp Nijjer · 23 July 2026

  Model-based reinforcement-learning agents of the DreamerV3 family forget catastrophically when trained on task sequences, even when an unbounded replay buffer preserves every earlier experience. We ask a question the continual-RL literature has assumed an answer to but never measured: which componen…

• SLPO: Scaling Latent Reasoning via a Surrogate Policy

  Runyang You, Zhiyuan Liu, Yongqi Li, Wenjie Li · 23 July 2026

  Reinforcement learning with verifiable rewards has become the predominant recipe for eliciting test-time scaling in explicit Chain-of-Thought reasoners. Yet this scaling path remains computationally costly, since every intermediate step must be decoded as a language token. Latent reasoning instead c…

• Dreamer-CPC: Message Learning with World Models for Decentralized Multi-agent Reinforcement Learning

  Taisuke Takayama, Naoto Yoshida, Tadahiro Taniguchi · 23 July 2026

  In multi-agent reinforcement learning (MARL), inter-agent communication is effective for improving performance under partial observability. Representation learning-based approaches enable decentralized agents to learn messages grounded in their own observations, but they rely only on current observa…

• S3: Stable Subgoal Selection by Constraining Uncertainty of Coarse Dynamics in Hierarchical Reinforcement Learning

  Kshitij Kumar Srivastava, Kshitij Jerath · 22 July 2026

  Hierarchical Reinforcement Learning (HRL) intends to separate strategic planning from primitive execution. It has been widely successful in solving long-horizon and complex tasks, where flat-RL algorithms have difficulty in learning. However, while the low-level agent in HRL benefits from dense feed…

• A Self-Evolving Default Action for Cooperative Tasks with Continuous Action Space

  Shuangyao Huang · 22 July 2026

  Counterfactual credit assignment has proven effective in multi-agent reinforcement learning (MARL) for discrete action spaces, yet its extension to continuous-action cooperative tasks remains challenging. Existing methods that approximate the counterfactual baseline via Monte Carlo sampling often in…

• Scalable Policy Optimization for Networked Multi-Agent Reinforcement Learning with Continuous State-Action Spaces

  Dongming Wang, Pengcheng Dai, Wenwu Yu, Wei Ren · 22 July 2026

  We develop the Continuous Distributed Coupled Policy Gradient (CDCPG) algorithm for cooperative reinforcement learning in networked Markov decision processes with continuous state and action spaces. Each agent maintains a local actor over a bounded graph neighborhood, and a localized least-squares t…

• Comparative Study of Multi-Agent Actor-Critic Algorithms in Parameterized Action Reinforcement Learning

  Ubayd Ali Bapoo, Clement N Nyirenda · 22 July 2026

  Parameterized action reinforcement learning has shown strong performance in environments requiring both discrete action selection and continuous parameterization. Prior work established the effectiveness of single-agent actor-critic algorithms - Greedy Actor-Critic (GAC), Soft Actor-Critic (SAC), an…

• The Open Ant: A Robot Platform for Reinforcement Learning Research

  Elena Sorina Lupu, Patrick Spieler, Khurram Javed, Kris De Asis, John D. Martin, Martha Steenstrup, Joseph Modayil · 22 July 2026

  Reinforcement learning (RL) research has demonstrated success in both physical and simulated domains; however, the predominant methodology remains rooted in simulations. The predominance of simulations makes translating research to physical reality uncertain for both algorithms and researchers. We p…

• Planning as Emergent Behavior in Reinforcement Learning with Relational Hidden States

  Armin Sommer · 22 July 2026

  Reinforcement learning is conventionally divided into model-based and model-free methods. In this taxonomy, model-based methods perform lookahead planning over a learned world model, whereas model-free methods learn a reactive state-action mapping. Recent work, however, has shown that planning can e…

• Strategy-Following Multi-Agent Deep Reinforcement Learning Considering Control Strategies Provided to Other Agents

  Yamato Takahagi, Gentoku Nakasone, Yoshinari Motokawa, Toshiharu Sugawara · 22 July 2026

  This study proposes a learning method for multi-agent systems that allows agents to be controlled through human manager instructions after learning and enables uninstructed agents to implicitly complement the overall work based on the actions of other agents. Multi-agent applications using deep lear…

• RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts

  Yuxin Xiong, Xunyi Jiang, Rohan Surana, Xintong Li, Sheldon Yu, Nikki Lijing Kuang, Ryan A. Rossi, Jingbo Shang, Tong Yu, Julian McAuley, Junda Wu · 22 July 2026

  Group Relative Policy Optimization (GRPO) has shown strong effectiveness in reinforcement learning from verifiable feedback, where sampled rollouts can be compared within a group using task-provided correctness signals. However, extending group-relative optimization beyond verifiable settings is cha…

• S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHF

  Wei Chen, Guanghui Zhu, Yafei Li, Limin Wang, Yihua Huang · 22 July 2026

  Reinforcement learning from human feedback (RLHF) with preference-based reward models often exhibits unstable training dynamics. A key contributing factor is that standard RLHF relies on a single sequence-level scalar reward, which is propagated to token-level policy updates and leaves credit assign…