Reinforcement Learning in Robotics

Latest papers

• Knowledge Reutilization in Meta-Reinforcement Learning

  Yuan Meng, Bo Wang, Juan de los Rios Ruiz, Xiangtong Yao, Zhenshan Bing, Fuchun Sun, Alois Knoll · 17 June 2026

  Meta-reinforcement learning enables fast adaptation by extracting shared structure from related tasks, but existing end-to-end methods often couple task inference with embodiment-specific control. This coupling can obscure non-parametric task semantics, reduce sample efficiency, and limit cross-agen…

• Performance-Driven Environment Abstraction with Multi-Timescale Learning

  Yue Guan, Dipankar Maity, Panagiotis Tsiotras · 17 June 2026

  We study performance-driven environment abstraction for decision-making in large Markov decision processes. Rather than preserving geometric or topological structure, we seek abstractions that directly optimize decision quality. We model abstraction as a controlled approximation obtained by aggregat…

• Reward hacking in physical reinforcement learning revealed by turbulent drag reduction

  Giorgio Maria Cavallazzi, Miguel P\'erez-Cuadrado, Alfredo Pinelli · 17 June 2026

  A reinforcement-learning agent maximises its reward, which can diverge from the outcome its designer intended. In physical control the reward rarely closes that gap, and drag reduction in wall turbulence makes it concrete. A mass-conservation projection couples agents' outputs and erases the per-age…

• PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation

  Anhao Zhao, Junlong Tong, Yingqi Fan, Ping Nie, Wenjie Li, Xiaoyu Shen · 17 June 2026

  Standard on-policy distillation (OPD) for large language models estimates the reverse-KL objective using student-sampled tokens, yielding an unbiased single-sample Monte Carlo estimator that avoids vocabulary-wide computation. However, we show that this estimator suffers from severe training patholo…

• Learning Fair Pareto-Optimal Policies in Multi-Objective Reinforcement Learning

  Umer Siddique, Peilang Li, Yongcan Cao · 17 June 2026

  Fairness is an important aspect of decision-making in multi-objective reinforcement learning (MORL), where policies must ensure both optimality and equity across multiple, potentially conflicting objectives. While single-policy MORL methods can learn fair policies for fixed user preferences using we…

• Reversal Q-Learning

  Aditya Oberai, Seohong Park, Sergey Levine · 17 June 2026

  Iterative generative modeling techniques, such as flow matching, provide powerful tools to model complex behaviors for effective offline reinforcement learning (RL). In this work, we propose a new off-policy RL algorithm that trains a flow policy based on prior data. Our idea starts from the "expand…

• Visual Verification Enables Inference-time Steering and Autonomous Policy Improvement

  Mingtong Zhang, Dhruv Shah · 17 June 2026

  Robots deployed in the real world should learn from their experience and improve over time. This requires a mechanism of practicing and learning from feedback. In this paper, we propose VERITAS, a generator-verifier framework for generalist robot policies for inference-time policy steering and self-…

• A Diffusion Approximation for Temporal-Difference Learning with Linear Features under Markovian Noise

  M. Forzo, E. Monzio Compagnoni, A. Russo, A. Pacchiano · 17 June 2026

  Temporal difference (TD) learning with linear function approximation is a core method for policy evaluation. Its classical continuous-time description is an ordinary differential equation (ODE), which captures the asymptotic mean dynamics but neglects stochastic fluctuations determining the error fl…

• Beyond the Sampled Token: Preserving Candidate Support in RLVR

  Ruotian Peng, Yi Ren, Zhouliang Yu, Weiyang Liu, Yandong Wen · 17 June 2026

  We revisit exploration collapse in reinforcement learning with verifiable rewards (RLVR), from the perspective of the \emph{candidate distribution} for next-token prediction. We formally show that as probability concentrates on the top-$1$ candidate, the expected number of distinct responses collaps…

• Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers

  Sajad Movahedi, Vera Milovanovi\'c, Shlomo Libo Feigin, Alexander Theus, Thomas Hofmann, Valentina Boeva, T. Konstantin Rusch, Antonio Orvieto · 17 June 2026

  Looped architectures provide an inductive bias toward learning step-by-step procedures for tasks that require compositional reasoning. The number of effective layers reached by looping determines the quality of the solution these models find. Like deep architectures, looped architectures are prone t…

• Where Should Action Generation Begin? A Learnable Source Prior for Generative Robot Policies

  Meipo Dai, Qiyuan Zhuang, He-Yang Xu, Ying-Jie Shuai, Yijun Wang, Qi Dou, Xiu-Shen Wei · 17 June 2026

  Generative robot policies typically begin action generation from an observation-independent standard Gaussian distribution, leaving the choice of source distribution underexplored. This work asks a simple question: where should action generation begin? We propose LeaP, a Learnable source Prior that …

• APEX: Adaptive Principle EXtraction A Three-Layer Self-Evolution Framework for Production AI Agents

  Ya-Chuan Chen, Tien-Jen Lai, Hsiang-Wei Hu · 16 June 2026

  Self-improvement in AI agents has emerged as a key research frontier: systems that modify their own prompts, workflows, and decision rules based on accumulated operational experience. The state-of-the-art Self-Harness framework [1] achieves 14--21% improvement on Terminal-Bench-2.0 by mining failure…

• Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data

  Yi Zhao, Aidan Scannell, Wenshuai Zhao, Yuxin Hou, Tianyu Cui, Le Chen, Dieter B\"uchler, Arno Solin, Juho Kannala, Joni Pajarinen · 16 June 2026

  Leveraging offline data is a promising way to improve the sample efficiency of online reinforcement learning (RL). This paper expands the pool of usable data for offline-to-online RL by leveraging abundant non-curated data that is reward-free, of mixed quality, and collected across multiple embodime…

• Hierarchical Advantage Weighting for Online RL Fine-Tuning of VLAs from Sparse Episode Outcomes

  Tongyan Fang, Siyuan Huang, Naiyu Fang, Ganlong Zhao, Zhongjin Luo, Jianbo Liu, Xiaogang Wang, Ying Dong, Hongsheng Li · 16 June 2026

  When pretrained VLA policies are fine-tuned through online RL, each rollout episode produces only a single binary outcome (success or failure), yet the actor update requires per-transition supervision. Existing approaches commonly reduce this sparse outcome to a single scalar reward or advantage sig…

• HiMPO: Hindsight-Informed Memory Policy Optimization for Less-Entangled Credit in Long-Horizon Agents

  Jiangze Yan, Yi Shen, Wenjing Zhang, Jieyun Huang, Zhaoxiang Liu, Ning Wang, Kai Wang, Shiguo Lian · 16 June 2026

  Long-horizon agents rely on memory mechanisms to compress interaction history, but optimizing memory writing faces a distinct credit assignment challenge: a memory update may be rewarded or penalized due to downstream tool failures, noisy observations, or reasoning errors rather than its own contrib…

• Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning

  Zirui Pang, Chenlong Zhang, Haosheng Tan, Zhuoran Jin, Jiaheng Wei, Zixin Zhong · 16 June 2026

  LLM unlearning has emerged as a cost-effective alternative to full retraining for removing hazardous knowledge from pretrained models while preserving general utility. Recent RL-based methods such as RULE reformulate unlearning as learning a refusal behavior, but their on-policy optimization repeate…

• Task-Error Residual Learning for Real-Robot Five-Ball Juggling

  Kai Ploeger, Jan Peters · 16 June 2026

  For residual learning that refines existing behavior, sample efficiency depends on two things: how much information each rollout returns, and how efficiently the learner uses that information. Reinforcement learning's standard scalar reward carries far less information than the directional task erro…

• On the Role of Computation in Reinforcement Learning

  Raj Ghugare, Micha{\l} Bortkiewicz, Alicja Ziarko, Benjamin Eysenbach · 16 June 2026

  How does the amount of compute available to a reinforcement learning (RL) policy affect its learning? Can policies using a fixed amount of parameters, still benefit from additional compute? The standard RL framework does not provide a language to answer these questions formally. Empirically, deep RL…

• HOLO-MPPI: Multi-Scenario Motion Planning via Hierarchical Policy Optimization

  Youngjae Min, Jovin D'sa, Faizan M. Tariq, David Isele, Navid Azizan, Sangjae Bae · 16 June 2026

  Robots deployed in the real world must plan motions across diverse scenarios without per-scenario retuning. End-to-end reinforcement learning (RL) can generalize across scenarios but often becomes brittle under distribution shift, reward misspecification, and stochastic interactions. Model predictiv…

• Evolutionary Bilevel Reward Shaping for Generalization in Reinforcement Learning

  Ekasit Usaratniwart, Xilin Gao, Marc Ong, Youhei Akimoto · 16 June 2026

  Reinforcement learning (RL) often suffers from performance degradation when deployed in environments that differ from those encountered during training. Existing techniques such as domain randomization (DR) mitigate this, but require access to diverse training environments and full trajectory observ…

• Infant Spontaneous Movement Noise Improves Exploration in Deep RL

  Francisco M. L\'opez, Markus R. Ernst, Francisco Cruz, Matej Hoffmann, and Jochen Triesch · 16 June 2026

  Exploration in deep reinforcement learning (RL) is commonly implemented as temporally uncorrelated white noise. However, recent works show that temporally correlated colored noise can improve exploration efficiency by producing smooth trajectories with better coverage of the state space. We inquire …

• Proximal Policy Optimization for Amortized Discrete Sampling

  Anna Zykova-Myzina, Timofei Gritsaev, Daniil Tiapkin, Nikita Morozov · 16 June 2026

  This paper explores policy gradient algorithms for training stochastic policies to sample from structured discrete probability distributions under the Generative Flow Network (GFlowNet) framework. Building on extensive theoretical connections between GFlowNets and entropy-regularized reinforcement l…

• From Noise to Intent: Anchoring Generative VLA Policies with Residual Bridges

  Yiming Zhong, Yaoyu He, Zemin Yang, Pengfei Tian, Yifan Huang, Qingqiu Huang, Xinge Zhu, Yuexin Ma · 16 June 2026

  Bridging high-level semantic understanding with low-level physical control remains a persistent challenge in embodied intelligence, stemming from the fundamental spatiotemporal scale mismatch between cognition and action. Existing generative VLA policies typically adopt a "Generation-from-Noise" par…

• Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games with Average Reward

  \c{S}evket Kaan Alk{\i}r, Naci Sald{\i}, Berkay Anahtarc{\i}, Can Deha Kar{\i}ks{\i}z · 16 June 2026

  We study inverse reinforcement learning for discrete-time, infinite-horizon mean-field games (MFGs) under an average-reward criterion. Expert demonstrations are assumed to arise from a stationary mean-field equilibrium under an unknown reward, and the goal is to recover a policy explaining the obser…

• PO-PDDL: Learning Symbolic POMDPs from Visual Demonstrations for Robot Planning Under Uncertainty

  Wenjing Tang, Xuanjin Jin, Yuan Liu, Renming Huang, Cewu Lu, Panpan Cai · 16 June 2026

  Real-world robot task planning must operate under both stochastic action execution and partial observability, yet constructing Partially Observable Markov Decision Process (POMDP) models for real robotics domains remains difficult and labor-intensive. We introduce PO-PDDL, a symbolic formulation of …