Reinforcement Learning in Robotics

Latest papers

• Offline Reinforcement Learning with Universal Horizon Models

  Hojun Chung, Junseo Lee, Songhwai Oh · 18 May 2026

  Model-based reinforcement learning (RL) offers a compelling approach to offline RL by enabling value learning on imagined on-policy trajectories. However, it often suffers from compounding errors due to repeated model inference on self-generated states. While geometric horizon models (GHM) alleviate…

• Imperfect World Models are Exploitable

  Logan Mondal Bhamidipaty (University of Edinburgh), Esmeralda S. Whitammer (University of Edinburgh), David Abel (University of Edinburgh), Mykel J. Kochenderfer (Stanford University), Subramanian Ramamoorthy (University of Edinburgh) · 18 May 2026

  We propose a novel definition of model exploitation in reinforcement learning. Informally, a world model is exploitable if it implies that one policy should be strictly preferred over another while the environment's true transition model implies the reverse. We analogize our definition with a prior …

• Lamarckian Inheritance in Dynamic Environments: How Key Variables Affect Evolutionary Dynamics

  K. Ege de Bruin, Kyrre Glette, Kai Olav Ellefsen · 18 May 2026

  The co-optimization of a robot's body and brain presents a coupled challenge: the morphology constrains which control strategies are effective, while the control determines how well the morphology performs. To address this, we combine morphology optimization as evolution with controller optimization…

• LEAP: Trajectory-Level Evaluation of LLMs in Iterative Scientific Design

  Marilyn Zhang, Tianfeng Chen, Fabi\'an Barzuna, Ankita Rathod, Mark E. Whiting · 18 May 2026

  LLMs are increasingly deployed in autonomous laboratories, under the assumption that their domain priors and reasoning over iterative feedback let them converge on good designs in fewer iterations than feedback-only baselines. Current iterative scientific design benchmarks, however, score only outco…

• Feedback World Model Enables Precise Guidance of Diffusion Policy

  Tuo An, Jindou Jia, Gen Li, Jingliang Li, Chuhao Zhou, Pengfei Liu, Bofan Lyu, Jiaqi Bai, Xinying Guo, Geng Li, Jianfei Yang · 18 May 2026

  World models aim to improve robotic decision making by predicting the consequences of actions. However, in practice, their predictions often become unreliable once the robot encounters states outside the training distribution, limiting their effectiveness at deployment. We observe that execution its…

• Deep Double Q-learning

  Prabhat Nagarajan, Martha White, Marlos C. Machado · 18 May 2026

  Double Q-learning is a classical control algorithm that mitigates the maximization bias of Q-learning. To do so, it explicitly trains two independent action-value functions and uses them to decouple action-selection and action-evaluation when computing bootstrap targets. Double DQN adapts target boo…

• BAPR: Bayesian amnesic piecewise-robust reinforcement learning for non-stationary continuous control

  Yifan Zhang, Liang Zheng · 18 May 2026

  Real-world control systems frequently operate under \emph{piecewise stationary} conditions, where dynamics remain stable for extended periods before undergoing abrupt regime changes. Standard robust RL methods face a fundamental dilemma: a globally conservative policy wastes performance during stabl…

• Learning Bilevel Policies over Symbolic World Models for Long-Horizon Planning

  Dillon Z. Chen, Till Hofmann, Toryn Q. Klassen, Sheila A. McIlraith · 18 May 2026

  We tackle the challenge of building embodied AI agents that can reliably solve long-horizon planning problems. Imitation learning from demonstrations has shown itself to be effective in training robots to solve a diversity of complex tasks requiring fine motor control and manipulation over low-level…

• Second-Order Actor-Critic Methods for Discounted MDPs via Policy Hessian Decomposition

  Sanjeev Manivannan, Shuban V · 15 May 2026

  We address the discounted reward setting in reinforcement learning (RL). To mitigate the value approximation challenges in policy gradient methods, actor-critic approaches have been developed and are known to converge to stationary points under suitable assumptions. However, these methods rely on fi…

• ROAD: Adaptive Data Mixing for Offline-to-Online Reinforcement Learning via Bi-Level Optimization

  Letian Yang (Shanghai Jiao Tong University, Shanghai, China), Xu Liu (Shanghai Jiao Tong University, Shanghai, China), Yiqiang Lu (Ant Group, Shanghai, China), Jian Liu (Ant Group, Shanghai, China), Weiqiang Wang (Ant Group, Shanghai, China), Shuai Li (Shanghai Jiao Tong University, Shanghai, China) · 15 May 2026

  Offline-to-online reinforcement learning harnesses the stability of offline pretraining and the flexibility of online fine-tuning. A key challenge lies in the non-stationary distribution shift between offline datasets and the evolving online policy. Common approaches often rely on static mixing rati…

• Self-Distilled Agentic Reinforcement Learning

  Zhengxi Lu, Zhiyuan Yao, Zhuowen Han, Zi-Han Wang, Jinyang Wu, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen · 15 May 2026

  Reinforcement learning (RL) has emerged as a central paradigm for post-training LLM agents, yet its trajectory-level reward signal provides only coarse supervision for long-horizon interaction. On-Policy Self-Distillation (OPSD) complements RL by introducing dense token-level guidance from a teacher…

• Distributionally Robust Multi-Task Reinforcement Learning via Adaptive Task Sampling

  Nicholas E. Corrado, Wenyuan Huang, Josiah P. Hanna · 15 May 2026

  Multi-task reinforcement learning (MTRL) aims to train a single agent to efficiently optimize performance across multiple tasks simultaneously. However, jointly optimizing all tasks often yields imbalanced learning: agents quickly solve easy tasks but learn slowly on harder ones. While prior work pr…

• R2R2: Robust Representation for Intensive Experience Reuse via Redundancy Reduction in Self-Predictive Learning

  Sanghyeob Song, Donghyeok Lee, Jinsik Kim, Sungroh Yoon · 15 May 2026

  For reinforcement learning in data-scarce domains like real-world robotics, intensive data reuse enhances efficiency but induces overfitting. While prior works focus on critic bias, representation-level instability in Self-Predictive Learning (SPL) under high Update-to-Data (UTD) regimes remains und…

• Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients

  Matias Alvo, Daniel Russo, Yash Kanoria · 15 May 2026

  We study reinforcement learning in hybrid discrete-continuous action spaces, such as settings where the discrete component selects a regime (or index) and the continuous component optimizes within it -- a structure common in robotics, control, and operations problems. Standard model-free policy grad…

• Collaborative Yet Personalized Policy Training: Single-Timescale Federated Actor-Critic

  Leo Muxing Wang, Pengkun Yang, Lili Su · 15 May 2026

  Despite the popularity of the actor-critic method and the practical needs of collaborative policy training, existing works typically either overlook environmental heterogeneity or give up personalization altogether by training a single shared policy across all agents. We consider a federated actor-c…

• Action-Conditioned Risk Gating for Safety-Critical Control under Partial Observability

  Yushen Liu, Yin-Jen Chen, Ziyi Chen, Tao Wang, Heng Huang, Xugui Zhou, Yanfu Zhang · 15 May 2026

  Many safety-critical control problems are modeled as risk-sensitive partially observable Markov decision processes, where the controller must make decisions from incomplete observations while balancing task performance against safety risk. Although belief-space planning provides a principled solutio…

• Temporal Fair Division in Multi-Agent Systems: From Precise Alternation Metrics to Scalable Coordination Proxies

  Nikolaos Al. Papadopoulos · 15 May 2026

  A plethora real-world environments require agents to compete repeatedly for the same limited resource, calling for a temporal notion of fairness judged across entire interaction histories. This paper advances the theory of temporal fair division by introducing Rotational Periodicity (RP), a family o…

• Dynamic Latent Routing

  Fangyuan Yu, Xin Su, Amir Abdullah · 15 May 2026

  We investigate the temporal concatenation of sub-policies in Markov Decision Processes (MDP) with time-varying reward functions. We introduce General Dijkstra Search (GDS), and prove that globally optimal goal-reaching policies can be recovered through temporal composition of intermediate optimal su…

• Diagnosing Training Inference Mismatch in LLM Reinforcement Learning

  Tianle Zhong, Neiwen Ling, Yifan Pi, Zijun Wei, Tianshu Yu, Geoffrey Fox, Peng Wu, Xiao Yu · 15 May 2026

  Modern LLM RL systems separate rollout generation from policy optimization. These two stages are expected to produce token probabilities that match exactly. However, implementation differences can make them assign different values to the same sequence under the same model weights, inducing Training-…

• Peng's Q($\lambda$) for Conservative Value Estimation in Offline Reinforcement Learning

  Byeongchan Kim, Min-hwan Oh · 15 May 2026

  We propose a model-free offline multi-step reinforcement learning (RL) algorithm, Conservative Peng's Q($\lambda$) (CPQL). Our algorithm adapts the Peng's Q($\lambda$) (PQL) operator for conservative value estimation as an alternative to the Bellman operator. To the best of our knowledge, this is th…

• Slot-MPC: Goal-Conditioned Model Predictive Control with Object-Centric Representations

  Jonathan Spieler, Angel Villar-Corrales, Sven Behnke · 15 May 2026

  Predictive world models enable agents to model scene dynamics and reason about the consequences of their actions. Inspired by human perception, object-centric world models capture scene dynamics using object-level representations, which can be used for downstream applications such as action planning…

• Fast Rates for Inverse Reinforcement Learning

  Andreas Schlaginhaufen, Maryam Kamgarpour · 15 May 2026

  We establish novel structural and statistical results for entropy-regularized min-max inverse reinforcement learning (Min-Max-IRL) with linear reward classes in finite-horizon MDPs with Borel state and action spaces. On the structural side, we show that maximum likelihood estimation (MLE) and Min-Ma…

• Ready from Day 1: Population-Aware Coordination for Large-Scale Constrained Multi-Agent Systems

  Angel Wang, Dominique Perrault-Joncas, Alvaro Maggiar, Carson Eisenach, Dean Foster · 15 May 2026

  In large-scale multi-agent systems with shared resource constraints, an upstream planner must iteratively evaluate candidate resource plans -- assessing feasibility, aggregate response, and marginal cost -- before committing to one. Lagrangian relaxation separates local decisions through a broadcast…

• Quantifying Potential Observation Missingness in Inverse Reinforcement Learning

  Leo Benac, Abhishek Sharma, Alihan Huyuk, Finale Doshi-Velez · 14 May 2026

  Inverse reinforcement learning (IRL), which infers reward functions from demonstrations, is a valuable tool for modeling and understanding decision-making behavior. Many variants of IRL have been developed to capture complexities of human decision-making, such as subjective beliefs, imperfect planni…

• Temper and Tilt Lead to SLOP: Reward Hacking Mitigation with Inference-Time Alignment

  Ye Wang, Jing Liu, Toshiaki Koike-Akino · 14 May 2026

  Inference-time alignment techniques offer a lightweight alternative or complement to costly reinforcement learning, while enabling continual adaptation as alignment objectives and reward targets evolve. Existing theoretical analyses justify these methods as approximations to sampling from distributi…