Reinforcement Learning in Robotics

Latest papers

• Incorporating Spatial Information into Goal-Conditioned Hierarchical Reinforcement Learning via Graph Representations

  Shuyuan Zhang, Zihan Wang, Xiao-Wen Chang, Doina Precup · 17 November 2025

  The integration of graphs with Goal-conditioned Hierarchical Reinforcement Learning (GCHRL) has recently gained attention, as intermediate goals (subgoals) can be effectively sampled from graphs that naturally represent the overall task structure in most RL tasks. However, existing approaches typica…

• Behaviour Policy Optimization: Provably Lower Variance Return Estimates for Off-Policy Reinforcement Learning

  Alexander W. Goodall, Edwin Hamel-De le Court, Francesco Belardinelli · 17 November 2025

  Many reinforcement learning algorithms, particularly those that rely on return estimates for policy improvement, can suffer from poor sample efficiency and training instability due to high-variance return estimates. In this paper we leverage new results from off-policy evaluation; it has recently be…

• Scalable Population Training for Zero-Shot Coordination

  Bingyu Hui, Lebin Yu, Quanming Yao, Yunpeng Qu, Xudong Zhang, Jian Wang · 17 November 2025

  Zero-shot coordination(ZSC) has become a hot topic in reinforcement learning research recently. It focuses on the generalization ability of agents, requiring them to coordinate well with collaborators that are not seen before without any fine-tuning. Population-based training has been proven to prov…

• DiAReL: Reinforcement Learning with Disturbance Awareness for Robust Sim2Real Policy Transfer in Robot Control

  Mohammadhossein Malmir (Department of Computer Engineering, School of Computation, Information and Technology, Technical University of Munich), Josip Josifovski (Department of Computer Engineering, School of Computation, Information and Technology, Technical University of Munich), Noah Klarmann (Rosenheim University of Applied Sciences), Alois Knoll (Department of Computer Engineering, School of Computation, Information and Technology, Technical University of Munich) · 17 November 2025

  Delayed Markov decision processes (DMDPs) fulfill the Markov property by augmenting the state space of agents with a finite time window of recently committed actions. In reliance on these state augmentations, delay-resolved reinforcement learning algorithms train policies to learn optimal interactio…

• Succeed or Learn Slowly: Sample Efficient Off-Policy Reinforcement Learning for Mobile App Control

  Georgios Papoudakis, Thomas Coste, Jianye Hao, Jun Wang, Kun Shao · 14 November 2025

  Reinforcement learning (RL) using foundation models for policy approximations in multi-turn tasks remains challenging. We identify two main limitations related to sparse reward settings and policy gradient updates, based on which we formulate a key insight: updates from positive samples with high re…

• Transfer in Reinforcement Learning via Regret Bounds for Learning Agents

  Adrienne Tuynman, Ronald Ortner · 14 November 2025

  We present an approach for the quantification of the usefulness of transfer in reinforcement learning via regret bounds for a multi-agent setting. Considering a number of $\aleph$ agents operating in the same Markov decision process, however possibly with different reward functions, we consider the …

• Harnessing Bounded-Support Evolution Strategies for Policy Refinement

  Ethan Hirschowitz, Fabio Ramos · 14 November 2025

  Improving competent robot policies with on-policy RL is often hampered by noisy, low-signal gradients. We revisit Evolution Strategies (ES) as a policy-gradient proxy and localize exploration with bounded, antithetic triangular perturbations, suitable for policy refinement. We propose Triangular-Dis…

• Out-of-Distribution Generalization with a SPARC: Racing 100 Unseen Vehicles with a Single Policy

  Bram Grooten, Patrick MacAlpine, Kaushik Subramanian, Peter Stone, Peter R. Wurman · 14 November 2025

  Generalization to unseen environments is a significant challenge in the field of robotics and control. In this work, we focus on contextual reinforcement learning, where agents act within environments with varying contexts, such as self-driving cars or quadrupedal robots that need to operate in diff…

• Multi-agent In-context Coordination via Decentralized Memory Retrieval

  Tao Jiang, Zichuan Lin, Lihe Li, Yi-Chen Li, Cong Guan, Lei Yuan, Zongzhang Zhang, Yang Yu, Deheng Ye · 14 November 2025

  Large transformer models, trained on diverse datasets, have demonstrated impressive few-shot performance on previously unseen tasks without requiring parameter updates. This capability has also been explored in Reinforcement Learning (RL), where agents interact with the environment to retrieve conte…

• Beyond Monotonicity: Revisiting Factorization Principles in Multi-Agent Q-Learning

  Tianmeng Hu, Yongzheng Cui, Rui Tang, Biao Luo, Ke Li · 14 November 2025

  Value decomposition is a central approach in multi-agent reinforcement learning (MARL), enabling centralized training with decentralized execution by factorizing the global value function into local values. To ensure individual-global-max (IGM) consistency, existing methods either enforce monotonici…

• Opinion: Towards Unified Expressive Policy Optimization for Robust Robot Learning

  Haidong Huang, Haiyue Zhu. Jiayu Song, Xixin Zhao, Yaohua Zhou, Jiayi Zhang, Yuze Zhai, Xiaocong Li · 14 November 2025

  Offline-to-online reinforcement learning (O2O-RL) has emerged as a promising paradigm for safe and efficient robotic policy deployment but suffers from two fundamental challenges: limited coverage of multimodal behaviors and distributional shifts during online adaptation. We propose UEPO, a unified …

• Optimistic Reinforcement Learning with Quantile Objectives

  Mohammad Alipour-Vaezi, Huaiyang Zhong, Kwok-Leung Tsui, Sajad Khodadadian · 14 November 2025

  Reinforcement Learning (RL) has achieved tremendous success in recent years. However, the classical foundations of RL do not account for the risk sensitivity of the objective function, which is critical in various fields, including healthcare and finance. A popular approach to incorporate risk sensi…

• Scaling Environments for LLM Agents in the Era of Learning from Interaction: A Survey

  Yuchen Huang, Sijia Li, Minghao Liu, Wei Liu, Shijue Huang, Zhiyuan Fan, Hou Pong Chan, Yi R. Fung · 14 November 2025

  LLM-based agents can autonomously accomplish complex tasks across various domains. However, to further cultivate capabilities such as adaptive behavior and long-term decision-making, training on static datasets built from human-level knowledge is insufficient. These datasets are costly to construct …

• Operator Models for Continuous-Time Offline Reinforcement Learning

  Nicolas Hoischen, Petar Bevanda, Max Beier, Stefan Sosnowski, Boris Houska, Sandra Hirche · 14 November 2025

  Continuous-time stochastic processes underlie many natural and engineered systems. In healthcare, autonomous driving, and industrial control, direct interaction with the environment is often unsafe or impractical, motivating offline reinforcement learning from historical data. However, there is limi…

• Tree-OPO: Off-policy Monte Carlo Tree-Guided Advantage Optimization for Multistep Reasoning

  Bingning Huang, Tu Nguyen, Matthieu Zimmer · 13 November 2025

  Recent advances in reasoning with large language models (LLMs) have shown the effectiveness of Monte Carlo Tree Search (MCTS) for generating high-quality intermediate trajectories, particularly in math and symbolic domains. Inspired by this, we explore how MCTS-derived trajectories-traditionally use…

• TIGER-MARL: Enhancing Multi-Agent Reinforcement Learning with Temporal Information through Graph-based Embeddings and Representations

  Nikunj Gupta, Ludwika Twardecka, James Zachary Hare, Jesse Milzman, Rajgopal Kannan, Viktor Prasanna · 13 November 2025

  In this paper, we propose capturing and utilizing \textit{Temporal Information through Graph-based Embeddings and Representations} or \textbf{TIGER} to enhance multi-agent reinforcement learning (MARL). We explicitly model how inter-agent coordination structures evolve over time. While most MARL app…

• Diffusion Policies with Value-Conditional Optimization for Offline Reinforcement Learning

  Yunchang Ma, Tenglong Liu, Yixing Lan, Xin Yin, Changxin Zhang, Xinglong Zhang, Xin Xu · 13 November 2025

  In offline reinforcement learning, value overestimation caused by out-of-distribution (OOD) actions significantly limits policy performance. Recently, diffusion models have been leveraged for their strong distribution-matching capabilities, enforcing conservatism through behavior policy constraints.…

• Latent Planning via Embedding Arithmetic: A Contrastive Approach to Strategic Reasoning

  Andrew Hamara, Greg Hamerly, Pablo Rivas, Andrew C. Freeman · 13 November 2025

  Planning in high-dimensional decision spaces is increasingly being studied through the lens of learned representations. Rather than training policies or value heads, we investigate whether planning can be carried out directly in an evaluation-aligned embedding space. We introduce SOLIS, which learns…

• Probabilistic Safety Guarantee for Stochastic Control Systems Using Average Reward MDPs

  Saber Omidi, Marek Petrik, Se Young Yoon, Momotaz Begum · 12 November 2025

  Safety in stochastic control systems, which are subject to random noise with a known probability distribution, aims to compute policies that satisfy predefined operational constraints with high confidence throughout the uncertain evolution of the state variables. The unpredictable evolution of state…

• Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learning

  Jiayu Chen, Le Xu, Wentse Chen, Jeff Schneider · 12 November 2025

  Offline RL is a powerful approach for data-driven decision-making and control. Compared to model-free methods, offline model-based RL (MBRL) explicitly learns world models from a static dataset and uses them as surrogate simulators, improving the data efficiency and enabling the learned policy to po…

• Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learning

  Jiayu Chen, Le Xu, Aravind Venugopal, Jeff Schneider · 12 November 2025

  Offline reinforcement learning (RL) offers a powerful paradigm for data-driven control. Compared to model-free approaches, offline model-based RL (MBRL) explicitly learns a world model from a static dataset and uses it as a surrogate simulator, improving data efficiency and enabling potential genera…

• ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learning

  Debamita Ghosh, George K. Atia, Yue Wang · 12 November 2025

  We investigate reinforcement learning (RL) in the presence of distributional mismatch between training and deployment, where policies trained in simulators often underperform in practice due to mismatches between training and deployment conditions, and thereby reliable guarantees on real-world perfo…

• Policy Transfer for Continuous-Time Reinforcement Learning: A (Rough) Differential Equation Approach

  Xin Guo, Zijiu Lyu · 12 November 2025

  This paper studies policy transfer, one of the well-known transfer learning techniques adopted in large language models, for two classes of continuous-time reinforcement learning problems. In the first class of continuous-time linear-quadratic systems with Shannon's entropy regularization (a.k.a. LQ…

• Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Options

  Joongkyu Lee, Seouh-won Yi, Min-hwan Oh · 12 November 2025

  We study online preference-based reinforcement learning (PbRL) with the goal of improving sample efficiency. While a growing body of theoretical work has emerged-motivated by PbRL's recent empirical success, particularly in aligning large language models (LLMs)-most existing studies focus only on pa…

• Uncertainty-driven Adaptive Exploration

  Leonidas Bakopoulos, Georgios Chalkiadakis · 12 November 2025

  Adaptive exploration methods propose ways to learn complex policies via alternating between exploration and exploitation. An important question for such methods is to determine the appropriate moment to switch between exploration and exploitation and vice versa. This is critical in domains that requ…