Reinforcement Learning in Robotics

Neueste Paper

• AdaGamma: State-Dependent Discounting for Temporal Adaptation in Reinforcement Learning

  Yaomin Wang, Jianting Pan, Ran Tian, Xiaoyang Li, Yu Zhang, Hengle Qin, Tianshu YU · 8. Mai 2026

  The discount factor in reinforcement learning controls both the effective planning horizon and the strength of bootstrapping, yet most deep RL methods use a single fixed value across all states. While state-dependent discounting is conceptually appealing, naive deep actor--critic implementations can…

• Operator-Guided Invariance Learning for Continuous Reinforcement Learning

  Zuyuan Zhang, Fei Xu Yu, Tian Lan · 8. Mai 2026

  Reinforcement learning (RL) with continuous time and state/action spaces is often data-intensive and brittle under nuisance variability and shift, motivating methods that exploit value-preserving structures to stabilize and improve learning. Most existing approaches focus on special cases, such as p…

• SOPE: Stabilizing Off-Policy Evaluation for Online RL with Prior Data

  Carlo Romeo, Girolamo Macaluso, Alessandro Sestini, Andrew D. Bagdanov · 8. Mai 2026

  Incorporating prior data into online reinforcement learning accelerates training but typically forces a difficult trade-off between high computational costs and long, multi-stage training pipelines. While fixed-length stabilization phases are significantly more computationally efficient than static …

• Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization

  Alireza Modirshanechi, Benjamin Eysenbach, Peter Dayan, Eric Schulz · 8. Mai 2026

  Unsupervised pretraining has driven empirical advances in goal-conditioned reinforcement learning (GCRL), but its theoretical foundations remain poorly understood. In particular, an influential class of methods, mutual information skill learning (MISL), discovers behaviorally diverse skills that can…

• A Unified Pair-GRPO Family: From Implicit to Explicit Preference Constraints for Stable and General RL Alignment

  Hao Yu · 8. Mai 2026

  Large language model (LLM) alignment via reinforcement learning from human preferences (RLHF) suffers from unstable policy updates, ambiguous gradient directions, poor interpretability, and high gradient variance in mainstream pairwise preference learning paradigms. To systematically address these l…

• Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level

  Nan Jia, Haojin Yang, Xing Ma, Jiesong Lian, Shuailiang Zhang, Weipeng Zhang, Ke Zeng, Xunliang Cai, Zequn Sun · 8. Mai 2026

  On-policy distillation (OPD) trains a student on its own trajectories with token-level teacher feedback and often outperforms off-policy distillation and standard reinforcement learning. However, we find that its standard advantage weighted policy gradient suffers from three structural weaknesses, i…

• Neural Co-state Policies: Structuring Hidden States in Recurrent Reinforcement Learning

  David Leeftink, Max Hinne, Marcel van Gerven · 8. Mai 2026

  A key capability of intelligent agents is operating under partial observability: reasoning and acting effectively despite missing or incomplete state observations. While recurrent (memory-based) policies learned via reinforcement learning address this by encoding history into latent state representa…

• Reward Shaping and Action Masking for Compositional Tasks using Behavior Trees and LLMs

  Nicholas Potteiger, Ankita Samaddar, Taylor T. Johnson, Xenofon Koutsoukos · 8. Mai 2026

  Decomposing complex tasks into a sequence of simpler subtasks can improve learning efficiency for an autonomous agent. Reinforcement learning (RL) can be used to optimize agent policies to complete subtasks, but requires well-defined subtask rewards and benefits from action masking. Recent work uses…

• Causal Reinforcement Learning for Complex Card Games: A Magic The Gathering Benchmark

  Cristiano da Costa Cunha, Ajmal Mian, Tim French, Wei Liu · 8. Mai 2026

  Causal reinforcement learning (RL) lacks benchmarks for complex systems that combine sequential decision making, hidden information, large masked action spaces, and explicit causal structure. We introduce MTG-Causal-RL, a Gymnasium benchmark built on Magic: The Gathering with a 3,077-dimensional par…

• Beyond Autoregressive RTG: Conditioning via Injection Outside Sequential Modeling in Decision Transformer

  Yongyi Wang, Hanyu Liu, Lingfeng Li, Bozhou Chen, Ang Li, Qirui Zheng, Xionghui Yang, Chucai Wang, Wenxin Li · 8. Mai 2026

  Decision Transformer (DT) formulates offline reinforcement learning as autoregressive sequence modeling, achieving promising results by predicting actions from a sequence of Return-to-Go (RTG), state, and action tokens. However, RTG is a scalar that summarizes future rewards, containing far less inf…

• Entropy-Regularized Adjoint Matching for Offline RL

  Abdelghani Ghanem, Mounir Ghogho · 8. Mai 2026

  Integrating expressive generative policies, such as flow-matching models, into offline reinforcement learning (RL) allows agents to capture complex, multi-modal behaviors. While Q-learning with Adjoint Matching (QAM) stabilizes policy optimization via the continuous adjoint method, it remains inhere…

• Q-MMR: Off-Policy Evaluation via Recursive Reweighting and Moment Matching

  Xiang Li, Nan Jiang · 8. Mai 2026

  We present a novel theoretical framework, Q-MMR, for off-policy evaluation in finite-horizon MDPs. Q-MMR learns a set of scalar weights, one for each data point, such that the reweighted rewards approximate the expected return under the target policy. The weights are learned inductively in a top-dow…

• Hitting Time Isomorphism for Multi-Stage Planning with Foundation Policies

  Magnus Victor Boock, Abdullah Akg\"ul, Mustafa Mert \c{C}elikok, Melih Kandemir · 8. Mai 2026

  We present a new operator-theoretic representation learning framework for offline reinforcement learning that recovers the directed temporal geometry of a controlled Markov process from hitting time observations. While prior art often produces symmetric distances or fails to satisfy the triangle ine…

• Adaptive Q-Chunking for Offline-to-Online Reinforcement Learning

  Nandiraju Gireesh, Yuanliang Ju, He Wang · 8. Mai 2026

  Offline-to-online reinforcement learning with action chunking eliminates multi-step off-policy bias and enables temporally coherent exploration, but all existing methods use a fixed chunk size across every state. This is suboptimal: near contact events the agent needs short chunks for reactive contr…

• Leveraging Analytic Gradients in Provably Safe Reinforcement Learning

  Tim Walter, Hannah Markgraf, Jonathan K\"ulz, Matthias Althoff · 8. Mai 2026

  The deployment of autonomous robots in safety-critical applications requires safety guarantees. Provably safe reinforcement learning is an active field of research that aims to provide such guarantees using safeguards. These safeguards should be integrated during training to reduce the sim-to-real g…

• Maximizing Rollout Informativeness under a Fixed Budget: A Submodular View of Tree Search for Tool-Use Agentic Reinforcement Learning

  Yuelin Hu, Zhenbo Yu, Zhengxue Cheng, Wei Liu, Li Song · 8. Mai 2026

  We formalize Rollout Informativeness under a Fixed Budget (RIFB) as the expected non-vanishing policy-gradient mass that a tool-use rollout set injects into Group Relative Policy Optimization (GRPO). We prove that any budget-agnostic independent sampler suffers a collapse rate bounded away from zero…

• Approximate Next Policy Sampling: Replacing Conservative Target Policy Updates in Deep RL

  Dillon Sandhu, Ronald Parr · 8. Mai 2026

  We revisit a classic "chicken-and-egg" problem in reinforcement learning: to safely improve a policy, the value function must be accurate on the state-visitation distribution of the updated policy. That distribution over states is unknown and cannot be sampled for the purposes of training the value …

• RVPO: Risk-Sensitive Alignment via Variance Regularization

  Ivan Montero, Tomasz Jurczyk, Bhuwan Dhingra · 8. Mai 2026

  Current critic-less RLHF methods aggregate multi-objective rewards via an arithmetic mean, leaving them vulnerable to constraint neglect: high-magnitude success in one objective can numerically offset critical failures in others (e.g., safety or formatting), masking low-performing "bottleneck" rewar…

• Near-Policy: Accelerating On-Policy Distillation via Asynchronous Generation and Selective Packing

  Miao Rang, Zhenni Bi, Hang Zhou, Kai Han, Xuechun Wang, An Xiao, Xinghao Chen, Yunhe Wang, Hanting Chen · 8. Mai 2026

  Standard knowledge distillation for autoregressive models often suffers from distribution mismatch. While on-policy methods mitigate this by leveraging student-generated outputs, they rely on computationally expensive Reinforcement Learning (RL) frameworks. To improve efficiency, we propose Near-Pol…

• On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR

  Hao Ye, Jisheng Dang, Junfeng Fang, Bimei Wang, Yizhou Zhang, Ning Lv, Wencan Zhang, Hong Peng, Bin Hu, Tat-Seng Chua · 8. Mai 2026

  Recent extensive research has demonstrated that the enhanced reasoning capabilities acquired by models through Reinforcement Learning with Verifiable Rewards (RLVR) are primarily concentrated within the rank-1 components. Predicated on this observation, we employed Periodic Rank-1 Substitution and i…

• Prediction-Based Markov Violation Scores for Detecting Non-Markovian Observations in Reinforcement Learning

  Naveen Mysore · 8. Mai 2026

  Reinforcement learning algorithms assume that observations satisfy the Markov property, yet real-world sensors frequently violate this assumption through correlated noise, latency, or partial observability. Standard performance metrics conflate Markov breakdowns with other sources of suboptimality, …

• A Measure-Theoretic Finite-Sample Theory for Adaptive-Data Fitted Q-Iteration

  Manuel Haussmann, Mustafa Mert \c{C}elikok, Melih Kandemir · 8. Mai 2026

  While reinforcement learning (RL) promises to revolutionize the control of complex nonlinear robotic systems, a profound gap persists between the heuristic success of model-free off-policy deep RL and the underlying theory, which remains largely confined to tabular or linearizable settings. We ident…

• Selective Rollout: Mid-Trajectory Termination for Multi-Sample Agent RL

  Zhiyuan Zhai, Xin Wang · 8. Mai 2026

  Group-relative RL training (GRPO) samples a small group of parallel rollouts for every training prompt and uses their within-group reward spread to compute per-trajectory advantages. In agentic environments each rollout is a long multi-turn dialogue with one LLM call per step, so this multi-sample m…

• Independent Learning of Nash Equilibria in Partially Observable Markov Potential Games with Decoupled Dynamics

  Philip Jordan, Maryam Kamgarpour · 8. Mai 2026

  We study Nash equilibrium learning in partially observable Markov games (POMGs), a multi-agent reinforcement learning framework in which agents cannot fully observe the underlying state. Prior work in this setting relies on centralization or information sharing, and suffers from sample and computati…

• Recursive Agent Optimization

  Apurva Gandhi, Satyaki Chakraborty, Xiangjun Wang, Aviral Kumar, Graham Neubig · 8. Mai 2026

  We introduce Recursive Agent Optimization (RAO), a reinforcement learning approach for training recursive agents: agents that can spawn and delegate sub-tasks to new instantiations of themselves recursively. Recursive agents implement an inference-time scaling algorithm that naturally allows agents …