Reinforcement Learning in Robotics

Neueste Paper

• Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL

  Ruiming Liang, Yi Zhong, Yizhen Yuan, Yinan Zheng, Tianyi Tan, Tianyue Wang, Haiyun Guo, Jinqiao Wang, Xianyuan Zhan · 3. August 2026

  Modern large language models (LLMs) are expected not just to answer correctly, but to adapt their behavior to different human values and use cases. As a result, multi-reward reinforcement learning (RL) has become an increasingly important problem for LLMs, where each reward captures a different aspe…

• When Does On-Policy Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning

  Luca Viano, Antoine Moulin, Audrey Huang, Volkan Cevher, Philip Amortila, Dylan J. Foster · 3. August 2026

  Imitation learning (IL)---training an agent to replicate expert behavior from demonstrations---underpins applications from robotics to language model training. Standard approaches such as Behavior Cloning (BC) are known to suffer from compounding errors and performance plateaus, particularly when th…

• LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback

  Manith Adikari, Bei Peng, Samuele Vinanzi, Angelo Cangelosi · 3. August 2026

  Reinforcement Learning (RL) systems are typically trained using a single, well-specified scalar reward function. However, real-world decision-making tasks often involve multiple, competing objectives, such as performance versus efficiency, where ground-truth reward functions are difficult to specify…

• DASH-OPD: Discrepancy-Aware Switching with Hysteresis for On-Policy Distillation

  Yuchen Xia, Qianguo Sun, Chao Song, Junlong Wu, Yiyan Qi, Yunjian Xu · 3. August 2026

  On-policy distillation (OPD) trains student models on their own rollouts to reduce exposure bias. However, in multi-turn agent scenarios, early student errors can lead a trajectory away from the teacher's familiar domain. Existing curriculum learning methods regulate how much teacher support is used…

• Hypergradient-based Bilevel Reinforcement Learning with Improved Sample Complexity

  Naman Saxena, Mudit Gaur, Vaneet Aggarwal · 3. August 2026

  Bilevel reinforcement learning (RL) is an important framework within the literature of RL that can be used to formalize various categories of problems, such as meta-learning, hierarchical task decomposition, and reinforcement learning from human feedback (RL-HF). Most of the bilevel RL algorithms ar…

• SAF-OPD: Stable Advantage Fusion for On-Policy Distillation

  Yifan Ding, Xincheng Wei, Yoshua Y. Li, Ziheng Li, Yuquan Lu, Siyu Zhang, Dongsheng Ma, Rongxiang Weng, Xunliang Cai, Yun Chen · 3. August 2026

  Reinforcement learning with verifiable rewards (RLVR) broadcasts a single response-level reward to every token, while on-policy distillation (OPD) scores each token against a stronger teacher for a dense advantage but caps performance at teacher quality and discourages exploration beyond it. Their c…

• PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization

  Wonjoong Kim, Yeonjun In, Sangwu Park, Dongha Lee, Chanyoung Park · 3. August 2026

  A significant hurdle for current LLMs is the execution of complex, multi-stage tasks. Group Relative Policy Optimization (GRPO) has been emerging as a leading choice, but its reliance on sparse outcome rewards severely limits credit assignment across intermediate steps. Existing remedies such as run…

• Adaptive Policy Backbone via Shared Network

  Bumgeun Park, Donghwan Lee · 3. August 2026

  Reinforcement learning (RL) has achieved impressive results across domains, yet learning an optimal policy typically requires extensive interaction data, limiting practical deployment. A common remedy is to leverage priors, such as pre-collected datasets or reference policies, but their utility degr…

• Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation

  Qian Tan, Huaifei Liang, Xuanyu Zhu, Lei Jiang, Yuqiang Li · 3. August 2026

  On-policy distillation (OPD) provides dense teacher supervision along student-generated trajectories, but its online rollout process incurs substantial computational cost, particularly when a few long responses delay batch completion. Existing acceleration methods typically control rollout length us…

• Sample Efficient Hierarchical Reinforcement Learning via Best Policy Identification

  Anders Jonsson, Emilie Kaufmann, Gianmarco Tedeschi, Lorenzo Steccanella · 3. August 2026

  We present HBPI-UCRL, a model-based algorithm for hierarchical reinforcement learning (HRL) that learns high-level and low-level policies in parallel. HBPI-UCRL exploits the fact that a high-level transition corresponds to a multi-step transition at the low level. We introduce two conditions on the …

• Explore Beyond the Boundary Using Entropic Information

  Bumgeun Park, Donghwan Lee · 3. August 2026

  In reinforcement learning, exploration with sparse and delayed rewards presents a significant challenge due to the limited feedback available for guiding the learning process. Addressing this issue requires extensive exploration in the state space to discover valuable reward signals. In this paper, …

• Gated Q-learning: Add Off-Policy Bias to Taste

  Brett Daley · 3. August 2026

  Multistep credit assignment is critical for sample-efficient reinforcement learning, yet managing off-policy bias in Q-learning remains a fundamental challenge. For 30 years, practitioners have been limited to a binary choice: eliminate the bias at the cost of severely truncated eligibility traces (…

• Beyond Black-Box Advice: Learning-Augmented Algorithms for MDPs with Q-Value Predictions

  Tongxin Li, Yiheng Lin, Shaolei Ren, Adam Wierman · 3. August 2026

  We study the tradeoff between consistency and robustness in the context of a single-trajectory time-varying Markov Decision Process (MDP) with untrusted machine-learned advice. Our work departs from the typical approach of treating advice as coming from black-box sources by instead considering a set…

• Temporally Centered SIGReg Improves Multi-Task LeWorldModel Learning: From Analysis to Method

  Chang Liu, Fei Suo, Yanzhou Jin, Yusuke Iwasawa, Yutaka Matsuo, Yaonan Zhu · 30. Juli 2026

  Recent work on LeWorldModel (LeWM) has shown that the Sketched Isotropic Gaussian Regularizer (SIGReg) enables stable end-to-end world-model learning from pixels by regularizing the latent marginal distribution toward an isotropic Gaussian, thereby preventing representation collapse. While effective…

• Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback

  Yunpeng Chu · 30. Juli 2026

  Reinforcement Learning from Human Feedback (RLHF) is the standard approach for aligning large language models with human preferences, but its quality is limited by static, task-agnostic reward models. This mismatch leads to sparse learning signals and suboptimal alignment. We introduce MeRLa (Meta-L…

• Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?

  Perry Dong, Ron Polonsky, Dorsa Sadigh, Chelsea Fin · 30. Juli 2026

  Pre-training followed by fine-tuning has become the dominant recipe for learning performant policies, and in value-based reinforcement learning (RL) this raises a natural question: given a pretrained policy, should the Q-function be pretrained on offline data too? Conventional wisdom suggests it sho…

• Q-Steer: Action-Value Guidance for Molecular Policy Optimization

  Xinyu Wang, Jinbo Bi, Minghu Song · 30. Juli 2026

  Oracle-limited molecular optimization gives reward only after a complete molecule is generated, while each rollout requires many local next-token decisions. This delayed-feedback interface makes molecular policy optimization myopic: an optimizer can learn that a molecule was good without knowing whi…

• Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR

  Pixel Nomand, Elena Voss, Marcus Hale, Sofia Reyes · 30. Juli 2026

  Reinforcement learning with verifiable rewards (RLVR) is bottlenecked by rollout generation, yet many sampled prompts produce saturated groups (all responses correct or all incorrect) whose zero reward variance yields no policy-gradient signal. Existing remedies either oversample a larger candidate …

• Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning

  Gong Gao, Xiao Lai, Ziqi Xie, Guojie Chen, Xianhui Liu, Weidong Zhao · 30. Juli 2026

  Deep off-policy reinforcement learning algorithms for continuous control typically rely on neural value function approximation to guide policy improvement. However, temporal-difference (TD) learning introduces noisy targets, resulting in non-stationary optimization, while greedy policy updates ampli…

• SCOUT: Per-Context Reset Curricula for Sparse-Reward Reinforcement Learning

  Siddharth Aphale, Ayushman Singh · 30. Juli 2026

  Sparse-reward reinforcement learning often fails because rollouts from the unassisted evaluation start rarely reach later task stages. Reset curricula address this by starting some training rollouts from easier intermediate states, called scaffolds. Such a curriculum faces two decisions: scaffold ac…

• Minimal Markovization via Stable Quotients in Holonomy-Cover Decision Processes

  Zuyuan Zhang, Yongshan Chen, Mahdi Imani, Tian Lan · 30. Juli 2026

  An agent acting under partial observability must retain a recursively updateable statistic of history that restores the Markov property, but the smallest such statistic is generally unknown. We characterize this minimal Markov sufficient statistic for holonomy-cover decision processes, a structured …

• CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization

  Bo-Wen Zhang, Junwei He, Wen Wang, Song-Lin Lv, Wentao Ma, Rongyi Lin, Shuhan Zhong, Lan-Zhe Guo · 29. Juli 2026

  Rubric-based reinforcement learning enriches language model training by evaluating model outputs against explicit criteria. Yet in GRPO-style pipelines, these structured judgments are reduced to a scalar response-level reward and converted into a response-level advantage, which is broadcast uniforml…

• Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning

  Luc McCutcheon, Evangelos Chatzaroulas, Saber Fallah · 29. Juli 2026

  Neural networks are hindered by accumulating dormant neurons and loss of expressivity throughout training, particularly in non-stationary data settings, such as continual supervised and reinforcement learning. Recently, neuron resets have been used to maintain gradient flow and restore plasticity. H…

• Reinformed Dreamer: An Asymmetric World Model Efficiently Trained through Latent Guidance

  Gaspard Lambrechts, Adrien Bolland, Daniel Ebi, Damien Ernst · 29. Juli 2026

  Much like humans benefit from guidance while learning, reinforcement learning algorithms may benefit from additional supervision beyond rewards. Leveraging additional information during training to learn better representations and behaviors has been the focus of asymmetric reinforcement learning. Th…

• Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions

  Zeyu Bian, Ying Zhou, Yifan Cui · 29. Juli 2026

  Standard offline reinforcement learning (RL) algorithms typically assume that the actions in the dataset are observed without error. However, in many real-world applications, the true actions are unobserved and only noisy proxies are available, causing existing RL methods to yield biased and potenti…