Reinforcement Learning in Robotics

Latest papers

• The Past Is Prologue: A Plug-in Controller for Selective Updates in Sequentially Evolving LLM Memory

  Zihan Chen, Songwei Dong, Chengshuai Shi, Peng Wang, Song Wang, Cong Shen, Jundong Li · 1 July 2026

  Sequentially evolving LLM memory enables agents to reuse past experience, but existing systems usually deploy each locally generated memory update without checking whether it improves future behavior. As a result, updates that help the current task may overwrite useful knowledge, introduce over-spec…

• Predictable GRPO: A Closed-Form Model of Training Dynamics

  Rajat Ghosh, Datta Nimmaturi, Aryan Singhal, Vaishnavi Bhargava, Henry Wong, Johnu George, Debojyoti Dutta · 1 July 2026

  Group Relative Policy Optimization (GRPO) has become a standard tool for improving the reasoning ability of large language models, yet its training dynamics are still described empirically: reward trajectories are fit with low-parameter functional forms whose constants carry no mechanistic meaning, …

• The Speedup Paradox: Rethinking Inference Speed-Quality Trade-off in Embodied Tasks

  Yujin Wang, Junli Chen, Yixuan Li, Shunan Dong, Huazhong Yang, Yongpan Liu, Hongyang Jia · 1 July 2026

  Embodied foundation models have recently been widely used to improve robot generalization and task success rates. Previous works apply lossy efficient-inference techniques such as quantization, pruning, and asynchronous inference, accepting small action quality degradation in exchange for lower per-…

• Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding

  Zhenghao Zhang, Yuanxiang Wang, Zhenyu Guan, Yujia Yang, Bingkang Shi, Tianyu Zong, Hongzhu Yi, Guoqing Chao, Xingchen Chen, Tiankun Yang, Chenxi Bao, Tao Yu, Jingjing Zhou, Jungang Xu · 1 July 2026

  Learning visual world models for planning requires compact latent dynamics that remain sensitive to actions, yet reconstruction-free joint-embedding objectives can collapse to action-insensitive representations. We propose Delta-JEPA, an end-to-end reconstruction-free world model that augments laten…

• QVal: Cheaply Evaluating Dense Supervision Signals for Long-Horizon LLM Agents

  Sergio Hern\'andez-Guti\'errez, Matteo Merler, Ilze Amanda Auzina, Joschka Str\"uber, Ameya Prabhu, Matthias Bethge · 1 July 2026

  LLM agents increasingly act over long horizons, where a single trajectory can contain hundreds or thousands of actions. In these settings, outcome-only rewards provide too sparse guidance, failing to inform the model about the goodness of intermediate actions. Dense supervision methods aim to solve …

• When Regulation Has Memory: Hysteresis and Control Burden in Artificial Agency

  Veronique Ziegler · 1 July 2026

  Adaptive agents are usually judged by what they do, but an agent can appear stable while the internal effort required to keep it stable is increasing. This hidden regulatory burden matters for artificial agents operating under noise, delay, or changing demands: two systems may reach similar internal…

• Warp RL: Reshaping Base Policy Distributions for Dynamics Adaptation

  Ethan Hirschowitz, Fabio Ramos · 1 July 2026

  Residual reinforcement learning adapts a pretrained robot policy by learning an additive correction to its actions. While effective when adaptation amounts to shifting the base policy's action distribution, additive corrections cannot change the distribution's shape, scale, or state-dependent geomet…

• TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning

  Yuanda Xu, Zhengze Zhou, Hejian Sang, Xiaomin Li, Jiaxin Zhang, Xinchen Du, Zhipeng Wang, Alborz Geramifard · 1 July 2026

  Agentic reinforcement learning requires assigning credit to environment-facing actions such as searches, clicks, edits, navigation commands, and object interactions. Standard GRPO uses the final verifier outcome as a uniform advantage over all action tokens. This outcome signal is useful but structu…

• Safe Online Learning via Smooth Safety-Structured Policy Composition

  Hongpeng Cao, Liqun Zhao, Yuliang Gu, Naira Hovakimyan, Lui Sha, Marco Caccamo · 1 July 2026

  Safe online reinforcement learning requires policies to respect safety constraints while maintaining smooth optimization dynamics. Existing approaches typically rely on either strict safety enforcement via action interventions, which introduce discontinuities in system interaction and learning, or s…

• Distributionally Robust Reinforcement Learning with Human Feedback

  Debmalya Mandal, Paulius Sasnauskas, Goran Radanovic · 30 June 2026

  Reinforcement learning from human feedback (RLHF) has evolved to be one of the main methods for fine-tuning large language models (LLMs). However, existing RLHF methods are non-robust, and their performance deteriorates if the downstream task differs significantly from the preference dataset used in…

• Dual-Flow Reinforcement Learning with State-Aware Exploration

  Qijun Li, Zheng Fu, Qi Song, Yifei He, Weitao Zhou, Kun Jiang, Diange Yang · 30 June 2026

  In complex continuous-control reinforcement learning tasks, multimodal optimal actions often coincide with uncertain, multimodal return distributions, making reliable value estimation and multimodal exploration challenging. Existing value estimation methods using unimodal Gaussians restrict expressi…

• Accelerating Q-learning through Efficient Value-Sharing across Actions

  Prabhat Nagarajan, Brett Daley, Martha White, Marlos C. Machado · 30 June 2026

  Action-values are foundational to many control algorithms such as Q-learning. Therefore learning action-values efficiently is central to reinforcement learning (RL). However, learning them can be slow, requiring many updates to move values from their initialization, typically near zero, to their tru…

• FlowAWR: Online Adaptive Flow Reinforcement via Advantage-Weighted Rectification

  Zheming Fu, Ruizhe He, Wei Shang, Xiaoxiao Ma, Lei Wang, Chang Liu, Siming Fu · 30 June 2026

  Aligning generative flow models on continuous spaces via online reinforcement learning is constrained by intractable trajectory likelihoods. Existing density-approximated policy gradient methods rely on stochastic SDE samplers to construct tractable transition kernels, which introduce training-infer…

• TERC: A Transfer Entropy Redundancy Criterion for State Variable Selection in Reinforcement Learning

  Charles Westphal, Stephen Hailes, Mirco Musolesi · 30 June 2026

  Identifying the most suitable variables to represent the state is a fundamental challenge in Reinforcement Learning (RL). These variables must efficiently capture the information necessary for making optimal decisions. In order to address this problem, in this paper, we introduce the Transfer Entrop…

• Robust Strategic Classification under Decision-Dependent Cost Uncertainty

  Sura Alhanouti, G\"uzin Bayraksan, Parinaz Naghizadeh · 30 June 2026

  Humans facing algorithmic decision systems have been found to ``game'' them by altering their input data (at a cost to them) in order to favorably change the algorithmic outcomes they receive (at a cost to the algorithm). The growing literature on strategic classification seeks to develop robust mac…

• Agile Reinforcement Learning through Separable Neural Architecture and Applications

  Rajib Mostakim, Reza T. Batley, Sourav Saha · 30 June 2026

  Deep reinforcement learning (RL) is increasingly deployed in resource-constrained environments, yet go-to function approximators - multilayer perceptrons (MLPs) - are often parameter-inefficient due to an imperfect inductive bias for the smooth structure of many value functions. This mismatch can al…

• On the Policy Gradient Foundations of Group Relative Policy Optimization: Credit Assignment, Gradient Sparsity, and Rank Collapse

  Amritansh Mishra, Supriyo Chakraborty, Berkcan Kapusuzoglu · 30 June 2026

  Group Relative Policy Optimization (GRPO) eliminates the learned critic in PPO by using the mean reward of grouped rollouts as a baseline. We provide a rigorous derivation of GRPO from first principles of the policy gradient theorem, revealing a fundamental credit assignment failure: under output-on…

• ProSpec RL: Plan Ahead, then Execute

  Liangliang Liu, Yi Guan, BoRan Wang, Rujia Shen, Yi Lin, Chaoran Kong, Lian Yan, Jingchi Jiang · 30 June 2026

  Imagining potential outcomes of actions before execution helps agents make more informed decisions, a prospective thinking ability fundamental to human cognition. However, mainstream model-free Reinforcement Learning (RL) methods lack the ability to proactively envision future scenarios, plan, and g…

• On the Emergence of Implicit Curriculum in RLVR Learning Dynamics

  Yu Huang, Zixin Wen, Yuejie Chi, Yuting Wei, Aarti Singh, Yingbin Liang, Yuxin Chen · 30 June 2026

  Reinforcement learning with verifiable rewards (RLVR) has been a main driver of recent breakthroughs in large reasoning models. Yet it remains a mystery how rewards based solely on final outcomes can help overcome the long-horizon barrier to extended reasoning. To understand this, we develop a theor…

• Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy

  Matthew Vandergrift, Esraa Elelimy, Martha White · 30 June 2026

  One goal in reinforcement learning (RL) research is to understand general-purpose sequential decision-making, using benchmark simulators as a proxy for learning in deployment settings. When running experiments, however, the goal of achieving high performance in the simulator can mutate into focusing…

• Complementary RL: Towards Efficient Experience-Driven Agent Learning

  Dilxat Muhtar, Jiashun Liu, Wei Gao, Weixun Wang, Shaopan Xiong, Ju Huang, Siran Yang, Wenbo Su, Jiamang Wang, Ling Pan, Bo Zheng · 30 June 2026

  Reinforcement Learning (RL) has emerged as a powerful paradigm for training LLM-based agents, yet remains limited by low sample efficiency, stemming not only from sparse outcome feedback but also from the agent's inability to leverage prior experience across episodes. While augmenting agents with hi…

• Modification-Considering Value Learning for Reward Hacking Mitigation in RL

  Evgenii Opryshko, Umangi Jain, Igor Gilitschenski · 30 June 2026

  Reinforcement learning agents can exploit misspecified reward signals to achieve high apparent returns while failing on the intended objective, a failure mode known as reward hacking. Existing practical defenses typically constrain policy updates to stay near a known safe reference, creating a tensi…

• Hierarchical Decision Making with Structured Policies: A Principled Design via Inverse Optimization

  Yuexuan Wang, Jingyuan Zhou, Kaidi Yang · 30 June 2026

  Hierarchical decision-making frameworks are pivotal for addressing complex control tasks, enabling agents to decompose intricate problems into manageable subgoals. Despite their promise, existing hierarchical policies face critical limitations: (i) reinforcement learning (RL)-based methods struggle …

• PS-PPO: Prefix-Sampling PPO for Critic-Free RLHF

  Doo Hwan Hwang, Kee-Eung Kim · 30 June 2026

  Reinforcement Learning from Human Feedback (RLHF) for Large Language Models increasingly relies on critic-free methods as a practical alternative to actor--critic training. Despite their simplicity, existing critic-free approaches propagate a trajectory-level learning signal uniformly across all tok…

• When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizon

  Huaqing Zhang, Jingchu Gai, Juno Kim, Bingbin Liu, Andrej Risteski · 30 June 2026

  Online imitation learning (IL), particularly on-policy distillation, has emerged as a strong LLM post-training approach, often outperforming offline supervised fine-tuning (SFT). Yet a principled understanding of when and why online interaction helps remains unclear. In this work, we challenge the v…