Reinforcement Learning in Robotics

Últimos artículos

• HPO: Hysteretic Policy Optimization for Stable and Efficient Training under Sparse-Reward Regime

  Mohamed Sana, Nicola Piovesan, Antonio De Domenico, Fadhel Ayed, Haozhe Zhang · 29 de mayo de 2026

  We investigate a narrow but common failure mode of GRPO-style reinforcement learning in the context of sparse verifiable rewards: early updates contain more responses with negative advantages than those with positive advantages, while response-level length normalization ties the magnitude of the upd…

• Kernel-based potential mean-field games with unbiased random Fourier $U$-statistics

  Yumiharu Nakano · 29 de mayo de 2026

  We study the subclass of potential mean-field games in which the running interaction cost and the terminal target cost are both expressed through reproducing-kernel maximum mean discrepancy (MMD) penalties, and develop a computational framework that exploits this kernel structure. Both costs are est…

• Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents

  Ziyan Liu, Zhezheng Hao, Yeqiu Chen, Hong Wang, Jingren Hou, Ruiyi Ding, Yongkang Yang, Wence Ji, Wei Xia, Feng Liu · 29 de mayo de 2026

  Memory-augmented LLM agents tackle complex long-horizon tasks by recursively summarizing interaction trajectories into compact memory. However, existing approaches typically train these memory policies using outcome-based reinforcement learning, failing to localize where intermediate memory quality …

• Discovering Cooperative Pipelines: Autoresearch for Sequential Social Dilemmas

  V\'ictor Gallego · 29 de mayo de 2026

  We study two-level autoresearch for cooperation: an outer-loop AI agent autonomously redesigns the inner-loop pipeline of an LLM policy-synthesis system for multi-agent Sequential Social Dilemmas (SSDs). A researcher agent $\mathcal{R}$ (run as a coding agent) reads the inner-loop source code, edits…

• Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learning

  Christoph Dann, Yishay Mansour, Mehryar Mohri · 29 de mayo de 2026

  Model-based reinforcement learning (MBRL) agents typically learn world models by minimizing predictive loss. However, powerful RL optimizers inevitably exploit minor model inaccuracies, leading to simulator exploitation and a reality gap where policies succeed in simulation but fail in the real worl…

• Mean-Field Diffuser: Scaling Offline MARL to Thousands of Agents

  Wenhao Li, Xiangfeng Wang, Bo Jin · 29 de mayo de 2026

  Diffusion-based planning has achieved strong results in single-agent offline reinforcement learning, yet scaling to many-agent systems remains intractable due to the curse of dimensionality in the joint trajectory space. We introduce MF-Diffuser, a framework that lifts trajectory planning to the Was…

• Differentiable Belief-based Opponent Shaping

  Aarav G Sane, Karthik Sivachandran, Rohan Paleja · 29 de mayo de 2026

  Human coordination often relies on the ability to influence the beliefs of others through strategic action. In multi-agent reinforcement learning, opponent shaping attempts to replicate this influence, though existing methods typically operate within an opponent's parameter, policy, or value space. …

• BenchTrace: A Benchmark for Testing Reflection Ability and Controlled Evolution in LLM Agents

  Jiahao Huang, Fei Cheng, Junfeng Jiang, Zefan Yu, Akiko Aizawa · 29 de mayo de 2026

  Self-evolving agents improve over time by reflecting on past failures, but existing evaluation is limited in two ways: it measures only task scores, leaving reflection quality unknown, and it relies on agents' own episode runs, offering no mechanism to target specific failure patterns. We present \t…

• OISD: On-Policy Internal Self-Distillation of Language Models

  Xinyu Liu, Darryl Cherian Jacob, Yang Zhou, Jindong Wang, Pan He · 29 de mayo de 2026

  Recent reinforcement learning (RL) post-training approaches primarily optimize the final output policy using sparse outcome-level rewards, while largely overlooking predictive signals encoded in intermediate representations. In this paper, we introduce a new paradigm called on-policy internal self-d…

• ESPO: Early-Stopping Proximal Policy Optimization

  Zihang Li, Rui Zhou, Yingcheng Shi, Wenhan Yu, Zhewen Tan, Zixiang Liu, Zeming Li, Binhua Li, Yongbin Li, Tong Yang, Jieping Ye · 29 de mayo de 2026

  When a large language model under reinforcement learning commits a wrong reasoning step early in a trajectory, standard algorithms force it to keep generating until the maximum horizon, spending compute on tokens that never receive positive reward and polluting advantage estimates with post-failure …

• Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence

  Xinyu Liu, Kechen Jiao, Chunyang Xiao, Runsong Zhao, Junhao Ruan, Bei Li, Jiahao Liu, Qifan Wang, Xin Chen, Jingang Wang, Chenglong Wang, Tong Xiao, JingBo Zhu · 29 de mayo de 2026

  On-policy distillation (OPD) has become a promising paradigm for reasoning-oriented post-training of large language models (LLMs), especially when combined with reinforcement learning from verifiable rewards (RLVR). Existing OPD methods rely on reverse KL (RKL)-based teacher supervision over traject…

• FedQHD: Closed-Form Function-Space Federated Reinforcement Learning

  Yuchen Hou, Yongshan Chen, Zhuowen Zou, Calvin Yeung, Mohsen Imani, Tian Lan, Mahdi Imani · 29 de mayo de 2026

  Federated reinforcement learning enables decentralized agents to collaboratively improve policies or value estimates without exchanging raw trajectories. However, FedAvg-style parameter averaging is not function-space consistent: when clients use heterogeneous encoders or even identical nonlinear ne…

• When RL Suppresses Its Own Vocabulary: Recovering Reasoning Diversity in Puzzle-to-Math Transfer

  Mayug Maniparambil, Arjun Karuvally, Terrence Sejnowski, Fergal Reid · 29 de mayo de 2026

  Reinforcement learning using verifiable rewards (RLVR) improves LLM reasoning, but the conditions under which it transfers across domains -- and why it does so -- remain under-explored. We study cross-domain transfer in a 7B model whose SFT and RL post-training stages use only constraint-satisfactio…

• RL2ML: Finite-Rollout Surrogate Objectives from Reinforcement Learning to Maximum Likelihood

  Yifu Zheng · 29 de mayo de 2026

  Correctness-based Reinforcement Learning with Verifiable Rewards (RLVR) trains language models from binary feedback on sampled outputs, but the objective optimized in expectation and the stochastic update geometry induced by finite rollout groups are often conflated. This paper develops RL2ML, a fam…

• Return-to-Go Is More Than a Number: Q-Guided Alignment for Return-Conditioned Supervised Learning

  Yuxiao Yang, Weitong Zhang · 29 de mayo de 2026

  Conditioned Sequence Models (CSMs) learn policies by treating return-to-go (RTG) as a control signal. However, existing CSMs often treat the RTGs as simple numerical inputs rather than aligning them with the performance of their policies. In this paper, we propose Q-ALIGN DT, a framework that enforc…

• Behavior-Aware Auxiliary Corrections for Off-Policy Temporal-Difference Prediction

  Xingguo Chen, Zhiang He, Yuchen Shen, Shangdong Yang, Chao Li, Guang Yang, Wenhao Wang · 29 de mayo de 2026

  Temporal-difference learning with function approximation can be unstable under off-policy sampling. TDC stabilizes off-policy TD through an auxiliary covariance correction, and TDRC further regularizes this correction in a single-timescale recursion. This paper studies a behavior-aware replacement o…

• When LLM Reward Design Fails: Diagnostic-Driven Refinement for Sparse Structured RL

  Youting Wang, Yuan Tang, Bowen Liu, Xuan Liu, Dingyan Shang · 29 de mayo de 2026

  For sparse, structured reinforcement-learning tasks with semantic reward-function interfaces, LLM-generated reward shaping is better framed as debugging than one-shot generation. We study PPO-trained agents using MiniGrid as core evaluation and MuJoCo as boundary stress test. Our audit finds two dom…

• Bridging the Sim-to-Real Gap in Reinforcement Learning-Based Industrial Dispatching through Execution Semantics

  Jonathan Hoss, Noah Klarmann · 29 de mayo de 2026

  Event-driven scheduling policies are increasingly deployed in industrial environments, where decisions are made under asynchronous and partially observed system states. As a result, decision states are not temporally consistent, action admissibility is not explicitly defined, and the origin of execu…

• Self-Play Reinforcement Learning under Imperfect Information in Big 2

  Aalok Patwa · 29 de mayo de 2026

  Imperfect-information multiplayer games test whether agents can act under hidden information, sparse rewards, and non-stationary opponents. We study these challenges in Big 2, a four-player imperfect-information card game. We develop a self-play RL framework for Big 2 that enables controlled compari…

• Moment Matching Q-Learning

  Yiyan (Edgar), Liang, Sifei Liu, Weitong Zhang · 29 de mayo de 2026

  Score-based and flow-based generative models exhibit remarkable expressive capacity in capturing complex distributions, and have been extensively deployed in tasks ranging from image generation to reinforcement learning. Nevertheless, these models suffer from prolonged inference latency, which impos…

• Training Deliberative Monitors for Black-Box Scheming Detection

  Aditya Sinha, Akshat Naik, Victor Gillioz, Simon Storf, Kilian Merkelbach, Rich Barton-Cooper, Axel H{\o}jmark, Marius Hobbhahn · 29 de mayo de 2026

  As autonomous agents become more capable of performing real-world tasks, distinguishing scheming behavior from benign task pursuit may become a central AI control problem. Existing monitors often rely on chain-of-thought access or internal activations, or use prompted frontier models, all of which c…

• Hista and Numca: Estimate State Value Effectively for LLM Reinforcement Learning

  Zizhe Chen, Jiqian Dong, Yizhou Tian, Garry Yang, Yongqiang Chen, Zhitang Chen, James Cheng · 29 de mayo de 2026

  Reinforcement learning (RL) refines large language models (LLMs) by directly optimizing model behavior through reward signals. While accurate state value estimation is critical for stable training in classical RL, it remains an underexplored challenge in LLM post-training. In this work, we introduce…

• Offline Reinforcement Learning with Generative Trajectory Policies

  Xinsong Feng, Leshu Tang, Chenan Wang, Haipeng Chen · 29 de mayo de 2026

  Generative models have emerged as a powerful class of policies for offline reinforcement learning (RL) due to their ability to capture complex, multi-modal behaviors. However, existing methods face a stark trade-off: slow, iterative models like diffusion policies are computationally expensive, while…

• Behavior-Induced Mirror-Prox Temporal-Difference Learning for Faster Off-Policy Prediction

  Xingguo Chen, Yuchen Shen, Shangdong Yang, Chao Li, Guang Yang, Wenhao Wang · 29 de mayo de 2026

  Gradient temporal-difference methods provide stable off-policy prediction with linear function approximation, but their practical performance is strongly affected by the geometry induced by the auxiliary-variable metric. Existing Mirror-Prox TD methods typically use the feature covariance metric, wh…

• Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance

  Shutong Ding, Zejia Zhong, Zhongyi Wang, Ke Hu, Bikang Pan, Jingya Wang, Ye Shi · 29 de mayo de 2026

  Recent advances in reinforcement learning (RL) have achieved great successes by leveraging the multimodality and exploration capability of diffusion policies. Among these approaches, one representative branch focuses on the sampling-based policy optimization. This design enables better exploration c…