Parallel Computing and Optimization Techniques

Derniers papiers

• MEMTIER: Tiered Memory Architecture and Retrieval Bottleneck Analysis for Long-Running Autonomous AI Agents

  Bronislav Sidik, Lior Rokach · 22 mai 2026

  Long-running autonomous AI agents suffer from a well-documented memory coherence problem: tool-execution success rates degrade 14 percentage points over 72-hour operation windows due to four compounding failure modes in existing flat-file memory systems. We present MEMTIER, a tripartite memory archi…

• Block-Wise Differentiable Sinkhorn Attention: Tail-Refinement Gradients with a Gap-Aware Dustbin Bridge

  Dylan Forde · 21 mai 2026

  We study long-context balanced entropic optimal transport (OT) attention on TPU hardware through a stopped-base, fixed-depth tail-refinement surrogate. After a stopped $T$-step Sinkhorn solve, we unroll a short refinement tail and differentiate that surrogate exactly. For the reported $R=2$ TPU path…

• ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuning

  Yongkang Liu, Zijing Wang, Mengjie Zhao, Ercong Nie, Mingyang Wang, Qian Li, Feiliang Ren, Shi Feng, Daling Wang, Hinrich Sch\"utze · 21 mai 2026

  This work presents \textsc{ChunkFT}, a memory-efficient fine-tuning framework that reformulates full-parameter fine-tuning around a dynamically activated working set. \textsc{ChunkFT} enables gradient computation for arbitrary sub-tensors without modifying the network architecture, providing an algo…

• Understanding and Improving Communication Performance in Multi-node LLM Inference

  Prajwal Singhania, Siddharth Singh, Lannie Dalton Hough, Akarsh Srivastava, Harshitha Menon, Charles Fredrick Jekel, Abhinav Bhatele · 21 mai 2026

  As large language models (LLMs) continue to grow in size, distributed inference has become increasingly important. Model-parallel strategies must now efficiently scale not only across multiple GPUs but also across multiple nodes. In this work, we present a detailed performance study of multi-node di…

• Frontier: Towards Comprehensive and Accurate LLM Inference Simulation

  Yicheng Feng, Xin Tan, Yangtao Deng, Yimin Jiang, Yibo Zhu, Hong Xu · 21 mai 2026

  Modern LLM serving is no longer homogeneous or monolithic. Production systems now combine disaggregated execution, complex parallelism, runtime optimizations, and stateful workloads such as reasoning, agents, and RL rollouts. Simulation is attractive for exploring this growing design space, yet exis…

• Runtime-Certified Bounded-Error Quantized Attention

  Dean Calver · 21 mai 2026

  KV cache quantization reduces the memory cost of long-context LLM inference, but introduces approximation error that is typically validated only empirically. Existing systems rely on average-case robustness, with no mechanism to detect or recover from failures at runtime. We present a tiered KV cach…

• Instant GPU Efficiency Visibility at Fleet Scale

  Connor Pedersen, Dong H. Ahn, Michel Migdal, Collin Neale, Nik Konyuchenko · 21 mai 2026

  We present Overall FLOP Utilization (OFU), a hardware-level, precision-agnostic GPU efficiency metric for AI workloads on HPC systems, derived from two on-chip performance counters: Tensor Pipe Activity and SM clock frequency. OFU requires no application instrumentation and works across GPU generati…

• OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond

  Zunhai Su, Rui Yang, Chao Zhang, Yaxiu Liu, Yifan Zhang, Wei Wu, Jing Xiong, Dayou Du, Xialie Zhuang, Yulei Qian, Yuchen Xie, Yik-Chung Wu, Hongxia Yang, Ngai Wong · 20 mai 2026

  The rapid advancement toward long-context reasoning and multi-modal intelligence has made the memory footprint of the Key-Value (KV) cache a dominant memory bottleneck for efficient deployment. While the established per-channel quantization effectively accommodates intrinsic channel-wise outliers in…

• Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Caches

  Shaoke Fang, Ziang Li, Wenfei Wu, Jiatong Ji, Qingsong Liu, Ruizhi Pu · 20 mai 2026

  Prefix caching is a key optimization in Large Language Model (LLM) serving, reusing attention Key-Value (KV) states across requests with shared prompt prefixes to reduce expensive prefill computation. However, its benefit depends critically on the eviction policy as GPU memory is scarce, and existin…

• Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption

  Mert Yildiz, Pietro Spadaccino, Alexey Rolich, Francesca Cuomo, Andrea Baiocchi · 20 mai 2026

  Modern deployments of Large Language Models (LLMs) increasingly require serving multiple models with diverse architectures, sizes, and specialization on shared, heterogeneous hardware. This setting introduces new challenges for resource allocation, dispatching, and scheduling, particularly under GPU…

• Learn-by-Wire Training Control Governance: Bounded Autonomous Training Under Stress for Stability and Efficiency

  Anis Radianis · 20 mai 2026

  Modern language-model training is increasingly exposed to instability, degraded runs, and wasted compute, especially under aggressive learning-rate, scale, and runtime-stress conditions. This paper introduces Learn-by-Wire Guard (LBW-Guard), a bounded autonomous training-control governance layer tha…

• Flash PD-SSM: Memory-Optimized Structured Sparse State-Space Models

  Aleksandar Terzi\'c, Francesco Carzaniga, Nicolas Menet, Yannick Biehl, Michael Hersche, Thomas Hofmann, Abbas Rahimi · 20 mai 2026

  State-space models (SSMs) face a fundamental trade-off between efficiency and expressivity that is mainly dictated by the structure of the model's transition matrix. Unstructured transition matrices enable maximal expressivity, as measured by their ability to model finite-state automaton (FSA) trans…

• CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs

  Han Guo, Jack Zhang, Arjun Menon, Driss Guessous, Vijay Thakkar, Yoon Kim, Tri Dao · 20 mai 2026

  Transformer training systems are built around dense linear algebra, yet a nontrivial fraction of end-to-end time is spent on surrounding memory-bound operators. Normalization, activations, residual updates, reductions, and related computations repeatedly move large intermediate tensors through globa…

• First-Passage Prediction of Grokking Delay: ACalibrated Law under AdamW with Causal Validation

  Truong Xuan Khanh, Truong Quynh Hoa, Luu Duc Trung, Phan Thanh Duc · 20 mai 2026

  We give the first quantitative prediction of grokking delay under AdamW. Treating the delay as a first-passage time, we derive a closed-form law T_grok - T_mem = (1 / 2 kappa_LL eta lambda) log(V_mem / V_star), where V_t = ||theta_t||^2 is the squared parameter norm, V_star is an architecture-depend…

• Parallel Recursive LSTM

  Tristan Gaudreault, Yongyi Mao · 19 mai 2026

  Transformers have become the dominant architecture for sequence modeling by using self-attention to enable expressive and highly parallel processing. However, the resulting quadratic time and memory costs limit efficiency in long-context settings. Recurrent models such as LSTMs provide explicit nonl…

• AgentKernelArena: Generalization-Aware Benchmarking of GPU Kernel Optimization Agents

  Sharareh Younesian, Wenwen Ouyang, Sina Rafati, Mehdi Rezagholizadeh, Sharon Zhou, Ji Liu, Yue Liu, Yuchen Yang, Hao Li, Ziqiong Liu, Dong Li, Vikram Appia, Zhenyu Gu, Emad Barsoum · 19 mai 2026

  GPU kernel optimization is increasingly critical for efficient deep learning systems, but writing high-performance kernels still requires substantial low-level expertise. Recent AI coding agents can iteratively read code, invoke compilers and profilers, and refine implementations, yet existing kerne…

• A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variability

  Ruitao Liu (Tsinghua University), Xinyang Tian (Tsinghua University), Shuo Chen (Tsinghua University), Tingrui Zhang (Tsinghua University), Guang Yang (Tsinghua University), Alan Zhao (Scitix AI), Wei Xu (Tsinghua University) · 19 mai 2026

  Pipeline parallelism is a key technique for scaling large-model training, but modern workloads exhibit runtime variability in computation and communication. Existing pipeline systems typically consume static, profiled, or adaptively generated schedules as pre-committed execution orders. When realize…

• VeriCache: Turning Lossy KV Cache into Lossless LLM Inference

  Jiayi Yao, Samuel Shen, Kuntai Du, Shaoting Feng, Dongjoo Seo, Rui Zhang, Yuyang Huang, Yuhan Liu, Shan Lu, Junchen Jiang · 19 mai 2026

  The large size of the KV cache has become a major bottleneck for serving LLMs with increasing context lengths. In response, many KV cache compression methods, such as token dropping and quantization, have been proposed. However, almost all of these methods are inherently lossy-despite minimal accura…

• Minimal-Intervention KV Retention via Set-Conditioned Diversity

  Libo Sun, Po-wei Harn, Peixiong He, Xiao Qin · 19 mai 2026

  KV-cache compression at small budgets is a crowded design space spanning cache representation, head-wise routing, compression cadence, decoding behavior, and within-budget scoring. We study seven mechanisms across these five families under matched mean cache on long-form mathematical reasoning (MATH…

• OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization

  Zhongzhu Zhou, Donglin Zhuang, Jisen Li, Ziyan Chen, Shuaiwen Leon Song, Ben Athiwaratkun, Xiaoxia Wu · 19 mai 2026

  INT2 KV-cache quantization is attractive for long-context LLM serving, but it remains difficult to make both accurate and deployable. Simple rotations such as Hadamard transforms reduce outliers, but still degrade at INT2 because they are not aligned with downstream attention. We propose OSCAR, an U…

• A Few GPUs, A Whole Lotta Scale: Faithful LLM Training Emulation with PrismLLM

  Shaoke Xi, ChonLam Lao, Boyi Jia, Jiaqi Gao, Zhipeng Zhang, Jiamin Cao, Brian Sutioso, Erci Xu, Minlan Yu, Kui Ren, Yong Li, Zhengping Qian, Ennan Zhai, Jingren Zhou · 18 mai 2026

  Large language model (LLM) training today runs on clusters spanning thousands of GPUs. While this scale enables rapid model advances, developing, debugging, and performance-tuning the training framework inevitably becomes complex and costly. This is because engineers often need to reproduce producti…

• MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Serving

  Zhaoyuan Su, Olatunji Ruwase, Karthik Ganesan, Aurick Qiao, Samyam Rajbhandari, Juncheng Yang, Yue Cheng, Yuxiong He · 18 mai 2026

  Production LLM workloads increasingly serve discriminative tasks, such as classification, recommendation, and verification, whose answers are read from the logits of a single prefill pass with no autoregressive decoding. Serving these prefill-only workloads on mixture-of-experts (MoE) models is bott…

• Transformer Scalability Crisis: The First Comprehensive Empirical Analysis of Performance Walls in Modern Language Models

  Mahdi Naser Moghadasi, Faezeh Ghaderi · 18 mai 2026

  Despite the remarkable success of transformer architectures in natural language processing, their scalability limitations remain poorly understood through systematic empirical analysis. This paper presents the first comprehensive large-scale evaluation of 118 transformer models across seven distinct…

• Searching on a Budget: HW-NAS with 10 Latency Probes

  Francesco Capuano, Gabriele Tiboni, Niccol\`o Cavagnero, Giuseppe Averta · 18 mai 2026

  Existing hardware-aware NAS (HW-NAS) methods typically assume access to precise information circa the target device, either via analytical approximations of the post-compilation latency model, or through learned latency predictors. Such approximate approaches risk introducing estimation errors that …

• Runtime-Structured Task Decomposition for Agentic Coding Systems

  Shubhi Asthana, Bing Zhang, Chad DeLuca, Hima Patel, Ruchi Mahindru · 18 mai 2026

  Agentic coding systems increasingly use large language models (LLMs) for software engineering tasks such as debugging, root cause analysis, and code review. However, many existing systems encode task logic, execution flow, and output generation inside monolithic prompts. This design creates brittle …