Parallel Computing and Optimization Techniques

Neueste Paper

• FASQ: Flexible Accelerated Subspace Quantization for Calibration-Free LLM Compression

  Ye Qiao, Yian Wang, Zhiheng Chen, Hyoukjun Kwon, Sitao Huang · 7. Mai 2026

  Compressing large language models (LLMs) for deployment on commodity GPUs remains challenging: conventional scalar quantization is limited to fixed bit-widths (e.g., 8/4/3-bit), offers only a few discrete compression points, and typically requires calibration data. We present FASQ (Flexible Accelera…

• Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism

  Sajal Dash, Feiyi Wang · 7. Mai 2026

  Frontier models increasingly adopt Mixture-of-Experts (MoE) architectures to achieve large-model performance at reduced cost. However, training MoE models on HPC platforms is hindered by large memory footprints, frequent large-scale communication across heterogeneous networks, and severe workload im…

• LCM: Lossless Context Management

  Clint Ehrlich, Theodore Blackman · 7. Mai 2026

  We introduce Lossless Context Management (LCM), a deterministic architecture for LLM memory that outperforms Claude Code on long-context tasks. When benchmarked using Opus 4.6, our LCM-augmented coding agent, Volt, achieves higher scores than Claude Code on the OOLONG long-context eval, including at…

• KernelBench-X: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels

  Han Wang, Jintao Zhang, Kai Jiang, Haoxu Wang, Jianfei Chen, Jun Zhu · 7. Mai 2026

  LLM-based Triton kernel generation has attracted significant interest, yet a fundamental empirical question remains unanswered: where does this capability break down, and why? We present KernelBench-X, a benchmark designed to answer this question through category-aware evaluation of correctness and …

• One Pool, Two Caches: Adaptive HBM Partitioning for Accelerating Generative Recommender Serving

  Wenjun Yu, Shuguang Han, Amelie Chi Zhou · 7. Mai 2026

  Generative Recommender (GR) inference places embedding hot caches (EMB) and KV caches in direct competition for limited GPU HBM: allocating more memory to one improves its efficiency but degrades the other. Existing systems optimize them in isolation, overlooking that the optimal EMB-KV allocation r…

• StateSMix: Online Lossless Compression via Mamba State Space Models and Sparse N-gram Context Mixing

  Roberto Tacconelli · 6. Mai 2026

  We present StateSMix, a fully self-contained lossless compressor that couples an online-trained Mamba-style State Space Model (SSM) with sparse n-gram context mixing and arithmetic coding. The model is initialised from scratch and trained token-by-token on the file being compressed, requiring no pre…

• AutoRAGTuner: A Declarative Framework for Automatic Optimization of RAG Pipelines

  Xintan Zeng, Yongchao Liu, Yice Luo, Jiajun Zhen · 6. Mai 2026

  Retrieval-Augmented Generation (RAG) enhances LLMs, but performance is highly sensitive to complex architecture designs and hyper-parameter configurations, which currently rely on inefficient manual tuning. We present AutoRAGTuner, a declarative, configuration-driven framework that automates the RAG…

• COREY: Entropy-Guided Runtime Chunk Scheduling for Selective Scan Kernels

  Bo Ma, Jinsong Wu, Weiqi Yan · 6. Mai 2026

  Mamba selective state space models (SSMs) provide linear-time sequence modeling but remain sensitive to selective-scan chunk scheduling. We present COREY, a \emph{concept-and-feasibility} runtime scheduler that maps fixed-bin activation entropy to chunk size. We evaluate COREY in three tiers: a prot…

• ZeRO-Prefill: Zero Redundancy Overheads in MoE Prefill Serving

  Zhaoyuan Su, Olatunji Ruwase, Karthik Ganesan, Aurick Qiao, Samyam Rajbhandari, Juncheng Yang, Yue Cheng, Yuxiong He · 6. Mai 2026

  Production LLM workloads increasingly serve discriminative tasks, such as classification, recommendation, and verification, whose answers are read from the logits of a single prefill pass with no autoregressive decoding. Serving these prefill-only workloads on mixture-of-experts (MoE) models is bott…

• DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning

  Hossein Entezari Zarch, Lei Gao, Chaoyi Jiang, Murali Annavaram · 5. Mai 2026

  Large reasoning models (LRMs) achieve state-of-the-art performance on challenging benchmarks by generating long chains of intermediate steps, but their inference cost is dominated by decoding, where each new token must attend to the entire growing sequence. One approach to reduce this latency is to …

• SURGE: SuperBatch Unified Resource-efficient GPU Encoding for Heterogeneous Partitioned Data

  Shashank Kapadia, Deep Narayan Mishra, Sujal Reddy Alugubelli, Ajay Kumar, Swapnil Yadav, Rishi Bhatia · 5. Mai 2026

  We present SURGE, a streaming GPU encoding system deployed in production to generate embeddings for over 800 million texts across 40,000 logical partitions. Production embedding pipelines face a tension between logical data partitioning and efficient GPU utilization: processing each partition indepe…

• LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference

  Shashank Kapadia, Deep Naryan Mishra, Sujal Reddy Alugubelli, Haoan Wang, Saipraveen Vabbilisetty, Rishi Bhatia, Anupriya Sharma · 5. Mai 2026

  Layer-aligned distillation and convergence-based early exit represent two predominant computational efficiency paradigms for transformer inference; yet we establish that they exhibit systematic incompatibility under standard deployment conditions for convergence-based early exit. Distillation object…

• SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving

  Yipin Guo, Siddharth Joshi · 5. Mai 2026

  Contemporary systems serving large language models (LLMs) have adopted prefill-decode disaggregation to better load-balance between the compute-bound prefill phase and the memory-bound decode phase. Under this design, prefill workers generate a KV cache that must be transferred to decode workers bef…

• AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments

  Zhijie Cai, Haolong Chen, Guangxu Zhu · 4. Mai 2026

  Fine-tuning LLMs is necessary for various dedicated downstream tasks, but classic backpropagation-based fine-tuning methods require substantial GPU memory. To this end, a recent work, MeZO, which relies solely on forward passes to fine-tune LLMs, significantly reduces GPU requirements at the cost of…

• SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters

  Dongxin Guo, Jikun Wu, Siu Ming Yiu · 4. Mai 2026

  AI agents execute tens to hundreds of chained LLM calls per task, yet GPU schedulers treat each call as independent, discarding gigabytes of intermediate state between steps and inflating end-to-end latency by 3-8x. We argue that this request-level abstraction is fundamentally mismatched to compound…

• Surprisingly High Redundancy in Electronic Structure Data Across Materials Explained by Low Intrinsic Dimensionality

  Sazzad Hossain, Ponkrshnan Thiagarajan, Shashank Pathrudkar, Stephanie Taylor, Abhijeet S. Gangan, Amartya S. Banerjee, Susanta Ghosh · 4. Mai 2026

  Machine learning (ML) models for electronic structure typically rely on large datasets generated by computationally expensive Kohn-Sham density functional theory calculations, as it is not known a priori which portions of the data are essential for accurate learning. Here, we reveal significant redu…

• Silicon Showdown: Performance, Efficiency, and Ecosystem Barriers in Consumer-Grade LLM Inference

  Allan Kazakov, Abdurrahman Javat · 4. Mai 2026

  The operational landscape of local Large Language Model (LLM) inference has shifted from lightweight models to datacenter-class weights exceeding 70B parameters, creating profound systems challenges for consumer hardware. This paper presents a systematic empirical analysis of the Nvidia and Apple Si…

• Efficient Training on Multiple Consumer GPUs with RoundPipe

  Yibin Luo, Shiwei Gao, Huichuan Zheng, Youyou Lu, Jiwu Shu · 1. Mai 2026

  Fine-tuning Large Language Models (LLMs) on consumer-grade GPUs is highly cost-effective, yet constrained by limited GPU memory and slow PCIe interconnects. Pipeline parallelism combined with CPU offloading mitigates these hardware bottlenecks by reducing communication overhead. However, existing PP…

• AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism

  Ahan Gupta, Zhihao Wang, Neel Dani, Masahiro Tanaka, Olatunji Ruwase, Minjia Zhang · 1. Mai 2026

  Large-language-models (LLMs) demonstrate enormous utility in long-context tasks which require processing prompts that consist of tens to hundreds of thousands of tokens. However, existing LLM training libraries do not provide easy to use abstractions to optimize for long-context training, instead fo…

• Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference

  Sanjeev Rao Ganjihal · 1. Mai 2026

  Key-value (KV) cache memory management is the primary bottleneck limiting throughput and cost-efficiency in large-scale GPU inference serving. Current systems suffer from three compounding inefficiencies: (1) the absence of unified KV cache sizing across all attention architectures--particularly mul…

• RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts

  Vyom Sharma, Debajyoti Datta · 30. April 2026

  The optimal kernel configuration for Mixture-of-Experts (MoE) inference depends on both batch size and the expert routing distribution, yet production systems dispatch from batch size alone, leaving 10-70% of kernel throughput unrealized. We present RaMP, a routing-aware dispatch framework. A perfor…

• CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Training

  Rezaul Karim, Austin Wen, Wang Zongzuo, Weiwei Zhang, Yang Liu, Walid Ahmed · 30. April 2026

  The rapid growth in the size of large language models has necessitated the partitioning of computational workloads across accelerators such as GPUs, TPUs, and NPUs. However, these parallelization strategies incur substantial data communication overhead significantly hindering computational efficienc…

• AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving

  Zhongkai Yu, Haotian Ye, Chenyang Zhou, Ohm Rishabh Venkatachalam, Zaifeng Pan, Zhengding Hu, Junsung Kim, Won Woo Ro, Po-An Tsai, Shuyi Pei, Yangwook Kang, Yufei Ding · 30. April 2026

  All current LLM serving systems place the GPU at the center, from production-level attention-FFN disaggregation to NVIDIA's Rubin GPU-LPU heterogeneous platform. Even academic PIM/PNM proposals still treat the GPU as the central hub for cross-device communication. Yet the GPU's compute-rich architec…

• Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving

  Zihan Zhao, Baotong Lu, Shengjie Lin, Yizou Chen, Jing Liu, Yanqi Zhang, Ziming Miao, Ming-Chang Yang, Haiying Shen, Qi Chen, Fan Yang · 30. April 2026

  Long-context LLM serving is bottlenecked by the cost of attending over ever-growing KV caches. Dynamic sparse attention promises relief by accessing only a small, query-dependent subset of the KV state per decoding step and extending the KV storage to CPU memory. In practice, however, these algorith…

• MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment

  Hanxian Huang, Igor Fedorov, Andrey Gromov, Bernard Beckerman, Naveen Suda, David Eriksson, Maximilian Balandat, Rylan Conway, Patrick Huber, Chinnadhurai Sankar, Ayushi Dalmia, Zechun Liu, Lemeng Wu, Tarek Elgamal, Adithya Sagar, Vikas Chandra, Raghuraman Krishnamoorthi · 29. April 2026

  Real-time AI experiences call for on-device large language models (OD-LLMs) optimized for efficient deployment on resource-constrained hardware. The most useful OD-LLMs produce near-real-time responses and exhibit broad hardware compatibility, maximizing user reach. We present a methodology for desi…