Parallel Computing and Optimization Techniques

Latest papers

• Breaking the Memory Wall: Exact Analytical Differentiation via Tiled Operator-Space Evolution

  Shuhuan Wang, Yuzhen Xie, Jiayi Li, Yinliang Diao · 30 December 2025

  Selective State Space Models (SSMs) achieve linear-time inference, yet their gradient-based sensitivity analysis remains bottlenecked by O(L) memory scaling during backpropagation. This memory constraint precludes genomic-scale modeling (L > 10^5) on consumer-grade hardware. We introduce Phase Gradi…

• AKG kernel Agent: A Multi-Agent Framework for Cross-Platform Kernel Synthesis

  Jinye Du, Quan Yuan, Zuyao Zhang, Yanzhi Yi, Jiahui Hu, Wangyi Chen, Yiyang Zhu, Qishui Zheng, Wenxiang Zou, Xiangyu Chang, Zuohe Zheng, Zichun Ye, Chao Liu, Shanni Li, Renwei Zhang, Yiping Deng, Xinwei Hu, Xuefeng Jin, Jie Zhao · 30 December 2025

  Modern AI models demand high-performance computation kernels. The growing complexity of LLMs, multimodal architectures, and recommendation systems, combined with techniques like sparsity and quantization, creates significant computational challenges. Moreover, frequent hardware updates and diverse c…

• Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels

  Maximilian Beck, Korbinian P\"oppel, Phillip Lippe, Sepp Hochreiter · 30 December 2025

  Linear RNNs with gating recently demonstrated competitive performance compared to Transformers in language modeling. Although their linear compute scaling in sequence length offers theoretical runtime advantages over Transformers, realizing these benefits in practice requires optimized custom kernel…

• GPU Kernel Optimization Beyond Full Builds: An LLM Framework with Minimal Executable Programs

  Ruifan Chu, Anbang Wang, Xiuxiu Bai, Shuai Liu, Xiaoshe Dong · 30 December 2025

  In high-performance computing, hotspot GPU kernels are primary bottlenecks, and expert manual tuning is costly and hard to port. Large language model methods often assume kernels can be compiled and executed cheaply, which fails in large applications where full builds and runs are expensive. We pres…

• LOOPer: A Learned Automatic Code Optimizer For Polyhedral Compilers

  Massinissa Merouani, Afif Boudaoud, Iheb Nassim Aouadj, Nassim Tchoulak, Islem Kara Bernou, Hamza Benyamina, Fatima Benbouzid-Si Tayeb, Karima Benatchba, Hugh Leather, Riyadh Baghdadi · 30 December 2025

  While polyhedral compilers have shown success in implementing advanced code transformations, they still face challenges in selecting the ones that lead to the most profitable speedups. This has motivated the use of machine learning based cost models to guide the search for polyhedral optimizations. …

• iOS as Acceleration

  Alexander K. Chen · 30 December 2025

  Practical utilization of large-scale machine learning requires a powerful compute setup, a necessity which poses a significant barrier to engagement with such artificial intelligence in more restricted system environments. While cloud computing offers a solution to weaker local environments, certain…

• LOOPerSet: A Large-Scale Dataset for Data-Driven Polyhedral Compiler Optimization

  Massinissa Merouani, Afif Boudaoud, Riyadh Baghdadi · 30 December 2025

  The advancement of machine learning for compiler optimization, particularly within the polyhedral model, is constrained by the scarcity of large-scale, public performance datasets. This data bottleneck forces researchers to undertake costly data generation campaigns, slowing down innovation and hind…

• GPU-Virt-Bench: A Comprehensive Benchmarking Framework for Software-Based GPU Virtualization Systems

  Jithin VG, Ditto PS · 30 December 2025

  The proliferation of GPU-accelerated workloads, particularly in artificial intelligence and large language model (LLM) inference, has created unprecedented demand for efficient GPU resource sharing in cloud and container environments. While NVIDIA's Multi-Instance GPU (MIG) technology provides hardw…

• Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2

  Pere Martra · 30 December 2025

  Structured width pruning of GLU-MLP layers, guided by the Maximum Absolute Weight (MAW) criterion, reveals a systematic dichotomy in how reducing the expansion ratio affects different model capabilities. While performance on tasks relying on parametric knowledge (e.g., MMLU, GSM8K) and perplexity me…

• Role-Based Fault Tolerance System for LLM RL Post-Training

  Zhenqian Chen, Baoquan Zhong, Xiang Li, Qing Dai, Xinkui Zhao, Miao Ye, Ren Cheng, Lufei Zhang, Jianwei Yin · 30 December 2025

  RL post-training for LLMs has been widely scaled to enhance reasoning and tool-using capabilities. However, RL post-training interleaves training and inference workloads, exposing the system to faults from both sides. Existing fault tolerance frameworks for LLMs target either training or inference, …

• Prefill vs. Decode Bottlenecks: SRAM-Frequency Tradeoffs and the Memory-Bandwidth Ceiling

  Hannah Atmer, Yuan Yao, Thiemo Voigt, Stefanos Kaxiras · 29 December 2025

  Energy consumption dictates the cost and environmental impact of deploying Large Language Models. This paper investigates the impact of on-chip SRAM size and operating frequency on the energy efficiency and performance of LLM inference, focusing on the distinct behaviors of the compute-bound prefill…

• nncase: An End-to-End Compiler for Efficient LLM Deployment on Heterogeneous Storage Architectures

  Hui Guo, Qihang Zheng, Chenghai Huo, Dongliang Guo, Haoqi Yang, Yang Zhang · 29 December 2025

  The efficient deployment of large language models (LLMs) is hindered by memory architecture heterogeneity, where traditional compilers suffer from fragmented workflows and high adaptation costs. We present nncase, an open-source, end-to-end compilation framework designed to unify optimization across…

• Deadline-Aware Online Scheduling for LLM Fine-Tuning with Spot Market Predictions

  Linggao Kong, Yuedong Xu, Lei Jiao, Chuan Xu · 25 December 2025

  As foundation models grow in size, fine-tuning them becomes increasingly expensive. While GPU spot instances offer a low-cost alternative to on-demand resources, their volatile prices and availability make deadline-aware scheduling particularly challenging. We tackle this difficulty by using a mix o…

• Per-Axis Weight Deltas for Frequent Model Updates

  Stefan Kuyumdzhiev, Radostin Cholakov · 24 December 2025

  Serving many task-specialized LLM variants is often limited by the large size of fine-tuned checkpoints and the resulting cold-start latency. Since fine-tuned weights differ from their base model by relatively small structured residuals, a natural approach is to represent them as compressed deltas. …

• UCCL-EP: Portable Expert-Parallel Communication

  Ziming Mao, Yihan Zhang, Chihan Cui, Kaichao You, Zhongjie Chen, Zhiying Xu, Scott Shenker, Costin Raiciu, Yang Zhou, Ion Stoica · 24 December 2025

  Mixture-of-Experts (MoE) workloads rely on expert parallelism (EP) to achieve high GPU efficiency. State-of-the-art EP communication systems such as DeepEP demonstrate strong performance but exhibit poor portability across heterogeneous GPU and NIC platforms. The poor portability is rooted in archit…

• Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures

  Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Huazuo Gao, Jiashi Li, Liyue Zhang, Panpan Huang, Shangyan Zhou, Shirong Ma, Wenfeng Liang, Ying He, Yuqing Wang, Yuxuan Liu, Y. X. Wei · 24 December 2025

  The rapid scaling of large language models (LLMs) has unveiled critical limitations in current hardware architectures, including constraints in memory capacity, computational efficiency, and interconnection bandwidth. DeepSeek-V3, trained on 2,048 NVIDIA H800 GPUs, demonstrates how hardware-aware mo…

• Surprisingly High Redundancy in Electronic Structure Data

  Sazzad Hossain, Ponkrshnan Thiagarajan, Shashank Pathrudkar, Stephanie Taylor, Abhijeet S. Gangan, Amartya S. Banerjee, Susanta Ghosh · 24 December 2025

  Accurate prediction of electronic structure underpins advances in chemistry, materials science, and condensed matter physics. In recent years, Machine Learning (ML) has enabled the development of powerful surrogate models that can enable the prediction of the ground state electron density and relate…

• CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs

  Gunho Park, Jeongin Bae, Byeongwook Kim, Baeseong park, Jiwon Ryu, Hoseung Kim, Se Jung Kwon, Dongsoo Lee · 23 December 2025

  Weight-only quantization is widely used to mitigate the memory-bound nature of LLM inference. Codebook-based methods extend this trend by achieving strong accuracy in the extremely low-bit regime (e.g., 2-bit). However, current kernels rely on dequantization, which repeatedly fetches centroids and r…

• ML Inference Scheduling with Predictable Latency

  Haidong Zhao, Nikolaos Georgantas · 23 December 2025

  Machine learning (ML) inference serving systems can schedule requests to improve GPU utilization and to meet service level objectives (SLOs) or deadlines. However, improving GPU utilization may compromise latency-sensitive scheduling, as concurrent tasks contend for GPU resources and thereby introdu…

• Small Language Models as Compiler Experts: Auto-Parallelization for Heterogeneous Systems

  Prathamesh Devadiga · 23 December 2025

  Traditional auto-parallelizing compilers, reliant on rigid heuristics, struggle with the complexity of modern heterogeneous systems. This paper presents a comprehensive evaluation of small (approximately 1B parameter) language-model-driven compiler auto-parallelization. We evaluate three models: gem…

• Efficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRA

  Allison Li, Kristjan Greenewald, Thomas Parnell, Navid Azizan · 23 December 2025

  Modern large language model (LLM) systems increasingly rely on multi-turn pipelines that are composed of multiple task-specific adapters, yet existing serving frameworks remain inefficient, incurring substantial recomputation overhead when switching between adapters. We present the first LLM serving…

• Optimal Software Pipelining and Warp Specialization for Tensor Core GPUs

  Rupanshu Soi, Rohan Yadav, Fredrik Kjolstad, Alex Aiken, Maryam Mehri Dehnavi, Michael Garland, Michael Bauer · 23 December 2025

  GPU architectures have continued to grow in complexity, with recent incarnations introducing increasingly powerful fixed-function units for matrix multiplication and data movement to accompany highly parallel general-purpose cores. To fully leverage these machines, software must use sophisticated sc…

• A Reinforcement Learning Environment for Automatic Code Optimization in the MLIR Compiler

  Mohammed Tirichine, Nassim Ameur, Nazim Bendib, Iheb Nassim Aouadj, Bouchama Djad, Rafik Bouloudene, Riyadh Baghdadi · 23 December 2025

  Code optimization is a crucial task that aims to enhance code performance. However, this process is often tedious and complex, highlighting the necessity for automatic code optimization techniques. Reinforcement Learning (RL) has emerged as a promising approach for tackling such complex optimization…

• Efficient Deep Learning Infrastructures for Embedded Computing Systems: A Comprehensive Survey and Future Envision

  Xiangzhong Luo, Di Liu, Hao Kong, Shuo Huai, Hui Chen, Guochu Xiong, Weichen Liu · 23 December 2025

  Deep neural networks (DNNs) have recently achieved impressive success across a wide range of real-world vision and language processing tasks, spanning from image classification to many other downstream vision tasks, such as object detection, tracking, and segmentation. However, previous well-establi…

• Libra: Unleashing GPU Heterogeneity for High-Performance Sparse Matrix Multiplication

  Jinliang Shi, Shigang Li, Youxuan Xu, Xueying Wang, Rongtian Fu, Zhi Ma, Tong Wu · 23 December 2025

  Sparse matrix multiplication operators (i.e., SpMM and SDDMM) are widely used in deep learning and scientific computing. Modern accelerators are commonly equipped with Tensor Core Units (TCUs) and CUDA cores to accelerate sparse operators. The former excels at structured matrix computations, whereas…