Parallel Computing and Optimization Techniques

Latest papers

• Self-Indexing KVCache: Predicting Sparse Attention from Compressed Keys

  Xu Yang, Jiapeng Zhang, Dongyang Zhao, Guo Chen, Zhuo Tang · 17 March 2026

  The KV cache in self-attention has emerged as a major bottleneck in long-context and large-batch inference for LLMs. Existing approaches often treat sparsity prediction and compression as separate modules, relying on auxiliary index structures to select relevant tokens, and on complex quantization s…

• The Big Send-off: Scalable and Performant Collectives for Deep Learning

  Siddharth Singh, Keshav Pradeep, Mahua Singh, Cunyang Wei, Abhinav Bhatele · 17 March 2026

  Collective communication is becoming increasingly important in data center and supercomputer workloads with an increase in distributed AI related jobs. However, existing libraries that provide collective support such as NCCL, RCCL, and Cray-MPICH exhibit several performance and scalability limitatio…

• Machine Learning-Driven Intelligent Memory System Design: From On-Chip Caches to Storage

  Rahul Bera, Rakesh Nadig, Onur Mutlu · 17 March 2026

  Despite the data-rich environment in which memory systems of modern computing platforms operate, many state-of-the-art architectural policies employed in the memory system rely on static, human-designed heuristics that fail to truly adapt to the workload and system behavior via principled learning m…

• NCCL EP: Towards a Unified Expert Parallel Communication API for NCCL

  Amos Goldman (NVIDIA Corporation), Nimrod Boker (NVIDIA Corporation), Maayan Sheraizin (NVIDIA Corporation), Nimrod Admoni (NVIDIA Corporation), Artem Polyakov (NVIDIA Corporation), Subhadeep Bhattacharya (NVIDIA Corporation), Fan Yu (NVIDIA Corporation), Kai Sun (NVIDIA Corporation), Georgios Theodorakis (NVIDIA Corporation), Hsin-Chun Yin (NVIDIA Corporation), Peter-Jan Gootzen (NVIDIA Corporation), Aamir Shafi (NVIDIA Corporation), Assaf Ravid (NVIDIA Corporation), Salvatore Di Girolamo (NVIDIA Corporation), Manjunath Gorentla Venkata (NVIDIA Corporation), Gil Bloch (NVIDIA Corporation) · 17 March 2026

  Mixture-of-Experts (MoE) architectures have become essential for scaling large language models, driving the development of specialized device-initiated communication libraries such as DeepEP, Hybrid-EP, and others. These libraries demonstrate the performance benefits of GPU-initiated RDMA for MoE di…

• KernelFoundry: Hardware-aware evolutionary GPU kernel optimization

  Nina Wiedemann, Quentin Leboutet, Michael Paulitsch, Diana Wofk, Benjamin Ummenhofer · 16 March 2026

  Optimizing GPU kernels presents a significantly greater challenge for large language models (LLMs) than standard code generation tasks, as it requires understanding hardware architecture, parallel optimization strategies, and performance profiling outputs. Most existing LLM-based approaches to kerne…

• OpenACMv2: An Accuracy-Constrained Co-Optimization Framework for Approximate DCiM

  Yiqi Zhou, Yue Yuan, Yikai Wang, Bohao Liu, Qinxin Mei, Zhuohua Liu, Shan Shen, Wei Xing, Daying Sun, Li Li, Guozhu Liu · 16 March 2026

  Digital Compute-in-Memory (DCiM) accelerates neural networks by reducing data movement. Approximate DCiM can further improve power-performance-area (PPA), but demands accuracy-constrained co-optimization across coupled architecture and transistor-level choices. Building on OpenYield, we introduce Ac…

• AdaBoN: Adaptive Best-of-N Alignment

  Vinod Raman, Hilal Asi, Satyen Kale · 16 March 2026

  Recent advances in test-time alignment methods, such as Best-of-N sampling, offer a simple and effective way to steer language models (LMs) toward preferred behaviors using reward models (RM). However, these approaches can be computationally expensive, especially when applied uniformly across prompt…

• Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity

  Donglin Yu · 16 March 2026

  Multimodal large language model (MLLM) inference splits into two phases with opposing hardware demands: vision encoding is compute-bound, while language generation is memory-bandwidth-bound. We show that under standard transformer KV caching, the modality boundary (between vision encoder and languag…

• Rethinking Attention: Polynomial Alternatives to Softmax in Transformers

  Hemanth Saratchandran, Jianqiao Zheng, Yiping Ji, Wenbo Zhang, Simon Lucey · 16 March 2026

  This paper questions whether the strong performance of softmax attention in transformers stems from producing a probability distribution over inputs. Instead, we argue that softmax's effectiveness lies in its implicit regularization of the Frobenius norm of the attention matrix, which stabilizes tra…

• LongFlow: Efficient KV Cache Compression for Reasoning M

  Yi Su, Zhenxu Tian, Dan Qiao, Yuechi Zhou, Juntao Li, Min Zhang · 13 March 2026

  Recent reasoning models such as OpenAI-o1 and DeepSeek-R1 have shown strong performance on complex tasks including mathematical reasoning and code generation. However, this performance gain comes with substantially longer output sequences, leading to significantly increased deployment costs. In part…

• H2LooP Spark Preview: Continual Pretraining of Large Language Models for Low-Level Embedded Systems Code

  Amit Singh, Vedant Nipane, Pulkit Agrawal, Jatin Kishnani · 13 March 2026

  Large language models (LLMs) demonstrate strong code generation abilities in general-purpose programming languages but remain limited in specialized domains such as low-level embedded systems programming. This domain involves hardware register manipulation, vendor-specific SDKs, real-time operating …

• The DMA Streaming Framework: Kernel-Level Buffer Orchestration for High-Performance AI Data Paths

  Marco Graziano · 12 March 2026

  AI transport libraries move bytes efficiently, but they commonly assume that buffers are already correctly allocated, placed, shared, registered, and safe under completion and teardown pressure. This paper presents dmaplane, a Linux kernel module that makes this missing layer explicit as buffer orch…

• Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study

  Athos Georgiou · 12 March 2026

  We present a cross-architecture evaluation of production LLM inference on AMD Instinct MI325X GPUs, benchmarking four models spanning 235B to 1 trillion parameters across three architectural families (MoE+MLA, Dense+GQA, MoE+GQA) on an 8-GPU cluster with 2TB aggregate HBM3e using vLLM v0.14.1. Our r…

• RedFuser: An Automatic Operator Fusion Framework for Cascaded Reductions on AI Accelerators

  Xinsheng Tang, Yangcheng Li, Nan Wang, Zhiyi Shu, Xingyu Ling, Junna Xing, Peng Zhou, Qiang Liu · 12 March 2026

  Operator fusion, as a key performance optimization technique in the deployment of AI models, significantly improves execution efficiency and has been widely adopted in modern AI compilers. However, for cascaded reduction operations involving multiple loops with inter-loop data dependencies, such as …

• Towards Cold-Start Drafting and Continual Refining: A Value-Driven Memory Approach with Application to NPU Kernel Synthesis

  Yujie Zheng, Zhuo Li, Shengtao Zhang, Hanjing Wang, Junjie Sheng, Jiaqian Wang, Junchi Yan, Weinan Zhang, Ying Wen, Bo Tang, Muning Wen · 12 March 2026

  Deploying Large Language Models to data-scarce programming domains poses significant challenges, particularly for kernel synthesis on emerging Domain-Specific Architectures where a "Data Wall" limits available training data. While models excel on data-rich platforms like CUDA, they suffer catastroph…

• Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttention

  Mengqi Liao, Lu Wang, Chaoyun Zhang, Bo Qiao, Si Qin, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Huaiyu Wan · 11 March 2026

  With reasoning becoming the generative paradigm for large language models (LLMs), the memory bottleneck caused by KV cache during the decoding phase has become a critical factor limiting high-concurrency service. Although existing KV cache eviction methods address the memory issue, most of them are …

• Compiler-First State Space Duality and Portable $O(1)$ Autoregressive Caching for Inference

  Cosmo Santoni · 11 March 2026

  State-space model releases are typically coupled to fused CUDA and Triton kernels, inheriting a hard dependency on NVIDIA hardware. We show that Mamba-2's state space duality algorithm -- diagonal state structure, chunkable recurrence, and einsum-dominated compute with static control flow -- maps cl…

• Unveiling the Potential of Quantization with MXFP4: Strategies for Quantization Error Reduction

  Jatin Chhugani, Geonhwa Jeong, Bor-Yiing Su, Yunjie Pan, Hanmei Yang, Aayush Ankit, Jiecao Yu, Summer Deng, Yunqing Chen, Nadathur Satish, Changkyu Kim · 11 March 2026

  Large Language Models (LLMs) have intensified the need for low-precision formats that enable efficient, large-scale inference. The Open Compute Project (OCP) Microscaling (MX) standard is attractive due to its favorable hardware efficiency, but its 4-bit variant (MXFP4) lags behind NVIDIA's NVFP4 in…

• KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware

  Jiayi Nie, Haoran Wu, Yao Lai, Zeyu Cao, Cheng Zhang, Binglei Lou, Erwei Wang, Jianyi Cheng, Timothy M. Jones, Robert Mullins, Rika Antonova, Yiren Zhao · 11 March 2026

  New AI accelerators with novel instruction set architectures (ISAs) often require developers to manually craft low-level kernels -- a time-consuming, laborious, and error-prone process that cannot scale across diverse hardware targets. This prevents emerging hardware platforms from reaching the mark…

• Diagnosing FP4 inference: a layer-wise and block-wise sensitivity analysis of NVFP4 and MXFP4

  Musa Cim, Burak Topcu, Mahmut Taylan Kandemir · 11 March 2026

  Quantization addresses the high resource demand for large language models (LLMs) by alleviating memory pressure and bandwidth congestion and providing significantly scaled compute power with a tolerable impact on accuracy. Four-bit floating point (FP4), the lowest-precision format that preserves ess…

• The Missing Memory Hierarchy: Demand Paging for LLM Context Windows

  Tony Mason · 11 March 2026

  The context window of a large language model is not memory. It is L1 cache: a small, fast, expensive resource that the field treats as the entire memory system. There is no L2, no virtual memory, no paging. Every tool definition, every system prompt, and every stale tool result occupies context for …

• Making LLMs Optimize Multi-Scenario CUDA Kernels Like Experts

  Yuxuan Han, Meng-Hao Guo, Zhengning Liu, Wenguang Chen, Shi-Min Hu · 10 March 2026

  Optimizing GPU kernels manually is a challenging and time-consuming task. With the rapid development of LLMs, automated GPU kernel optimization is gradually becoming a tangible reality. However, current LLM-driven automated optimization methods narrowly focus on machine learning applications, such a…

• PolyBlocks: A Compiler Infrastructure for AI Chips and Programming Frameworks

  Uday Bondhugula, Akshay Baviskar, Navdeep Katel, Vimal Patel, Anoop JS, Arnab Dutta · 10 March 2026

  We present the design and implementation of PolyBlocks, a modular and reusable MLIR-based compiler infrastructure for AI programming frameworks and AI chips. PolyBlocks is based on pass pipelines that compose transformations on loop nests and SSA, primarily relying on lightweight affine access analy…

• Mitigating the Memory Bottleneck with Machine Learning-Driven and Data-Aware Microarchitectural Techniques

  Rahul Bera · 10 March 2026

  Modern applications process massive data volumes that overwhelm the storage and retrieval capabilities of memory systems, making memory the primary performance and energy-efficiency bottleneck of computing systems. Although many microarchitectural techniques attempt to hide or tolerate long memory a…

• EROICA: Online Performance Troubleshooting for Large-scale Model Training

  Yu Guan, Zhiyu Yin, Haoyu Chen, Sheng Cheng, Chaojie Yang, Kun Qian, Tianyin Xu, Pengcheng Zhang, Yang Zhang, Hanyu Zhao, Yong Li, Wei Lin, Dennis Cai, Ennan Zhai · 10 March 2026

  Troubleshooting performance problems of large model training (LMT) is immensely challenging, due to unprecedented scales of modern GPU clusters, the complexity of software-hardware interactions, and the data intensity of the training process. Existing troubleshooting approaches designed for traditio…