Parallel Computing and Optimization Techniques

Latest papers

• Library Liberation: Competitive Performance Matmul Through Compiler-composed Nanokernels

  Arun Thangamani, Md Asghar Ahmad Shahid, Adam Siemieniuk, Rolf Morel, Renato Golin, Alexander Heinecke · 19 November 2025

  The rapidly evolving landscape of AI and machine learning workloads has widened the gap between high-level domain operations and efficient hardware utilization. Achieving near-peak performance still demands deep hardware expertise-experts either handcraft target-specific kernels (e.g., DeepSeek) or …

• CLO: Efficient LLM Inference System with CPU-Light KVCache Offloading via Algorithm-System Co-Design

  Jiawei Yi, Ping Gong, Youhui Bai, Jiaqi Ruan, Shengnan Wang, Pengcheng Wang, Haibo Wang, Weiguang Wang, Xia Zhu, Feng Wu, Cheng Li · 19 November 2025

  The growth of million-token LLMs exposes the scalability limits of inference systems, where the KVCache dominates memory usage and data transfer overhead. Recent offloading systems migrate the KVCache to CPU memory and incorporate top-k attention to reduce the volume of data transferred from the CPU…

• 10Cache: Heterogeneous Resource-Aware Tensor Caching and Migration for LLM Training

  Sabiha Afroz, Redwan Ibne Seraj Khan, Hadeel Albahar, Jingoo Han, Ali R. Butt · 19 November 2025

  Training large language models (LLMs) in the cloud faces growing memory bottlenecks due to the limited capacity and high cost of GPUs. While GPU memory offloading to CPU and NVMe has made large-scale training more feasible, existing approaches suffer from high tensor migration latency and suboptimal…

• From Legacy Fortran to Portable Kokkos: An Autonomous Agentic AI Workflow

  Sparsh Gupta, Kamalavasan Kamalakkannan, Maxim Moraru, Galen Shipman, Patrick Diehl · 19 November 2025

  Scientific applications continue to rely on legacy Fortran codebases originally developed for homogeneous, CPU-based systems. As High-Performance Computing (HPC) shifts toward heterogeneous GPU-accelerated architectures, many accelerators lack native Fortran bindings, creating an urgent need to mode…

• ParallelKittens: Systematic and Practical Simplification of Multi-GPU AI Kernels

  Stuart H. Sul, Simran Arora, Benjamin F. Spector, Christopher R\'e · 19 November 2025

  Inter-GPU communication has become a major bottleneck for modern AI workloads as models scale and improvements in hardware compute throughput outpace improvements in interconnect bandwidth. Existing systems mitigate this through compute-communication overlap but often fail to meet theoretical peak p…

• Maya: Optimizing Deep Learning Training Workloads using GPU Runtime Emulation

  Srihas Yarlagadda, Amey Agrawal, Elton Pinto, Hakesh Darapaneni, Mitali Meratwal, Shivam Mittal, Pranavi Bajjuri, Srinivas Sridharan, Alexey Tumanov · 18 November 2025

  Training large foundation models costs hundreds of millions of dollars, making deployment optimization critical. Current approaches require machine learning engineers to manually craft training recipes through error-prone trial-and-error on expensive compute clusters. To enable efficient exploration…

• MACKO: Sparse Matrix-Vector Multiplication for Low Sparsity

  Vladim\'ir Macko, Vladim\'ir Bo\v{z}a · 18 November 2025

  Sparse Matrix-Vector Multiplication (SpMV) is a fundamental operation in the inference of sparse Large Language Models (LLMs). Because existing SpMV methods perform poorly under the low and unstructured sparsity (30-90%) commonly observed in pruned LLMs, unstructured pruning provided only limited me…

• QUILL: An Algorithm-Architecture Co-Design for Cache-Local Deformable Attention

  Hyunwoo Oh, Hanning Chen, Sanggeon Yun, Yang Ni, Wenjun Huang, Tamoghno Das, Suyeon Jang, Mohsen Imani · 18 November 2025

  Deformable transformers deliver state-of-the-art detection but map poorly to hardware due to irregular memory access and low arithmetic intensity. We introduce QUILL, a schedule-aware accelerator that turns deformable attention into cache-friendly, single-pass work. At its core, Distance-based Out-o…

• KForge: Program Synthesis for Diverse AI Hardware Accelerators

  Taras Sereda, Tom St. John, Burak Bartan, Natalie Serrino, Sachin Katti, Zain Asgar · 18 November 2025

  GPU kernels are critical for ML performance but difficult to optimize across diverse accelerators. We present KForge, a platform-agnostic framework built on two collaborative LLM-based agents: a generation agent that produces and iteratively refines programs through compilation and correctness feedb…

• Mind the Gap: Revealing Inconsistencies Across Heterogeneous AI Accelerators

  Elliott Wen, Sean Ma, Ewan Tempero, Jens Dietrich, Daniel Luo, Jiaxing Shen, Kaiqi Zhao, Bruce Sham, Yousong Song, Jiayi Hua, Jia Hong · 18 November 2025

  While NVIDIA remains the dominant provider of AI accelerators within cloud data center, emerging vendors such as AMD, Intel, Mac, and Huawei offer cost-effective alternatives with claims of compatibility and performance. This paper presents the first empirical study investigating divergence in machi…

• FERMI-ML: A Flexible and Resource-Efficient Memory-In-Situ SRAM Macro for TinyML acceleration

  Mukul Lokhande, Akash Sankhe, S. V. Jaya Chand, Santosh Kumar Vishvakarma · 18 November 2025

  The growing demand for low-power and area-efficient TinyML inference on AIoT devices necessitates memory architectures that minimise data movement while sustaining high computational efficiency. This paper presents FERMI-ML, a Flexible and Resource-Efficient Memory-In-Situ (MIS) SRAM macro designed …

• LMM-IR: Large-Scale Netlist-Aware Multimodal Framework for Static IR-Drop Prediction

  Kai Ma, Zhen Wang, Hongquan He, Qi Xu, Tinghuan Chen, Hao Geng · 18 November 2025

  Static IR drop analysis is a fundamental and critical task in the field of chip design. Nevertheless, this process can be quite time-consuming, potentially requiring several hours. Moreover, addressing IR drop violations frequently demands iterative analysis, thereby causing the computational burden…

• Iris: First-Class Multi-GPU Programming Experience in Triton

  Muhammad Awad, Muhammad Osama, Brandon Potter · 18 November 2025

  Multi-GPU programming traditionally requires developers to navigate complex trade-offs between performance and programmability. High-performance implementations typically rely on low-level HIP/CUDA communication libraries that demand substantial engineering effort for even basic overlap patterns, wh…

• MMA-Sim: Bit-Accurate Reference Model of Tensor Cores and Matrix Cores

  Peichen Xie, Yang Wang, Fan Yang, Mao Yang · 17 November 2025

  The rapidly growing computation demands of deep neural networks (DNNs) have driven hardware vendors to integrate matrix multiplication accelerators (MMAs), such as NVIDIA Tensor Cores and AMD Matrix Cores, into modern GPUs. However, due to distinct and undocumented arithmetic specifications for floa…

• Event-Driven Digital-Time-Domain Inference Architectures for Tsetlin Machines

  Tian Lan, Rishad Shafik, Alex Yakovlev · 13 November 2025

  Machine learning fits model parameters to approximate input-output mappings, predicting unknown samples. However, these models often require extensive arithmetic computations during inference, increasing latency and power consumption. This paper proposes a digital-time-domain computing approach for …

• LLM Inference Beyond a Single Node: From Bottlenecks to Mitigations with Fast All-Reduce Communication

  Prajwal Singhania, Siddharth Singh, Lannie Dalton Hough, Akarsh Srivastava, Harshitha Menon, Charles Fredrick Jekel, Abhinav Bhatele · 13 November 2025

  As large language models (LLMs) continue to grow in size, distributed inference has become increasingly important. Model-parallel strategies must now efficiently scale not only across multiple GPUs but also across multiple nodes. In this work, we present a detailed performance study of multi-node di…

• Streaming Tensor Program: A streaming abstraction for dynamic parallelism

  Gina Sohn, Genghan Zhang, Konstantin Hossfeld, Jungwoo Kim, Nathan Sobotka, Nathan Zhang, Olivia Hsu, Kunle Olukotun · 12 November 2025

  Dynamic behaviors are becoming prevalent in many tensor applications. In machine learning, for example, the input tensors are dynamically shaped or ragged, and data-dependent control flow is widely used in many models. However, the limited expressiveness of prior programming abstractions for spatial…

• HipKittens: Fast and Furious AMD Kernels

  William Hu, Drew Wadsworth, Sean Siddens, Stanley Winata, Daniel Y. Fu, Ryann Swann, Muhammad Osama, Christopher R\'e, Simran Arora · 12 November 2025

  AMD GPUs offer state-of-the-art compute and memory bandwidth; however, peak performance AMD kernels are written in raw assembly. To address the difficulty of mapping AI algorithms to hardware, recent work proposes C++ embedded and PyTorch-inspired domain-specific languages like ThunderKittens (TK) t…

• Accelerating LLM Inference Throughput via Asynchronous KV Cache Prefetching

  Yanhao Dong, Yubo Miao, Weinan Li, Xiao Zheng, Chao Wang, Jiesheng Wu, Feng Lyu · 11 November 2025

  Large Language Models (LLMs) exhibit pronounced memory-bound characteristics during inference due to High Bandwidth Memory (HBM) bandwidth constraints. In this paper, we propose an L2 Cache-oriented asynchronous KV Cache prefetching method to break through the memory bandwidth bottleneck in LLM infe…

• Stateful KV Cache Management for LLMs: Balancing Space, Time, Accuracy, and Positional Fidelity

  Pratik Poudel · 10 November 2025

  The Key-Value (KV) cache is integral to efficient autoregressive inference in large language models (LLMs), yet its unbounded growth in stateful multi-turn scenarios presents major challenges. This paper examines the interplay between KV cache management strategies, the architectural context limits …

• DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing

  Lei Gao, Chaoyi Jiang, Hossein Entezari Zarch, Daniel Wong, Murali Annavaram · 10 November 2025

  Modern LLM serving systems must sustain high throughput while meeting strict latency SLOs across two distinct inference phases: compute-intensive prefill and memory-bound decode phases. Existing approaches either (1) aggregate both phases on shared GPUs, leading to interference between prefill and d…

• Inference-Time Hyper-Scaling with KV Cache Compression

  Adrian {\L}a\'ncucki, Konrad Staniszewski, Piotr Nawrot, Edoardo M. Ponti · 10 November 2025

  Inference-time scaling trades efficiency for increased reasoning accuracy by generating longer or more parallel sequences. However, in Transformer LLMs, generation cost is bottlenecked by the size of the key-value (KV) cache, rather than the number of generated tokens. Hence, we explore inference-ti…

• How Memory in Optimization Algorithms Implicitly Modifies the Loss

  Matias D. Cattaneo, Boris Shigida · 7 November 2025

  In modern optimization methods used in deep learning, each update depends on the history of previous iterations, often referred to as memory, and this dependence decays fast as the iterates go further into the past. For example, gradient descent with momentum has exponentially decaying memory throug…

• PDE-SHARP: PDE Solver Hybrids through Analysis and Refinement Passes

  Shaghayegh Fazliani, Madeleine Udell · 6 November 2025

  Current LLM-driven approaches using test-time computing to generate PDE solvers execute a large number of solver samples to identify high-accuracy solvers. These paradigms are especially costly for complex PDEs requiring substantial computational resources for numerical evaluation. We introduce PDE-…

• Personalized Interpolation: Achieving Efficient Conversion Estimation with Flexible Optimization Windows

  Xin Zhang, Weiliang Li, Rui Li, Zihang Fu, Tongyi Tang, Zhengyu Zhang, Wen-Yen Chen, Nima Noorshams, Nirav Jasapara, Xiaowen Ding, Ellie Wen, Xue Feng · 5 November 2025

  Optimizing conversions is crucial in modern online advertising systems, enabling advertisers to deliver relevant products to users and drive business outcomes. However, accurately predicting conversion events remains challenging due to variable time delays between user interactions (e.g., impression…