Parallel Computing and Optimization Techniques

Latest papers

• Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Eviction

  Ziyao Tang, Pengkun Jiao, Xinhang Chen, Wei Liu, Shiyong Li, Jingjing Chen · 10 February 2026

  Given the quadratic complexity of attention, KV cache eviction is vital to accelerate model inference. Current KV cache eviction methods typically rely on instantaneous heuristic metrics, implicitly assuming that score magnitudes are consistent proxies for importance across all heads. However, this …

• Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization

  Chong Wang, Nan Du, Tom Gunter, Tao Lei, Kulin Seth, Senyu Tong, Jianyu Wang, Guoli Yin, Xiyou Zhou, Kelvin Zou, Ruoming Pang · 10 February 2026

  Efficient large-scale inference of transformer-based large language models (LLMs) remains a fundamental systems challenge, frequently requiring multi-GPU parallelism to meet stringent latency and throughput targets. Conventional tensor parallelism decomposes matrix operations across devices but intr…

• DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity

  Jitai Hao, Qiang Huang, Yaowei Wang, Min Zhang, Jun Yu · 10 February 2026

  The deployment of efficient long-context LLMs in applications like autonomous agents, long-chain reasoning, and creative writing is fundamentally bottlenecked by the linear growth of KV cache memory. Existing compression and eviction methods often struggle to balance accuracy, compression ratio, and…

• Compiler-Assisted Speculative Sampling for Accelerated LLM Inference on Heterogeneous Edge Devices

  Alejandro Ruiz y Mesa, Guilherme Korol, Moritz Riesteter, Jo\~ao Paulo Cardoso de Lima, Jeronimo Castrillon · 10 February 2026

  LLM deployment on resource-constrained edge devices faces severe latency constraints, particularly in real-time applications where delayed responses can compromise safety or usability. Among many approaches to mitigate the inefficiencies of sequential token-by-token generation, Speculative Decoding …

• ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Models

  Raghav Singhal, Kaustubh Ponkshe, Rohit Vartak, Praneeth Vepakomma · 10 February 2026

  Large Language Models have demonstrated strong performance across a wide range of tasks, but adapting them efficiently to new domains remains a key challenge. Parameter-Efficient Fine-Tuning (PEFT) methods address this by introducing lightweight, trainable modules while keeping most pre-trained weig…

• Canzona: A Unified, Asynchronous, and Load-Balanced Framework for Distributed Matrix-based Optimizers

  Liangyu Wang, Siqi Zhang, Junjie Wang, Yiming Dong, Bo Zheng, Zihan Qiu, Shengkun Tang, Di Wang, Rui Men, Dayiheng Liu · 9 February 2026

  The scaling of Large Language Models (LLMs) drives interest in matrix-based optimizers (e.g., Shampoo, Muon, SOAP) for their convergence efficiency; yet their requirement for holistic updates conflicts with the tensor fragmentation in distributed frameworks like Megatron. Existing solutions are subo…

• TimelyFreeze: Adaptive Parameter Freezing Mechanism for Pipeline Parallelism

  Seonghye Cho, Jaemin Han, Hyunjin Kim, Euisoo Jung, Jae-Gil Lee · 6 February 2026

  Pipeline parallelism enables training models that exceed single-device memory, but practical throughput remains limited by pipeline bubbles. Although parameter freezing can improve training throughput by adaptively skipping backward computation, existing methods often over-freeze parameters, resulti…

• VibeCodeHPC: An Agent-Based Iterative Prompting Auto-Tuner for HPC Code Generation Using LLMs

  Shun-ichiro Hayashi, Koki Morita, Daichi Mukunoki, Tetsuya Hoshino, Takahiro Katagiri · 6 February 2026

  We propose VibeCodeHPC, an automatic tuning system for HPC programs based on multi-agent LLMs for code generation. VibeCodeHPC tunes programs through multi-agent role allocation and iterative prompt refinement. We describe the system configuration with four roles: Project Manager (PM), System Engine…

• Hybrid Gated Flow (HGF): Stabilizing 1.58-bit LLMs via Selective Low-Rank Correction

  David Alejandro Trejo Pizzo · 6 February 2026

  The deployment of Large Language Models (LLMs) on edge devices is fundamentally constrained by the "Memory Wall" -- a hardware limitation where memory bandwidth, not compute, becomes the bottleneck. Recent 1.58-bit quantization techniques (e.g., BitNet b1.58) dramatically reduce memory footprint but…

• P-Tensors: a General Formalism for Constructing Higher Order Message Passing Networks

  Andrew Hands, Tianyi Sun, Risi Kondor · 5 February 2026

  Several recent papers have proposed increasing the expressive power of graph neural networks by exploiting subgraphs or other topological structures. In parallel, researchers have investigated higher order permutation equivariant networks. In this paper we tie these two threads together by providing…

• DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs

  Zeyu Zhu, Gang Li, Peisong Wang, Zitao Mo, Minnan Pei, Zhuoran Song, Xiaoyao Liang, Jian Cheng · 4 February 2026

  Mixture of Experts (MoE) architectures significantly enhance the capacity of LLMs without proportional increases in computation, but at the cost of a vast parameter size. Offloading MoE expert parameters to host memory and leveraging both CPU and GPU computation has recently emerged as a promising d…

• Every Bit Counts: A Theoretical Study of Precision-Expressivity Tradeoffs in Quantized Transformers

  Sayak Chakrabarti, Toniann Pitassi, Josh Alman · 4 February 2026

  Quantization reduces the numerical precision of Transformer computations and is widely used to accelerate inference, yet its effect on expressivity remains poorly characterized. We demonstrate a fine-grained theoretical tradeoff between expressivity and precision: For every p we exhibit a function {…

• Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotation

  Hyunji Jung, Sungbin Shin, Namhoon Lee · 4 February 2026

  Asynchronous pipeline parallelism maximizes hardware utilization by eliminating the pipeline bubbles inherent in synchronous execution, offering a path toward efficient large-scale distributed training. However, this efficiency gain can be compromised by gradient staleness, where the immediate model…

• Dissecting Outlier Dynamics in LLM NVFP4 Pretraining

  Peijie Dong, Ruibo Fan, Yuechen Tao, Di Mou, Wenhu Hu, Zhenheng Tang, Yinghao Yu, Jiamang Wang, Wenbo Su, Guodong Yang, Liping Zhang, Xiaowen Chu, Baochun Li, Bo Li · 3 February 2026

  Training large language models using 4-bit arithmetic enhances throughput and memory efficiency. Yet, the limited dynamic range of FP4 increases sensitivity to outliers. While NVFP4 mitigates quantization error via hierarchical microscaling, a persistent loss gap remains compared to BF16. This study…

• HyperOffload: Graph-Driven Hierarchical Memory Management for Large Language Models on SuperNode Architectures

  Fangxin Liu, Qinghua Zhang, Hanjing Shen, Qinghua Zhang, Zhibo Liang, Li Jiang, Haibing Guan, Chong Bao, Xuefeng Jin · 3 February 2026

  The rapid evolution of Large Language Models (LLMs) towards long-context reasoning and sparse architectures has pushed memory requirements far beyond the capacity of individual device HBM. While emerging supernode architectures offer terabyte-scale shared memory pools via high-bandwidth interconnect…

• Self-Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation

  Franz A. Heinsen, Leo Kozachkov · 3 February 2026

  The most widely used artificial intelligence (AI) models today are Transformers employing self-attention. In its standard form, self-attention incurs costs that increase with context length, driving demand for storage, compute, and energy that is now outstripping society's ability to provide them. T…

• Message Passing on the Edge: Towards Scalable and Expressive GNNs

  Pablo Barcel\'o, Fabian Jogl, Alexander Kozachinskiy, Matthias Lanzinger, Stefan Neumann, Crist\'obal Rojas · 3 February 2026

  Graph neural networks (GNNs) are widely used in graph learning and most architectures propagate information by passing messages between vertices. In this work, we shift our attention to GNNs that perform message passing on edges and introduce EB-1WL, an edge-based color-refinement test, and a corres…

• JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulation

  Yebin Yang, Huaijin Wu, Fu Guo, Lin Yao, Xiaohan Qin, Jingzhi Wang, Debing Zhang, Junchi Yan · 3 February 2026

  LLMs have traditionally scaled along dense dimensions, where performance is coupled with near-linear increases in computational cost. While MoE decouples capacity from compute, it introduces large memory overhead and hardware efficiency challenges. To overcome these, we propose token-indexed paramet…

• COLT: Lightweight Multi-LLM Collaboration through Shared MCTS Reasoning for Model Compilation

  Annabelle Sujun Tang, Christopher Priebe, Lianhui Qin, Hadi Esmaeilzadeh · 3 February 2026

  Model serving costs dominate AI systems, making compiler optimization essential for scalable deployment. Recent works show that a large language model (LLM) can guide compiler search by reasoning over program structure and optimization history. However, using a single large model throughout the sear…

• AnyBCQ: Hardware Efficient Flexible Binary-Coded Quantization for Multi-Precision LLMs

  Gunho Park, Jeongin Bae, Beomseok Kwon, Byeongwook Kim, Se Jung Kwon, Dongsoo Lee · 3 February 2026

  The deployment of large language models (LLMs) is increasingly constrained by memory and latency bottlenecks, motivating the need for quantization techniques that flexibly balance accuracy and efficiency. Recent work has introduced multi-precision models, which enable inference at multiple precision…

• BAPS: A Fine-Grained Low-Precision Scheme for Softmax in Attention via Block-Aware Precision reScaling

  Zisheng Ye, Xiaoyu He, Maoyuan Song, Guoliang Qiu, Chao Liao, Chen Wu, Yonggang Sun, Zhichun Li, Xiaoru Xie, Yuanyong Luo, Hu Liu, Pinyan Lu, Heng Liao · 3 February 2026

  As the performance gains from accelerating quantized matrix multiplication plateau, the softmax operation becomes the critical bottleneck in Transformer inference. This bottleneck stems from two hardware limitations: (1) limited data bandwidth between matrix and vector compute cores, and (2) the sig…

• MiniTensor: A Lightweight, High-Performance Tensor Operations Library

  Soumyadip Sarkar · 3 February 2026

  We present MiniTensor, an open source tensor operations library that focuses on minimalism, correctness, and performance. MiniTensor exposes a familiar PyTorch-like Python API while it executes performance critical code in a Rust engine. The core supports dense $n$ dimensional tensors, broadcasting,…

• SuperCoder: Assembly Program Superoptimization with Large Language Models

  Anjiang Wei, Tarun Suresh, Huanmi Tan, Yinglun Xu, Gagandeep Singh, Ke Wang, Alex Aiken · 2 February 2026

  Superoptimization is the task of transforming a program into a faster one while preserving its input-output behavior. In this work, we investigate whether large language models (LLMs) can serve as superoptimizers, generating assembly programs that outperform code already optimized by industry-standa…

• HetCCL: Accelerating LLM Training with Heterogeneous GPUs

  Heehoon Kim, Jaehwan Lee, Taejeoung Kim, Jongwon Park, Jinpyo Kim, Pyongwon Suh, Ryan H. Choi, Sangwoo Lee, Jaejin Lee · 2 February 2026

  The rapid growth of large language models is driving organizations to expand their GPU clusters, often with GPUs from multiple vendors. However, current deep learning frameworks lack support for collective communication across heterogeneous GPUs, leading to inefficiency and higher costs. We present …

• Introducing Instruction-Accurate Simulators for Performance Estimation of Autotuning Workloads

  Rebecca Pelke, Nils Bosbach, Lennart M. Reimann, Rainer Leupers · 30 January 2026

  Accelerating Machine Learning (ML) workloads requires efficient methods due to their large optimization space. Autotuning has emerged as an effective approach for systematically evaluating variations of implementations. Traditionally, autotuning requires the workloads to be executed on the target ha…