Parallel Computing and Optimization Techniques

Neueste Paper

• LUMINA: LLM-Guided GPU Architecture Exploration via Bottleneck Analysis

  Tao Zhang, Rui Ma, Shuotao Xu, Peng Cheng, Yongqiang Xiong · 9. März 2026

  GPU design space exploration (DSE) for modern AI workloads, such as Large-Language Model (LLM) inference, is challenging because of GPUs' vast, multi-modal design spaces, high simulation costs, and complex design optimization objectives (e.g. performance, power and area trade-offs). Existing automat…

• A Persistent-State Dataflow Accelerator for Memory-Bound Linear Attention Decode on FPGA

  Neelesh Gupta, Peter Wang, Rajgopal Kannan, Viktor K. Prasanna · 9. März 2026

  Gated DeltaNet (GDN) is a linear attention mechanism that replaces the growing KV cache with a fixed-size recurrent state. Hybrid LLMs like Qwen3-Next use 75% GDN layers and achieve competitive accuracy to attention-only models. However, at batch-1, GDN decode is memory-bound on GPUs since the full …

• Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks

  Burak Topcu, Musa Oguzhan Cim, Poovaiah Palangappa, Meena Arunachalam, Mahmut Taylan Kandemir · 9. März 2026

  Breakthroughs in the generative AI domain have fueled an explosion of large language model (LLM)-powered applications, whose workloads fundamentally consist of sequences of inferences through transformer architectures. Within this rapidly expanding ecosystem, dense LLMs--those that activate all mode…

• Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices

  Yakov Pyotr Shkolnikov · 6. März 2026

  Multi-agent LLM systems on edge devices face a memory management problem: device RAM is too small to hold every agent's KV cache simultaneously. On Apple M4 Pro with 10.2 GB of cache budget, only 3 agents fit at 8K context in FP16. A 10-agent workflow must constantly evict and reload caches. Without…

• Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection

  Hengshuai Yao, Guan Wang · 6. März 2026

  Standard transformer attention uses identical dimensionality for queries, keys, and values ($d_q = d_k = d_v = \dmodel$). Our insight is that these components serve fundamentally different roles, and this symmetry is unnecessary. Queries and keys produce scalar attention weights (\emph{selection}), …

• Why Are Linear RNNs More Parallelizable?

  William Merrill, Hongjian Jiang, Yanhong Li, Ashish Sabharwal · 5. März 2026

  The community is increasingly exploring linear RNNs (LRNNs) as language models, motivated by their expressive power and parallelizability. While prior work establishes the expressivity benefits of LRNNs over transformers, it is unclear what makes LRNNs -- but not traditional, nonlinear RNNs -- as ea…

• ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problems

  Egor Cherepanov, Alexey K. Kovalev, Aleksandr I. Panov · 5. März 2026

  Real-world robotic agents must act under partial observability and long horizons, where key cues may appear long before they affect decision making. However, most modern approaches rely solely on instantaneous information, without incorporating insights from the past. Standard recurrent or transform…

• InstMeter: An Instruction-Level Method to Predict Energy and Latency of DL Model Inference on MCUs

  Hao Liu, Qing Wang, Marco Zuniga · 5. März 2026

  Deep learning (DL) models can now run on microcontrollers (MCUs). Through neural architecture search (NAS), we can search DL models that meet the constraints of MCUs. Among various constraints, energy and latency costs of the model inference are critical metrics. To predict them, existing research r…

• Joint Hardware-Workload Co-Optimization for In-Memory Computing Accelerators

  Olga Krestinskaya, Mohammed E. Fouda, Ahmed Eltawil, Khaled N. Salama · 5. März 2026

  Software-hardware co-design is essential for optimizing in-memory computing (IMC) hardware accelerators for neural networks. However, most existing optimization frameworks target a single workload, leading to highly specialized hardware designs that do not generalize well across models and applicati…

• CUCo: An Agentic Framework for Compute and Communication Co-design

  Bodun Hu, Yoga Sri Varshan V, Saurabh Agarwal, Aditya Akella · 4. März 2026

  Custom CUDA kernel development is essential for maximizing GPU utilization in large-scale distributed LLM training and inference, yet manually writing kernels that jointly leverage both computation and communication remains a labor-intensive and error-prone process. Prior work on kernel optimization…

• SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Serving

  Sunghyeon Woo, Ahreum Seo, Jaegwang Lee, Jaeeun Kil, Hanbae Seo, Joonghoon Kim, Baeseong Park, Se Jung Kwon, Dongsoo Lee · 4. März 2026

  In multi-model LLM serving, decode execution remains inefficient due to model-specific resource partitioning: since cross-model batching is not possible, memory-bound decoding often suffers from severe GPU underutilization, especially under skewed workloads. We propose Shared Use of Next-token Predi…

• ParEVO: Synthesizing Code for Irregular Data: High-Performance Parallelism through Agentic Evolution

  Liu Yang, Zeyu Nie, Andrew Liu, Felix Zou, Deniz Altinb\"uken, Amir Yazdanbakhsh, Quanquan C. Liu · 4. März 2026

  The transition from sequential to parallel computing is essential for modern high-performance applications but is hindered by the steep learning curve of concurrent programming. This challenge is magnified for irregular data structures (such as sparse graphs, unbalanced trees, and non-uniform meshes…

• PiKV: KV Cache Management System for Mixture of Experts

  Dong Liu, Yanxuan Yu, Ben Lengerich, Ying Nian Wu, Xuhong Wang · 3. März 2026

  As large language models continue to scale up in both size and context length, the memory and communication cost of key-value (KV) cache storage has become a major bottleneck in multi-GPU and multi-node inference. While MoE-based architectures sparsify computation across experts, the corresponding K…

• Motivating Next-Gen Accelerators with Flexible (N:M) Activation Sparsity via Benchmarking Lightweight Post-Training Sparsification Approaches

  Shirin Alanova, Kristina Kazistova, Ekaterina Galaeva, Alina Kostromina, Vladimir Smirnov, Redko Dmitry, Alexey Dontsov, Maxim Zhelnin, Evgeny Burnaev, Egor Shvetsov · 3. März 2026

  The demand for efficient large language model (LLM) inference has intensified the focus on sparsification techniques. While semi-structured (N:M) pruning is well-established for weights, its application to activation pruning remains underexplored despite its potential for dynamic, input-adaptive com…

• ROMA: a Read-Only-Memory-based Accelerator for QLoRA-based On-Device LLM

  Wenqiang Wang, Yijia Zhang, Zikai Zhang, Guanting Huo, Hao Liang, Shijie Cao, Ningyi Xu · 3. März 2026

  As large language models (LLMs) demonstrate powerful capabilities, deploying them on edge devices has become increasingly crucial, offering advantages in privacy and real-time interaction. QLoRA has emerged as the standard approach for on-device LLMs, leveraging quantized models to reduce memory and…

• MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core

  Dennis Liu, Zijie Yan, Xin Yao, Tong Liu, Vijay Korthikanti, Evan Wu, Shiqing Fan, Gao Deng, Hongxiao Bai, Jianbin Chang, Ashwath Aithal, Michael Andersch, Mohammad Shoeybi, Jiajie Yao, Chandler Zhou, David Wu, Xipeng Li, June Yang · 3. März 2026

  Mixture of Experts (MoE) models enhance neural network scalability by dynamically selecting relevant experts per input token, enabling larger model sizes while maintaining manageable computation costs. However, efficient training of large-scale MoE models across thousands of GPUs presents significan…

• OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration

  Xinyue Ma, Heelim Hong, Taegeon Um, Jongseop Lee, Seoyeong Choy, Woo-Yeon Lee, Myeongjae Jeon · 3. März 2026

  Serving long-context LLMs is challenging because request lengths and batch composition vary during token generation, causing the memory footprint to fluctuate significantly at runtime. Offloading KV caches to host memory limits effective memory usage, but existing static and predetermined offloading…

• CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation

  Weinan Dai, Hanlin Wu, Qiying Yu, Huan-ang Gao, Jiahao Li, Chengquan Jiang, Weiqiang Lou, Yufan Song, Hongli Yu, Jiaze Chen, Wei-Ying Ma, Ya-Qin Zhang, Jingjing Liu, Mingxuan Wang, Xin Liu, Hao Zhou · 2. März 2026

  GPU kernel optimization is fundamental to modern deep learning but remains a highly specialized task requiring deep hardware expertise. Despite strong performance in general programming, large language models (LLMs) remain uncompetitive with compiler-based systems such as torch.compile for CUDA kern…

• InnerQ: Hardware-aware Tuning-free Quantization of KV Cache for Large Language Models

  Sayed Mohammadreza Tayaranian Hosseini, Amir Ardakani, Warren J. Gross · 27. Februar 2026

  Reducing the hardware footprint of large language models (LLMs) during decoding is critical for efficient long-sequence generation. A key bottleneck is the key-value (KV) cache, whose size scales with sequence length and easily dominates the memory footprint of the model. Previous work proposed quan…

• ArchAgent: Agentic AI-driven Computer Architecture Discovery

  Raghav Gupta, Akanksha Jain, Abraham Gonzalez, Alexander Novikov, Po-Sen Huang, Matej Balog, Marvin Eisenberger, Sergey Shirobokov, Ng\^an V\~u, Martin Dixon, Borivoje Nikoli\'c, Parthasarathy Ranganathan, Sagar Karandikar · 27. Februar 2026

  Agile hardware design flows are a critically needed force multiplier to meet the exploding demand for compute. Recently, agentic generative AI systems have demonstrated significant advances in algorithm design, improving code efficiency, and enabling discovery across scientific domains. Bridging t…

• S2O: Early Stopping for Sparse Attention via Online Permutation

  Yu Zhang, Songwei Liu, Chenqian Yan, Sheng Lin, Beichen Ning, Fangmin Chen, Xing Wang · 27. Februar 2026

  Attention scales quadratically with sequence length, fundamentally limiting long-context inference. Existing block-granularity sparsification can reduce latency, but coarse blocks impose an intrinsic sparsity ceiling, making further improvements difficult even with carefully engineered designs. We p…

• LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure

  Jaehong Cho, Hyunmin Choi, Guseul Heo, Jongse Park · 27. Februar 2026

  Large language model (LLM) serving infrastructures are undergoing a shift toward heterogeneity and disaggregation. Modern deployments increasingly integrate diverse accelerators and near-memory processing technologies, introducing significant hardware heterogeneity, while system software increasingl…

• AutoQRA: Joint Optimization of Mixed-Precision Quantization and Low-rank Adapters for Efficient LLM Fine-Tuning

  Changhai Zhou, Shiyang Zhang, Yuhua Zhou, Qian Qiao, Jun Gao, Cheng Jin, Kaizhou Qin, Weizhong Zhang · 27. Februar 2026

  Quantization followed by parameter-efficient fine-tuning has emerged as a promising paradigm for downstream adaptation under tight GPU memory constraints. However, this sequential pipeline fails to leverage the intricate interaction between quantization bit-width and LoRA rank. Specifically, a caref…

• Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking

  Ravi Ghadia, Maksim Abraham, Sergei Vorobyov, Max Ryabinin · 25. Februar 2026

  Efficiently processing long sequences with Transformer models usually requires splitting the computations across accelerators via context parallelism. The dominant approaches in this family of methods, such as Ring Attention or DeepSpeed Ulysses, enable scaling over the context dimension but do not …

• Scaling State-Space Models on Multiple GPUs with Tensor Parallelism

  Anurag Dutt, Nimit Shah, Hazem Masarani, Anshul Gandhi · 25. Februar 2026

  Selective state space models (SSMs) have rapidly become a compelling backbone for large language models, especially for long-context workloads. Yet in deployment, their inference performance is often bounded by the memory capacity, bandwidth, and latency limits of a single GPU, making multi-GPU exec…