Parallel Computing and Optimization Techniques

Neueste Paper

• GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model Compression

  Kainan Liu, Yong Zhang, Ning Cheng, Zhitao Li, Shaojun Wang, Jing Xiao · 24. Februar 2026

  Recent studies have demonstrated that many layers are functionally redundant in large language models (LLMs), enabling model compression by removing these layers to reduce inference cost. While such approaches can improve efficiency, indiscriminate layer pruning often results in significant performa…

• Beyond Pass-by-Pass Optimization: Intent-Driven IR Optimization with Large Language Models

  Lei Qiu, Zi Yang, Fang Lyu, Ming Zhong, Huimin Cui, Xiaobing Feng · 24. Februar 2026

  Modern compilers optimize programs through a sequence of modular passes over intermediate representations (IR). While this pass-by-pass paradigm offers engineering benefits, it suffers from a pass coordination problem: locally beneficial transformations may block more profitable optimizations in lat…

• RPU -- A Reasoning Processing Unit

  Matthew Adiletta, Gu-Yeon Wei, David Brooks · 24. Februar 2026

  Large language model (LLM) inference performance is increasingly bottlenecked by the memory wall. While GPUs continue to scale raw compute throughput, they struggle to deliver scalable performance for memory bandwidth bound workloads. This challenge is amplified by emerging reasoning LLM application…

• Hardware-Friendly Input Expansion for Accelerating Function Approximation

  Hu Lou, Yin-Jun Gao, Dong-Xiao Zhang, Tai-Jiao Du, Jun-Jie Zhang, Jia-Rui Zhang · 23. Februar 2026

  One-dimensional function approximation is a fundamental problem in scientific computing and engineering applications. While neural networks possess powerful universal approximation capabilities, their optimization process is often hindered by flat loss landscapes induced by parameter-space symmetrie…

• A Case Study of Selected PTQ Baselines for Reasoning LLMs on Ascend NPU

  Yuchen Luo, Fangyue Zhu, Ruining Zhou, Mingzhe Huang, Jian Zhu, Fanyu Fan, Wei Shao · 23. Februar 2026

  Post-Training Quantization (PTQ) is crucial for efficient model deployment, yet its effectiveness on Ascend NPU remains under-explored compared to GPU architectures. This paper presents a case study of representative PTQ baselines applied to reasoning-oriented models such as DeepSeek-R1-Distill-Qwen…

• Pareto Optimal Benchmarking of AI Models on ARM Cortex Processors for Sustainable Embedded Systems

  Pranay Jain, Maximilian Kasper, G\"oran K\"ober, Axel Plinge, Dominik Seu{\ss} · 20. Februar 2026

  This work presents a practical benchmarking framework for optimizing artificial intelligence (AI) models on ARM Cortex processors (M0+, M4, M7), focusing on energy efficiency, accuracy, and resource utilization in embedded systems. Through the design of an automated test bench, we provide a systemat…

• The Compute ICE-AGE: Invariant Compute Envelope under Addressable Graph Evolution

  Raymond Jay Martin II · 20. Februar 2026

  This paper presents empirical results from a production-grade C++ implementation of a deterministic semantic state substrate derived from prior formal work on Bounded Local Generator Classes (Martin, 2026). The system was mathematically specified prior to implementation and realized as a CPU-residen…

• Energy Concerns with HPC Systems and Applications

  Roblex Nana, Claude Tadonki, Petr Dokladal, Youssef Mesri · 18. Februar 2026

  For various reasons including those related to climate changes, {\em energy} has become a critical concern in all relevant activities and technical designs. For the specific case of computer activities, the problem is exacerbated with the emergence and pervasiveness of the so called {\em intelligent…

• KernelBlaster: Continual Cross-Task CUDA Optimization via Memory-Augmented In-Context Reinforcement Learning

  Kris Shengjun Dong, Sahil Modi, Dima Nikiforov, Sana Damani, Edward Lin, Siva Kumar Sastry Hari, Christos Kozyrakis · 17. Februar 2026

  Optimizing CUDA code across multiple generations of GPU architectures is challenging, as achieving peak performance requires an extensive exploration of an increasingly complex, hardware-specific optimization space. Traditional compilers are constrained by fixed heuristics, whereas finetuning Large …

• QuEPT: Quantized Elastic Precision Transformers with One-Shot Calibration for Multi-Bit Switching

  Ke Xu, Yixin Wang, Zhongcheng Li, Hao Cui, Jinshui Hu, Xingyi Zhang · 16. Februar 2026

  Elastic precision quantization enables multi-bit deployment via a single optimization pass, fitting diverse quantization scenarios.Yet, the high storage and optimization costs associated with the Transformer architecture, research on elastic quantization remains limited, particularly for large langu…

• Multipole Semantic Attention: A Fast Approximation of Softmax Attention for Pretraining

  Rupert Mitchell, Kristian Kersting · 16. Februar 2026

  Pretraining transformers on long sequences (entire code repositories, collections of related documents) is bottlenecked by quadratic attention costs. We present Multipole Semantic Attention (MuSe), which accelerates 64k-context pretraining by 36% while matching baseline loss, requiring no architectu…

• CacheMind: From Miss Rates to Why -- Natural-Language, Trace-Grounded Reasoning for Cache Replacement

  Kaushal Mhapsekar, Azam Ghanbari, Bita Aslrousta, Samira Mirbagher-Ajorpaz · 16. Februar 2026

  Cache replacement remains a challenging problem in CPU microarchitecture, often addressed using hand-crafted heuristics, limiting cache performance. Cache data analysis requires parsing millions of trace entries with manual filtering, making the process slow and non-interactive. To address this, we …

• Unleashing Low-Bit Inference on Ascend NPUs: A Comprehensive Evaluation of HiFloat Formats

  Pengxiang Zhao, Hui-Ling Zhen, Xing Li, Han Bao, Weizhe Lin, Zhiyuan Yang, Ziwei Yu, Xin Wang, Mingxuan Yuan, Xianzhi Yu, Zhenhua Dong · 16. Februar 2026

  As LLMs scale, low-bit floating-point formats like MXFP and NVFP4 offer new opportunities for precision and efficiency. In this work, we evaluate HiFloat (HiF8 and HiF4), a family of formats tailored for Ascend NPUs. Through rigorous comparison across weight-activation and KV-cache tasks, we provide…

• OptiML: An End-to-End Framework for Program Synthesis and CUDA Kernel Optimization

  Arijit Bhattacharjee, Heng Ping, Son Vu Le, Paul Bogdan, Nesreen K. Ahmed, Ali Jannesari · 16. Februar 2026

  Generating high-performance CUDA kernels remains challenging due to the need to navigate a combinatorial space of low-level transformations under noisy and expensive hardware feedback. Although large language models can synthesize functionally correct CUDA code, achieving competitive performance req…

• HiFloat4 Format for Language Model Inference

  Yuanyong Luo, Jing Huang, Yu Cheng, Ziwei Yu, Kaihua Zhang, Kehong Hong, Xinda Ma, Xin Wang, Anping Tong, Guipeng Hu, Yun Xu, Mehran Taghian, Peng Wu, Guanglin Li, Yunke Peng, Tianchi Hu, Minqi Chen, Michael Bi Mi, Hu Liu, Xiping Zhou, Junsong Wang, Qiang Lin, Heng Liao · 13. Februar 2026

  This paper introduces HiFloat4 (HiF4), a block floating-point data format tailored for deep learning. Each HiF4 unit packs 64 4-bit elements with 32 bits of shared scaling metadata, averaging 4.5 bits per value. The metadata specifies a three-level scaling hierarchy, capturing inter- and intra-group…

• Towards Compressive and Scalable Recurrent Memory

  Yunchong Song, Jushi Kai, Liming Lu, Kaixi Qiu, Zhouhan Lin · 13. Februar 2026

  Transformers face a quadratic bottleneck in attention when scaling to long contexts. Recent approaches introduce recurrent memory to extend context beyond the current window, yet these often face a fundamental trade-off between theoretical principles and practical scalability. To address this, we in…

• PASCAL: A Phase-Aware Scheduling Algorithm for Serving Reasoning-based Large Language Models

  Eunyeong Cho, Jehyeon Bang, Ranggi Hwang, Minsoo Rhu · 13. Februar 2026

  The emergence of reasoning-based LLMs leveraging Chain-of-Thought (CoT) inference introduces new serving challenges, as their extended reasoning phases delay user-visible output and inflate Time-To-First-Token (TTFT). Existing LLM serving frameworks fail to distinguish between reasoning and answerin…

• EmbBERT: Attention Under 2 MB Memory

  Riccardo Bravin, Massimo Pavan, Hazem Hesham Yousef Shalby, Fabrizio Pittorino, Manuel Roveri · 12. Februar 2026

  Transformer architectures based on the attention mechanism have revolutionized natural language processing (NLP), driving major breakthroughs across virtually every NLP task. However, their substantial memory and computational requirements still hinder deployment on ultra-constrained devices such as…

• QTALE: Quantization-Robust Token-Adaptive Layer Execution for LLMs

  Kanghyun Noh, Jinheon Choi, Yulwha Kim · 12. Februar 2026

  Large language models (LLMs) demand substantial computational and memory resources, posing challenges for efficient deployment. Two complementary approaches have emerged to address these issues: token-adaptive layer execution, which reduces floating-point operations (FLOPs) by selectively bypassing …

• SnapMLA: Efficient Long-Context MLA Decoding via Hardware-Aware FP8 Quantized Pipelining

  Yifan Zhang, Zunhai Su, Shuhao Hu, Rui Yang, Wei Wu, Yulei Qian, Yuchen Xie, Xunliang Cai · 12. Februar 2026

  While FP8 attention has shown substantial promise in innovations like FlashAttention-3, its integration into the decoding phase of the DeepSeek Multi-head Latent Attention (MLA) architecture presents notable challenges. These challenges include numerical heterogeneity arising from the decoupling of …

• PRISM: Parallel Residual Iterative Sequence Model

  Jie Jiang, Ke Cheng, Xin Xu, Mengyang Pang, Tianhao Lu, Jiaheng Li, Yue Liu, Yuan Wang, Jun Zhang, Huan Yu, Zhouchen Lin · 12. Februar 2026

  Generative sequence modeling faces a fundamental tension between the expressivity of Transformers and the efficiency of linear sequence models. Existing efficient architectures are theoretically bounded by shallow, single-step linear updates, while powerful iterative methods like Test-Time Training …

• Fine-Tuning GPT-5 for GPU Kernel Generation

  Ali Tehrani, Yahya Emara, Essam Wissam, Wojciech Paluch, Waleed Atallah, {\L}ukasz Dudziak, Mohamed S. Abdelfattah · 12. Februar 2026

  Developing efficient GPU kernels is essential for scaling modern AI systems, yet it remains a complex task due to intricate hardware architectures and the need for specialized optimization expertise. Although Large Language Models (LLMs) demonstrate strong capabilities in general sequential code gen…

• Flash-SD-KDE: Accelerating SD-KDE with Tensor Cores

  Elliot L. Epstein, Rajat Vadiraj Dwaraknath, John Winnicki · 12. Februar 2026

  Score-debiased kernel density estimation (SD-KDE) achieves improved asymptotic convergence rates over classical KDE, but its use of an empirical score has made it significantly slower in practice. We show that by re-ordering the SD-KDE computation to expose matrix-multiplication structure, Tensor Co…

• RuleFlow : Generating Reusable Program Optimizations with LLMs

  Avaljot Singh, Dushyant Bharadwaj, Stefanos Baziotis, Kaushik Varadharajan, Charith Mendis · 11. Februar 2026

  Optimizing Pandas programs is a challenging problem. Existing systems and compiler-based approaches offer reliability but are either heavyweight or support only a limited set of optimizations. Conversely, using LLMs in a per-program optimization methodology can synthesize nontrivial optimizations, b…

• LLM Serving Optimization with Variable Prefill and Decode Lengths

  Meixuan Wang, Yinyu Ye, Zijie Zhou · 11. Februar 2026

  We study offline scheduling for large language model (LLM) serving under a fixed KV-cache memory budget, where requests have heterogeneous prompt (prefill) and response (decode) lengths. Prompt tokens determine initial KV usage, and each generated token increases memory by one unit. Given a backlog …