Parallel Computing and Optimization Techniques

Últimos artículos

• CARMA: Collocation-Aware Resource Manager

  Ehsan Yousefzadeh-Asl-Miandoab, Reza Karimzadeh, Bulat Ibragimov, Florina M. Ciorba, P{\i}nar T\"oz\"un · 4 de noviembre de 2025

  GPUs running deep learning (DL) workloads are frequently underutilized. Collocating multiple DL training tasks on the same GPU can improve utilization but introduces two key risks: (1) out-of-memory (OOM) crashes for newly scheduled tasks, and (2) severe performance interference among co-running tas…

• FlexQ: Efficient Post-training INT6 Quantization for LLM Serving via Algorithm-System Co-Design

  Hao Zhang, Aining Jia, Weifeng Bu, Yushu Cai, Kai Sheng, Hao Chen, Xin He · 4 de noviembre de 2025

  Large Language Models (LLMs) demonstrate exceptional performance but entail significant memory and computational costs, restricting their practical deployment. While existing INT4/INT8 quantization reduces these costs, they often degrade accuracy or lack optimal efficiency. INT6 quantization offers …

• AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUs

  Ran Yan, Youhe Jiang, Tianyuan Wu, Jiaxuan Gao, Zhiyu Mei, Wei Fu, Haohui Mai, Wei Wang, Yi Wu, Binhang Yuan · 4 de noviembre de 2025

  Maximizing training throughput and cost-efficiency of RL for LLMs is essential to democratize this advanced technique. One promising but challenging approach is to deploy such a computational workflow over heterogeneous GPUs. Unlike conventional large-scale LLM pretraining, RL training generally dec…

• Agentic Auto-Scheduling: An Experimental Study of LLM-Guided Loop Optimization

  Massinissa Merouani, Islem Kara Bernou, Riyadh Baghdadi · 4 de noviembre de 2025

  Automatic code optimization remains a difficult challenge, particularly for complex loop nests on modern hardware. This paper investigates a novel approach to code optimization where Large Language Models (LLMs) guide the process through a closed-loop interaction with a compiler. We present ComPilot…

• On the Structure of Floating-Point Noise in Batch-Invariant GPU Matrix Multiplication

  Tadisetty Sai Yashwanth · 4 de noviembre de 2025

  Floating-point non-associativity makes fundamental deep learning operations, such as matrix multiplication (matmul) on GPUs, inherently non-deterministic. Despite this, the statistical structure of the resulting numerical error remains poorly understood. A common working assumption is that these err…

• Memory-Efficient Training with In-Place FFT Implementation

  Xinyu Ding, Bangtian Liu, Siyu Liao, Zhongfeng Wang · 4 de noviembre de 2025

  Fast Fourier Transforms (FFT) are widely used to reduce memory and computational costs in deep learning. However, existing implementations, including standard FFT and real FFT (rFFT), cannot achieve true in-place computation. In particular, rFFT maps an input of size n to a complex output of size n/…

• FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Management

  Nazmul Takbir, Hamidreza Alikhani, Nikil Dutt, Sangeetha Abdu Jyothi · 4 de noviembre de 2025

  Large Language Model (LLM) serving is increasingly constrained by the growing size of the key-value (KV) cache, which scales with both context length and generation length. Prior work shows that attention is dominated by a small subset of critical tokens, yet existing systems struggle to exploit thi…

• PDE-SHARP: PDE Solver Hybrids Through Analysis & Refinement Passes

  Shaghayegh Fazliani, Madeleine Udell · 4 de noviembre de 2025

  Current LLM-driven approaches using test-time computing to generate PDE solvers execute a large number of solver samples to identify high-accuracy solvers. These paradigms are especially costly for complex PDEs requiring substantial computational resources for numerical evaluation. We introduce PDE-…

• Resource-Adaptive Successive Doubling for Hyperparameter Optimization with Large Datasets on High-Performance Computing Systems

  Marcel Aach, Rakesh Sarma, Helmut Neukirchen, Morris Riedel, Andreas Lintermann · 3 de noviembre de 2025

  On High-Performance Computing (HPC) systems, several hyperparameter configurations can be evaluated in parallel to speed up the Hyperparameter Optimization (HPO) process. State-of-the-art HPO methods follow a bandit-based approach and build on top of successive halving, where the final performance o…

• Kimi Linear: An Expressive, Efficient Attention Architecture

  Kimi Team, Yu Zhang, Zongyu Lin, Xingcheng Yao, Jiaxi Hu, Fanqing Meng, Chengyin Liu, Xin Men, Songlin Yang, Zhiyuan Li, Wentao Li, Enzhe Lu, Weizhou Liu, Yanru Chen, Weixin Xu, Longhui Yu, Yejie Wang, Yu Fan, Longguang Zhong, Enming Yuan, Dehao Zhang, Yizhi Zhang, T. Y. Liu, Haiming Wang, Shengjun Fang, Weiran He, Shaowei Liu, Yiwei Li, Jianlin Su, Jiezhong Qiu, Bo Pang, Junjie Yan, Zhejun Jiang, Weixiao Huang, Bohong Yin, Jiacheng You, Chu Wei, Zhengtao Wang, Chao Hong, Yutian Chen, Guanduo Chen, Yucheng Wang, Huabin Zheng, Feng Wang, Yibo Liu, Mengnan Dong, Zheng Zhang, Siyuan Pan, Wenhao Wu, Yuhao Wu, Longyu Guan, Jiawen Tao, Guohong Fu, Xinran Xu, Yuzhi Wang, Guokun Lai, Yuxin Wu, Xinyu Zhou, Zhilin Yang, Yulun Du · 31 de octubre de 2025

  We introduce Kimi Linear, a hybrid linear attention architecture that, for the first time, outperforms full attention under fair comparisons across various scenarios -- including short-context, long-context, and reinforcement learning (RL) scaling regimes. At its core lies Kimi Delta Attention (KDA)…

• 3D Optimization for AI Inference Scaling: Balancing Accuracy, Cost, and Latency

  Minseok Jung, Abhas Ricky, Muhammad Rameez Chatni · 30 de octubre de 2025

  AI inference scaling is often tuned through 1D heuristics (a fixed reasoning passes) or 2D bivariate trade-offs (e.g., performance vs. compute), which fail to consider cost and latency constraints. We introduce a 3D optimization framework that jointly calibrates accuracy, cost, and latency within a …

• Trade-offs in Data Memorization via Strong Data Processing Inequalities

  Vitaly Feldman, Guy Kornowski, Xin Lyu · 29 de octubre de 2025

  Recent research demonstrated that training large language models involves memorization of a significant fraction of training data. Such memorization can lead to privacy violations when training on sensitive user data and thus motivates the study of data memorization's role in learning. In this work,…

• Efficient Low Rank Attention for Long-Context Inference in Large Language Models

  Tenghui Li, Guoxu Zhou, Xuyang Zhao, Yuning Qiu, Qibin Zhao · 29 de octubre de 2025

  As the length of input text grows, the key-value (KV) cache in LLMs imposes prohibitive GPU memory costs and limits long-context inference on resource constrained devices. Existing approaches, such as KV quantization and pruning, reduce memory usage but suffer from numerical precision loss or subopt…

• Block-Diagonal LoRA for Eliminating Communication Overhead in Tensor Parallel LoRA Serving

  Xinyu Wang, Jonas M. K\"ubler, Kailash Budhathoki, Yida Wang, Matth\"aus Kleindessner · 28 de octubre de 2025

  When serving a single base LLM with several different LoRA adapters simultaneously, the adapters cannot simply be merged with the base model's weights as the adapter swapping would create overhead and requests using different adapters could not be batched. Rather, the LoRA computations have to be se…

• Transformer Based Linear Attention with Optimized GPU Kernel Implementation

  Armin Gerami, Ramani Duraiswami · 28 de octubre de 2025

  The original softmax-based attention mechanism (regular attention) in the extremely successful Transformer architecture computes attention between $N$ tokens, each embedded in a $D$-dimensional head, with a time complexity of $O(N^2D)$. Given the success of Transformers, improving their runtime duri…

• Batch Speculative Decoding Done Right

  Ranran Haoran Zhang, Soumik Dey, Ashirbad Mishra, Hansi Wu, Binbin Li, Rui Zhang · 28 de octubre de 2025

  Speculative decoding speeds up LLM inference by using a small draft model to propose multiple tokens that a target model verifies in parallel. Extending this idea to batches is essential for production serving, but it introduces the ragged tensor problem: sequences in the same batch accept different…

• Backward-Friendly Optimization: Training Large Language Models with Approximate Gradients under Memory Constraints

  Jing Yang, Kaitong Cai, Yijia Fan, Yufeng Yang, Keze Wang · 28 de octubre de 2025

  Full fine-tuning of Large Language Models (LLMs) is notoriously memory-intensive, primarily because conventional optimizers such as SGD or Adam assume access to exact gradients derived from cached activations. Existing solutions either alter the model architecture (e.g., reversible networks) or trad…

• Performance Trade-offs of Optimizing Small Language Models for E-Commerce

  Josip Tomo Licardo, Nikola Tankovic · 28 de octubre de 2025

  Large Language Models (LLMs) offer state-of-the-art performance in natural language understanding and generation tasks. However, the deployment of leading commercial models for specialized tasks, such as e-commerce, is often hindered by high computational costs, latency, and operational expenses. Th…

• xMem: A CPU-Based Approach for Accurate Estimation of GPU Memory in Deep Learning Training Workloads

  Jiabo Shi, Dimitrios Pezaros, Yehia Elkhatib · 27 de octubre de 2025

• Accelerating Mobile Inference through Fine-Grained CPU-GPU Co-Execution

  Zhuojin Li, Marco Paolieri, Leana Golubchik · 27 de octubre de 2025

• Sparser Block-Sparse Attention via Token Permutation

  Xinghao Wang, Pengyu Wang, Dong Zhang, Chenkun Tan, Shaojun Zhou, Zhaoxiang Liu, Shiguo Lian, Fangxu Liu, Kai Song, Xipeng Qiu · 27 de octubre de 2025

• GPU Memory Requirement Prediction for Deep Learning Task Based on Bidirectional Gated Recurrent Unit Optimization Transformer

  Chao Wang, Zhizhao Wen, Ruoxin Zhang, Puyang Xu, Yifan Jiang · 27 de octubre de 2025

• REx86: A Local Large Language Model for Assisting in x86 Assembly Reverse Engineering

  Darrin Lea, James Ghawaly, Golden Richard III, Aisha Ali-Gombe, Andrew Case · 27 de octubre de 2025