Parallel Computing and Optimization Techniques

Últimos artículos

• STEM: Scaling Transformers with Embedding Modules

  Ranajoy Sadhukhan, Sheng Cao, Harry Dong, Changsheng Zhao, Attiano Purpura-Pontoniere, Yuandong Tian, Zechun Liu, Beidi Chen · 16 de enero de 2026

  Fine-grained sparsity promises higher parametric capacity without proportional per-token compute, but often suffers from training instability, load balancing, and communication overhead. We introduce STEM (Scaling Transformers with Embedding Modules), a static, token-indexed approach that replaces t…

• Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implications

  Jiaxi Li, Yue Zhu, Eun Kyung Lee, Klara Nahrstedt · 15 de enero de 2026

  Different from traditional Large Language Model (LLM) serving that colocates the prefill and decode stages on the same GPU, disaggregated serving dedicates distinct GPUs to prefill and decode workload. Once the prefill GPU completes its task, the KV cache must be transferred to the decode GPU. While…

• ELUTQ: Optimizing Quantization Accuracy under LUT-Based Computation for Edge LLMs

  Xin Nie, Liang Dong, Haicheng Zhang, Jiawang Xiao, G. Sun · 15 de enero de 2026

  Weight quantization effectively reduces memory consumption and enable the deployment of Large Language Models on edge devices, yet existing hardware-friendly methods often rely on uniform quantization, which suffers from poor weight-distribution fitting and high dequantization overhead under low-bit…

• Bridging the Gap: Empowering Small Models in Reliable OpenACC-based Parallelization via GEPA-Optimized Prompting

  Samyak Jhaveri, Cristina V. Lopes · 15 de enero de 2026

  OpenACC lowers the barrier to GPU offloading, but writing high-performing pragma remains complex, requiring deep domain expertise in memory hierarchies, data movement, and parallelization strategies. Large Language Models (LLMs) present a promising potential solution for automated parallel code gene…

• EvoFSM: Controllable Self-Evolution for Deep Research with Finite State Machines

  Shuo Zhang, Chaofa Yuan, Ryan Guo, Xiaomin Yu, Rui Xu, Zhangquan Chen, Zinuo Li, Zhi Yang, Shuhao Guan, Zhenheng Tang, Sen Hu, Liwen Zhang, Ronghao Chen, Huacan Wang · 15 de enero de 2026

  While LLM-based agents have shown promise for deep research, most existing approaches rely on fixed workflows that struggle to adapt to real-world, open-ended queries. Recent work therefore explores self-evolution by allowing agents to rewrite their own code or prompts to improve problem-solving abi…

• A Machine Learning Approach Towards Runtime Optimisation of Matrix Multiplication

  Yufan Xia, Marco De La Pierre, Amanda S. Barnard, Giuseppe Maria Junior Barca · 15 de enero de 2026

  The GEneral Matrix Multiplication (GEMM) is one of the essential algorithms in scientific computing. Single-thread GEMM implementations are well-optimised with techniques like blocking and autotuning. However, due to the complexity of modern multi-core shared memory systems, it is challenging to det…

• Parallelizable memory recurrent units

  Florent De Geeter, Gaspard Lambrechts, Damien Ernst, Guillaume Drion · 15 de enero de 2026

  With the emergence of massively parallel processing units, parallelization has become a desirable property for new sequence models. The ability to parallelize the processing of sequences with respect to the sequence length during training is one of the main factors behind the uprising of the Transfo…

• KVzap: Fast, Adaptive, and Faithful KV Cache Pruning

  Simon Jegou, Maximilian Jeblick · 14 de enero de 2026

  Growing context lengths in transformer-based language models have made the key-value (KV) cache a critical inference bottleneck. While many KV cache pruning methods have been proposed, they have not yet been adopted in major inference engines due to speed--accuracy trade-offs. We introduce KVzap, a …

• Reducing Compute Waste in LLMs through Kernel-Level DVFS

  Jeffrey Spaan, Kuan-Hsun Chen, Ana-Lucia Varbanescu · 14 de enero de 2026

  The rapid growth of AI has fueled the expansion of accelerator- or GPU-based data centers. However, the rising operational energy consumption has emerged as a critical bottleneck and a major sustainability concern. Dynamic Voltage and Frequency Scaling (DVFS) is a well-known technique used to reduce…

• ZeroDVFS: Zero-Shot LLM-Guided Core and Frequency Allocation for Embedded Platforms

  Mohammad Pivezhandi, Mahdi Banisharif, Abusayeed Saifullah, Ali Jannesari · 14 de enero de 2026

  Dynamic voltage and frequency scaling (DVFS) and task-to-core allocation are critical for thermal management and balancing energy and performance in embedded systems. Existing approaches either rely on utilization-based heuristics that overlook stall times, or require extensive offline profiling for…

• Mosaic: Unlocking Long-Context Inference for Diffusion LLMs via Global Memory Planning and Dynamic Peak Taming

  Liang Zheng, Bowen Shi, Yitao Hu, Jiawei Zhang, Ruofan Li, Sheng Chen, Wenxin Li, Keqiu Li · 13 de enero de 2026

  Diffusion-based large language models (dLLMs) have emerged as a promising paradigm, utilizing simultaneous denoising to enable global planning and iterative refinement. While these capabilities are particularly advantageous for long-context generation, deploying such models faces a prohibitive memor…

• AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving

  Tianhao Xu, Yiming Liu, Xianglong Lu, Yijia Zhao, Xuting Zhou, Aichen Feng, Yiyi Chen, Yi Shen, Qin Zhou, Xumeng Chen, Ilya Sherstyuk, Haorui Li, Rishi Thakkar, Ben Hamm, Yuanzhe Li, Xue Huang, Wenpeng Wu, Anish Shanbhag, Harry Kim, Chuan Chen, Junjie Lai · 13 de enero de 2026

  Optimizing Large Language Model (LLM) inference in production systems is increasingly difficult due to dynamic workloads, stringent latency/throughput targets, and a rapidly expanding configuration space. This complexity spans not only distributed parallelism strategies (tensor/pipeline/expert) but …

• HiDVFS: A Hierarchical Multi-Agent DVFS Scheduler for OpenMP DAG Workloads

  Mohammad Pivezhandi, Abusayeed Saifullah, Ali Jannesari · 13 de enero de 2026

  With advancements in multicore embedded systems, leakage power, exponentially tied to chip temperature, has surpassed dynamic power consumption. Energy-aware solutions use dynamic voltage and frequency scaling (DVFS) to mitigate overheating in performance-intensive scenarios, while software approach…

• MoEBlaze: Breaking the Memory Wall for Efficient MoE Training on Modern GPUs

  Jiyuan Zhang, Yining Liu, Siqi Yan, Lisen Deng, Jennifer Cao, Shuqi Yang, Min Ni, Bi Xue, Shen Li · 12 de enero de 2026

  The pervasive "memory wall" bottleneck is significantly amplified in modern large-scale Mixture-of-Experts (MoE) architectures. MoE's inherent architectural sparsity leads to sparse arithmetic compute and also introduces substantial activation memory overheads -- driven by large token routing buffer…

• MaxCode: A Max-Reward Reinforcement Learning Framework for Automated Code Optimization

  Jiefu Ou, Sapana Chaudhary, Kaj Bostrom, Nathaniel Weir, Shuai Zhang, Huzefa Rangwala, George Karypis · 12 de enero de 2026

  Large Language Models (LLMs) demonstrate strong capabilities in general coding tasks but encounter two key challenges when optimizing code: (i) the complexity of writing optimized code (such as performant CUDA kernels and competition-level CPU code) requires expertise in systems, algorithms and spec…

• ParaCodex: A Profiling-Guided Autonomous Coding Agent for Reliable Parallel Code Generation and Translation

  Erel Kaplan, Tomer Bitan, Lian Ghrayeb, Le Chen, Tom Yotam, Niranjan Hasabnis, Gal Oren · 9 de enero de 2026

  Parallel programming is central to HPC and AI, but producing code that is correct and fast remains challenging, especially for OpenMP GPU offload, where data movement and tuning dominate. Autonomous coding agents can compile, test, and profile on target hardware, but outputs are brittle without doma…

• GPU-Accelerated INT8 Quantization for KV Cache Compression in Large Language Models

  Maanas Taneja, Purab Shingvi · 9 de enero de 2026

  The key-value (KV) cache in large language models presents a significant memory bottleneck during inference, growing linearly with sequence length and often exceeding the memory footprint of model weights themselves. We implement and evaluate GPU-accelerated INT8 quantization for KV cache compressio…

• Bare-Metal Tensor Virtualization: Overcoming the Memory Wall in Edge-AI Inference on ARM64

  Bugra Kilictas, Faruk Alpay · 8 de enero de 2026

  The deployment of Large Language Models (LLMs) on edge devices is fundamentally constrained by the "Memory Wall" the bottleneck where data movement latency outstrips arithmetic throughput. Standard inference runtimes often incur significant overhead through high-level abstractions, dynamic dispatch,…

• Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoC

  Xinming Wei, Jiahao Zhang, Haoran Li, Jiayu Chen, Haoning Guan, Rui Qu, Maoliang Li, Xiang Chen, Guojie Luo · 7 de enero de 2026

  Personal LLM agents increasingly combine foreground reactive interactions with background proactive monitoring, forming long-lived, stateful LLM flows that interleave prefill and token-by-token decode. While modern heterogeneous SoCs integrate CPUs, iGPUs, and NPUs to support on-device intelligence,…

• Chronicals: A High-Performance Framework for LLM Fine-Tuning with 3.51x Speedup over Unsloth

  Arjun S. Nair · 7 de enero de 2026

  Large language model fine-tuning is bottlenecked by memory: a 7B parameter model requires 84GB--14GB for weights, 14GB for gradients, and 56GB for FP32 optimizer states--exceeding even A100-40GB capacity. We present Chronicals, an open-source training framework achieving 3.51x speedup over Unsloth t…

• KVCrush: Key value cache size-reduction using similarity in head-behaviour

  Gopi Krishna Jha, Sameh Gobriel, Liubov Talamanova, Nilesh Jain · 6 de enero de 2026

  Key-value (KV) caching has emerged as a crucial optimization technique for accelerating inference in large language models (LLMs). By allowing the attention operation to scale linearly rather than quadratically with the total sequence length, KV caching significantly enhances generation throughput. …

• SwiftEmbed: Ultra-Fast Text Embeddings via Static Token Lookup for Real-Time Applications

  Edouard Lansiaux, Antoine Simonet, Eric Wiel · 6 de enero de 2026

  We present a static token lookup methodology for text embedding generation that achieves 1.12 ms p50 latency for single text embeddings while maintaining 60.6 MTEB average score across 8 representative tasks, corresponding to 89% of contextual model quality. The Rust implementation delivers 50,000 r…

• Revati: Transparent GPU-Free Time-Warp Emulation for LLM Serving

  Amey Agrawal, Mayank Yadav, Sukrit Kumar, Anirudha Agrawal, Garv Ghai, Souradeep Bera, Elton Pinto, Sirish Gambhira, Mohammad Adain, Kasra Sohrab, Chus Antonanzas, Alexey Tumanov · 5 de enero de 2026

  Deploying LLMs efficiently requires testing hundreds of serving configurations, but evaluating each one on a GPU cluster takes hours and costs thousands of dollars. Discrete-event simulators are faster and cheaper, but they require re-implementing the serving system's control logic -- a burden that …

• 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)

  Tianyi Zhang, Mohsen Hariri, Shaochen Zhong, Vipin Chaudhary, Yang Sui, Xia Hu, Anshumali Shrivastava · 5 de enero de 2026

  Large-scale AI models, such as Large Language Models (LLMs) and Diffusion Models (DMs), have grown rapidly in size, creating significant challenges for efficient deployment on resource-constrained hardware. In this paper, we introduce Dynamic-Length Float (DFloat11), a lossless compression framework…

• Orchid: Flexible and Data-Dependent Convolution for Sequence Modeling

  Mahdi Karami, Ali Ghodsi · 1 de enero de 2026

  In the rapidly evolving field of deep learning, the demand for models that are both expressive and computationally efficient has never been more critical. This paper introduces Orchid, a novel architecture designed to address the quadratic complexity of traditional attention mechanisms without compr…