[2603.19338] DAPA: Distribution Aware Piecewise Activation Functions for On-Device Transformer Inference and Training

Computer Science > Machine Learning arXiv:2603.19338 (cs) [Submitted on 19 Mar 2026] Title:DAPA: Distribution Aware Piecewise Activation Functions for On-Device Transformer Inference and Training Authors:Maoyang Xiang, Bo Wang View a PDF of the paper titled DAPA: Distribution Aware Piecewise Activation Functions for On-Device Transformer Inference and Training, by Maoyang Xiang and 1 other authors View PDF HTML (experimental) Abstract:Non-linear activation functions play a pivotal role in on-device inference and training, as they not only consume substantial hardware resources but also impose a significant impact on system performance and energy efficiency. In this work, we propose Distribution-Aware Piecewise Activation (DAPA), a differentiable and hardware-friendly activation function for Transformer architectures by exploiting the distribution of pre-activation data. DAPA employs a non-uniform piecewise approximation that allocates finer segments to high-probability regions of the distribution, improving generalizability over prior piecewise linear methods. The resulting approximation is further quantized using Distribution-Weighted Mean Square Error to reduce latency and resource utilization for hardware deployment. Our HLS implementation demonstrates that DAPA speeds up GELU computation by 16$\times$ and decreases DSP utilization by 16$\times$ while maintaining comparable or better performance across vision Transformers and GPT-2 models. Subjects: Machine Learning (cs...