![[2512.12132] Approximation with SiLU Networks: Constant Depth and Exponential Rates for Basic Operations](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2512.12132] Approximation with SiLU Networks: Constant Depth and Exponential Rates for Basic Operations
Summary
This paper presents SiLU network constructions that optimize approximation efficiency for basic operations, particularly the square function, using constant depth and specific hyperparameter tuning.
Why It Matters
Understanding the efficiency of SiLU networks is crucial for advancing neural network architectures. This research highlights the balance between network depth and parameter optimization, which can lead to improved performance in machine learning applications.
Key Takeaways
- SiLU networks can achieve efficient approximations with constant depth.
- Optimal hyperparameter tuning is critical for minimizing approximation error.
- The research extends to Sobolev spaces, enhancing the applicability of SiLU networks.
Computer Science > Machine Learning arXiv:2512.12132 (cs) [Submitted on 13 Dec 2025 (v1), last revised 21 Feb 2026 (this version, v2)] Title:Approximation with SiLU Networks: Constant Depth and Exponential Rates for Basic Operations Authors:Koffi O. Ayena View a PDF of the paper titled Approximation with SiLU Networks: Constant Depth and Exponential Rates for Basic Operations, by Koffi O. Ayena View PDF HTML (experimental) Abstract:We present SiLU network constructions whose approximation efficiency depends critically on proper hyperparameter tuning. For the square function $x^2$, with optimally chosen shift $a$ and scale $\beta$, we achieve approximation error $\varepsilon$ using a two-layer network of constant width, where weights scale as $\beta^{\pm k}$ with $k = \mathcal{O}(\ln(1/\varepsilon))$. We then extend this approach through functional composition to Sobolev spaces, we obtain networks with depth $\mathcal{O}(1)$ and $\mathcal{O}(\varepsilon^{-d/n})$ parameters under optimal hyperparameters settings. Our work highlights the trade-off between architectural depth and activation parameter optimization in neural network approximation theory. Comments: Subjects: Machine Learning (cs.LG); Numerical Analysis (math.NA) Cite as: arXiv:2512.12132 [cs.LG] (or arXiv:2512.12132v2 [cs.LG] for this version) https://doi.org/10.48550/arXiv.2512.12132 Focus to learn more arXiv-issued DOI via DataCite Submission history From: Koffi Ognandon Ayena [view email] [v1] Sat, 13 Dec ...
