[2603.24647] Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch

Computer Science > Machine Learning arXiv:2603.24647 (cs) [Submitted on 25 Mar 2026] Title:Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch Authors:Fabio Ferreira, Lucca Wobbe, Arjun Krishnakumar, Frank Hutter, Arber Zela View a PDF of the paper titled Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch, by Fabio Ferreira and 4 other authors View PDF HTML (experimental) Abstract:The autoresearch repository enables an LLM agent to search for optimal hyperparameter configurations on an unconstrained search space by editing the training code directly. Given a fixed compute budget and constraints, we use \emph{autoresearch} as a testbed to compare classical hyperparameter optimization (HPO) algorithms against LLM-based methods on tuning the hyperparameters of a small language model. Within a fixed hyperparameter search space, classical HPO methods such as CMA-ES and TPE consistently outperform LLM-based agents. However, an LLM agent that directly edits training source code in an unconstrained search space narrows the gap to classical methods substantially despite using only a self-hosted open-weight 27B model. Methods that avoid out-of-memory failures outperform those with higher search diversity, suggesting that reliability matters more than exploration breadth. While small and mid-sized LLMs struggle to track optimization state across trials, classical methods lack domain knowledge. To bridge this gap...