![[2602.18800] Operational Robustness of LLMs on Code Generation](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.18800] Operational Robustness of LLMs on Code Generation
Summary
This article evaluates the operational robustness of large language models (LLMs) in code generation, proposing a new method to assess their sensitivity to variations in task descriptions.
Why It Matters
As LLMs become integral in software development, understanding their robustness is crucial for ensuring reliable code generation. This research highlights the limitations of current evaluation methods and introduces a novel approach that can help improve LLM performance in real-world applications.
Key Takeaways
- Introduces scenario domain analysis for evaluating LLM robustness.
- Findings indicate LLMs are less robust with complex coding tasks.
- Robustness varies significantly based on the topic and complexity of tasks.
- Ranks four state-of-the-art LLMs in terms of their robustness.
- Highlights the need for improved evaluation techniques in AI code generation.
Computer Science > Software Engineering arXiv:2602.18800 (cs) [Submitted on 21 Feb 2026] Title:Operational Robustness of LLMs on Code Generation Authors:Debalina Ghosh Paul, Hong Zhu, Ian Bayley View a PDF of the paper titled Operational Robustness of LLMs on Code Generation, by Debalina Ghosh Paul and Hong Zhu and Ian Bayley View PDF HTML (experimental) Abstract:It is now common practice in software development for large language models (LLMs) to be used to generate program code. It is desirable to evaluate the robustness of LLMs for this usage. This paper is concerned in particular with how sensitive LLMs are to variations in descriptions of the coding tasks. However, existing techniques for evaluating this robustness are unsuitable for code generation because the input data space of natural language descriptions is discrete. To address this problem, we propose a robustness evaluation method called scenario domain analysis, which aims to find the expected minimal change in the natural language descriptions of coding tasks that would cause the LLMs to produce incorrect outputs. We have formally proved the theoretical properties of the method and also conducted extensive experiments to evaluate the robustness of four state-of-the-art art LLMs: Gemini-pro, Codex, Llamma2 and Falcon 7B, and have found that we are able to rank these with confidence from best to worst. Moreover, we have also studied how robustness varies in different scenarios, including the variations with th...