![[2602.21800] An Evaluation of Context Length Extrapolation in Long Code via Positional Embeddings and Efficient Attention](https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png){kind=logo}
[2602.21800] An Evaluation of Context Length Extrapolation in Long Code via Positional Embeddings and Efficient Attention
Summary
This paper evaluates methods for context length extrapolation in long code using positional embeddings and efficient attention mechanisms, addressing limitations in large language models (LLMs) for software engineering tasks.
Why It Matters
As LLMs become integral to software engineering, understanding their limitations in handling long code sequences is crucial. This research aims to enhance the effectiveness of code generation and completion tools, potentially leading to more robust automated coding solutions.
Key Takeaways
- Current LLMs face challenges with fixed context lengths in long code.
- The study investigates zero-shot methods to improve position encodings.
- Optimizing attention mechanisms can enhance long code completion tasks.
- Understanding these methods can lead to better automated coding tools.
- The findings could influence future research in software engineering and AI.
Computer Science > Software Engineering arXiv:2602.21800 (cs) [Submitted on 25 Feb 2026] Title:An Evaluation of Context Length Extrapolation in Long Code via Positional Embeddings and Efficient Attention Authors:Madhusudan Ghosh, Rishabh Gupta View a PDF of the paper titled An Evaluation of Context Length Extrapolation in Long Code via Positional Embeddings and Efficient Attention, by Madhusudan Ghosh and Rishabh Gupta View PDF HTML (experimental) Abstract:The rapid advancement of large language models (LLMs) has led to a significant increase in automated tools in the software engineering, capable of performing various code-related tasks such as code generation, completion, and translation. Despite these advancements, its effectiveness is constrained by fixed context lengths, limiting its ability to generalize across long, domain-specific code sequences. To address this challenge, we investigate zero-shot, inference-only methods aimed at improving position encodings and optimizing attention mechanisms. Our goal is to provide a thorough analysis of current approaches that facilitate context length extrapolation in code, particularly in the context of long code completion tasks. Subjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI) Cite as: arXiv:2602.21800 [cs.SE] (or arXiv:2602.21800v1 [cs.SE] for this version) https://doi.org/10.48550/arXiv.2602.21800 Focus to learn more arXiv-issued DOI via DataCite (pending registration) Submission history From: Ri...
