[2604.07320] Evaluating In-Context Translation with Synchronous Context-Free Grammar Transduction

Computer Science > Computation and Language arXiv:2604.07320 (cs) [Submitted on 8 Apr 2026] Title:Evaluating In-Context Translation with Synchronous Context-Free Grammar Transduction Authors:Jackson Petty, Jaulie Goe, Tal Linzen View a PDF of the paper titled Evaluating In-Context Translation with Synchronous Context-Free Grammar Transduction, by Jackson Petty and 2 other authors View PDF HTML (experimental) Abstract:Low-resource languages pose a challenge for machine translation with large language models (LLMs), which require large amounts of training data. One potential way to circumvent this data dependence is to rely on LLMs' ability to use in-context descriptions of languages, like textbooks and dictionaries. To do so, LLMs must be able to infer the link between the languages' grammatical descriptions and the sentences in question. Here we isolate this skill using a formal analogue of the task: string transduction based on a formal grammar provided in-context. We construct synchronous context-free grammars which define pairs of formal languages designed to model particular aspects of natural language grammar, morphology, and written representation. Using these grammars, we measure how well LLMs can translate sentences from one formal language into another when given both the grammar and the source-language sentence. We vary the size of the grammar, the lengths of the sentences, the syntactic and morphological properties of the languages, and their written script. We ...