[2601.14958] Script Sensitivity: Benchmarking Language Models on Unicode, Romanized and Mixed-Script Sinhala

Computer Science > Computation and Language arXiv:2601.14958 (cs) [Submitted on 21 Jan 2026 (v1), last revised 21 Mar 2026 (this version, v2)] Title:Script Sensitivity: Benchmarking Language Models on Unicode, Romanized and Mixed-Script Sinhala Authors:Minuri Rajapakse, Ruvan Weerasinghe View a PDF of the paper titled Script Sensitivity: Benchmarking Language Models on Unicode, Romanized and Mixed-Script Sinhala, by Minuri Rajapakse and Ruvan Weerasinghe View PDF HTML (experimental) Abstract:The performance of Language Models (LMs) on low-resource, morphologically rich languages like Sinhala remains largely unexplored, particularly regarding script variation in digital communication. Sinhala exhibits script duality, with Unicode used in formal contexts and Romanized text dominating social media, while mixed-script usage is common in practice. This paper benchmarks 24 open-source LMs on Unicode, Romanized and mixed-script Sinhala using perplexity evaluation across diverse text sources. Results reveal substantial script sensitivity, with median performance degradation exceeding 300 times from Unicode to Romanized text. Critically, model size shows no correlation with script-handling competence, as smaller models often outperform architectures 28 times larger. Unicode performance strongly predicts mixed-script robustness but not Romanized capability, demonstrating that single-script evaluation substantially underestimates real-world deployment challenges. These findings estab...