[2503.18018] Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural Contexts?

Computer Science > Artificial Intelligence arXiv:2503.18018 (cs) [Submitted on 23 Mar 2025 (v1), last revised 8 Apr 2026 (this version, v2)] Title:Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural Contexts? Authors:Aabid Karim, Abdul Karim, Bhoomika Lohana, Matt Keon, Jaswinder Singh, Abdul Sattar View a PDF of the paper titled Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural Contexts?, by Aabid Karim and 5 other authors View PDF HTML (experimental) Abstract:We demonstrate that large language models' (LLMs) mathematical reasoning is culturally sensitive: testing 14 models from Anthropic, OpenAI, Google, Meta, DeepSeek, Mistral, and Microsoft across six culturally adapted variants of the GSM8K benchmark, we find accuracy drops ranging from 0.3% (Claude 3.5 Sonnet) to 5.9% (LLaMA 3.1-8B) when math problems are embedded in unfamiliar cultural contexts--even when the underlying mathematical logic remains unchanged. These statistically significant performance reductions (p < 0.01, confirmed through McNemar tests) reveal that mathematical reasoning in LLMs is not culturally neutral. To create these variants for Haiti, Moldova, Pakistan, Solomon Islands, Somalia, and Suriname, we systematically replaced cultural entities (names, foods, places, etc.) in 1,198 GSM8K questions while preserving all mathematical operations and numerical values. Our quantitative error analysis of 18,887 instances reveals that cultural adaptation affects br...