[2603.00359] How Large Language Models Get Stuck: Early structure with persistent errors

Computer Science > Computation and Language arXiv:2603.00359 (cs) [Submitted on 27 Feb 2026] Title:How Large Language Models Get Stuck: Early structure with persistent errors Authors:Alokesh Manna, William Snyder, Whitney Tabor View a PDF of the paper titled How Large Language Models Get Stuck: Early structure with persistent errors, by Alokesh Manna and 2 other authors View PDF HTML (experimental) Abstract:Linguistic insights may help make Large Language Model (LLM) training more efficient. We trained Meta's OPT model on the 100M word BabyLM dataset, and evaluated it on the BLiMP benchmark, which consists of 67 classes, each defined by sentence pairs that differ in a targeted syntactic or semantic rule violation. We tested the model's preference for grammatical over ungrammatical sentences across training iterations and grammatical types. In nearly one-third of the BLiMP classes, OPT fails to consistently assign a higher likelihood to grammatical sentences, even after extensive training. When it fails, it often establishes a clear (erroneous) separation of the likelihoods at an early stage of processing and sustains this to the end of our training phase. We hypothesize that this mis-categorization is costly because it creates entrenched biases that must, eventually, be reversed in order for the model to perform well. We probe this phenomenon using a mixture of qualitative (based on linguistic theory and the theory of Deep Learning) and quantitative (based on numerical tes...