[2509.24207] Humanline: Online Alignment as Perceptual Loss

Computer Science > Artificial Intelligence arXiv:2509.24207 (cs) [Submitted on 29 Sep 2025 (v1), last revised 27 Mar 2026 (this version, v2)] Title:Humanline: Online Alignment as Perceptual Loss Authors:Sijia Liu, Niklas Muennighoff, Kawin Ethayarajh View a PDF of the paper titled Humanline: Online Alignment as Perceptual Loss, by Sijia Liu and 2 other authors View PDF HTML (experimental) Abstract:Online alignment (e.g., GRPO) is generally more performant than offline alignment (e.g., DPO) -- but why? Drawing on prospect theory from behavioral economics, we propose a human-centric explanation. We prove that online on-policy sampling better approximates the human-perceived distribution of what the model can produce, and PPO/GRPO-style clipping -- originally introduced to just stabilize training -- recovers a perceptual bias in how humans perceive probability. In this sense, PPO/GRPO act as perceptual losses already. Our theory further suggests that the online/offline dichotomy is itself incidental to maximizing human utility, since we can achieve the same effect by selectively training on any data in a manner that mimics human perception, rather than restricting ourselves to online on-policy data. Doing so would allow us to post-train more quickly, cheaply, and flexibly without sacrificing performance. To this end, we propose a design pattern that explicitly incorporates perceptual distortions of probability into objectives like DPO/KTO/GRPO, creating humanline variants of ...