AI assistants are optimized to seem helpful. That is not the same thing as being helpful.

About this article

RLHF trains models on human feedback. Humans rate responses they like. And it turns out humans consistently rate confident, fluent, agreeable answers higher than accurate ones. The result: every major AI assistant has been optimized, at scale, to produce responses that feel good rather than responses that are true. The training signal is user satisfaction, not correctness. This shows up in concrete ways: Ask the same factual question three different ways and you will often get three different...