[R] Causal self-attention as a probabilistic model over embeddings

About this article

We’ve been working on a probabilistic interpretation of causal self-attention where token embeddings are treated as latent variables. In that view, the attention map induces a change-of-variables term, which leads to a barrier / degeneracy boundary in embedding space. The resulting picture is: a stability-margin interpretation of causal attention “support tokens,” i.e. the positions closest to the degeneracy boundary a simple MAP-style training penalty: standard cross-entropy plus a smooth lo...