Google, OpenAI, Microsoft, and Anthropic will honor a shared robots.txt policy token that lets site owners opt out of AI training across all four.