I spent years building a 103B-token Usenet corpus (1980–2013) and finally documented it [P]

About this article

For the past several years I've been quietly assembling and processing what I believe is one of the larger privately held pretraining corpora around... a complete Usenet archive spanning 1980 to 2013. Here's what it ended up being: 103.1 billion tokens (cl100k_base) 408 million posts across 9 newsgroup hierarchies 18,347 newsgroups covered 33 years of continuous coverage The processing pipeline included full deduplication, binary removal (alt.binaries.* excluded at the hierarchy level before ...