For more than a decade, the nonprofit common crawl has been scraping billions of webpages to build a massive archive of the internet, notes the atlantic, making it freely available for research Despite claims of compliance with publishers' requests to remove their articles, investigations reveal that many remain in the archive In recent years, however, this archive has been put to a controversial purpose
Mali J | Instagram, TikTok | Linktree
Common crawl’s massive internet archive may be giving ai companies access to paywalled journalism, according to a new report.
A nonprofit organization has been systematically supplying paywalled news articles to major ai companies for training large language models, according to an investigation published november 4, 2025, by the atlantic's alex reisner
Common crawl maintains archives containing millions of articles from major news organizations that readers typically must pay to access, enabling ai developers. The atlantic on common crawl, the nonprofit funneling paywalled articles to ai companies a brutally efficient exposé, alex reisner caught them in several lies by simply looking at their crawl data (via) At the request of brein, common crawl has removed over two million news articles belonging to popular dutch news outlets from its ai training dataset The common crawl foundation has been scraping the internet for over a decade, creating a vast archive used by ai companies to train models, including paywalled content
