The Free and Open Web Is Under Attack at the IETF

The ability to access publicly available information using automated tools is a central value and benefit of a free and open internet. Automated access—often called crawling or scraping—powers important, useful tools for locating, preserving, and analyzing online information. For example, crawling and scraping helps
journalists, researchers, and watchdog organizations
report the news, find security flaws, and investigate discrimination. Crawling the web allows non-profits like the Internet Archive to
preserve historical copies
of websites. Tools for automated
comparison shopping
allow consumers to find the best deals on items they want to buy. And so on.

Yet the open internet access is increasingly under threat from publishers and Big Tech companies alike. Fearing lost advertising and licensing revenues, website operators increasingly claim that they need to lock down their sites from bots that crawl public web content to train or operate AI models. Some companies are even trying to embed their business models into internet standards by
changing Internet Engineering Task Force (IETF) technical standards
that shape much of the internet.

Many of their economic anxieties are understandable. AI bots can strain
websites’

infrastructure, in some cases, degrading site performance or taking them offline altogether. Upgrading systems costs money that some sites may not have. And AI is likely to disrupt the business models many publishers adopted in response to the rise of the internet, if users rely on AI overviews instead of visiting source websites.

However reasonable these fears may be, the answer is not to
chang

the IETF standards from neutral protocols that

encourage openness to restrictive requirements designed to monetize internet access.

The worst of these proposed standards would give websites far greater ability to automatically block legitimate, lawful scraping and crawling. For example, the
AI Preferences

working group is working on pr