OpenAI, Google, Microsoft and Anthropic adopt unified AI scraping opt-out

OpenAI, Google, Microsoft, and Anthropic have agreed to adopt a new shared standard that lets website publishers opt out of having their content scraped for AI training across multiple companies at once. The move, reported earlier by Search Engine Roundtable, addresses a growing frustration among publishers who currently must set separate preferences for each AI company's crawler. Meta has also signaled tentative support for the idea.

The new mechanism is a policy token proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). It lives inside a website's robots.txt file, the long-standing protocol that tells web crawlers which parts of a site they may or may not access. Until now, publishers had to list each AI company's crawler individually—for example, blocking OpenAI's GPTBot, Google's Google-Extended, and Anthropic's ClaudeBot one by one. The shared token lets them express a single preference that all participating AI companies will honor.

A single signal for multiple AI crawlers

The standard is designed to simplify the opt-out process for publishers who want to keep their content out of AI training datasets. Instead of maintaining separate rules for each company, a website can include the shared token in its robots.txt file, and any AI company that supports the standard will restrict its crawler accordingly. The token does not affect search indexing; sites can still appear in search results even if they block AI training use.

Microsoft says it will offer granular controls that let publishers distinguish between opting out of AI training and opting out of AI grounding—the practice of pointing an AI model to a verified source to reduce hallucinations. Google has already introduced similar controls that allow sites to manage whether their content appears in AI search features or is used for training, without affecting visibility in standard search. Anthropic has already implemented the new signal, while Google and Microsoft plan to support it next year. OpenAI says it will implement it in the coming months.

Industry support and timeline

The announcement comes as publishers struggle with a rising tide of AI crawlers that harvest content for model training, often without explicit permission. Some organizations have tried other approaches: the Really Simple Licensing (RSL) standard lets publishers charge crawlers for access, and Cloudflare has begun blocking AI crawlers by default. The new shared token offers a lighter-weight, standardized alternative that works across multiple providers.

Meta's communications head, Andy Stone, expressed support for the concept but did not confirm whether Meta will honor the token. “We believe creating a common signal that conveys such preferences to multiple providers at once is an important step,” Stone said. “And we look forward to working with other AI companies on the standards and their implementation across the industry.”

What it means for publishers and AI companies

For publishers, the shared token reduces administrative burden and provides a clearer, more consistent way to assert control over their content. For AI companies, adopting a common standard could help ease tensions with content creators and avoid legal disputes over unauthorized scraping. The move also reflects a broader industry push toward more transparent and consent-based data practices.

The standard is still under development within the IETF, and its effectiveness will depend on widespread adoption. While the four major AI companies have committed, other players—including smaller AI startups and search engines—have yet to announce support. Still, the coordination among leading firms marks a significant step toward a more unified approach to AI data collection, giving publishers a practical tool to protect their work in an increasingly AI-driven web.