Google just signed on to the biggest thing to happen to robots.txt since Google-Extended. On Thursday the company announced it will voluntarily adopt a shared AI training control built on the IETF's AIPREF working group — a single "policy token" a publisher drops into robots.txt to express training preferences across multiple AI providers at once, instead of naming GPTBot, Google-Extended, and every other crawler individually, according to Search Engine Roundtable.
Google plans to implement the signal in 2027, "along with other AI providers," and Bing, OpenAI, and Anthropic have announced similar adoption. The mechanics matter: a site that uses the policy token to opt out of training — and doesn't name Google-Extended separately — will be treated as if it had disallowed Google-Extended. But Google is explicit that existing company-specific controls keep working alongside it, so publishers can opt out broadly while still carving out individual relationships. Sites that explicitly allow Google-Extended stay eligible for Gemini Apps grounding.
John Mueller, Google Search Advocate, framed it as the culmination of pressure from "publishers, civil society groups, regulators and other stakeholders that have called for industry-wide controls on AI training," adding that the proposal is "consistent with the EU Code of Practice for General Purpose AI (GPAI) Model Providers under the EU AI Act that recognizes robots.txt as a primary method for web publishers to express machine-readable preferences."
Why this matters: publishers have been living a token-management nightmare. OpenAI's GPTBot, OAI-SearchBot, Common Crawl, Perplexity, Meta's crawlers — each with different tokens, different jobs, and spotty compliance. HasData testing found hundreds of sites disallowing GPTBot still serving it live pages. A single cross-industry signal is the first plausible answer to that mess — if everyone actually honors it.
The catches are real. It's voluntary, it's not arriving until 2027, and as Google's Gary Illyes reminded everyone just last week, robots.txt governs whether a crawler enters a page — not what an AI system does with content it already collected. This doesn't un-train a model.
What to watch: the IETF working group still has to standardize the rest of the AIPREF vocabulary, and Google named no other participating providers or dates. For publishers, the practical move is to keep Google-Extended and friends in place until 2027 actually delivers — and for everyone else, to watch whether a voluntary 2027 pledge survives contact with the next training-data lawsuit.
— via Search Engine Roundtable