SAN FRANCISCO, Calif. - OpenAI says it will honour a common signal in robots.txt, the file websites use to express their preferences about automated crawlers, letting site owners state one preference about the use of their content for foundational generative AI training instead of writing separate rules for every provider.
Until now, a website that wanted to keep its pages out of AI training had to identify individual crawlers and maintain separate rules for each provider. A shared signal, OpenAI says, reduces that work for publishers and website operators by letting them communicate the same preference to multiple participating providers at once.
The effort is voluntary, not a law and not a finished standard. It draws on the ongoing work of the Internet Engineering Task Force’s AI Preferences working group, and OpenAI says it is joining other participants in that working group in the same-day announcement of the shared signal. OpenAI aims to implement the signal in the coming months and will share implementation details in a further update.
Crucially for site owners, the new signal will not replace the controls OpenAI already provides through robots.txt. Those existing crawler controls will continue to apply and be respected, OpenAI says. The signal also covers training only: it is a preference about using content to train foundation models, not a block on crawling or indexing, and it does not remove content that has already been collected.
For publishers, including news outlets, the practical takeaway is patience. The shared signal is a promise of simpler controls, not a switch that works everywhere today. Site owners should keep their existing per-provider rules in place until each provider confirms it is reading the shared signal, and should check each provider’s documentation before relying on the new line.
“We believe this is a practical step toward an open web where people can publish and discover information, and website owners have clearer, simpler ways to communicate their choices about how their content is used,” OpenAI said, encouraging other AI providers to honour the signal.
(Source: OpenAI announcement, October 30, 2026)
