AI CRAWLERS
AI crawler access without the confusion
Different crawlers can have different purposes. One robots.txt rule does not necessarily control every use of your content.
Search and training can be separate. OpenAI uses OAI-SearchBot for ChatGPT search and GPTBot for model-training crawling, with independent controls.
robots.txt is only one layer. Firewalls, CDNs, bot protection, authentication and JavaScript challenges can also block access.
Use deliberate rules. Decide what you want discoverable in search separately from what you want available for training where a provider exposes separate controls.
llms.txt is optional. Some systems may choose to use it; Google says it is not required for Google Search and does not improve or reduce visibility there.