New robots.txt opt-out for AI training: a group with User-agent: __usage_ai-training and Disallow: /. OpenAI, Google, Microsoft and Anthropic said on 30 Oct they'll honour it, on different dates.We tried Disallow: / then Allow: /press/ in Python 3.9.6's robotparser. /press/ still came back refused. Put Allow lines first.https://proxymint.com/robots-txt-ai-training-token/?utm_source=mastodon&utm_medium=social&utm_campaign=news-robots-txt-ai-training-token#robotstxt #AI #WebDev
New robots.txt opt-out for AI training: a group with User-agent: __usage_ai-training and Disallow: /. OpenAI, Google, Mi...
