The Official Internet Standard for Robots.txt (RFC 9309)
In 2022, the IETF ratified RFC 9309, officially standardizing the Robots Exclusion Protocol after nearly 30 years of informal conventions. Understanding these rules is essential to prevent crawler blocking disasters.
Key RFC 9309 Directives:
- 500-Error Handling: When a robots.txt returns a 5xx HTTP status code, major search engines halt crawling across the entire website for up to 24 hours.
- File Size Constraints: Parsers are only mandated to inspect the first 500 KiB of a robots.txt file. Any rules below this threshold are ignored.
- Blocking AI Scrapers: Safely disallow automated training crawlers like GPTBot, CCBot, and ClaudeBot while keeping Googlebot and Bingbot completely unobstructed.