robots.txt
Sign in to saveAlso known as robots.txt protocol, Robots Exclusion Protocol
robots.txt is the filename used for implementing the Robots Exclusion Protocol, a standard used by websites to indicate to visiting web crawlers and other web robots which portions of the website they are allowed to visit.
Key facts
- Technology standard.caption
- Example of a simple robots.txt file, indicating that a user-agent called "Mallorybot" is not allowed to crawl any of the website's pages, and that other user-agents cannot crawl more than one page every 20 seconds, and are not allowed to crawl the "secret" folder
- Technology standard.class
- skin-invert-image
- Technology standard.status
- Proposed Standard
- Technology standard.first_published
- 1994 published, formally standardized in 2022
- Technology standard.website
- ,
- Technology standard.title
- robots.txt
- Technology standard.long_name
- Robots Exclusion Protocol
- Technology standard.image
- Robots txt.svg
via Wikipedia infobox
Wikidata facts
- Official website
- www.robotstxt.org
- Image
- Robots txt.svg
Show 3 more facts
- Stack Exchange tag
- webmasters.stackexchange.com/tags/robots.txt
- described at URL
- wiki.archiveteam.org/index.php/Robots.txt
- Commons category
- Robots.txt
Sources (1)
via Wikidata · CC0
~15 min read
Article
23 sectionsContents
- History
- Standard
- Compliance
- Search engines
- Archival sites
- Artificial intelligence
- Security
- Alternatives
- Examples
- The use of the wildcard * in rules
- Nonstandard extensions
- Crawl-delay directive
- Sitemap
- Universal "*" match
- Content-Signal
- Meta tags and headers
- A "noindex" meta tag
- A "noindex" HTTP response header
- Maximum size of a robots.txt file
- See also
- References
- Further reading
- External links
robots.txt is the filename used for implementing the Robots Exclusion Protocol, a standard used by websites to indicate to visiting web crawlers and other web robots which portions of the website they are allowed to visit.
The standard, developed in 1994, relies on voluntary compliance. Malicious bots can use the file as a directory of which pages to visit, though standards bodies discourage countering this with security through obscurity. Some archival sites ignore robots.txt. The standard was used in the 1990s to mitigate server overload. In the 2020s, websites began denying bots that collect information for generative artificial intelligence.