The short answer
robots.txt lives at https://example.com/robots.txt. It lists rules such as User-agent: * and Disallow: /checkout/, telling automated visitors which parts of the site the owner does not want crawled. It became an official internet standard, RFC 9309, in 2022.
What it is and is not
It is a request, not a lock: nothing technically stops a bot from ignoring it. But ignoring it is bad manners, can breach a site's terms, and can be used against you in a dispute. Ethical scrapers read it, honour it, and also respect any crawl-delay hints.
Also known as
Robots exclusion protocol, robots exclusion standard, REP.
Learn more
- Ethical web scraping
- Are proxies legal?
- Web scraping proxies
- What is rate limiting?
- RFC 9309, the Robots Exclusion Protocol
- Google's introduction to robots.txt
- Common Crawl, a crawler that respects robots.txt
- Web scraping overview and legal history
Affiliate disclosure: Team of Monkeys is an affiliate of Geonode, Webshare and Rayobyte. Links and banners marked as sponsored, and links to Geonode, Webshare and Rayobyte in the text, use our referral links, and we may earn a commission if you buy through them, at no extra cost to you. Our explanations and comparisons are written to be useful whichever provider you choose.
Editorial note: Proxy providers change their pools, features and prices often, so treat any figures here as typical ranges at the time of writing and confirm current terms with the provider before you buy.
Editorial note: Proxy providers change their pools, features and prices often, so treat any figures here as typical ranges at the time of writing and confirm current terms with the provider before you buy.