What Is Robots.txt?
Robots.txt is a text file that gives automated crawlers instructions about which website paths they may request. Website owners place the file at the site’s root address, where crawlers can find it before exploring pages. The file guides crawling behaviour, but it does not remove pages from search results or control access for people.
Quick Facts About Robots.txt
Category
Technical SEO file
Used for
Crawler access instructions
Common confusion
Not a page removal tool
Also called
Robots Exclusion File, robots exclusion protocol
Often discussed with
Technical SEO
Key Takeaways About Robots.txt
- It controls crawler requests, not human access.
- Rules apply to paths within a website.
- Blocking a URL does not guarantee removal from search results.
How Robots.txt Works
Robots.txt contains plain-text rules for automated crawlers. Each rule can identify a crawler and allow or disallow access to selected website paths. Crawlers read the file before requesting those paths, so the instructions can reduce unwanted crawling or keep internal areas out of routine exploration.
Related glossary terms: Crawlability, Crawl Budget, XML Sitemap.
Search platforms may still discover a blocked URL through links or other signals. A disallowed path can remain known to a search engine even when its content is not fetched. XML Sitemap files serve a different purpose by listing URLs that a site wants crawlers to find. Clear rules support Crawlability, while page indexing needs separate controls such as page content and indexing directives.
Expert Note
When we review Robots.txt, we check the file at the correct root address and test whether its rules match the website’s important paths. We compare blocked areas with crawl data and the XML Sitemap. Our team also checks that developers have not used Robots.txt as a substitute for page removal or access control.
Robots.txt in Practice: A Real-World Example
A local trades business may use Robots.txt to keep a staging folder and internal search URLs from routine crawling. The file leaves public service pages available while directing crawlers away from areas that do not support organic search.
Related Services
Related Terms
Crawlability
Crawlability is the ability of search engine crawlers to access, follow and retrieve a website’s pages…
Crawl Budget
Crawl Budget is the amount of attention a search engine crawler allocates to a website during…
XML Sitemap
An XML Sitemap is a structured file that lists important URLs on a website for search…
Index Coverage
Index Coverage is a report of how search engines process a website’s URLs for inclusion in…
SEO-Adelaide.au
Have Questions About Robots.txt?
Contact SEO-Adelaide.au for practical guidance on Robots.txt and related seo services work in Adelaide.