β’ User-agent: Specifies which crawler the rules apply to (* = all crawlers)
β’ Disallow: Paths the crawler should not access
β’ Allow: Paths the crawler can access (can override Disallow)
β’ Sitemap: Declares the sitemap URL for crawlers
β’ Crawl-delay: Sets the request interval in seconds
β’ File location: Should be at the root of the website (/robots.txt)