Aura Dev

robots.txt Generator

Generate and validate robots.txt files for your website.

Rules

User-agent

Allow (comma separated)

Disallow (comma separated)

Sitemap URL (Optional)

User-agent: *
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

A robots.txt file tells search engine crawlers which pages or files they can or cannot request from your site.

It is a fundamental part of technical SEO, helping to manage crawl budget and keep private or irrelevant directories out of search engine indexes.