Robots.txt Generator
Choose how search engine crawlers should treat your site and copy or download the robots.txt file.
User-agent: * Allow: /
What robots.txt controls
Robots.txt is a plain text file that tells well-behaved search engine crawlers which parts of your site they are allowed to visit, helping you save crawl budget on pages that don't need to be indexed, like admin panels or internal search results.
Frequently asked questions
Where should I put the robots.txt file?
It must be placed at the root of your domain, so it is reachable at https://yourdomain.com/robots.txt.
Does robots.txt guarantee a page will not appear in search results?
No. It only asks well-behaved crawlers not to access certain paths; a disallowed page can still be indexed if it is linked from elsewhere. Use a noindex meta tag for that instead.
Related tools
You might also need these while working on the same task.
Generate SEO, Open Graph and Twitter Card meta tags and copy ready-to-use HTML.
Open toolTurn any title into a clean, SEO-friendly URL slug.
Open toolAdd utm_source, utm_medium and utm_campaign parameters to any URL.
Open toolCreate a Brazilian PIX copy-and-paste QR code in seconds.
Open tool